← 研踪 PaperTrailarXiv:2602.16169 · v3
Unlocking Continuous Diffusion Language Models via Discrete Stochastic Localization
首次提交 2026-02-18 12:05
版本更新 2026-11-18 14:49
雷达观测 2026-11-19 19:08
摘要
Continuous diffusion is a natural framework for non-autoregressive generation but has generally lagged behind masked discrete diffusion models (MDMs) on discrete sequence generation. We argue that continuous diffusion can effectively model discrete sequences through the joint design of noise parameterization and token representation geometry. We introduce \emph{Discrete Stochastic Localization} (DSL), a continuous-state framework that combines a stochastic localization channel with unit-sphere token embeddings, resulting in a Bayes-optimal denoiser that depends only on the noisy state, without additional timestep conditioning. A single trained denoising network then supports multiple sampling methods, from continuous denoising to masked diffusion, random-order autoregressive generation and hybrid continuous-then-discrete generation. Parallel generation in this continuous-state framework, therefore, need not imply synchronized denoising: some tokens can establish context earlier while others remain open to refinement. Fine-tuning a pretrained MDLM checkpoint with DSL substantially improves generation quality, as measured by MAUVE, on OpenWebText across all step budgets from to . The same checkpoint supports random-order autoregressive sampling, and a hybrid continuous-then-discrete sampler using as few as total steps, without distillation or further training.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-18 14:49 · 最新观测版本
Unlocking Continuous Diffusion Language Models via Discrete Stochastic Localization
Continuous diffusion is a natural framework for non-autoregressive generation but has generally lagged behind masked discrete diffusion models (MDMs) on discrete sequence generation. We argue that continuous diffusion can effectively model discrete sequences through the joint design of noise parameterization and token representation geometry. We introduce \emph{Discrete Stochastic Localization} (DSL), a continuous-state framework that combines a stochastic localization channel with unit-sphere token embeddings, resulting in a Bayes-optimal denoiser that depends only on the noisy state, without additional timestep conditioning. A single trained denoising network then supports multiple sampling methods, from continuous denoising to masked diffusion, random-order autoregressive generation and hybrid continuous-then-discrete generation. Parallel generation in this continuous-state framework, therefore, need not imply synchronized denoising: some tokens can establish context earlier while others remain open to refinement. Fine-tuning a pretrained MDLM checkpoint with DSL substantially improves generation quality, as measured by MAUVE, on OpenWebText across all step budgets from to . The same checkpoint supports random-order autoregressive sampling, and a hybrid continuous-then-discrete sampler using as few as total steps, without distillation or further training.
查看此版本解读 →查看此版本原文 ↗