← 研踪 PaperTrailarXiv:2604.11748 · v4
LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
首次提交 2026-04-14 01:21
版本更新 2026-11-06 06:09
雷达观测 2026-11-09 10:16
摘要
Continuous diffusion has served as a foundation for high-fidelity, controllable, and few-step generation across continuous data modalities such as images, videos, and molecular structures. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts. Existing categorical and simplex-based approaches operate over extremely large and sparse language spaces, while prior embedding-space approaches avoid this sparsity but lack a well-selected design space. In this work, we close this gap with LangFlow by connecting embedding-space DLMs to Flow Matching, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, which improves both likelihood and sample quality for embedding-space DLMs and behaves differently from its use in discrete diffusion. Putting everything together, LangFlow is competitive with top discrete DLMs on generative quality, reaching a generative perplexity of 92.3 on LM1B and 36.5 on OpenWebText. It also outperforms discrete counterparts on reasoning tasks like Sudoku. As the first continuous DLM shown to rival discrete diffusion in generative quality, LangFlow provides clear evidence that continuous diffusion is a promising paradigm for language modeling. Code: https://github.com/nealchen2003/LangFlow.
作者备注：NeurIPS 2026
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v4 · 2026-11-06 06:09 · 最新观测版本
LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
Continuous diffusion has served as a foundation for high-fidelity, controllable, and few-step generation across continuous data modalities such as images, videos, and molecular structures. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts. Existing categorical and simplex-based approaches operate over extremely large and sparse language spaces, while prior embedding-space approaches avoid this sparsity but lack a well-selected design space. In this work, we close this gap with LangFlow by connecting embedding-space DLMs to Flow Matching, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, which improves both likelihood and sample quality for embedding-space DLMs and behaves differently from its use in discrete diffusion. Putting everything together, LangFlow is competitive with top discrete DLMs on generative quality, reaching a generative perplexity of 92.3 on LM1B and 36.5 on OpenWebText. It also outperforms discrete counterparts on reasoning tasks like Sudoku. As the first continuous DLM shown to rival discrete diffusion in generative quality, LangFlow provides clear evidence that continuous diffusion is a promising paradigm for language modeling. Code: https://github.com/nealchen2003/LangFlow.
查看此版本解读 →查看此版本原文 ↗