← 研踪 PaperTrailarXiv:2603.01367 · v3
DUEL: Exact Likelihood Reshapes Masked Diffusion Scaling Laws and Beyond
首次提交 2026-03-02 09:56
版本更新 2026-11-05 03:24
雷达观测 2026-11-06 11:06
摘要
Masked diffusion models (MDMs) are trained to predict tokens in every order but generate in one. As such, MDM likelihood is an intractable sum over all orders. The evidence lower bound (ELBO) is used as a proxy for likelihood, but it is a loose bound and assumes positions are unmasked uniformly at random. Frontier MDMs, however, select positions with deterministic rules such as confidence. To resolve this mismatch we introduce DUEL, a framework that unifies common deterministic MDM samplers. DUEL computes the likelihood exactly, under the sampler that is actually used during generation. DUEL is a simple plug-and-play algorithm applicable to any pretrained MDM with no retraining. To compute the likelihood of a text sequence, simply replay generation, reveal the true tokens instead of sampled ones, and sum their log-probabilities. Evaluating with DUEL instead of the ELBO changes what we conclude about MDMs. The perplexity gap between MDMs and autoregressive (AR) models shrinks by up to 32% in-domain and 82% zero-shot. In scaling laws, the reported 16x MDM-AR compute gap falls to 9x under DUEL, showing over 1/3 of it is an evaluation artifact. Scoring multiple-choice answers with likelihood via DUEL instead of the ELBO improves LLaDA-8B accuracy on all six tasks, by up to 9.7 points. DUEL also selects samplers correctly: it matches a gold-standard LLM judge on every decided comparison, which generative perplexity and MAUVE fail to do. DUEL reveals that MDMs are more competitive with AR models than the ELBO suggests.
作者备注：36 pages, 4 figures, 21 tables. Code: https://github.com/kuleshov-group/duel. Project page: https://giladturok.github.io/duel/
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-05 03:24 · 最新观测版本
DUEL: Exact Likelihood Reshapes Masked Diffusion Scaling Laws and Beyond
Masked diffusion models (MDMs) are trained to predict tokens in every order but generate in one. As such, MDM likelihood is an intractable sum over all orders. The evidence lower bound (ELBO) is used as a proxy for likelihood, but it is a loose bound and assumes positions are unmasked uniformly at random. Frontier MDMs, however, select positions with deterministic rules such as confidence. To resolve this mismatch we introduce DUEL, a framework that unifies common deterministic MDM samplers. DUEL computes the likelihood exactly, under the sampler that is actually used during generation. DUEL is a simple plug-and-play algorithm applicable to any pretrained MDM with no retraining. To compute the likelihood of a text sequence, simply replay generation, reveal the true tokens instead of sampled ones, and sum their log-probabilities. Evaluating with DUEL instead of the ELBO changes what we conclude about MDMs. The perplexity gap between MDMs and autoregressive (AR) models shrinks by up to 32% in-domain and 82% zero-shot. In scaling laws, the reported 16x MDM-AR compute gap falls to 9x under DUEL, showing over 1/3 of it is an evaluation artifact. Scoring multiple-choice answers with likelihood via DUEL instead of the ELBO improves LLaDA-8B accuracy on all six tasks, by up to 9.7 points. DUEL also selects samplers correctly: it matches a gold-standard LLM judge on every decided comparison, which generative perplexity and MAUVE fail to do. DUEL reveals that MDMs are more competitive with AR models than the ELBO suggests.
查看此版本解读 →查看此版本原文 ↗