Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards
Shaoan Xie Carnegie Mellon University, Lingjing Kong Computer Science Department, School of Computer Science, Xiangchen Song Carnegie Mellon University, Xinshuai Dong, Guangyi Chen Mohamed bin Zayed University of Artificial Intelligence and Carnegie Mellon University, Eric P. Xing, Kun Zhang Carnegie Mellon University and Mohamed bin Zayed University of Artificial Intelligence
Diffusion-based large language models offer a non-autoregressive alternative for text generation, but enabling them to perform complex reasoning remains challenging. Reinforcement learning has recently emerged as an effective post-training strategy for improving their performance; however, existing methods rely primarily on outcome-based rewards, which provide no direct supervision over the denoising process and often result in poorly structured reasoning that is difficult to interpret and inconsistently supports the final prediction. To address this limitation, we introduce denoising process reward, a process-level reinforcement signal defined over the denoising trajectory of diffusion language models. This reward is obtained by estimating the contribution of intermediate denoising intervals to the final task outcome, encouraging the model to favor reasoning trajectories that consistently guide generation toward correct predictions. We further propose an efficient stochastic estimator that reuses standard training rollouts, enabling practical process-level supervision at scale. Experiments on challenging reasoning benchmarks demonstrate that our approach yields consistent improvements in reasoning stability, interpretability, and overall task performance.
尚无 AI 解读，展示原文摘要
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表