← 研踪 PaperTrailarXiv:2609.03688 · v2
ToPO: Token-Oriented Preference Optimization for Text-to-Image Diffusion Models
首次提交 2026-09-03 19:24
版本更新 2026-10-27 19:36
雷达观测 2026-10-29 00:35
摘要
Diffusion models have achieved remarkable success in text-to-image generation, yet aligning their outputs with human preferences remains challenging. Diffusion-DPO learns from image preference pairs, but their global labels do not specify how supervision should be allocated across spatial locations and denoising timesteps. The semantic roles of prompt tokens are also not explicitly considered when assigning these local weights. To address these limitations, we propose Token-Oriented Preference Optimization (ToPO), which builds a token-conditioned spatial-temporal weighting map for each pair. A frozen reference denoiser obtains spatial and temporal weights from local noise-prediction error differences under shared noise, while preferred-branch cross-attention links spatial evidence to content tokens. The map reweights Diffusion-DPO and supports an auxiliary pixel-midpoint ordering loss. It requires neither local preference annotations nor a separately trained reward model. Extensive experiments across multiple preference and compositional evaluations show that ToPO achieves state-of-the-art performance among the compared methods with only 500 training updates, demonstrating the potential of token-conditioned preference allocation for high-quality diffusion alignment within a limited update budget.
作者备注：28 pages, 11 figures. Revised manuscript with updated presentation and analyses
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-27 19:36 · 最新观测版本
ToPO: Token-Oriented Preference Optimization for Text-to-Image Diffusion Models
Diffusion models have achieved remarkable success in text-to-image generation, yet aligning their outputs with human preferences remains challenging. Diffusion-DPO learns from image preference pairs, but their global labels do not specify how supervision should be allocated across spatial locations and denoising timesteps. The semantic roles of prompt tokens are also not explicitly considered when assigning these local weights. To address these limitations, we propose Token-Oriented Preference Optimization (ToPO), which builds a token-conditioned spatial-temporal weighting map for each pair. A frozen reference denoiser obtains spatial and temporal weights from local noise-prediction error differences under shared noise, while preferred-branch cross-attention links spatial evidence to content tokens. The map reweights Diffusion-DPO and supports an auxiliary pixel-midpoint ordering loss. It requires neither local preference annotations nor a separately trained reward model. Extensive experiments across multiple preference and compositional evaluations show that ToPO achieves state-of-the-art performance among the compared methods with only 500 training updates, demonstrating the potential of token-conditioned preference allocation for high-quality diffusion alignment within a limited update budget.
查看此版本解读 →查看此版本原文 ↗