← 研踪 PaperTrailarXiv:2603.13389 · v2
Logit-Conditioned Diffusion Decoding for Frozen Discrete-Token VLMs
首次提交 2026-03-11 15:02
版本更新 2026-11-09 15:29
雷达观测 2026-11-10 13:18
摘要
Pre-trained discrete-token VLMs generate images by sampling sequences of codebook indices, but their visual fidelity is bounded by the VQ-VAE decoding stage. Prior work addresses this bottleneck by replacing the discrete tokenizer with continuous or hybrid visual token representations, or by integrating diffusion into the generation pipeline; both require foundation-scale VLM retraining or realignment. We show this is unnecessary: before each token is sampled, discrete-token VLMs already compute a full logit distribution over the codebook, but standard decoding discards it after selecting a single index. Under Otsu's threshold, the pre-sampling logits of capable VLMs assign statistically meaningful probability to at least two codebook entries per token, and recovering this distribution can substantially close the fidelity gap. We propose DCDD, a post-hoc framework that conditions a diffusion decoder on this discarded distribution while leaving the VLM frozen. At inference, DCDD converts the VLM's pre-sampling logits into distribution-weighted code vectors; during training, it uses VQ-VAE encoder-derived proxy logits calibrated to match the distributional statistics of the VLM's inference-time logits. The decoder maps these representations to high-fidelity images as an alternative to the native VQ-VAE decoder. Trained on ImageNet-1K for 50K steps, DCDD reduces reconstruction FID by 74% and Janus-Pro 7B's generation FID on MJHQ-30K by 34%, while semantic alignment across GenEval, DPG-Bench, and WISE remain consistent. Controlled comparisons with diffusion models from the same backbone family, applied either as an image-to-image refiner or as an identically trained decoder on the sampled codes, confirm that the gain stems from conditioning on the VLM's pre-sampling logit distribution rather than from the diffusion model itself.
作者备注：Accepted at NeurIPS 2026. Revised title, author order, and manuscript. Code: https://github.com/jiwoohong93/dcdd_code
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-09 15:29 · 最新观测版本
Logit-Conditioned Diffusion Decoding for Frozen Discrete-Token VLMs
Pre-trained discrete-token VLMs generate images by sampling sequences of codebook indices, but their visual fidelity is bounded by the VQ-VAE decoding stage. Prior work addresses this bottleneck by replacing the discrete tokenizer with continuous or hybrid visual token representations, or by integrating diffusion into the generation pipeline; both require foundation-scale VLM retraining or realignment. We show this is unnecessary: before each token is sampled, discrete-token VLMs already compute a full logit distribution over the codebook, but standard decoding discards it after selecting a single index. Under Otsu's threshold, the pre-sampling logits of capable VLMs assign statistically meaningful probability to at least two codebook entries per token, and recovering this distribution can substantially close the fidelity gap. We propose DCDD, a post-hoc framework that conditions a diffusion decoder on this discarded distribution while leaving the VLM frozen. At inference, DCDD converts the VLM's pre-sampling logits into distribution-weighted code vectors; during training, it uses VQ-VAE encoder-derived proxy logits calibrated to match the distributional statistics of the VLM's inference-time logits. The decoder maps these representations to high-fidelity images as an alternative to the native VQ-VAE decoder. Trained on ImageNet-1K for 50K steps, DCDD reduces reconstruction FID by 74% and Janus-Pro 7B's generation FID on MJHQ-30K by 34%, while semantic alignment across GenEval, DPG-Bench, and WISE remain consistent. Controlled comparisons with diffusion models from the same backbone family, applied either as an image-to-image refiner or as an identically trained decoder on the sampled codes, confirm that the gain stems from conditioning on the VLM's pre-sampling logit distribution rather than from the diffusion model itself.
查看此版本解读 →查看此版本原文 ↗