RepFusion:利用多模态先验在表示空间中进行去噪
RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space
Xichen Pan, Aashu Singh, Satya Narayan Shukla, Xiangjun Fan, Shlok Kumar Mishra, Saining Xie
AI 速读 基于摘要
将多模态大语言模型(MLLM)复用为含噪视觉表示的编码器,以其输出作为扩散变换器的条件信号,在表示空间完成去噪生成。
cs.CV
查看原文摘要
Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative backbones. The emergence of representation autoencoders (RAEs) shifts the generation target toward semantically structured visual representations, creating a latent space that is more compatible with pretrained LLM priors. Inspired by multimodal LLMs (MLLMs), where an MLP projector is sufficient to align clean visual representations with a pretrained LLM, we repurpose the MLLM itself as a noisy representation encoder, extending this mechanism from clean to noisy inputs. We present RepFusion, which uses the resulting MLLM outputs as the conditioning signal for a diffusion transformer. In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that devote comparable capacity to newly initialized denoisers. Reusing MLLM outputs across denoising steps further accelerates inference while retaining performance comparable to per-step MLLM recomputation. These results demonstrate that MLLMs provide strong priors for denoising visual representations.