JVT:仅用视频Transformer即可实现简单的像素空间视频扩散
JVT: Just Video Transformer makes pixel-space video diffusion simple
Jianjun Zhou, Hua Geng, Haoyu Guo
AI 速读 基于摘要
作者提出JVT,一种无需预训练自编码器、辅助细节器、级联阶段或U-Net式跳跃连接的简单大块DiT式Transformer,用于像素空间视频扩散,并在类别条件与无条件生成上取得与潜在扩散基线相当的质量。
cs.CV
查看原文摘要
Direct diffusion modeling of high-dimensional raw video data is challenging, and existing pixel-space methods often rely on specialized architectures. We introduce JVT (Just Video Transformer), a simple large-patch DiT-style Transformer for pixel-space video diffusion, without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net-style skip connections. We systematically examine video patch embeddings, register tokens, and adaptive layer normalization to identify design choices that balance generation quality and computational efficiency. The resulting JVT design achieves generation quality competitive with latent diffusion baselines in both class-conditioned and unconditional settings. The pixel-space formulation of JVT also naturally supports joint generation of pixel-aligned modalities. We demonstrate this capability through camera-conditioned RGB-disparity generation, which improves camera following relative to RGB-only baselines and yields explicit scene geometry without an additional reconstruction model at inference. These results highlight the potential of simple DiT-style Transformers for pixel-space video diffusion, and we hope our findings offer useful insights and practical guidance for future research.