K2V:面向物理合理视频生成的生成式关键帧方法
K2V: Generative Keyframing For Physically Plausible Video Generation
Felix O'Mahony, Daniel Olmeda Reino, Roberto Cipolla, Ayush Tewari
AI 速读 基于摘要
将物理推理交给语言模型驱动的关键帧生成模块,再让视频扩散模型只负责关键帧间插值,以提升生成视频的物理合理性。
cs.CV
查看原文摘要
Video diffusion models are able to generate samples with high-quality appearance and textural qualities. However, content produced by these methods is known to exhibit significant physical errors. This is because, in a classical video generation setting, all physical reasoning must take place simultaneously and exclusively within the diffusion process.We propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which allows for explicit reasoning in language, and fine-grained control over the appearance of the video at important moments in the video. We then condition a video generation model with these keyframes, reducing the role of the diffusion model to the simpler task of interpolating between physically-correct intermediate states, rather than inferring an entire sequence with diffusion. Results demonstrate that our approach significantly outperforms existing approaches, achieving state-of-the-art performance on the Physics-IQ Verified benchmark and PhyGenBench.