← 研踪 PaperTrailarXiv:2510.07441 · v3
A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion
首次提交 2025-10-09 02:41
版本更新 2026-10-27 04:02
雷达观测 2026-10-29 00:35
摘要
Existing text-to-video (T2V) evaluation benchmarks, such as VBench and EvalCrafter, predominantly contain either subject-centric prompts or static camera scenes, overlooking the importance of camera motion, essential for producing cinematic shots with realistic scenes. Due to this, the metrics developed in these works often produce incorrect results in the presence of camera motion. To fill this gap, we introduce DynamicEval, a benchmark comprising of systematically curated prompts that emphasize dynamic camera motion, paired with 45k human annotations on video pairs. DynamicEval evaluates two key dimensions of video quality: the consistency of the foreground subjects and the consistency of the background scene. To address the shortcomings of existing metrics under camera motion, we introduce two novel metrics: 1) Tracker-FG, which evaluates subject consistency by assessing the keypoint tracks of the subject. 2) Debiased Motion Smoothness, for background consistency, devoid from issues of occlusions/disocclusions arising from camera and foreground subject movements. Extensive experiments on the large-scale DynamicEval benchmark demonstrate that our proposed metrics achieve stronger correlations with human preferences at both the video level, and the model level with an increase of 0.32 in average Pearson Linear Correlation.
作者备注：Accepted at WACV 2027. 22 pages, 15 figures, 13 tables. Access the project page in https://nithincbabu7.github.io/DynamicEval
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-10-27 04:02 · 最新观测版本
A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion
Existing text-to-video (T2V) evaluation benchmarks, such as VBench and EvalCrafter, predominantly contain either subject-centric prompts or static camera scenes, overlooking the importance of camera motion, essential for producing cinematic shots with realistic scenes. Due to this, the metrics developed in these works often produce incorrect results in the presence of camera motion. To fill this gap, we introduce DynamicEval, a benchmark comprising of systematically curated prompts that emphasize dynamic camera motion, paired with 45k human annotations on video pairs. DynamicEval evaluates two key dimensions of video quality: the consistency of the foreground subjects and the consistency of the background scene. To address the shortcomings of existing metrics under camera motion, we introduce two novel metrics: 1) Tracker-FG, which evaluates subject consistency by assessing the keypoint tracks of the subject. 2) Debiased Motion Smoothness, for background consistency, devoid from issues of occlusions/disocclusions arising from camera and foreground subject movements. Extensive experiments on the large-scale DynamicEval benchmark demonstrate that our proposed metrics achieve stronger correlations with human preferences at both the video level, and the model level with an increase of 0.32 in average Pearson Linear Correlation.
查看此版本解读 →查看此版本原文 ↗