← 研踪 PaperTrailarXiv:2607.12304 · v2
What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation
首次提交 2026-07-14 11:26
版本更新 2026-11-18 13:07
雷达观测 2026-11-19 19:08
摘要
A score on a temporal video question answering benchmark is meant to measure that a model has temporal understanding, but it conflates two questions. (1)~is the question even temporal, does it need several frames and their order? And (2)~when it does, does the model recover the order from the pixels, or read it off the positional encoding (RoPE)? Most of a temporal score answers neither, since a single frame and answer priors often carry it, and the field's validity checks, frame-shuffle sensitivity and multi-frame gain, speak only to (1). For (2) we score a reversal conflict, frames reversed while RoPE stays forward, two ways: label-free, as the accuracy lost against the forward labels, and against reversed labels wherever labels transform deterministically under time reversal. This separates a model that follows the reversed content from one merely disrupted. The two readings dissociate two similarly scoring models at both model scales. Qwen3-VL answers the reversed event it actually sees, reading the visual sequence. Molmo2 correctly reports a backwards video when positions agree, and flipping only the RoPE indices flips its prediction to the forward events, i.e. it reads positions. We call these regimes \emph{visual-sequence-dominant} and \emph{position-dominant}. Controls show reversed content is processable whenever the channels agree, and that both families share a bias toward describing events as running forward (arrow-of-time prior) that, being similar in strength, cannot by itself account for the dissociation. The distinction matters because the two channels fail on opposite inputs: two models with the same score are not interchangeable.
作者备注:Accepted at the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2027, Datasets and Evaluation Track
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-19 13:14 · 最新观测版本
What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation
A score on a temporal video question answering benchmark is meant to measure that a model has temporal understanding, but it conflates two questions. (1) is the question even temporal, does it need several frames and their order? And (2) when it does, does the model recover the order from the pixels, or read it off the positional encoding (RoPE)? Most of a temporal score answers neither, since a single frame and answer priors often carry it, and the field's validity checks, frame-shuffle sensitivity and multi-frame gain, speak only to (1). For (2) we score a reversal conflict, frames reversed while RoPE stays forward, two ways: label-free, as the accuracy lost against the forward labels, and against reversed labels wherever labels transform deterministically under time reversal. This separates a model that follows the reversed content from one merely disrupted. The two readings dissociate two similarly scoring models at both model scales. Qwen3-VL answers the reversed event it actually sees, reading the visual sequence. Molmo2 correctly reports a backwards video when positions agree, and flipping only the RoPE indices flips its prediction to the forward events, i.e. it reads positions. We call these regimes visual-sequence-dominant and position-dominant. Controls show reversed content is processable whenever the channels agree, and that both families share a bias toward describing events as running forward (arrow-of-time prior) that, being similar in strength, cannot by itself account for the dissociation. The distinction matters because the two channels fail on opposite inputs: two models with the same score are not interchangeable.
查看此版本解读 →查看此版本原文 ↗v2 · 2026-11-18 13:07
What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation
A score on a temporal video question answering benchmark is meant to measure that a model has temporal understanding, but it conflates two questions. (1)~is the question even temporal, does it need several frames and their order? And (2)~when it does, does the model recover the order from the pixels, or read it off the positional encoding (RoPE)? Most of a temporal score answers neither, since a single frame and answer priors often carry it, and the field's validity checks, frame-shuffle sensitivity and multi-frame gain, speak only to (1). For (2) we score a reversal conflict, frames reversed while RoPE stays forward, two ways: label-free, as the accuracy lost against the forward labels, and against reversed labels wherever labels transform deterministically under time reversal. This separates a model that follows the reversed content from one merely disrupted. The two readings dissociate two similarly scoring models at both model scales. Qwen3-VL answers the reversed event it actually sees, reading the visual sequence. Molmo2 correctly reports a backwards video when positions agree, and flipping only the RoPE indices flips its prediction to the forward events, i.e. it reads positions. We call these regimes \emph{visual-sequence-dominant} and \emph{position-dominant}. Controls show reversed content is processable whenever the channels agree, and that both families share a bias toward describing events as running forward (arrow-of-time prior) that, being similar in strength, cannot by itself account for the dissociation. The distinction matters because the two channels fail on opposite inputs: two models with the same score are not interchangeable.
查看此版本解读 →查看此版本原文 ↗