← 研踪 PaperTrailarXiv:2610.41983 · v2
TwinPast: Benchmarking Historical Evidence Use in Streaming Video Understanding with Evidence Contrastive Video Pairs
首次提交 2026-10-31 00:31
版本更新 2026-11-15 15:09
雷达观测 2026-11-17 19:15
摘要
High accuracy on streaming video benchmarks does not show that a model uses visual history. Auditing OVO-Bench and StreamingBench, we find that recent frames alone suffice for 32% of sampled historical-task questions, and that the question and options alone also yield many correct answers. We introduce TwinPast, a benchmark of 350 human-reviewed video pairs spanning five historical tasks. The two videos in a pair share the recent video context, question, and ordered options, and differ in earlier events and therefore in the correct answer. Each video is evaluated independently and PairAcc requires both answers to be correct, so an answer determined by the shared inputs cannot solve a pair. Masking the annotated critical evidence lowers PairAcc from as high as 91% to at most 19%, whereas equal-area masking elsewhere changes it by at most 3.4 points. The strongest streaming system reaches 49% PairAcc, while general-purpose MLLMs given each complete video reach 67-91%, leaving substantial headroom. Streaming systems fail mainly by giving both histories the same answer. They give different valid answers on at most 57% of pairs, while the general-purpose MLLMs do so on 79-94%. Case studies further show wrong answers when the deciding evidence is still available at the final query.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-15 15:09 · 最新观测版本
TwinPast: Benchmarking Historical Evidence Use in Streaming Video Understanding with Evidence Contrastive Video Pairs
High accuracy on streaming video benchmarks does not show that a model uses visual history. Auditing OVO-Bench and StreamingBench, we find that recent frames alone suffice for 32% of sampled historical-task questions, and that the question and options alone also yield many correct answers. We introduce TwinPast, a benchmark of 350 human-reviewed video pairs spanning five historical tasks. The two videos in a pair share the recent video context, question, and ordered options, and differ in earlier events and therefore in the correct answer. Each video is evaluated independently and PairAcc requires both answers to be correct, so an answer determined by the shared inputs cannot solve a pair. Masking the annotated critical evidence lowers PairAcc from as high as 91% to at most 19%, whereas equal-area masking elsewhere changes it by at most 3.4 points. The strongest streaming system reaches 49% PairAcc, while general-purpose MLLMs given each complete video reach 67-91%, leaving substantial headroom. Streaming systems fail mainly by giving both histories the same answer. They give different valid answers on at most 57% of pairs, while the general-purpose MLLMs do so on 79-94%. Case studies further show wrong answers when the deciding evidence is still available at the final query.
查看此版本解读 →查看此版本原文 ↗