← 研踪 PaperTrailarXiv:2611.22329 · v1
M$^3$-Test: Coverage Gaps in Joint-Perturbation Testing of Visual Question Answering
首次提交 2026-11-17 15:27
版本更新 2026-11-17 15:27
雷达观测 2026-11-18 19:43
摘要
A visual question answering (VQA) model can pass every image-only and question-only test yet fail when the same changes occur together. Standard robustness reports average over conditions and therefore cannot say which accepted requests remain vulnerable. We introduce M3-Test, a coverage audit that aligns each clean input with its single-modality variants and a complete 5x5 family, 4x4 strength joint library, so that any reduced test suite can be replayed on the same requests. Across 2.6 million responses from four vision-language models, joint failures affect 26-29% of the Qwen questions that pass all 40 single-modality tests, although fewer than 0.6% of their joint records fail. Passing the 25 maximum-strength corners does not close this gap: 11-14% of the accepted questions still fail elsewhere in the library. Re-executing the archived failures reproduces about 90% of them, against a 0.3-0.6% flip rate among matched passing controls, and re-checking the most fragile failure records supports 80-89% of them. On the evidence examined, neither execution noise nor grading error can fully account for the gap. The same aligned grids make previous evaluations reusable. On held-out images, a risk order learned from other models discovers about 80% of affected Qwen questions within 25 additional queries, compared with 65-68% for maximum-first and 53-57% for pairwise covering arrays. M3-Test thus turns a passing robustness suite into a request-level account of what remains untested, supporting acceptance, regression review and test allocation.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-17 15:27 · 最新观测版本
M$^3$-Test: Coverage Gaps in Joint-Perturbation Testing of Visual Question Answering
A visual question answering (VQA) model can pass every image-only and question-only test yet fail when the same changes occur together. Standard robustness reports average over conditions and therefore cannot say which accepted requests remain vulnerable. We introduce M3-Test, a coverage audit that aligns each clean input with its single-modality variants and a complete 5x5 family, 4x4 strength joint library, so that any reduced test suite can be replayed on the same requests. Across 2.6 million responses from four vision-language models, joint failures affect 26-29% of the Qwen questions that pass all 40 single-modality tests, although fewer than 0.6% of their joint records fail. Passing the 25 maximum-strength corners does not close this gap: 11-14% of the accepted questions still fail elsewhere in the library. Re-executing the archived failures reproduces about 90% of them, against a 0.3-0.6% flip rate among matched passing controls, and re-checking the most fragile failure records supports 80-89% of them. On the evidence examined, neither execution noise nor grading error can fully account for the gap. The same aligned grids make previous evaluations reusable. On held-out images, a risk order learned from other models discovers about 80% of affected Qwen questions within 25 additional queries, compared with 65-68% for maximum-first and 53-57% for pairwise covering arrays. M3-Test thus turns a passing robustness suite into a request-level account of what remains untested, supporting acceptance, regression review and test allocation.
查看此版本解读 →查看此版本原文 ↗