← 研踪 PaperTrailarXiv:2607.14256 · v2
Disagreement-Guided Agentic Data Curation for Multimodal Safety
首次提交 2026-07-16 02:13
版本更新 2026-11-18 12:04
雷达观测 2026-11-19 19:08
摘要
Written safety policies in multimodal settings define categories whose visual boundaries span many contexts. We study disagreement as a signal for identifying useful synthetic boundary cases. Our pipeline generates policy-conditioned images, selects cases with target--rater or inter-rater disagreement, conditionally escalates unresolved cases, and retrieves the resulting rater-labeled examples as in-context demonstrations. On a failure-enriched multimodal safety benchmark, policy grounding reduces false negative rate (FNR) from 0.875 to 0.413 and raises F1 from 0.220 to 0.731; retrieval from 50 disagreement-selected examples further reduces FNR to 0.245, compared with 0.297 from a 350-example Regular pool. The selected memory also improves a different target-model family, with a smaller gain on a proprietary benchmark with a refined policy. In a blinded human study, experts disagree on 24% of disagreement-guided images and 17% of baseline-curated images. These findings support traceable, disagreement-guided curation for multimodal safety evaluation and example selection.
作者备注:9 pages; published in 2026 IEEE International Conference on Big Data
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-18 12:04 · 最新观测版本
Disagreement-Guided Agentic Data Curation for Multimodal Safety
Written safety policies in multimodal settings define categories whose visual boundaries span many contexts. We study disagreement as a signal for identifying useful synthetic boundary cases. Our pipeline generates policy-conditioned images, selects cases with target--rater or inter-rater disagreement, conditionally escalates unresolved cases, and retrieves the resulting rater-labeled examples as in-context demonstrations. On a failure-enriched multimodal safety benchmark, policy grounding reduces false negative rate (FNR) from 0.875 to 0.413 and raises F1 from 0.220 to 0.731; retrieval from 50 disagreement-selected examples further reduces FNR to 0.245, compared with 0.297 from a 350-example Regular pool. The selected memory also improves a different target-model family, with a smaller gain on a proprietary benchmark with a refined policy. In a blinded human study, experts disagree on 24% of disagreement-guided images and 17% of baseline-curated images. These findings support traceable, disagreement-guided curation for multimodal safety evaluation and example selection.
查看此版本解读 →查看此版本原文 ↗