← 研踪 PaperTrailarXiv:2611.10541 · v1
AFVE-Bench: Measuring Abandoned-Frame Verbalization Errors in Multi-Turn LLM Revision
首次提交 2026-11-07 15:17
版本更新 2026-11-07 15:17
雷达观测 2026-11-10 11:14
摘要
Multi-turn revision often asks a model to replace an earlier organizing frame while preserving still-valid facts and artifact requirements. Existing instruction-following evaluations mainly assess the latest target and may therefore score an artifact as successful even when it adopts the new frame but exposes the superseded one. We call this failure an abandoned-frame verbalization error (AFVE). AFVE-Bench evaluates silent frame replacement in English and Chinese using 192 matched four-turn dialogues across eight writing domains. Each seed has zero, one, or two pre-target user requests to generate under the source frame. AFVE eligibility depends on the user-request history and applies to the one- and two-request conditions; the zero-request condition serves as a frame-bearing baseline. Eleven model configurations yielded 4,224 expected post-target response units. Two annotators independently scored 4,204 scorable responses for target adoption, source-frame verbalization, and direct usability; model and condition metadata were hidden, and disagreements were adjudicated. AFVE occurs in 25.54% (715/2,799) of history-conditioned responses. Of 2,388 target-passing history-conditioned responses, 472 (19.77%) still expose the abandoned source frame, most often through contrastive framing (423/472). For these standalone revision tasks, target adoption does not ensure source-frame silence.
作者备注:18 pages, 4 figures
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-07 15:17 · 最新观测版本
AFVE-Bench: Measuring Abandoned-Frame Verbalization Errors in Multi-Turn LLM Revision
Multi-turn revision often asks a model to replace an earlier organizing frame while preserving still-valid facts and artifact requirements. Existing instruction-following evaluations mainly assess the latest target and may therefore score an artifact as successful even when it adopts the new frame but exposes the superseded one. We call this failure an abandoned-frame verbalization error (AFVE). AFVE-Bench evaluates silent frame replacement in English and Chinese using 192 matched four-turn dialogues across eight writing domains. Each seed has zero, one, or two pre-target user requests to generate under the source frame. AFVE eligibility depends on the user-request history and applies to the one- and two-request conditions; the zero-request condition serves as a frame-bearing baseline. Eleven model configurations yielded 4,224 expected post-target response units. Two annotators independently scored 4,204 scorable responses for target adoption, source-frame verbalization, and direct usability; model and condition metadata were hidden, and disagreements were adjudicated. AFVE occurs in 25.54% (715/2,799) of history-conditioned responses. Of 2,388 target-passing history-conditioned responses, 472 (19.77%) still expose the abandoned source frame, most often through contrastive framing (423/472). For these standalone revision tasks, target adoption does not ensure source-frame silence.
查看此版本解读 →查看此版本原文 ↗