← 研踪 PaperTrailarXiv:2611.10040 · v1
SPRING-5: Measuring Tool Documentation Repair for LLM Agents by Efficiency
首次提交 2026-11-06 03:26
版本更新 2026-11-06 03:26
雷达观测 2026-11-10 10:12
摘要
Methods that repair tool documentation automatically are mostly evaluated on task accuracy. We report a benchmark that scores them on cost. SPRING-5 is 20 tool families and 344 tasks, each with a deliberately degraded specification, a hand-written honest one, a deterministic backend and an answer key, scored on calls charged against the ideal call path. A wrong answer is charged the full call budget, so this cost index largely tracks the completion rate, and we report calls per correct task beside it, which carries no such charge. Over 2,644 baseline sessions, 54% of wrong answers arrive with no rejected call, no refusal and no retry. The benchmark carries three validity instruments. A build-time ceiling assertion requires the honest specification to reach the ideal path and fails systematically on only one family. A dud rule names the families a stronger model solves cold and catches four. A directly measured noise floor shows that two evaluations of a byte-identical specification differ by a 95% factor of 0.64 to 1.56, so about one comparison in eight crosses a conventional win threshold on noise alone, and the cheap way of estimating that floor is biased by which families happen to go unedited. Against these instruments, a schema-only method closes 1% of the baseline-to-ideal cost gap, an online self-evolution loop 13%, one batch rewrite over every training transcript 33%, and a variant given correctness labels 58%. The batch advantage over the loop is the evidence it reads: held to the loop's exposure it is no longer detectable. With five and ten tools in context the effect survives, the baseline cost does not change, and only machine-written descriptions sent calls to the wrong tool. On BFCL, 62% of held-out queries are already answered from the shipped documentation, which explains a pooled null there. Code, data and runs are released.
作者备注：25 pages, 5 figures. Code and data: https://github.com/CoronRing/DATE
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-06 03:26 · 最新观测版本
SPRING-5: Measuring Tool Documentation Repair for LLM Agents by Efficiency
Methods that repair tool documentation automatically are mostly evaluated on task accuracy. We report a benchmark that scores them on cost. SPRING-5 is 20 tool families and 344 tasks, each with a deliberately degraded specification, a hand-written honest one, a deterministic backend and an answer key, scored on calls charged against the ideal call path. A wrong answer is charged the full call budget, so this cost index largely tracks the completion rate, and we report calls per correct task beside it, which carries no such charge. Over 2,644 baseline sessions, 54% of wrong answers arrive with no rejected call, no refusal and no retry. The benchmark carries three validity instruments. A build-time ceiling assertion requires the honest specification to reach the ideal path and fails systematically on only one family. A dud rule names the families a stronger model solves cold and catches four. A directly measured noise floor shows that two evaluations of a byte-identical specification differ by a 95% factor of 0.64 to 1.56, so about one comparison in eight crosses a conventional win threshold on noise alone, and the cheap way of estimating that floor is biased by which families happen to go unedited. Against these instruments, a schema-only method closes 1% of the baseline-to-ideal cost gap, an online self-evolution loop 13%, one batch rewrite over every training transcript 33%, and a variant given correctness labels 58%. The batch advantage over the loop is the evidence it reads: held to the loop's exposure it is no longer detectable. With five and ten tools in context the effect survives, the baseline cost does not change, and only machine-written descriptions sent calls to the wrong tool. On BFCL, 62% of held-out queries are already answered from the shipped documentation, which explains a pooled null there. Code, data and runs are released.
查看此版本解读 →查看此版本原文 ↗