跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.20732 · v1

Routing Beyond Task Categories: A Cost-Aware LAB-Bench Study

首次提交 2026-11-16 16:38
版本更新 2026-11-16 16:38
雷达观测 2026-11-17 13:07
arXiv ↗PDF ↗

摘要

We tested whether features of biology questions or of a model's first answer can improve accuracy within an inference budget. In a LAB-Bench pilot, we compared a category router with one that adds six structural features. Neither reached the primary 80% accuracy target on development. At the 50% and 60% secondary targets, both chose Haiku for all 120 held-out questions and scored 71/120. We then froze a new comparison before collecting responses on 144 additional questions. Semantic routing scored 86/144. A policy that could defer after reading Haiku's answer scored 87/144. Both stayed within the inference budget of USD 0.678 per 1,000 queries. Against a 95% Haiku and 5% GPT mixture, the accuracy gains were 0.66 percentage points for semantic routing and 1.35 points for deferral. The paired 95% intervals were [-0.28,2.25] and [-0.14,3.47] points. A simple rule that escalated Haiku nonanswers scored 92/144 at USD 0.710 per 1,000 queries, above the budget. A retrospective nonanswer rule with a calibration-selected escalation probability gave an expected 85.63/144. At the learned deferral policy's observed inference cost, post hoc interpolation gave an expected 86.14/144. In a later development check, Astra and Gemini each scored 30/40. The best tested policy in each router family matched that score. The pair covered 33/40 in hindsight. The routers missed the 32/40 criterion, so the proposed 120-question evaluation did not start.

作者备注:21 pages, 3 figures. Reproduction archive at https://doi.org/10.5281/zenodo.24220501

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-16 16:38 · 最新观测版本

Routing Beyond Task Categories: A Cost-Aware LAB-Bench Study

We tested whether features of biology questions or of a model's first answer can improve accuracy within an inference budget. In a LAB-Bench pilot, we compared a category router with one that adds six structural features. Neither reached the primary 80% accuracy target on development. At the 50% and 60% secondary targets, both chose Haiku for all 120 held-out questions and scored 71/120. We then froze a new comparison before collecting responses on 144 additional questions. Semantic routing scored 86/144. A policy that could defer after reading Haiku's answer scored 87/144. Both stayed within the inference budget of USD 0.678 per 1,000 queries. Against a 95% Haiku and 5% GPT mixture, the accuracy gains were 0.66 percentage points for semantic routing and 1.35 points for deferral. The paired 95% intervals were [-0.28,2.25] and [-0.14,3.47] points. A simple rule that escalated Haiku nonanswers scored 92/144 at USD 0.710 per 1,000 queries, above the budget. A retrospective nonanswer rule with a calibration-selected escalation probability gave an expected 85.63/144. At the learned deferral policy's observed inference cost, post hoc interpolation gave an expected 86.14/144. In a later development check, Astra and Gemini each scored 30/40. The best tested policy in each router family matched that score. The pair covered 33/40 in hindsight. The routers missed the 32/40 criterion, so the proposed 120-question evaluation did not start.

查看此版本解读 →查看此版本原文 ↗