跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.24159 · v1

Fast Bayesian Methods for Uncertainty Quantification in Sparse AI Evaluation

首次提交 2026-11-18 18:02
版本更新 2026-11-18 18:02
雷达观测 2026-11-19 15:04
arXiv ↗PDF ↗

摘要

Estimating how an AI system will perform on a new benchmark need not require another large model. We use public evaluation counts to choose a weak Beta prior, then update it exactly from at most 31 revealed outcomes. The predictor uses no item semantics, pretrained weights, or inference API. Its single online evaluation in the AIMS competition returned a team-reported Brier ALC of 0.188477, a 24.6% reduction from the constant-0.5 reference. The complete submission is 7,754 bytes; a server microbenchmark measures 12.71 microseconds per prediction with 31 matching labels. We describe how source evaluations are compressed into 301 prior means and show why a concentration of two lets new evidence rapidly determine the prediction. The resulting posterior gives exact uncertainty about the modelled success rate without numerical integration or neural inference. A public evaluation covers 79 system-benchmark pairs and eight seeds, reaching Brier ALC 0.200250 for the submitted method. Accompanying code and sufficient statistics reconstruct the fitted prior byte for byte and replay all 632 evaluation trajectories. The result demonstrates how much predictive value a compact Bayesian model can extract from sparse target feedback at very low online cost.

作者备注:13 pages, 5 figures. Accepted at NeurIPS 2026 AI Measurement Science Workshop

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-18 18:02 · 最新观测版本

Fast Bayesian Methods for Uncertainty Quantification in Sparse AI Evaluation

Estimating how an AI system will perform on a new benchmark need not require another large model. We use public evaluation counts to choose a weak Beta prior, then update it exactly from at most 31 revealed outcomes. The predictor uses no item semantics, pretrained weights, or inference API. Its single online evaluation in the AIMS competition returned a team-reported Brier ALC of 0.188477, a 24.6% reduction from the constant-0.5 reference. The complete submission is 7,754 bytes; a server microbenchmark measures 12.71 microseconds per prediction with 31 matching labels. We describe how source evaluations are compressed into 301 prior means and show why a concentration of two lets new evidence rapidly determine the prediction. The resulting posterior gives exact uncertainty about the modelled success rate without numerical integration or neural inference. A public evaluation covers 79 system-benchmark pairs and eight seeds, reaching Brier ALC 0.200250 for the submitted method. Accompanying code and sufficient statistics reconstruct the fitted prior byte for byte and replay all 632 evaluation trajectories. The result demonstrates how much predictive value a compact Bayesian model can extract from sparse target feedback at very low online cost.

查看此版本解读 →查看此版本原文 ↗