跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.26413 · v1

Sequential Anytime Valid Tests for Efficient Inference-Time Alignment

首次提交 2026-11-20 01:00
版本更新 2026-11-20 01:00
雷达观测 2026-11-20 12:52
arXiv ↗PDF ↗

摘要

Inference-time alignment methods, such as Best-of-N (BoN), improve language model outputs by generating multiple candidates per prompt and selecting among them using a reward model. Much of this cost is wasted: a fixed budget is applied to every prompt, although the computation required by prompts varies widely. Adaptive methods halt once sampled answers appear settled, but their thresholds are tuned heuristics with no statistical control over early-stopping errors. We instead seek an anytime-valid stopping rule that takes an error tolerance αα as input and halts only once an answer is certified to hold a majority (θa>1/2θ_a > 1/2) under the generator's distribution. We aggregate candidates by final answer with a reward-weighted self-consistency (RWSC) score and treat stopping as a sequential test over the resulting clusters. Applying Wald's sequential probability ratio test to these clusters yields SPRT-RWSC, which performs well empirically, but its guarantee fails because the leading answer shifts during sampling, and the run is continuously monitored. Our main contribution, e-RWSC, replaces Wald's test with a betting supermartingale (an e-process) for every answer the policy produces, bounding the probability of ever certifying a non-dominant answer by αα, uniformly over time and across all answers. The RWSC scores drive the bets, so they affect only how quickly a dominant answer a⋆a^\star is certified, not validity; certification takes O(log⁡(1/α)/dKL(θa⋆∥1/2))O(\log(1/α)/d_{\mathrm{KL}}(θ_{a^\star}\|1/2)) samples. Experiments across several language models, datasets, and reward models confirm the predicted 1/dKL1/d_{\mathrm{KL}} scaling of cost and show that accuracy improves over baselines at substantially lower token cost.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-20 01:00 · 最新观测版本

Sequential Anytime Valid Tests for Efficient Inference-Time Alignment

Inference-time alignment methods, such as Best-of-N (BoN), improve language model outputs by generating multiple candidates per prompt and selecting among them using a reward model. Much of this cost is wasted: a fixed budget is applied to every prompt, although the computation required by prompts varies widely. Adaptive methods halt once sampled answers appear settled, but their thresholds are tuned heuristics with no statistical control over early-stopping errors. We instead seek an anytime-valid stopping rule that takes an error tolerance αα as input and halts only once an answer is certified to hold a majority (θa>1/2θ_a > 1/2) under the generator's distribution. We aggregate candidates by final answer with a reward-weighted self-consistency (RWSC) score and treat stopping as a sequential test over the resulting clusters. Applying Wald's sequential probability ratio test to these clusters yields SPRT-RWSC, which performs well empirically, but its guarantee fails because the leading answer shifts during sampling, and the run is continuously monitored. Our main contribution, e-RWSC, replaces Wald's test with a betting supermartingale (an e-process) for every answer the policy produces, bounding the probability of ever certifying a non-dominant answer by αα, uniformly over time and across all answers. The RWSC scores drive the bets, so they affect only how quickly a dominant answer a⋆a^\star is certified, not validity; certification takes O(log⁡(1/α)/dKL(θa⋆∥1/2))O(\log(1/α)/d_{\mathrm{KL}}(θ_{a^\star}\|1/2)) samples. Experiments across several language models, datasets, and reward models confirm the predicted 1/dKL1/d_{\mathrm{KL}} scaling of cost and show that accuracy improves over baselines at substantially lower token cost.

查看此版本解读 →查看此版本原文 ↗