← 研踪 PaperTrailarXiv:2611.26592 · v1
Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning
实现可靠性 vs. 总体可靠性:相关推理下任意时间有效的自一致性认证
首次提交 2026-11-20 02:47
版本更新 2026-11-20 02:47
雷达观测 2026-11-20 12:52
摘要
Self-consistency certification treats repeated sampling from a language model as sequential evidence accumulation and stops when an anytime-valid e-process crosses a threshold, guaranteeing finite-sample Type I error control by Ville's inequality. Prior constructions assume sampled rollouts are conditionally independent. We show this assumption fails in practice: reasoning streams are positively correlated through shared reasoning prefixes or across-prompt heterogeneity, changing not merely certificate efficiency but the hypothesis certified. Modeling dependence via a per-unit latent state (the random-effects / intraclass-correlation structure) induces exchangeable, equi-correlated increments with correlation . Correlation splits the single null into a realized null about the drawn belief and a population null about average model behavior, which coincide only when . We show: (i) the standard betting e-process remains anytime-valid for the realized null, not the population null; (ii) read as a population-null test, a suitable mixture has asymptotic crossing probability at least , potentially exceeding nominal ; (iii) the within-unit mean's effective sample size saturates at , so extra within-unit rollouts cannot restore population validity; and (iv) a prompt-level e-process restores population validity, with optimal allocation at one rollout per prompt under a pure rollout budget. Simulations and Claude Haiku 4.5 rollouts confirm this regime dependence: on GSM8K (, ) the gap is dormant, whereas on AIME (, ) the single-unit rule crosses on 24% of prompts at nominal , while the prompt-level procedure declines (0.00). This correctness result identifies which reliability claim an anytime-valid certificate actually supports under dependence.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-20 02:47 · 最新观测版本
Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning
Self-consistency certification treats repeated sampling from a language model as sequential evidence accumulation and stops when an anytime-valid e-process crosses a threshold, guaranteeing finite-sample Type I error control by Ville's inequality. Prior constructions assume sampled rollouts are conditionally independent. We show this assumption fails in practice: reasoning streams are positively correlated through shared reasoning prefixes or across-prompt heterogeneity, changing not merely certificate efficiency but the hypothesis certified. Modeling dependence via a per-unit latent state (the random-effects / intraclass-correlation structure) induces exchangeable, equi-correlated increments with correlation . Correlation splits the single null into a realized null about the drawn belief and a population null about average model behavior, which coincide only when . We show: (i) the standard betting e-process remains anytime-valid for the realized null, not the population null; (ii) read as a population-null test, a suitable mixture has asymptotic crossing probability at least , potentially exceeding nominal ; (iii) the within-unit mean's effective sample size saturates at , so extra within-unit rollouts cannot restore population validity; and (iv) a prompt-level e-process restores population validity, with optimal allocation at one rollout per prompt under a pure rollout budget. Simulations and Claude Haiku 4.5 rollouts confirm this regime dependence: on GSM8K (, ) the gap is dormant, whereas on AIME (, ) the single-unit rule crosses on 24% of prompts at nominal , while the prompt-level procedure declines (0.00). This correctness result identifies which reliability claim an anytime-valid certificate actually supports under dependence.
查看此版本解读 →查看此版本原文 ↗