跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.26592 · v1

Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

实现可靠性 vs. 总体可靠性:相关推理下任意时间有效的自一致性认证

首次提交 2026-11-20 02:47
版本更新 2026-11-20 02:47
雷达观测 2026-11-20 12:52
arXiv ↗PDF ↗

摘要

Self-consistency certification treats repeated sampling from a language model as sequential evidence accumulation and stops when an anytime-valid e-process crosses a threshold, guaranteeing finite-sample Type I error control by Ville's inequality. Prior constructions assume sampled rollouts are conditionally independent. We show this assumption fails in practice: reasoning streams are positively correlated through shared reasoning prefixes or across-prompt heterogeneity, changing not merely certificate efficiency but the hypothesis certified. Modeling dependence via a per-unit latent state (the random-effects / intraclass-correlation structure) induces exchangeable, equi-correlated increments with correlation ρρ. Correlation splits the single null into a realized null about the drawn belief and a population null about average model behavior, which coincide only when ρ=0ρ=0. We show: (i) the standard betting e-process remains anytime-valid for the realized null, not the population null; (ii) read as a population-null test, a suitable mixture has asymptotic crossing probability at least π=P(μ(Z)>0)π=P(μ(Z)>0), potentially exceeding nominal εε; (iii) the within-unit mean's effective sample size saturates at 1/ρ1/ρ, so extra within-unit rollouts cannot restore population validity; and (iv) a prompt-level e-process restores population validity, with optimal allocation at one rollout per prompt under a pure rollout budget. Simulations and Claude Haiku 4.5 rollouts confirm this regime dependence: on GSM8K (ρ^=0.435\hatρ=0.435, θ^=+0.96\hatθ=+0.96) the gap is dormant, whereas on AIME (ρ^=0.480\hatρ=0.480, θ^=−0.15\hatθ=-0.15) the single-unit rule crosses on 24% of prompts at nominal ε=0.05ε=0.05, while the prompt-level procedure declines (0.00). This correctness result identifies which reliability claim an anytime-valid certificate actually supports under dependence.

AI 论文解读

基于摘要 · 非全文审阅

论文指出自一致性认证中样本独立假设在相关推理下失效,并区分实现零假设与总体零假设,提出提示级 e-process 恢复总体有效性。

正在读取账号与研究记录…

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-20 02:47 · 最新观测版本

Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

Self-consistency certification treats repeated sampling from a language model as sequential evidence accumulation and stops when an anytime-valid e-process crosses a threshold, guaranteeing finite-sample Type I error control by Ville's inequality. Prior constructions assume sampled rollouts are conditionally independent. We show this assumption fails in practice: reasoning streams are positively correlated through shared reasoning prefixes or across-prompt heterogeneity, changing not merely certificate efficiency but the hypothesis certified. Modeling dependence via a per-unit latent state (the random-effects / intraclass-correlation structure) induces exchangeable, equi-correlated increments with correlation ρρ. Correlation splits the single null into a realized null about the drawn belief and a population null about average model behavior, which coincide only when ρ=0ρ=0. We show: (i) the standard betting e-process remains anytime-valid for the realized null, not the population null; (ii) read as a population-null test, a suitable mixture has asymptotic crossing probability at least π=P(μ(Z)>0)π=P(μ(Z)>0), potentially exceeding nominal εε; (iii) the within-unit mean's effective sample size saturates at 1/ρ1/ρ, so extra within-unit rollouts cannot restore population validity; and (iv) a prompt-level e-process restores population validity, with optimal allocation at one rollout per prompt under a pure rollout budget. Simulations and Claude Haiku 4.5 rollouts confirm this regime dependence: on GSM8K (ρ^=0.435\hatρ=0.435, θ^=+0.96\hatθ=+0.96) the gap is dormant, whereas on AIME (ρ^=0.480\hatρ=0.480, θ^=−0.15\hatθ=-0.15) the single-unit rule crosses on 24% of prompts at nominal ε=0.05ε=0.05, while the prompt-level procedure declines (0.00). This correctness result identifies which reliability claim an anytime-valid certificate actually supports under dependence.

查看此版本解读 →查看此版本原文 ↗