Quantifying uncertainty over an entire LLM completion is critical for selective prediction and adaptive compute. Zero-cost signals such as mean token entropy and log-probability are local to tokens or to one completion, and self-consistency needs many extra completions. We show that Sequential Monte Carlo (SMC) decoding for power sampling already estimates the normalizing constant of the power distribution, which gives the sequence-level R'enyi entropy (SRE) of the model's completion distribution at zero marginal cost, with a guarantee at every particle budget. A R'enyi chain rule extends this readout to a family of stepwise entropies of any order. We prove that orders above one are tied to the most likely token of each step, which explains why they track log-probability, while the order-1/2 member accumulates the log of per-step search effort. This member is the best zero-cost selective-prediction signal in 10 of 13 settings. On MATH500 its AUROC is 0.022 to 0.044 above log-probability with all four models and within length quartiles, and the same order is selected when each setting is held out. An 8-particle probe identifies the problems that benefit from more particles, and a two-pass policy matches 64-particle accuracy on MATH500 with 34.2% fewer particles over three seeds.
Ege C. Kaya, Aliasghar Pourghani, Vijay Gupta, Abolfazl Hashemi
原文摘要
Average-reward reinforcement learning requires estimating the gain and a state-dependent bias, which solves the Poisson equation only up to an additive constant. Distributional analogues of the bias inherit this same ambiguity. We introduce a quotient-space formulation in which state-indexed laws are identified up to a common translation, together with a categorical parameterization that respects this symmetry. On this space, we define a projected average-reward distributional operator and show that it is well-defined, non-expansive in a coordinate Cramér metric, and admits fixed points. We interpret the projected fixed points through geometrically-stopped differential returns, whose categorical operators are contractive and whose fixed points converge in the Abel sense to a canonical fixed point of the projected undiscounted operator. We show that the means of the projected fixed points match the scalar bias up to an error caused by endpoint clipping of the categorical projections, and we bound this error. Finally, we establish almost-sure convergence and finite-iteration residual bounds for temporal-difference recursions with gain-centered rewards under i.i.d. and Markovian sampling. When the gain is unknown, we couple the recursion with an online gain estimator and prove the same guarantees under Markovian sampling, with a bound on the gain error.