When Does Exploration Prevent Collapse? Three Regimes and the Path of the Reward in Self-Consistency Training
Jaeho Hwang, Uiwon Hwang
原文摘要
Test-time reinforcement learning (TTRL) adapts a model without labels by rewarding answers that agree with the majority, but it collapses the policy onto its majority answer and can remove the correct answer from the samples. Exploration can prevent this collapse, yet its strength is chosen by trial and error, and how the training algorithm changes the required strength is unclear. We model training on a single problem as a continuous flow of the logits of a softmax policy and analyze which policies are stable as the exploration coefficient varies. Only collapsed policies are stable below a lower boundary and only policies spread over many answers above an upper boundary, and we derive these boundaries in closed form for an agreement reward with smoothing toward the uniform policy, compute them for the majority reward of TTRL on policies with one leading answer and bound them with interval arithmetic for actions on ordered bins. We prove that under exact gradients, the learning rate and, in a mean-field model, the normalization of GRPO leave these boundaries unchanged when exploration passes through them together with the reward but shift them when exploration bypasses them. Exact sums on these policies then show that a bonus for rare answers needs a coefficient above $1/\log(G-1)$ for groups of $G$ samples before the normalization and about two to three times that after it. In free-form TTRL on MATH-500, a bonus placed before the normalization prevented collapse and kept more correct answers among the samples, although it lowered pass@1.
cs.LG