← 研踪 PaperTrailarXiv:2611.15779 · v1
Honesty Is a Race: Predicting When Verifiable-Reward RL Teaches a Model to Bluff
首次提交 2026-11-12 01:54
版本更新 2026-11-12 01:54
雷达观测 2026-11-13 14:16
摘要
Two rewards with the same expected payoff at the start of training can end a reinforcement-learning run in opposite places, with an honest model or one that bluffs. We study rewards that pay for a verdict that a checker cannot verify ("this program is correct") and for a certificate that it accepts (an input revealing a bug). To first order, a GRPO step makes the model more honest exactly when . Here is the rate at which the checker accepts the model's certificates, weighted toward the items on which the model still gives both verdicts, and is the pull of the items on which the unverifiable verdict is right. Both terms move during training. A checker raises , so near the threshold honesty is a race, which a run just below the threshold can win and a run under a lottery (random pay with the same mean) cannot. The term starts large and falls as the other items settle, so a run just above the threshold first dips toward bluffing, on code nearly to zero, before climbing back. Paying the honest verdict also pulls the other items toward it, which predicts a ceiling on the payoff ratio, above which the model gives the honest verdict where it is wrong. On propositional Datalog, two code models, and abstention on grade-school math, with the model's success rate before RL, no run above ends collapsed, and runs under the checker can survive slightly below it. At , above the ceiling, both code models call about half of correct programs buggy. On depth-3 Datalog, training generously under the checker and then cutting the payoff keeps the model at least 67% honest in three seeds, at a ratio where the same model trained from the start falls to 4-15% and is at most 47% honest after 160 steps. On depth-2 Datalog, when honesty collapses, a linear probe fit before RL still reads the truth.
作者备注：34 pages, 10 figures
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-12 01:54 · 最新观测版本
Honesty Is a Race: Predicting When Verifiable-Reward RL Teaches a Model to Bluff
Two rewards with the same expected payoff at the start of training can end a reinforcement-learning run in opposite places, with an honest model or one that bluffs. We study rewards that pay for a verdict that a checker cannot verify ("this program is correct") and for a certificate that it accepts (an input revealing a bug). To first order, a GRPO step makes the model more honest exactly when . Here is the rate at which the checker accepts the model's certificates, weighted toward the items on which the model still gives both verdicts, and is the pull of the items on which the unverifiable verdict is right. Both terms move during training. A checker raises , so near the threshold honesty is a race, which a run just below the threshold can win and a run under a lottery (random pay with the same mean) cannot. The term starts large and falls as the other items settle, so a run just above the threshold first dips toward bluffing, on code nearly to zero, before climbing back. Paying the honest verdict also pulls the other items toward it, which predicts a ceiling on the payoff ratio, above which the model gives the honest verdict where it is wrong. On propositional Datalog, two code models, and abstention on grade-school math, with the model's success rate before RL, no run above ends collapsed, and runs under the checker can survive slightly below it. At , above the ceiling, both code models call about half of correct programs buggy. On depth-3 Datalog, training generously under the checker and then cutting the payoff keeps the model at least 67% honest in three seeds, at a ratio where the same model trained from the start falls to 4-15% and is at most 47% honest after 160 steps. On depth-2 Datalog, when honesty collapses, a linear probe fit before RL still reads the truth.
查看此版本解读 →查看此版本原文 ↗