arXiv:2608.02985 · v2
Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores
摘要
The standard check for contamination in LLM backtests is simple: compare scores before and after the training cutoff. We show this check is uninformative. Four flagship models fail it on questions they cannot have memorized: every scored question resolved after their cutoffs. The reason is structural. Models legitimately know more about times near their cutoff, so recency mimics leakage, and we prove no passive backtest can separate the two from genuine skill. Measurement, not just detection, requires information from outside the backtest. We supply it in two forms. A known cutoff identifies leakage at the boundary; a matched clean control identifies it globally and yields a leakage-adjusted score. Under a convex-pull model, leakage concentrates on outcomes that surprised the crowd and that training covered well. Partial memorization gives a large part of the full inflation. In twin models with planted leakage, the estimators detect and localize it, track its dose, and return null on clean questions. On frontier models, the estimators detect one cutoff-localized signature. They find no significant leakage at the tested boundaries in five models with positive naive gaps. Backtests need not be discarded; they need one defensible reference.
作者备注：Published in Transactions on Machine Learning Research (11/2026). Reviewed on OpenReview: https://openreview.net/forum?id=x3qagPELrp. Code: https://github.com/ZeyuZhang1901/Temporal-Leakage-Backtesting
期刊信息：Transactions on Machine Learning Research, 2026