← 研踪 PaperTrailarXiv:2607.11653 · v2
Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters
首次提交 2026-07-13 23:04
版本更新 2026-10-28 16:13
雷达观测 2026-10-30 10:22
摘要
Black-box conditional quantile forecasts are widely used for sequential decisions under asymmetric costs. Once deployed, such forecasters must be monitored continuously as data streams drift and regimes change; this invalidates standard, fixed-horizon backtests for calibration. Further, existing backtests do not take into account that the notion of calibration is, in fact, information-dependent: forecasts can look calibrated to an auditor with coarse information while being miscalibrated to an auditor with richer information. We develop a distribution-free and game-theoretic testing framework for continuously auditing such forecasters with dependent, potentially nonstationary observations. We first formalize notions of conditional quantile calibration relative to the contextual features available to the auditor, showing how coarser information can limit detection. We then identify detectable alternative classes and, for contextual bets linear in the features, derive finite-time detection guarantees without an i.i.d. assumption. The resulting evidence processes provide interpretable, "feature-aware" evidence for miscalibration. We empirically validate these methods on simulated and real data, finding that a popular time series forecaster (Chronos-2) is highly miscalibrated w.r.t. multiple relevant features.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-28 16:13 · 最新观测版本
Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters
Black-box conditional quantile forecasts are widely used for sequential decisions under asymmetric costs. Once deployed, such forecasters must be monitored continuously as data streams drift and regimes change; this invalidates standard, fixed-horizon backtests for calibration. Further, existing backtests do not take into account that the notion of calibration is, in fact, information-dependent: forecasts can look calibrated to an auditor with coarse information while being miscalibrated to an auditor with richer information. We develop a distribution-free and game-theoretic testing framework for continuously auditing such forecasters with dependent, potentially nonstationary observations. We first formalize notions of conditional quantile calibration relative to the contextual features available to the auditor, showing how coarser information can limit detection. We then identify detectable alternative classes and, for contextual bets linear in the features, derive finite-time detection guarantees without an i.i.d. assumption. The resulting evidence processes provide interpretable, "feature-aware" evidence for miscalibration. We empirically validate these methods on simulated and real data, finding that a popular time series forecaster (Chronos-2) is highly miscalibrated w.r.t. multiple relevant features.
查看此版本解读 →查看此版本原文 ↗