← 研踪 PaperTrailarXiv:2611.20325 · v1
Measuring Coherence in Predictive Models
首次提交 2026-11-16 07:39
版本更新 2026-11-16 07:39
雷达观测 2026-11-17 17:12
摘要
Many sequential decision-making procedures, such as active learning, implicitly assume that a model's current predictions agree with those it expects to make after updating on new data, a property we call coherence. Most models in use do not update by exact conditioning and are not guaranteed to be coherent, which can lead to worse decisions. Yet there is currently no general way to measure incoherence or to tell when it matters. We develop tools to measure incoherence and characterise when it affects decisions. We introduce the incoherence ratio, an MMD-based measure of how far any updating procedure departs from coherence, computed only from the procedure's own predictions and updates, and accompany it with an anytime-valid test of coherence. In Bayesian experimental design, acquisition objectives that are equivalent for coherent models choose different data under every approximate model we study, and the incoherence ratio tracks how far incoherence distorts them. We also introduce action coherence, which captures only the incoherence that changes a decision. What a resulting disagreement costs, however, is determined primarily by predictive accuracy. Sequential decisions depend on how a model updates as well as on what it predicts, so coherence, like accuracy, should be checked rather than assumed.
作者备注:Accepted at NeurIPS 2026
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-16 07:39 · 最新观测版本
Measuring Coherence in Predictive Models
Many sequential decision-making procedures, such as active learning, implicitly assume that a model's current predictions agree with those it expects to make after updating on new data, a property we call coherence. Most models in use do not update by exact conditioning and are not guaranteed to be coherent, which can lead to worse decisions. Yet there is currently no general way to measure incoherence or to tell when it matters. We develop tools to measure incoherence and characterise when it affects decisions. We introduce the incoherence ratio, an MMD-based measure of how far any updating procedure departs from coherence, computed only from the procedure's own predictions and updates, and accompany it with an anytime-valid test of coherence. In Bayesian experimental design, acquisition objectives that are equivalent for coherent models choose different data under every approximate model we study, and the incoherence ratio tracks how far incoherence distorts them. We also introduce action coherence, which captures only the incoherence that changes a decision. What a resulting disagreement costs, however, is determined primarily by predictive accuracy. Sequential decisions depend on how a model updates as well as on what it predicts, so coherence, like accuracy, should be checked rather than assumed.
查看此版本解读 →查看此版本原文 ↗