跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2610.26325 · v2

Typed Decision Models under Prevalence Shift and Label Budgets: Decision Training, Calibration, and Implicit Priors

首次提交 2026-10-19 21:01
版本更新 2026-11-18 05:12
雷达观测 2026-11-19 19:08
arXiv ↗PDF ↗

摘要

Typed decision models (TDMs) return probabilities over caller-defined options, and Jev's vendor describes it as calibrated. We ask whether vendor decision training improves a model over its backbone, whether a TDM's calibration and thresholds withstand prevalence (label) shift like a fine-tuned classifier's, and how many labels a trained model needs to overtake a TDM. We compare three TDMs (Jev, Tev1-4B, Nimble-9B), their base models and trained classifiers on identical candidate-set decisions from five public datasets (on intent datasets, within the gold label's intent group) under simulated shift, with 35 paired tests frozen before the confirmatory analysis (earlier test-data checks disclosed), and estimate label-budget curves. First, decision training is not always beneficial: Nimble-9B, which differs from Qwen3.5-9B only by a LoRA adapter, is better calibrated on intent tasks but 12 Macro-F1 points worse on financial sentiment, a loss that persists across prompt formats; Tev1-4B's largest gain over its backbone is a prompt-format effect. Second, on 50 fixed moderate shifts, Jev's top-label calibration error and threshold risk grow significantly more than a fine-tuned DeBERTa's only on financial sentiment, sensitive to the prior set; averaged over 2,000 shifts the gap is at most one point. Third, label-free EM correction with the model's implicit prior lowers Jev's shifted calibration error where it is highest, but labelled recalibration lowers it far more. Fourth, DeBERTa needs 100-250 labels to overtake Jev on hate speech and does not clearly overtake it on CLINC150 at any budget; a QLoRA adapter on Qwen3.5-4B overtakes Jev with at most 2,500 labels on every dataset, on CLINC150 only marginally and not under in-budget selection. The TDMs tested are neither uniformly better than their backbones nor uniquely fragile under shift; their lead over trained models depends on the task.

作者备注:44 pages, 14 figures. Code, protocol and per-example outputs: https://github.com/AbdulRahmanIqbal/tdm-prevalence-shift

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-18 05:12 · 最新观测版本

Typed Decision Models under Prevalence Shift and Label Budgets: Decision Training, Calibration, and Implicit Priors

Typed decision models (TDMs) return probabilities over caller-defined options, and Jev's vendor describes it as calibrated. We ask whether vendor decision training improves a model over its backbone, whether a TDM's calibration and thresholds withstand prevalence (label) shift like a fine-tuned classifier's, and how many labels a trained model needs to overtake a TDM. We compare three TDMs (Jev, Tev1-4B, Nimble-9B), their base models and trained classifiers on identical candidate-set decisions from five public datasets (on intent datasets, within the gold label's intent group) under simulated shift, with 35 paired tests frozen before the confirmatory analysis (earlier test-data checks disclosed), and estimate label-budget curves. First, decision training is not always beneficial: Nimble-9B, which differs from Qwen3.5-9B only by a LoRA adapter, is better calibrated on intent tasks but 12 Macro-F1 points worse on financial sentiment, a loss that persists across prompt formats; Tev1-4B's largest gain over its backbone is a prompt-format effect. Second, on 50 fixed moderate shifts, Jev's top-label calibration error and threshold risk grow significantly more than a fine-tuned DeBERTa's only on financial sentiment, sensitive to the prior set; averaged over 2,000 shifts the gap is at most one point. Third, label-free EM correction with the model's implicit prior lowers Jev's shifted calibration error where it is highest, but labelled recalibration lowers it far more. Fourth, DeBERTa needs 100-250 labels to overtake Jev on hate speech and does not clearly overtake it on CLINC150 at any budget; a QLoRA adapter on Qwen3.5-4B overtakes Jev with at most 2,500 labels on every dataset, on CLINC150 only marginally and not under in-budget selection. The TDMs tested are neither uniformly better than their backbones nor uniquely fragile under shift; their lead over trained models depends on the task.

查看此版本解读 →查看此版本原文 ↗