← 研踪 PaperTrailarXiv:2606.15964 · v2
PromptShift-CRC: What Drift-Aware Weighting Adds to Online Conformal Risk Control for Foundation Models
首次提交 2026-06-15 02:45
版本更新 2026-11-07 07:12
雷达观测 2026-11-10 10:11
摘要
A risk threshold calibrated on old prompts can stop being valid when prompts to a foundation model drift. We study PromptShift-CRC, a rule that weights past prompts by feature similarity, falls back to recent prompts when the current prompt is unlike the past, and updates its level online. We prove that the update controls the long-run violation rate for any weights, so marginal coverage cannot separate weighting schemes. We give a conditional bound at a fixed level whose cost is a weighted total variation distance, bounded by a computable prompt-drift term up to an unknown Lipschitz constant. We propose a comparison protocol: a matched short-window baseline, a validity and sharpness frontier, 30-seed synthetic streams, and real-output streams ordered without using the model's output. On MMLU, TruthfulQA (Qwen2.5-1.5B) and RealToxicityPrompts (GPT-2, replicated with a second generator and classifier), every online method reaches the target. PromptShift-CRC stays within 0.0055 of the Adaptive CRC frontier and avoids the worst window, but DtACI matches or beats it in most settings. The rule performs like a mid-ranked window without having to choose one. Tests fixed in advance to favor the weights show no consistent gain over the frontier. Code, outputs and streams are released.
作者备注：36 pages. Code and outputs: https://github.com/jopoku16/promptshift-crc
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 07:12 · 最新观测版本
PromptShift-CRC: What Drift-Aware Weighting Adds to Online Conformal Risk Control for Foundation Models
A risk threshold calibrated on old prompts can stop being valid when prompts to a foundation model drift. We study PromptShift-CRC, a rule that weights past prompts by feature similarity, falls back to recent prompts when the current prompt is unlike the past, and updates its level online. We prove that the update controls the long-run violation rate for any weights, so marginal coverage cannot separate weighting schemes. We give a conditional bound at a fixed level whose cost is a weighted total variation distance, bounded by a computable prompt-drift term up to an unknown Lipschitz constant. We propose a comparison protocol: a matched short-window baseline, a validity and sharpness frontier, 30-seed synthetic streams, and real-output streams ordered without using the model's output. On MMLU, TruthfulQA (Qwen2.5-1.5B) and RealToxicityPrompts (GPT-2, replicated with a second generator and classifier), every online method reaches the target. PromptShift-CRC stays within 0.0055 of the Adaptive CRC frontier and avoids the worst window, but DtACI matches or beats it in most settings. The rule performs like a mid-ranked window without having to choose one. Tests fixed in advance to favor the weights show no consistent gain over the frontier. Code, outputs and streams are released.
查看此版本解读 →查看此版本原文 ↗