← 研踪 PaperTrailarXiv:2611.11114 · v1
Compositional Conformal Risk Control for Multi-Stage LLM Cascades
首次提交 2026-11-08 11:55
版本更新 2026-11-08 11:55
雷达观测 2026-11-10 12:16
摘要
Multi-stage large language model (LLM) cascades balance accuracy and inference cost, but reliable deployment also requires controlling error risk and identifying its sources across stages. We propose Compositional Conformal Risk Control (CompCRC), a risk-control framework for pre-specified multi-stage LLM cascades. We decompose the risk of accepting incorrect answers into additive stage-wise contributions over the full query population, linking local risk budgets to a system-level guarantee. CompCRC constructs and freezes candidate budget allocations and acceptance thresholds, selects one candidate on independent data, and certifies its stage-wise risk contributions on separate held-out data. Under conditional exchangeability, we establish a finite-sample guarantee: the marginal expected accepted-error risk of the complete deployment procedure does not exceed the global risk budget. At the reported comparison budget of $α=0.45$, all three stages of CompCRC pass their local checks on MMLU, GSM8K, and HellaSwag. Compared with Global-CRC, which passes system-level risk certification at the same budget, CompCRC reduces observed accepted-error risk by 1.33--2.00 percentage points, improves final accuracy by 0.67--2.00 percentage points, and reduces weighted inference cost by 30.37\% on HellaSwag. The selected cost-based budget allocation also improves accuracy per unit cost by 10.6\%--46.7\% over uniform allocation. Further experiments show that stage-wise statistics identify stages that fail local certification checks, providing diagnostic information alongside system-level risk control.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-08 11:55 · 最新观测版本
Compositional Conformal Risk Control for Multi-Stage LLM Cascades
Multi-stage large language model (LLM) cascades balance accuracy and inference cost, but reliable deployment also requires controlling error risk and identifying its sources across stages. We propose Compositional Conformal Risk Control (CompCRC), a risk-control framework for pre-specified multi-stage LLM cascades. We decompose the risk of accepting incorrect answers into additive stage-wise contributions over the full query population, linking local risk budgets to a system-level guarantee. CompCRC constructs and freezes candidate budget allocations and acceptance thresholds, selects one candidate on independent data, and certifies its stage-wise risk contributions on separate held-out data. Under conditional exchangeability, we establish a finite-sample guarantee: the marginal expected accepted-error risk of the complete deployment procedure does not exceed the global risk budget. At the reported comparison budget of $α=0.45$, all three stages of CompCRC pass their local checks on MMLU, GSM8K, and HellaSwag. Compared with Global-CRC, which passes system-level risk certification at the same budget, CompCRC reduces observed accepted-error risk by 1.33--2.00 percentage points, improves final accuracy by 0.67--2.00 percentage points, and reduces weighted inference cost by 30.37\% on HellaSwag. The selected cost-based budget allocation also improves accuracy per unit cost by 10.6\%--46.7\% over uniform allocation. Further experiments show that stage-wise statistics identify stages that fail local certification checks, providing diagnostic information alongside system-level risk control.
查看此版本解读 →查看此版本原文 ↗