跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2608.23986 · v4

The Shadow Price of Intelligence: Quality Degradation and Retry Feedback in LLM Inference

首次提交 2026-08-25 10:26
版本更新 2026-11-16 04:05
雷达观测 2026-11-17 17:12
arXiv ↗PDF ↗

摘要

Large language model providers are compute constrained, and a common response to congestion is to degrade service: route queries to smaller models, cut reasoning budgets, or shorten context windows. A user who receives an unsatisfactory answer either retries, adding load while the system is busy, or abandons, possibly as churn. We model inference serving as a control problem in which service quality is the provider's decision. A two-regime fluid approximation with retries folded into the service rate gives closed-form trajectories and crossing times, so the dynamic program over quality tiers needs no simulation. We characterize a region in which a weaker tier uses less energy per task but more capacity per task, and show that in the fluid model reactive throttling to such a tier can keep the queue from draining. With heterogeneous customers the problem is a transportation problem with retry-inflated capacity coefficients; optimal routing follows a cμcμ-type index, and the dual of the capacity constraint gives the marginal cost of one more query by class and time. We illustrate the method on five instances calibrated to public benchmark data.

作者备注:28 pages, 12 figures. Numerical instances calibrated to public benchmark data for five LLM providers

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v4,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v4 · 2026-11-16 04:05 · 最新观测版本

The Shadow Price of Intelligence: Quality Degradation and Retry Feedback in LLM Inference

Large language model providers are compute constrained, and a common response to congestion is to degrade service: route queries to smaller models, cut reasoning budgets, or shorten context windows. A user who receives an unsatisfactory answer either retries, adding load while the system is busy, or abandons, possibly as churn. We model inference serving as a control problem in which service quality is the provider's decision. A two-regime fluid approximation with retries folded into the service rate gives closed-form trajectories and crossing times, so the dynamic program over quality tiers needs no simulation. We characterize a region in which a weaker tier uses less energy per task but more capacity per task, and show that in the fluid model reactive throttling to such a tier can keep the queue from draining. With heterogeneous customers the problem is a transportation problem with retry-inflated capacity coefficients; optimal routing follows a cμcμ-type index, and the dual of the capacity constraint gives the marginal cost of one more query by class and time. We illustrate the method on five instances calibrated to public benchmark data.

查看此版本解读 →查看此版本原文 ↗