← 研踪 PaperTrailarXiv:2604.26963 · v3
MARS: Towards High Goodput and Low KV Thrashing in Agent Serving under GPU Memory Constraints
首次提交 2026-04-14 13:15
版本更新 2026-11-11 07:48
雷达观测 2026-11-13 11:12
摘要
Self-hosted LLM agents offer greater control over sensitive data and model customization while avoiding per-request API fees. However, serving concurrent, multi-turn workflows within limited GPU memory makes it difficult to achieve high workflow-level goodput. In particular, as workflows accumulate context, their reusable key-value (KV) caches can easily outgrow GPU capacity, triggering repeated eviction and recovery, known as KV cache thrashing. This process consumes resources and delays workflow completion. Although existing approaches propose various methods, our motivation study shows that the evaluated baselines still incur substantial thrashing overhead or fail to address the workflow-level goodput issue. We present MARS, an agent serving system that coordinates workflow admission and in-engine scheduling to achieve high workflow-level goodput on memory-constrained devices while avoiding KV cache thrashing. Specifically, outside the engine, an admission controller selects root workflows using initial prefix footprint estimates and live KV availability, adapting admissions to memory pressure. Inside, an agent-centric scheduler prioritizes turn completion and coordinates prefill allocation with KV reclamation to reduce disruptive evictions. Across two models and three GPU configurations, MARS achieves 1.59x-7.56x mean end-to-end latency speedups over the fastest baseline. Integrated as the serving backend for OpenCode, Codex, and OpenHands, MARS further improves mean workflow completion time by 1.17x-1.87x. Our source code is publicly available at https://github.com/Afterglow231/MARS_preview .
作者备注：13 pages, 11 figures. Major revision with updated title, system design, and expanded evaluation. Preprint
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-11 07:48 · 最新观测版本
MARS: Towards High Goodput and Low KV Thrashing in Agent Serving under GPU Memory Constraints
Self-hosted LLM agents offer greater control over sensitive data and model customization while avoiding per-request API fees. However, serving concurrent, multi-turn workflows within limited GPU memory makes it difficult to achieve high workflow-level goodput. In particular, as workflows accumulate context, their reusable key-value (KV) caches can easily outgrow GPU capacity, triggering repeated eviction and recovery, known as KV cache thrashing. This process consumes resources and delays workflow completion. Although existing approaches propose various methods, our motivation study shows that the evaluated baselines still incur substantial thrashing overhead or fail to address the workflow-level goodput issue. We present MARS, an agent serving system that coordinates workflow admission and in-engine scheduling to achieve high workflow-level goodput on memory-constrained devices while avoiding KV cache thrashing. Specifically, outside the engine, an admission controller selects root workflows using initial prefix footprint estimates and live KV availability, adapting admissions to memory pressure. Inside, an agent-centric scheduler prioritizes turn completion and coordinates prefill allocation with KV reclamation to reduce disruptive evictions. Across two models and three GPU configurations, MARS achieves 1.59x-7.56x mean end-to-end latency speedups over the fastest baseline. Integrated as the serving backend for OpenCode, Codex, and OpenHands, MARS further improves mean workflow completion time by 1.17x-1.87x. Our source code is publicly available at https://github.com/Afterglow231/MARS_preview .
查看此版本解读 →查看此版本原文 ↗