← 研踪 PaperTrailarXiv:2606.00866 · v2
Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI
首次提交 2026-05-31 03:44
版本更新 2026-11-11 08:49
雷达观测 2026-11-13 11:11
摘要
Agentic programs alternate between LLM inference and tool calls, each request extending the previous one's context. Program-aware schedulers pin a program's KV cache in GPU HBM across tool calls to avoid re-prefilling it at every step, but as contexts grow, pinned KV cache exceeds HBM, raising program eviction: deciding when a program releases GPU memory and when it returns. Offloading to CPU DRAM lets an evicted program wait out a tool call instead of recomputing its context, but existing schedulers leave DRAM to LRU, so offloaded programs thrash. Making these decisions is hard: tool-call durations vary by orders of magnitude and are unknown in advance. We observe that programs nevertheless stay busy (short tool calls) or idle (long ones) for extended periods, and use a program's idleness, its recent fraction of time in tool calls, as a continuous, relative metric. MORI ranks programs by idleness, keeps the least idle in HBM and the next in DRAM, and drops the rest, enforcing admission control at both tiers under any GPU-to-CPU capacity ratio; it also proactively demotes idle programs to make room for ready ones. On real coding-agent traces, MORI achieves on average 64% higher throughput and 72% lower p90 TTFT than the best offloading baseline under high load.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-11 08:49 · 最新观测版本
Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI
Agentic programs alternate between LLM inference and tool calls, each request extending the previous one's context. Program-aware schedulers pin a program's KV cache in GPU HBM across tool calls to avoid re-prefilling it at every step, but as contexts grow, pinned KV cache exceeds HBM, raising program eviction: deciding when a program releases GPU memory and when it returns. Offloading to CPU DRAM lets an evicted program wait out a tool call instead of recomputing its context, but existing schedulers leave DRAM to LRU, so offloaded programs thrash. Making these decisions is hard: tool-call durations vary by orders of magnitude and are unknown in advance. We observe that programs nevertheless stay busy (short tool calls) or idle (long ones) for extended periods, and use a program's idleness, its recent fraction of time in tool calls, as a continuous, relative metric. MORI ranks programs by idleness, keeps the least idle in HBM and the next in DRAM, and drops the rest, enforcing admission control at both tiers under any GPU-to-CPU capacity ratio; it also proactively demotes idle programs to make room for ready ones. On real coding-agent traces, MORI achieves on average 64% higher throughput and 72% lower p90 TTFT than the best offloading baseline under high load.
查看此版本解读 →查看此版本原文 ↗