← 研踪 PaperTrailarXiv:2605.06914 · v2
Pacing Branch Parallelism in LLM Serving
首次提交 2026-05-08 04:23
版本更新 2026-11-02 07:01
雷达观测 2026-11-04 23:42
摘要
Recent methods expose intra-request parallelism in LLM outputs, allowing independent branches to decode concurrently. Existing serving systems execute these branches eagerly or under fixed caps. We show that both are brittle: eager admission inflates the shared decode step, degrading co-batched requests in serial stages, while conservative fixed caps forgo the throughput that motivated exposing branches in the first place. We call the excess step latency caused by admitted branches the \emph{branch externality} and show that the safe width depends on batch composition, context lengths, and accumulated slack, all of which change continuously over a workload trace. We introduce PACE, a per-step admission controller that treats extra branches as opportunistic width, admitted only when the predicted branch externality fits within the batch's current slack budget. Per-step regulation is practical because branch-level scheduling decouples compute from memory: branches share the request's prefix KV, so expanding or contracting width requires no memory reclamation. On Qwen3-32B, PACE improves goodput by $1.77\times$ over serving without branch parallelism and by $1.48\times$ over eager admission while maintaining 99\% SLO attainment. Comparable gains ($1.72$--$2.15\times$) hold on five additional dense and mixture-of-experts models with up to 671B parameters. Code is available at https://github.com/Swapnil-Gandhi/PACE
作者备注：NeurIPS'26 | Camera-Ready
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-02 07:01 · 最新观测版本
Pacing Branch Parallelism in LLM Serving
Recent methods expose intra-request parallelism in LLM outputs, allowing independent branches to decode concurrently. Existing serving systems execute these branches eagerly or under fixed caps. We show that both are brittle: eager admission inflates the shared decode step, degrading co-batched requests in serial stages, while conservative fixed caps forgo the throughput that motivated exposing branches in the first place. We call the excess step latency caused by admitted branches the \emph{branch externality} and show that the safe width depends on batch composition, context lengths, and accumulated slack, all of which change continuously over a workload trace. We introduce PACE, a per-step admission controller that treats extra branches as opportunistic width, admitted only when the predicted branch externality fits within the batch's current slack budget. Per-step regulation is practical because branch-level scheduling decouples compute from memory: branches share the request's prefix KV, so expanding or contracting width requires no memory reclamation. On Qwen3-32B, PACE improves goodput by $1.77\times$ over serving without branch parallelism and by $1.48\times$ over eager admission while maintaining 99\% SLO attainment. Comparable gains ($1.72$--$2.15\times$) hold on five additional dense and mixture-of-experts models with up to 671B parameters. Code is available at https://github.com/Swapnil-Gandhi/PACE
查看此版本解读 →查看此版本原文 ↗