arXiv:2611.06133 · v1

Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining

首次提交 2026-11-04 16:18

版本更新 2026-11-04 16:18

雷达观测 2026-11-05 11:10

摘要

The full-attention layers of large language models (LLMs) make every generated token more expensive than the previous one, slowing models down on long-context tasks. Hybrid models reduce this cost by replacing most full-attention layers with linear-attention layers, but the first still dominate decoding at long contexts. We accelerate these remaining layers without any training or weight changes using combined sparse attention: each layer attends to a sliding window of recent tokens, a pinned prefix, and a fixed budget of most relevant tokens in the context. The window size and budget are found on a per-layer basis using a multi-objective search algorithm. Across six hybrid models from three families, from 4B to 397B parameters, the selected configurations preserve their baseline model's accuracy on 21 long-context datasets. Decoding 64k/256k tokens becomes up to 1.26x/2.22x faster on Qwen3.5-4B, 1.20x/2.23x on Qwen3.5-122B-A10B, and 1.16x/2.06x on Ornith-1.0-397B.

作者备注：19 pages (main body: 10 pages, references: 3 pages, appendix: 6 pages), 8 figures, 6 tables