Express 语言建模
Express Language Modeling
Albert Gong, Annabelle Michael Carrell, Raaz Dwivedi, Lester Mackey
AI 速读 基于摘要
提出 Express,将非因果注意力近似转换为具有匹配保证的因果近似,并结合 Thinformer 与 Triton 实现加速语言建模流程。
cs.LG · cs.DS · math.ST · stat.ME · stat.ML
查看原文摘要
We introduce a new tool, Express, for converting a non-causal attention approximation into a causal approximation with matching approximation guarantees. When combined with the state-of-the-art Thinformer approximation, Express improves upon the best known causal attention guarantees, delivering $\log^{3/2}(n)/s$ approximation error with only $O(s)$ memory and $O(s^2 \log^2(n))$ compression overhead for a sequence of length $n$. We pair these developments with an efficient I/O-aware Triton implementation, demonstrate substantial speedups over FlashAttention 2, and use Express to overcome four resource bottlenecks in the language modeling pipeline: long-context prefill, KV cache compression, long-form memory-constrained decoding, and long-form compute-constrained decoding.