跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2510.05364 · v2

Challenging Quadratic Attention$\unicode{x2013}$A Holistic View on the Rise of Alternative Language Model Architectures

首次提交 2025-10-07 04:45
版本更新 2026-11-18 18:44
雷达观测 2026-11-19 15:04
arXiv ↗PDF ↗

摘要

Transformers have dominated sequence processing tasks for the past seven years$\unicode{x2013}$most notably language modeling. However, the inherent quadratic complexity of their attention mechanism remains a significant bottleneck as context length increases. We review and distill the recent efforts to overcome this bottleneck, including advances in (sub-quadratic) attention variants, recurrent neural networks, state space models, and hybrid architectures. We critically analyze approaches regarding compute and memory complexity, benchmark results, and fundamental limitations to assess whether the dominance of pure-attention transformers may soon be challenged, which we consider possible, particularly in domain-specific and edge-device applications.

作者备注:22 pages, 2 figures, 2 tables; replaced the preprint with the peer-reviewed conference version of the paper, including the new title

期刊信息:In Proceedings of The Big Picture v2 - Crafting a Research Narrative, pages 60-81, San Diego, CA, USA. Association for Computational Linguistics. 2026

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-18 18:44 · 最新观测版本

Challenging Quadratic Attention$\unicode{x2013}$A Holistic View on the Rise of Alternative Language Model Architectures

Transformers have dominated sequence processing tasks for the past seven years$\unicode{x2013}$most notably language modeling. However, the inherent quadratic complexity of their attention mechanism remains a significant bottleneck as context length increases. We review and distill the recent efforts to overcome this bottleneck, including advances in (sub-quadratic) attention variants, recurrent neural networks, state space models, and hybrid architectures. We critically analyze approaches regarding compute and memory complexity, benchmark results, and fundamental limitations to assess whether the dominance of pure-attention transformers may soon be challenged, which we consider possible, particularly in domain-specific and edge-device applications.

查看此版本解读 →查看此版本原文 ↗