← 研踪 PaperTrailarXiv:2605.30202 · v2
A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
首次提交 2026-05-29 00:41
版本更新 2026-11-04 21:31
雷达观测 2026-11-05 12:13
摘要
Looped transformers apply a shared block multiple times and have emerged as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs a looped model has strictly less capacity than a baseline transformer. We argue that compute, the number of sequential operations applied to a hidden state, and capacity, the parameters available at a single step, are distinct scaling axes that standard transformer layers conflate. We propose a dual-path block that exposes both axes as parallel pathways within a single layer: a deep sublayer re-applied $K$ times with shared parameters, and a wide sublayer with an enlarged feed-forward network applied once, combined by independent per-token gates. At two FLOP budgets, the best dual-path configuration surpasses both iso-FLOP single-axis controls on aggregate language modelling, commonsense, and math metrics, while using 17-33% fewer parameters than the width-scaled baseline. The learned gates are directly interpretable and show systematic per-token allocation: function words and lexical content trend wide, while punctuation, symbols, and arithmetic tokens trend deep. Our results suggest that compute and capacity can be scaled separately, with the model balancing the two token by token.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-04 21:31 · 最新观测版本
A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
Looped transformers apply a shared block multiple times and have emerged as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs a looped model has strictly less capacity than a baseline transformer. We argue that compute, the number of sequential operations applied to a hidden state, and capacity, the parameters available at a single step, are distinct scaling axes that standard transformer layers conflate. We propose a dual-path block that exposes both axes as parallel pathways within a single layer: a deep sublayer re-applied $K$ times with shared parameters, and a wide sublayer with an enlarged feed-forward network applied once, combined by independent per-token gates. At two FLOP budgets, the best dual-path configuration surpasses both iso-FLOP single-axis controls on aggregate language modelling, commonsense, and math metrics, while using 17-33% fewer parameters than the width-scaled baseline. The learned gates are directly interpretable and show systematic per-token allocation: function words and lexical content trend wide, while punctuation, symbols, and arithmetic tokens trend deep. Our results suggest that compute and capacity can be scaled separately, with the model balancing the two token by token.
查看此版本解读 →查看此版本原文 ↗