跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2609.02986 · v4

Two Principles for Autoregressive Language Modeling

首次提交 2026-09-02 22:49
版本更新 2026-11-20 00:38
雷达观测 2026-11-20 12:52
arXiv ↗PDF ↗

摘要

We investigate the functional differentiation learned by RoPE Transformers to derive principles for autoregressive language modeling. We introduce two intervention-based metrics: RoPE Frequency Importance Score (RFIS) measures how removing a frequency contribution changes a head's full attention distribution, and RoPE Positional Dependence (RPD) isolates the effect of rotary modulation. In Qwen3 and Llama3.1, these metrics distinguish retrieval from positional modeling, while individual heads can exhibit both functions. A mid-low-frequency Global Positional Band (GPBand) separates their frequency dependencies and tracks the training sequence length in controlled experiments, suggesting that global positional fitting can hinder zero-shot length extrapolation. These findings motivate two principles. (1) Functional Organization: retrieval and positional functions should be assigned separately and mixed within layers with layer-specific allocation; and (2) Positional Dependence: positional modeling should operate only locally, while global access should rely on position-independent retrieval. Matched pretraining from scratch and targeted architectural ablations at 380M and 1.4B parameters support both principles and highlight the importance of mechanism choice in realizing each function. Together, these findings suggest a promising route toward architectural innovation grounded in the functional requirements of autoregressive language modeling.

作者备注:29 pages, 16 figures, 10 tables

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

仅对比实际观测的元数据,不代表 PDF 正文或实验变化。版本可能不连续;带删除线为移除,下划线为新增。

这两个版本的已观测标题、作者、分类、摘要和备注没有变化。正文是否变化请核对原文。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v4 · 2026-11-20 00:38 · 最新观测版本

Two Principles for Autoregressive Language Modeling

We investigate the functional differentiation learned by RoPE Transformers to derive principles for autoregressive language modeling. We introduce two intervention-based metrics: RoPE Frequency Importance Score (RFIS) measures how removing a frequency contribution changes a head's full attention distribution, and RoPE Positional Dependence (RPD) isolates the effect of rotary modulation. In Qwen3 and Llama3.1, these metrics distinguish retrieval from positional modeling, while individual heads can exhibit both functions. A mid-low-frequency Global Positional Band (GPBand) separates their frequency dependencies and tracks the training sequence length in controlled experiments, suggesting that global positional fitting can hinder zero-shot length extrapolation. These findings motivate two principles. (1) Functional Organization: retrieval and positional functions should be assigned separately and mixed within layers with layer-specific allocation; and (2) Positional Dependence: positional modeling should operate only locally, while global access should rely on position-independent retrieval. Matched pretraining from scratch and targeted architectural ablations at 380M and 1.4B parameters support both principles and highlight the importance of mechanism choice in realizing each function. Together, these findings suggest a promising route toward architectural innovation grounded in the functional requirements of autoregressive language modeling.

查看此版本解读 →查看此版本原文 ↗
v3 · 2026-11-18 22:40

Two Principles for Autoregressive Language Modeling

We investigate the functional differentiation learned by RoPE Transformers to derive principles for autoregressive language modeling. We introduce two intervention-based metrics: RoPE Frequency Importance Score (RFIS) measures how removing a frequency contribution changes a head's full attention distribution, and RoPE Positional Dependence (RPD) isolates the effect of rotary modulation. In Qwen3 and Llama3.1, these metrics distinguish retrieval from positional modeling, while individual heads can exhibit both functions. A mid-low-frequency Global Positional Band (GPBand) separates their frequency dependencies and tracks the training sequence length in controlled experiments, suggesting that global positional fitting can hinder zero-shot length extrapolation. These findings motivate two principles. (1) Functional Organization: retrieval and positional functions should be assigned separately and mixed within layers with layer-specific allocation; and (2) Positional Dependence: positional modeling should operate only locally, while global access should rely on position-independent retrieval. Matched pretraining from scratch and targeted architectural ablations at 380M and 1.4B parameters support both principles and highlight the importance of mechanism choice in realizing each function. Together, these findings suggest a promising route toward architectural innovation grounded in the functional requirements of autoregressive language modeling.

查看此版本解读 →查看此版本原文 ↗