跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.13543 · v1

Sanity Checks for Attention Surgery: Null Models and Matched Interventions for Representation-Motivated Design

首次提交 2026-11-10 12:06
版本更新 2026-11-10 12:06
雷达观测 2026-11-11 13:08
arXiv ↗PDF ↗

摘要

Transformer architectural modifications frequently measure a statistic on internal representations, observe that it is large, remove it architecturally, and report an improvement. Interpreting that sequence requires evaluating whether the statistic exceeds its noise floor, whether it survives a geometric null, and whether the training gain requires the specific direction removed. We formulate these requirements as three reusable checks and apply them to self-attention removal, which deletes the component of an attention head's output aligned with its own value vector. Across nine multi-head pretrained models (5,408 heads, 124M to 6.9B parameters), an earlier-position causal null accounts for 48% to 63% of the head-value cosine alignment, with no monotone scale trend. Under grouped-query attention, this alignment is specific to each head's own key-value group and reverses across groups (+0.24 to +0.27 within-group, -0.11 to -0.19 across-group). Split-half reliability demonstrates that the per-head intervention effect is resolvable in GPT-2 (rΔ=0.80r_Δ= 0.80) but attenuated in Pythia-160M (0.42) and Pythia-410M (0.53), whereas structural self-cosine reliability is near-perfect in all three (rstat>0.99r_{\text{stat}} > 0.99); which statistic predicts its own intervention is model-dependent. In a pre-registered 3-arm by 8-seed paired factorial at 4.0×1084.0 \times 10^8 tokens per run (399,900,672 tokens), a fixed arbitrary rank-one direction does not differ from baseline (+0.0011 nats, p=0.197p = 0.197), while self-directed removal lowers validation loss (-0.0029 nats, Holm-corrected p=0.0023p = 0.0023) and outperforms the fixed arbitrary control directly (-0.0040 nats, p=0.0018p = 0.0018). At 50.9M scale the training benefit requires the own-value direction rather than a fixed arbitrary one, but a large cosine alignment alone does not establish redundancy in frozen models.

作者备注:13 pages, 7 figures, 4 tables. Code: https://github.com/Basavarajearlybird/SANITY-CHECKS-FOR-ATTENTION-SURGERY_Original (archived at https://doi.org/10.5281/zenodo.24096410)

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-10 12:06 · 最新观测版本

Sanity Checks for Attention Surgery: Null Models and Matched Interventions for Representation-Motivated Design

Transformer architectural modifications frequently measure a statistic on internal representations, observe that it is large, remove it architecturally, and report an improvement. Interpreting that sequence requires evaluating whether the statistic exceeds its noise floor, whether it survives a geometric null, and whether the training gain requires the specific direction removed. We formulate these requirements as three reusable checks and apply them to self-attention removal, which deletes the component of an attention head's output aligned with its own value vector. Across nine multi-head pretrained models (5,408 heads, 124M to 6.9B parameters), an earlier-position causal null accounts for 48% to 63% of the head-value cosine alignment, with no monotone scale trend. Under grouped-query attention, this alignment is specific to each head's own key-value group and reverses across groups (+0.24 to +0.27 within-group, -0.11 to -0.19 across-group). Split-half reliability demonstrates that the per-head intervention effect is resolvable in GPT-2 (rΔ=0.80r_Δ= 0.80) but attenuated in Pythia-160M (0.42) and Pythia-410M (0.53), whereas structural self-cosine reliability is near-perfect in all three (rstat>0.99r_{\text{stat}} > 0.99); which statistic predicts its own intervention is model-dependent. In a pre-registered 3-arm by 8-seed paired factorial at 4.0×1084.0 \times 10^8 tokens per run (399,900,672 tokens), a fixed arbitrary rank-one direction does not differ from baseline (+0.0011 nats, p=0.197p = 0.197), while self-directed removal lowers validation loss (-0.0029 nats, Holm-corrected p=0.0023p = 0.0023) and outperforms the fixed arbitrary control directly (-0.0040 nats, p=0.0018p = 0.0018). At 50.9M scale the training benefit requires the own-value direction rather than a fixed arbitrary one, but a large cosine alignment alone does not establish redundancy in frozen models.

查看此版本解读 →查看此版本原文 ↗