← 研踪 PaperTrailarXiv:2606.31963 · v2
Signed-Permutation Coordinate Transport for RMSNorm Transformers
首次提交 2026-07-01 01:02
版本更新 2026-11-07 18:52
雷达观测 2026-11-10 11:14
摘要
Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top-$k$ neuron sets, attribution lists, LoRA updates, optimizer states, and merge alignments. This requires fixing the model's residual-stream gauge. We show that the native coordinate-preserving orthogonal gauge is architecture-dependent: LayerNorm residual charts have permutation gauge $S_d$ up to a global sign flip, while RMSNorm charts with generic gains have signed-permutation gauge $B_d = S_d \ltimes \{\pm 1\}^d$. Permutation-only alignment is thus symmetry-incomplete for RMSNorm models. We introduce sign-marginalized Hungarian matching and prove that with decorrelated coordinates, signed-correlation matching has a permutation-accuracy ceiling equal to the true gauge's positive-sign fraction up to $O(d 2^{-d})$, which sign-marginalized matching removes. Omitting signs induces expected squared error $2\|ΔW\|_F^2$ on weight updates and loses orientation on vectors and AdamW first moments, while RMSNorm gains and AdamW second moments permute without signs. Controlled tests confirm these laws: a Qwen2.5-1.5B SST-2 LoRA adapter retains 0.944 accuracy (from 0.945) under $B_d$ transport vs. 0.523 under $S_d$, and steering preserves effects to $<10^{-4}$ at zero raw top-10 overlap. Across natural checkpoints, inherited slots suffice for same-base descendants (96.2%-100% retention), whereas cross-generation pairs support sparse prompt-stable subsets with causally decisive negative signs ($+0.185$ vs. $-0.010$ at $8σ$ on Qwen2/2.5-0.5B; $+0.189$ vs. $-0.021$ on 1.5B). Along fine-tuning trajectories, composing local $B_d$ gauges recovers 91.1% of cross-run Qwen2.5-1.5B coordinates at 1,500 steps vs. 60.3% for endpoint matching. Merge and interpretability audits confirm that coordinate names are reproducible only relative to an explicit $B_d$ gauge.
作者备注：Accepted at NeurIPS 2026. 37 pages, 2 figures, 28 tables
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 18:52 · 最新观测版本
Signed-Permutation Coordinate Transport for RMSNorm Transformers
Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top-$k$ neuron sets, attribution lists, LoRA updates, optimizer states, and merge alignments. This requires fixing the model's residual-stream gauge. We show that the native coordinate-preserving orthogonal gauge is architecture-dependent: LayerNorm residual charts have permutation gauge $S_d$ up to a global sign flip, while RMSNorm charts with generic gains have signed-permutation gauge $B_d = S_d \ltimes \{\pm 1\}^d$. Permutation-only alignment is thus symmetry-incomplete for RMSNorm models. We introduce sign-marginalized Hungarian matching and prove that with decorrelated coordinates, signed-correlation matching has a permutation-accuracy ceiling equal to the true gauge's positive-sign fraction up to $O(d 2^{-d})$, which sign-marginalized matching removes. Omitting signs induces expected squared error $2\|ΔW\|_F^2$ on weight updates and loses orientation on vectors and AdamW first moments, while RMSNorm gains and AdamW second moments permute without signs. Controlled tests confirm these laws: a Qwen2.5-1.5B SST-2 LoRA adapter retains 0.944 accuracy (from 0.945) under $B_d$ transport vs. 0.523 under $S_d$, and steering preserves effects to $<10^{-4}$ at zero raw top-10 overlap. Across natural checkpoints, inherited slots suffice for same-base descendants (96.2%-100% retention), whereas cross-generation pairs support sparse prompt-stable subsets with causally decisive negative signs ($+0.185$ vs. $-0.010$ at $8σ$ on Qwen2/2.5-0.5B; $+0.189$ vs. $-0.021$ on 1.5B). Along fine-tuning trajectories, composing local $B_d$ gauges recovers 91.1% of cross-run Qwen2.5-1.5B coordinates at 1,500 steps vs. 60.3% for endpoint matching. Merge and interpretability audits confirm that coordinate names are reproducible only relative to an explicit $B_d$ gauge.
查看此版本解读 →查看此版本原文 ↗