arXiv:2609.40063 · v2
LARC: Low-Rank Adaptive Residual Connections for Learning in Frozen Models
摘要
Low-Rank Adaptive Residual Connections (LARC) give a frozen model a compact numerical state that can learn from feedback. The map changes the hidden input to a pretrained mixer. An episode copies a learned initialization $\ρ$ into a private state $\Φ$, updates its factors when feedback arrives, and resets to the bound $\ρ$. We analyze the factor-space updates and trace this input-side policy carrier into Orbit, the post-training realization of Memory-Mediated Learning Architecture studied here. In Orbit's released default, the learned residual shapes predictions while memory adaptation changes the retained evidence.\n\nThe controlled studies use a rank-4 residual with 12,288 factor entries on frozen MiniCPM5-1B-SFT. Two feedback updates reduce expected query execution error by 24.65 and 36.65 percentage points relative to the respective trained reset states in a four-candidate development task with 16 parameter groups and three paired seeds. A direct support-loss rule reaches 0.78125\\% error. In repository-balanced chronological replay of public continuous-integration jobs, retaining updates raises half-Brier loss from 0.1274 to 0.1808. A follow-up intervention observes same-batch non-descent and inconsistent future benefit from shrinking steps. These comparisons measure when numerical state helps a later read and when the trained reference is more effective.
作者备注：20 pages, 7 figures, 15 tables. Technical report of MMLA. The authors contributed equally. Version 2 adds the Orbit realization, a factor-state counterexample, and updated public component access