跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.23040 · v1

Variance Restoration over Covariance Rank: Memory-Budgeted Analytic Continual Learning

首次提交 2026-11-18 00:39
版本更新 2026-11-18 00:39
雷达观测 2026-11-18 16:38
arXiv ↗PDF ↗

摘要

Analytic continual learning replaces iterative gradient updates with closed-form ridge regression over frozen pre-trained representations, summarizing all past sessions in two running sufficient statistics: the feature autocorrelation matrix ΣtΣ_t and the class cross-correlation matrix ΓtΓ_t. Lifting backbone features into a wide random code of dimension M≫DM \gg D substantially improves linear separability across tasks, but forces ΣtΣ_t to grow quadratically as O(M2)\mathcal{O}(M^2). Under a fixed persistent memory envelope $\Bcal$, a learner faces a fundamental structural trade-off: store exact pairwise co-activations inside a narrower feature code, or expand to a wide code while compressing cross-coordinate interactions. We observe that the coordinate self-variances $v_t = \diag(Σ_t)$ and class statistics ΓtΓ_t grow only linearly with MM, whereas off-diagonal interactions dominate memory. We introduce \method (\textbf{Co}ordinate-\textbf{Re}stored \textbf{A}nalytic \textbf{C}ontinual \textbf{L}earning), which retains (vt,Γt)(v_t, Γ_t) exactly, maintains off-diagonal co-activations through a streaming rank-qq factor FtF_t, restores exact diagonal variances via $\Sighat_t = F_t^\top F_t + \Diag(v_t - \diag(F_t^\top F_t))$, and solves the full M×CM\times C normal equations via the Woodbury identity. Across five class-incremental benchmarks on frozen ViT-B/16, \method matches dense full-covariance readouts within 0.090.09 points while cutting persistent state by 90.7%90.7\% (35.935.9 vs.\ 385.3385.3 MiB). Under equal memory envelopes of 1616--8080 MiB, prioritizing expansion width MM over covariance rank qq outperforms exact narrow readouts in 1717 of 1818 configurations by up to +2.19+2.19 points.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-18 00:39 · 最新观测版本

Variance Restoration over Covariance Rank: Memory-Budgeted Analytic Continual Learning

Analytic continual learning replaces iterative gradient updates with closed-form ridge regression over frozen pre-trained representations, summarizing all past sessions in two running sufficient statistics: the feature autocorrelation matrix ΣtΣ_t and the class cross-correlation matrix ΓtΓ_t. Lifting backbone features into a wide random code of dimension M≫DM \gg D substantially improves linear separability across tasks, but forces ΣtΣ_t to grow quadratically as O(M2)\mathcal{O}(M^2). Under a fixed persistent memory envelope $\Bcal$, a learner faces a fundamental structural trade-off: store exact pairwise co-activations inside a narrower feature code, or expand to a wide code while compressing cross-coordinate interactions. We observe that the coordinate self-variances $v_t = \diag(Σ_t)$ and class statistics ΓtΓ_t grow only linearly with MM, whereas off-diagonal interactions dominate memory. We introduce \method (\textbf{Co}ordinate-\textbf{Re}stored \textbf{A}nalytic \textbf{C}ontinual \textbf{L}earning), which retains (vt,Γt)(v_t, Γ_t) exactly, maintains off-diagonal co-activations through a streaming rank-qq factor FtF_t, restores exact diagonal variances via $\Sighat_t = F_t^\top F_t + \Diag(v_t - \diag(F_t^\top F_t))$, and solves the full M×CM\times C normal equations via the Woodbury identity. Across five class-incremental benchmarks on frozen ViT-B/16, \method matches dense full-covariance readouts within 0.090.09 points while cutting persistent state by 90.7%90.7\% (35.935.9 vs.\ 385.3385.3 MiB). Under equal memory envelopes of 1616--8080 MiB, prioritizing expansion width MM over covariance rank qq outperforms exact narrow readouts in 1717 of 1818 configurations by up to +2.19+2.19 points.

查看此版本解读 →查看此版本原文 ↗