← 研踪 PaperTrailarXiv:2611.23040 · v1
Variance Restoration over Covariance Rank: Memory-Budgeted Analytic Continual Learning
首次提交 2026-11-18 00:39
版本更新 2026-11-18 00:39
雷达观测 2026-11-18 16:38
摘要
Analytic continual learning replaces iterative gradient updates with closed-form ridge regression over frozen pre-trained representations, summarizing all past sessions in two running sufficient statistics: the feature autocorrelation matrix and the class cross-correlation matrix . Lifting backbone features into a wide random code of dimension substantially improves linear separability across tasks, but forces to grow quadratically as . Under a fixed persistent memory envelope $\Bcal$, a learner faces a fundamental structural trade-off: store exact pairwise co-activations inside a narrower feature code, or expand to a wide code while compressing cross-coordinate interactions. We observe that the coordinate self-variances $v_t = \diag(Σ_t)$ and class statistics grow only linearly with , whereas off-diagonal interactions dominate memory. We introduce \method (\textbf{Co}ordinate-\textbf{Re}stored \textbf{A}nalytic \textbf{C}ontinual \textbf{L}earning), which retains exactly, maintains off-diagonal co-activations through a streaming rank- factor , restores exact diagonal variances via $\Sighat_t = F_t^\top F_t + \Diag(v_t - \diag(F_t^\top F_t))$, and solves the full normal equations via the Woodbury identity. Across five class-incremental benchmarks on frozen ViT-B/16, \method matches dense full-covariance readouts within points while cutting persistent state by ( vs.\ MiB). Under equal memory envelopes of -- MiB, prioritizing expansion width over covariance rank outperforms exact narrow readouts in of configurations by up to points.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-18 00:39 · 最新观测版本
Variance Restoration over Covariance Rank: Memory-Budgeted Analytic Continual Learning
Analytic continual learning replaces iterative gradient updates with closed-form ridge regression over frozen pre-trained representations, summarizing all past sessions in two running sufficient statistics: the feature autocorrelation matrix and the class cross-correlation matrix . Lifting backbone features into a wide random code of dimension substantially improves linear separability across tasks, but forces to grow quadratically as . Under a fixed persistent memory envelope $\Bcal$, a learner faces a fundamental structural trade-off: store exact pairwise co-activations inside a narrower feature code, or expand to a wide code while compressing cross-coordinate interactions. We observe that the coordinate self-variances $v_t = \diag(Σ_t)$ and class statistics grow only linearly with , whereas off-diagonal interactions dominate memory. We introduce \method (\textbf{Co}ordinate-\textbf{Re}stored \textbf{A}nalytic \textbf{C}ontinual \textbf{L}earning), which retains exactly, maintains off-diagonal co-activations through a streaming rank- factor , restores exact diagonal variances via $\Sighat_t = F_t^\top F_t + \Diag(v_t - \diag(F_t^\top F_t))$, and solves the full normal equations via the Woodbury identity. Across five class-incremental benchmarks on frozen ViT-B/16, \method matches dense full-covariance readouts within points while cutting persistent state by ( vs.\ MiB). Under equal memory envelopes of -- MiB, prioritizing expansion width over covariance rank outperforms exact narrow readouts in of configurations by up to points.
查看此版本解读 →查看此版本原文 ↗