← 研踪 PaperTrailarXiv:2509.22075 · v7
CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
首次提交 2025-09-26 16:55
版本更新 2026-10-27 01:53
雷达观测 2026-10-29 00:35
摘要
Post-training LLM compression often relies on low-rank approximations, which force all columns of a projection matrix to share a single low-dimensional subspace. We propose CoSpaDi, a training-free compression framework that replaces this single-subspace assumption with a union-of-subspaces model via sparse dictionary learning. CoSpaDi factorizes each weight matrix into a dense dictionary and column-sparse coefficients, allowing different columns to select different subsets of dictionary atoms at the same storage budget. To preserve model behavior, we use calibration activations to transform functional reconstruction into a standard dictionary learning problem. Across LLaMA and Qwen models, CoSpaDi improves accuracy-compression and perplexity-compression trade-offs over SVD-based and structured pruning baselines at 20-40\% compression ratios, while naturally supporting sparse-dense computation and post-training quantization of sparse coefficients.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v8 · 2026-11-03 02:38 · 最新观测版本
CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
Post-training LLM compression often relies on low-rank approximations, which force all columns of a projection matrix to share a single low-dimensional subspace. We propose CoSpaDi, a training-free compression framework that replaces this single-subspace assumption with a union-of-subspaces model via sparse dictionary learning. CoSpaDi factorizes each weight matrix into a dense dictionary and column-sparse coefficients, allowing different columns to select different subsets of dictionary atoms at the same storage budget. To preserve model behavior, we use calibration activations to transform functional reconstruction into a standard dictionary learning problem. Across LLaMA and Qwen models, CoSpaDi improves accuracy-compression and perplexity-compression trade-offs over SVD-based and structured pruning baselines at 20-40\% compression ratios, while naturally supporting sparse-dense computation and post-training quantization of sparse coefficients.
查看此版本解读 →查看此版本原文 ↗v7 · 2026-10-27 01:53
CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
Post-training LLM compression often relies on low-rank approximations, which force all columns of a projection matrix to share a single low-dimensional subspace. We propose CoSpaDi, a training-free compression framework that replaces this single-subspace assumption with a union-of-subspaces model via sparse dictionary learning. CoSpaDi factorizes each weight matrix into a dense dictionary and column-sparse coefficients, allowing different columns to select different subsets of dictionary atoms at the same storage budget. To preserve model behavior, we use calibration activations to transform functional reconstruction into a standard dictionary learning problem. Across LLaMA and Qwen models, CoSpaDi improves accuracy-compression and perplexity-compression trade-offs over SVD-based and structured pruning baselines at 20-40\% compression ratios, while naturally supporting sparse-dense computation and post-training quantization of sparse coefficients.
查看此版本解读 →查看此版本原文 ↗