← 研踪 PaperTrailarXiv:2608.30741 · v2
Functional Degeneracy in Neural Networks: Measurement and Pruning
首次提交 2026-08-31 21:09
版本更新 2026-11-16 17:13
雷达观测 2026-11-17 17:11
摘要
A central question in modern machine learning is how much a trained model can be compressed without changing its behavior, to reduce the memory, compute and energy required to deploy it. To study this, we quantify functional degeneracy through the behavioral recovery rank, the number of leading behavioral-Hessian eigendirections required to recover a model's performance to a given tolerance. Using it as a geometric benchmark for compression, we find that the rank flattens once the task is saturated, while the number of parameters retained by one-shot pruning keeps growing. This gap suggests that functional degeneracy is distributed across parameter directions and not exposed by individual weights or neurons.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-16 17:13 · 最新观测版本
Functional Degeneracy in Neural Networks: Measurement and Pruning
A central question in modern machine learning is how much a trained model can be compressed without changing its behavior, to reduce the memory, compute and energy required to deploy it. To study this, we quantify functional degeneracy through the behavioral recovery rank, the number of leading behavioral-Hessian eigendirections required to recover a model's performance to a given tolerance. Using it as a geometric benchmark for compression, we find that the rank flattens once the task is saturated, while the number of parameters retained by one-shot pruning keeps growing. This gap suggests that functional degeneracy is distributed across parameter directions and not exposed by individual weights or neurons.
查看此版本解读 →查看此版本原文 ↗