GeoPair:保持几何结构的跨层分解用于免训练 Transformer 压缩
GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression
Baher Mohammad, Ammar Ali, Stamatios Lefkimmiatis
AI 速读 基于摘要
提出一种免训练框架,通过优化跨层权重配对与共享字典分解,在保留各层校准几何的前提下压缩 Transformer。
cs.LG
查看原文摘要
Transformer architectures exhibit cross-layer redundancies, yet post-training compression pipelines typically optimize layers in isolation or rely on heuristic grouping strategies that disregard layer-specific activation geometries. We introduce a principled, training-free framework that sequentially optimizes cross-layer weight pairings and shared-dictionary factorizations. Rather than forcing weights of adjacent layers to share a basis or heuristically merging activation statistics, our approach identifies structurally compatible projections and learns a shared representation that better preserves each layer's distinct calibration geometry. Coupled with structured sparsity, this yields highly efficient weight decompositions without sacrificing functional fidelity. Across diverse architectures, scales, and modalities, our method achieves state-of-the-art results, consistently outperforming independent structured weight decompositions and alternative pairwise weight factorizations, which operate under heuristic grouping strategies. By replacing heuristic engineering strategies with a convergent, optimization-driven pipeline, we establish a theoretically grounded foundation for scalable, transformer compression across different modalities.