← 研踪 PaperTrailarXiv:2512.21651 · v4
Beyond local reconstruction: Geometry-preserving output alignment for binary LLM quantization
首次提交 2025-12-25 20:39
版本更新 2026-11-10 02:49
雷达观测 2026-11-11 11:04
摘要
Recent advances in post-training quantization for large language models (LLMs) have demonstrated that low-bit quantization methods can maintain most of the original model performance. Despite such successes, 1-bit quantization remains particularly challenging. A common strategy in 1-bit quantization is to determine binary weights by matching full-precision parameters, following a weight-driven criterion. However, this objective is not directly aligned with the quantized model's objective, which is to preserve the model's output behavior under the impact of quantization. A natural alternative is to adopt output-driven criteria that minimize discrepancies in model outputs using calibration data. Surprisingly, naive output-driven approaches often perform even worse in the 1-bit regime. In this paper, we show that this failure arises from two fundamental issues: error accumulation across layers and, more critically, anisotropic distortion of the representation space. Based on this observation, we introduce a geometry-preserving output-alignment framework for 1-bit LLM quantization. First, we formulate output alignment as a constrained optimization problem that improves reconstruction while explicitly controlling changes in token similarity, preventing reconstruction gains from being achieved at the expense of representation geometry. Second, we propose a geometry-aware salient-mask selection strategy that identifies quantization configurations according to their ability to preserve token relations, rather than relying solely on reconstruction error. These components enable effective output-driven optimization while retaining the structural information lost by naive alignment. Extensive experiments across multiple LLM families and downstream tasks demonstrate consistent improvements over existing 1-bit PTQ methods.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v4 · 2026-11-10 02:49 · 最新观测版本
Beyond local reconstruction: Geometry-preserving output alignment for binary LLM quantization
Recent advances in post-training quantization for large language models (LLMs) have demonstrated that low-bit quantization methods can maintain most of the original model performance. Despite such successes, 1-bit quantization remains particularly challenging. A common strategy in 1-bit quantization is to determine binary weights by matching full-precision parameters, following a weight-driven criterion. However, this objective is not directly aligned with the quantized model's objective, which is to preserve the model's output behavior under the impact of quantization. A natural alternative is to adopt output-driven criteria that minimize discrepancies in model outputs using calibration data. Surprisingly, naive output-driven approaches often perform even worse in the 1-bit regime. In this paper, we show that this failure arises from two fundamental issues: error accumulation across layers and, more critically, anisotropic distortion of the representation space. Based on this observation, we introduce a geometry-preserving output-alignment framework for 1-bit LLM quantization. First, we formulate output alignment as a constrained optimization problem that improves reconstruction while explicitly controlling changes in token similarity, preventing reconstruction gains from being achieved at the expense of representation geometry. Second, we propose a geometry-aware salient-mask selection strategy that identifies quantization configurations according to their ability to preserve token relations, rather than relying solely on reconstruction error. These components enable effective output-driven optimization while retaining the structural information lost by naive alignment. Extensive experiments across multiple LLM families and downstream tasks demonstrate consistent improvements over existing 1-bit PTQ methods.
查看此版本解读 →查看此版本原文 ↗