跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2610.22771 · v2

Cross-Precision Transfer, Non-Local Context, and Capacity-Readout Boundaries of Scale Calibration in Low-Bit LLMs

首次提交 2026-10-16 08:21
版本更新 2026-11-16 07:18
雷达观测 2026-11-17 17:12
arXiv ↗PDF ↗

摘要

Low-bit post-training quantization freezes integer codes and zero points, leaving groupwise scales as a compact adaptation interface. We study cross-precision transfer, non-local context, optimizer capacity, and readout behavior on Qwen3-1.7B and Llama-3.2-3B. First, transferring relative scale gains between floating-point and INT3 Qwen models with zero recipient training retains 76.9%-90.5% of self-calibration NLL reductions on fresh WikiText and C4, alongside resolved own-source preferences. Group permutation destroys transfer, and norm-matched layer withdrawal degrades all 16 layer-domain comparisons. A Llama control with identical initial tensors at all calibrated projections isolates preferences induced by surrounding network context. Second, expanding Llama calibration from 4 to 196 projections (0.79M to 22.02M scales) makes learning rate 0.01 win a 50-step training-loss pilot. Continuing to 200 steps memorizes the 28-block WikiText pool, inflates unseen perplexity by 2.06-2.28 times relative to 4-site calibration, and degrades mixed-action PIQA accuracy; learning rate 0.001 prevents memorization and restores accuracy. Third, across 24 Qwen ARC readout endpoints, advancing from 200 to 600 steps worsens INT3 teacher KL on plain-text readouts yet improves it under chat wrapping. Cross-fitted four-parameter static option bias explains 71.4%-75.4% of quantized Chat-Space score-shift SSE, versus less than 2.3% on floating point. Subtracting this bias raises INT3 chat teacher agreement from 33.3%-33.9% to 56.6%-57.5% and cuts KL from 4.15-4.42 to 1.18-1.24 nats.

作者备注:26 pages, 4 figures, 18 tables. Revised and expanded version with an updated title

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-16 07:18 · 最新观测版本

Cross-Precision Transfer, Non-Local Context, and Capacity-Readout Boundaries of Scale Calibration in Low-Bit LLMs

Low-bit post-training quantization freezes integer codes and zero points, leaving groupwise scales as a compact adaptation interface. We study cross-precision transfer, non-local context, optimizer capacity, and readout behavior on Qwen3-1.7B and Llama-3.2-3B. First, transferring relative scale gains between floating-point and INT3 Qwen models with zero recipient training retains 76.9%-90.5% of self-calibration NLL reductions on fresh WikiText and C4, alongside resolved own-source preferences. Group permutation destroys transfer, and norm-matched layer withdrawal degrades all 16 layer-domain comparisons. A Llama control with identical initial tensors at all calibrated projections isolates preferences induced by surrounding network context. Second, expanding Llama calibration from 4 to 196 projections (0.79M to 22.02M scales) makes learning rate 0.01 win a 50-step training-loss pilot. Continuing to 200 steps memorizes the 28-block WikiText pool, inflates unseen perplexity by 2.06-2.28 times relative to 4-site calibration, and degrades mixed-action PIQA accuracy; learning rate 0.001 prevents memorization and restores accuracy. Third, across 24 Qwen ARC readout endpoints, advancing from 200 to 600 steps worsens INT3 teacher KL on plain-text readouts yet improves it under chat wrapping. Cross-fitted four-parameter static option bias explains 71.4%-75.4% of quantized Chat-Space score-shift SSE, versus less than 2.3% on floating point. Subtracting this bias raises INT3 chat teacher agreement from 33.3%-33.9% to 56.6%-57.5% and cuts KL from 4.15-4.42 to 1.18-1.24 nats.

查看此版本解读 →查看此版本原文 ↗