← 研踪 PaperTrailarXiv:2603.16382 · v3
Rotated Robustness: A Training-Free Defense against Bit-Flip Attacks on Large Language Models
首次提交 2026-03-17 19:11
版本更新 2026-11-10 21:42
雷达观测 2026-11-11 13:08
摘要
Bit-flip corruption of quantized weights poses a serious reliability threat to Large Language Models (LLMs), as even a small number of weight faults can trigger catastrophic model degradation. We show that such failures can be strongly amplified when corrupted weights are coupled to high-sensitivity activation channels. Based on this observation, we propose Rotated Robustness (RoR), a training-free sensitivity-aware hierarchical rotation that applies matched orthogonal transformations to activations and weights, redistributing sensitive channel contributions across feature dimensions while preserving the underlying linear mapping in exact arithmetic. Across eight evaluated LLMs, RoR matches or exceeds all compared methods in the number of cumulative PBS flips sustained before PPL exceeds 100 within the 100-flip evaluation horizon. Under BER-based random bit-flip injection, no RoR failures are observed in the evaluated 1,000-trial experiments for the tested models. Under defense-aware SPoF reconstruction, RoR increases the cost of reproducing specified one-bit catastrophic failures to thousands of deployed INT8 bit changes. RoR also preserves downstream-task utility under PBS perturbations. On NVIDIA H200 at batch size 8, RoR incurs 0.097\%--0.584\% persistent storage overhead, 5.2\%--14.1\% Prefill overhead, and 0.5\%--9.8\% Decode overhead across four representative models. These results show that sensitivity-aware hierarchical rotation provides a practical approach to improving the bit-flip robustness of quantized LLM Linear weights.
作者备注：15 pages, 8 figures. Accepted by TIFS 2026
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-10 21:42 · 最新观测版本
Rotated Robustness: A Training-Free Defense against Bit-Flip Attacks on Large Language Models
Bit-flip corruption of quantized weights poses a serious reliability threat to Large Language Models (LLMs), as even a small number of weight faults can trigger catastrophic model degradation. We show that such failures can be strongly amplified when corrupted weights are coupled to high-sensitivity activation channels. Based on this observation, we propose Rotated Robustness (RoR), a training-free sensitivity-aware hierarchical rotation that applies matched orthogonal transformations to activations and weights, redistributing sensitive channel contributions across feature dimensions while preserving the underlying linear mapping in exact arithmetic. Across eight evaluated LLMs, RoR matches or exceeds all compared methods in the number of cumulative PBS flips sustained before PPL exceeds 100 within the 100-flip evaluation horizon. Under BER-based random bit-flip injection, no RoR failures are observed in the evaluated 1,000-trial experiments for the tested models. Under defense-aware SPoF reconstruction, RoR increases the cost of reproducing specified one-bit catastrophic failures to thousands of deployed INT8 bit changes. RoR also preserves downstream-task utility under PBS perturbations. On NVIDIA H200 at batch size 8, RoR incurs 0.097\%--0.584\% persistent storage overhead, 5.2\%--14.1\% Prefill overhead, and 0.5\%--9.8\% Decode overhead across four representative models. These results show that sensitivity-aware hierarchical rotation provides a practical approach to improving the bit-flip robustness of quantized LLM Linear weights.
查看此版本解读 →查看此版本原文 ↗