Rotated Robustness: A Training-Free Defense against Bit-Flip Attacks on Large Language Models
Deng Liu, Song Chen
原文摘要
Bit-flip corruption of quantized weights poses a serious reliability threat to Large Language Models (LLMs), as even a small number of weight faults can trigger catastrophic model degradation. We show that such failures can be strongly amplified when corrupted weights are coupled to high-sensitivity activation channels. Based on this observation, we propose Rotated Robustness (RoR), a training-free sensitivity-aware hierarchical rotation that applies matched orthogonal transformations to activations and weights, redistributing sensitive channel contributions across feature dimensions while preserving the underlying linear mapping in exact arithmetic. Across eight evaluated LLMs, RoR matches or exceeds all compared methods in the number of cumulative PBS flips sustained before PPL exceeds 100 within the 100-flip evaluation horizon. Under BER-based random bit-flip injection, no RoR failures are observed in the evaluated 1,000-trial experiments for the tested models. Under defense-aware SPoF reconstruction, RoR increases the cost of reproducing specified one-bit catastrophic failures to thousands of deployed INT8 bit changes. RoR also preserves downstream-task utility under PBS perturbations. On NVIDIA H200 at batch size 8, RoR incurs 0.097\%--0.584\% persistent storage overhead, 5.2\%--14.1\% Prefill overhead, and 0.5\%--9.8\% Decode overhead across four representative models. These results show that sensitivity-aware hierarchical rotation provides a practical approach to improving the bit-flip robustness of quantized LLM Linear weights.
cs.CR