← 研踪 PaperTrailarXiv:2611.10971 · v1
When Safety Becomes An Outlier: Understanding the Retention of LLM Safety Behaviors
首次提交 2026-11-08 05:21
版本更新 2026-11-08 05:21
雷达观测 2026-11-10 12:16
摘要
Safety-aligned large language models often lose their safety behaviors after fine-tuning, even when safety data are included. While this fragility is well documented, its underlying cause remains unclear. In this paper, we propose that formulaic refusal strings can function as outlier modes: narrow, weakly grounded response patterns that are easy to fit but vulnerable to subsequent updates. Controlled experiments support this account: fixed refusals are acquired much like arbitrary constant strings, in contrast to prompt-grounded natural responses. Motivated by these findings, we introduce Naturalness-Enhanced Safety Targets (NEST), a safety alignment framework that improves safety retention by constructing natural, context-aware safety targets that keep safety responses within the model's existing output distribution and grounded in semantics. We instantiate NEST in both supervised and preference-based alignment settings, using perplexity under the original model as a proxy for distributional support. Across multiple models and alignment regimes, natural safety targets improve both immediate safety and its retention under continued fine-tuning without sacrificing helpfulness. These findings suggest a practical principle for more durable safety alignment: construct safety targets that draw on the model's existing language capabilities and remain grounded in the meaning of each request.
作者备注：Accepted at NeurIPS 2026
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-08 05:21 · 最新观测版本
When Safety Becomes An Outlier: Understanding the Retention of LLM Safety Behaviors
Safety-aligned large language models often lose their safety behaviors after fine-tuning, even when safety data are included. While this fragility is well documented, its underlying cause remains unclear. In this paper, we propose that formulaic refusal strings can function as outlier modes: narrow, weakly grounded response patterns that are easy to fit but vulnerable to subsequent updates. Controlled experiments support this account: fixed refusals are acquired much like arbitrary constant strings, in contrast to prompt-grounded natural responses. Motivated by these findings, we introduce Naturalness-Enhanced Safety Targets (NEST), a safety alignment framework that improves safety retention by constructing natural, context-aware safety targets that keep safety responses within the model's existing output distribution and grounded in semantics. We instantiate NEST in both supervised and preference-based alignment settings, using perplexity under the original model as a proxy for distributional support. Across multiple models and alignment regimes, natural safety targets improve both immediate safety and its retention under continued fine-tuning without sacrificing helpfulness. These findings suggest a practical principle for more durable safety alignment: construct safety targets that draw on the model's existing language capabilities and remain grounded in the meaning of each request.
查看此版本解读 →查看此版本原文 ↗