← 研踪 PaperTrailarXiv:2610.38001 · v1
Preserving Safety During Sequential Fine-Tuning: A Factorial and Geometric Analysis of Replay
首次提交 2026-10-28 05:52
版本更新 2026-10-28 05:52
雷达观测 2026-10-30 10:23
摘要
Benign fine-tuning can erode safety alignment in large language models, and sequential adaptation substantially compounds this risk. Prior work mitigates this erosion by replaying harmful-prompt refusals and benign-compliance examples, while other work instead matches base-model logits via distillation. However, the individual contributions of supervised exposure and distillation to safety behavior and refusal geometry during sequential adaptation remain unclear. We factor these two strategies apart and study their behavioral and geometric effects during sequential LoRA fine-tuning of Mistral-7B and Llama-3-8B across diverse task sequences (code generation, German-language summarization, and science question-answering). Behaviorally, supervised replay exposure yields the largest reduction in HarmBench attack success rate (ASR) on Mistral, averaging 51.1 points below base, but at a substantial over-refusal cost, averaging 53.4 points above base on OR-Bench-hard. Logit distillation, without any supervised replay loss, largely prevents safety erosion at a fraction of this cost: an average 5.4-point ASR reduction against only a 5.1-point rise in over-refusal. Llama shows the same asymmetry between the two strategies. Combining exposure and distillation moderates exposure's over-refusal while leaving downstream task scores largely unaffected. Geometrically, the two models diverge: on Mistral, distillation preserves the base refusal direction more closely than exposure does, whereas on Llama, refusal behavior after exposure largely survives ablation of the extracted refusal direction. This dissociation between models indicates that preserving the base refusal direction is not a reliable proxy for behavioral safety.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v1 · 2026-10-28 05:52 · 最新观测版本
Preserving Safety During Sequential Fine-Tuning: A Factorial and Geometric Analysis of Replay
Benign fine-tuning can erode safety alignment in large language models, and sequential adaptation substantially compounds this risk. Prior work mitigates this erosion by replaying harmful-prompt refusals and benign-compliance examples, while other work instead matches base-model logits via distillation. However, the individual contributions of supervised exposure and distillation to safety behavior and refusal geometry during sequential adaptation remain unclear. We factor these two strategies apart and study their behavioral and geometric effects during sequential LoRA fine-tuning of Mistral-7B and Llama-3-8B across diverse task sequences (code generation, German-language summarization, and science question-answering). Behaviorally, supervised replay exposure yields the largest reduction in HarmBench attack success rate (ASR) on Mistral, averaging 51.1 points below base, but at a substantial over-refusal cost, averaging 53.4 points above base on OR-Bench-hard. Logit distillation, without any supervised replay loss, largely prevents safety erosion at a fraction of this cost: an average 5.4-point ASR reduction against only a 5.1-point rise in over-refusal. Llama shows the same asymmetry between the two strategies. Combining exposure and distillation moderates exposure's over-refusal while leaving downstream task scores largely unaffected. Geometrically, the two models diverge: on Mistral, distillation preserves the base refusal direction more closely than exposure does, whereas on Llama, refusal behavior after exposure largely survives ablation of the extracted refusal direction. This dissociation between models indicates that preserving the base refusal direction is not a reliable proxy for behavioral safety.
查看此版本解读 →查看此版本原文 ↗