← 研踪 PaperTrailarXiv:2609.38379 · v2
Aligned Data Can Induce Misalignment via Context Confusion
首次提交 2026-09-30 02:35
版本更新 2026-11-02 05:53
雷达观测 2026-11-04 23:42
摘要
Large language models are frequently updated for various use cases, where filtering out misaligned training samples is a common practice for preventing post-update misalignment. However, alignment is inherently context-dependent: a behavior that is aligned in one context may be inappropriate in another. For example, recommending that researchers retain their data for reproducibility is aligned, whereas recommending that a mobile app retain users' sensitive data may violate users' privacy. Starting from this observation, we identify a posttraining phenomenon where aligned training induces misaligned behavior in other contexts. We call this phenomenon context confusion. We demonstrate context confusion across three domains: (1) Gender Equality, (2) Privacy, and (3) Physical Safety. Unlike the broad misalignment seen in emergent misalignment, context confusion causes domain-specific misalignment that evades general alignment evaluations. Further, it is not effectively reduced by injecting general alignment training data, but can be substantially reduced by including targeted alignment data for the misaligned domain or providing in-context learning examples during inference. Lastly, we provide a mechanistic explanation of context confusion. We observe that queries from different domains can undergo similar representational shifts during the fine-tuning. Consequently, a query from a different domain may activate the same behavioral feature learned during fine-tuning, which causes the behavior to transfer to a context where it is misaligned. Based on our findings, we argue that it is difficult to predict the alignment state of a model after training by inspecting the training data alone, which highlights the importance of comprehensive post-training alignment evaluations.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-02 05:53 · 最新观测版本
Aligned Data Can Induce Misalignment via Context Confusion
Large language models are frequently updated for various use cases, where filtering out misaligned training samples is a common practice for preventing post-update misalignment. However, alignment is inherently context-dependent: a behavior that is aligned in one context may be inappropriate in another. For example, recommending that researchers retain their data for reproducibility is aligned, whereas recommending that a mobile app retain users' sensitive data may violate users' privacy. Starting from this observation, we identify a posttraining phenomenon where aligned training induces misaligned behavior in other contexts. We call this phenomenon context confusion. We demonstrate context confusion across three domains: (1) Gender Equality, (2) Privacy, and (3) Physical Safety. Unlike the broad misalignment seen in emergent misalignment, context confusion causes domain-specific misalignment that evades general alignment evaluations. Further, it is not effectively reduced by injecting general alignment training data, but can be substantially reduced by including targeted alignment data for the misaligned domain or providing in-context learning examples during inference. Lastly, we provide a mechanistic explanation of context confusion. We observe that queries from different domains can undergo similar representational shifts during the fine-tuning. Consequently, a query from a different domain may activate the same behavioral feature learned during fine-tuning, which causes the behavior to transfer to a context where it is misaligned. Based on our findings, we argue that it is difficult to predict the alignment state of a model after training by inspecting the training data alone, which highlights the importance of comprehensive post-training alignment evaluations.
查看此版本解读 →查看此版本原文 ↗