Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
Baris Askin, Muhammed Ustaomeroglu, Anupam Nayak, Gauri Joshi, Guannan Qu, Carlee Joe-Wong
原文摘要
Post-training a language model can induce surprising behavioral changes far beyond the intended training target. We study two particularly striking examples: emergent misalignment (EM), where fine-tuning on a narrow dataset leads to misaligned behavior in unrelated contexts, and subliminal learning (SL), where a student model inherits a teacher's misalignment through seemingly benign training data. We examine how the training data, evaluation data, and training method shape where and how strongly misalignment transfers between data and models. We characterize the data by the domain of a prompt (its topical context) and its task (the function the model performs), and we build natural-language and synthetic datasets organized as domain-task grids. We find that EM transfers more strongly to prompts that share the task of the fine-tuning data than to prompts that share its domain. In the synthetic setting, tasks that the model has learned well spread misalignment more broadly. For SL, we go beyond SFT (supervised fine-tuning) to off-policy distillation on the teacher's full next-token distribution and, to our knowledge for the first time, on-policy distillation, where the teacher only scores tokens sampled by the student. Misalignment transfers even under on-policy distillation, more strongly than under SFT and at a level comparable to off-policy distillation. Unlike SFT, where the teacher both generates the data and supervises the student, distillation lets us separate the two. With the teacher fixed, the data change how much misalignment the student inherits, but the teacher determines to which domains and tasks it transfers most strongly. Together, these results show how data and training influence the extent and patterns of emergent and subliminal misalignment.
cs.LG · cs.AI · cs.CL