← 研踪 PaperTrailarXiv:2606.23700 · v2
Self-Modeling Interventions can Modulate Emergent Misalignment
首次提交 2026-06-04 08:04
版本更新 2026-11-10 06:42
雷达观测 2026-11-11 11:04
摘要
Emergent misalignment (EM) has been linked to the activation of misaligned persona vectors and evil character traits, suggesting that EM operates through disruption of the model's aligned character rather than direct learning of harmful content. Motivated by this connection, we study self-modeling interventions: specifically self-recognition and self-reports as character-targeted interventions that are distinct from existing in-training defenses. We conduct two-stage finetuning experiments across three models (GPT-4.1, Qwen2.5-32B-Instruct, Seed-OSS-36B-Instruct) and multiple EM datasets to compare self-recognition finetuning against benign finetuning baselines (correct domain-specific data, general knowledge, and word counting), finding it an effective defense in both reversal and prevention settings. For prevention, only self-recognition finetuning consistently reduces misalignment without exacerbating any individual metric, suggesting that character fortification specifically drives prevention. We also conduct in-training experiments across these models to compare the effectiveness of self-reports with benign interleaving and inoculation prompting. We provide further evidence for EM's relation to the LLM's default character by showing that EM finetuning leads to fragmentation of LLM's identity self-reports, reduces self-recognition performance to about chance, artificially corrupting self-recognition exacerbates misalignment caused by EM finetuning, and finetuning on self-reports from EM finetuned models can transfer misalignment. Together, these findings reframe EM not as the adoption of a coherent misaligned persona but as the destabilization of aligned character.
作者备注：18 pages, 11 figures. Accepted to NeurIPS 2026
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-10 06:42 · 最新观测版本
Self-Modeling Interventions can Modulate Emergent Misalignment
Emergent misalignment (EM) has been linked to the activation of misaligned persona vectors and evil character traits, suggesting that EM operates through disruption of the model's aligned character rather than direct learning of harmful content. Motivated by this connection, we study self-modeling interventions: specifically self-recognition and self-reports as character-targeted interventions that are distinct from existing in-training defenses. We conduct two-stage finetuning experiments across three models (GPT-4.1, Qwen2.5-32B-Instruct, Seed-OSS-36B-Instruct) and multiple EM datasets to compare self-recognition finetuning against benign finetuning baselines (correct domain-specific data, general knowledge, and word counting), finding it an effective defense in both reversal and prevention settings. For prevention, only self-recognition finetuning consistently reduces misalignment without exacerbating any individual metric, suggesting that character fortification specifically drives prevention. We also conduct in-training experiments across these models to compare the effectiveness of self-reports with benign interleaving and inoculation prompting. We provide further evidence for EM's relation to the LLM's default character by showing that EM finetuning leads to fragmentation of LLM's identity self-reports, reduces self-recognition performance to about chance, artificially corrupting self-recognition exacerbates misalignment caused by EM finetuning, and finetuning on self-reports from EM finetuned models can transfer misalignment. Together, these findings reframe EM not as the adoption of a coherent misaligned persona but as the destabilization of aligned character.
查看此版本解读 →查看此版本原文 ↗