Elias Sandmann, Sebastian Lapuschkin, Wojciech Samek
Recent mechanistic work has uncovered learned algorithms within neural networks, from modular arithmetic to search and planning in game-playing agents. But does algorithmic structure guarantee algorithmic behavior? We investigate this in Leela Chess Zero, the strongest neural chess engine, where prior work identified learned look-ahead. By extending the logit lens to its move-selecting policy network, we discover that correct puzzle solutions--including immediate checkmates--often appear in intermediate layers but are systematically overridden in the final output, a phenomenon we term "forgotten puzzles". Replicating prior analyses on these positions, we find that look-ahead operates normally--future moves of the correct continuation are represented, causally important, and linearly decodable--ruling out a failure of the algorithm. Instead, late layers increasingly shift toward safe play over aggression. To test whether this shift drives the override, we steer the model against these preferences and recover 71.7% of forgotten puzzles, providing causal evidence that safety priors override algorithmically computed solutions. These findings show that algorithmic structure does not guarantee algorithmic behavior: a model can internally solve a problem and still output the wrong answer.
Simon Ostermann German Research Center for AI and Universität des Saarlandes, Daniil Gurgurov, Tanja Baeumel, Michael A. Hedderich Ludwig-Maximilians-Universität München, Sebastian Lapuschkin Technological University Dublin and Fraunhofer HHI, Wojciech Samek TU Berlin and Fraunhofer HHI, Vera Schmitt Technische Universität Berlin
Post-training adaptation of large language models is commonly achieved through parameter updates or input based methods such as fine-tuning, parameter-efficient adaptation, and prompting. In parallel, a growing body of work modifies internal activations at inference time to influence model behavior, an approach known as steering. Despite increasing use, steering is rarely analyzed within the same conceptual framework as established adaptation methods.In this work, we argue that steering should be regarded as a form of model adaptation. We introduce a set of functional criteria for adaptation methods and use them to compare steering approaches with classical alternatives. This analysis positions steering as a distinct adaptation paradigm based on targeted interventions in activation space, enabling local and reversible behavioral change without parameter updates. The resulting framing clarifies how steering relates to existing methods, motivating a unified taxonomy for model adaptation.
尚无 AI 解读，展示原文摘要
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表