The Token Before the Value Is the Key
Ke Cheng, Xin Xu, Zhichao Chen, Yixiao Chen, Lei Xin, Jianbo Zhao, Fanhu Zeng, Yue Liu 等 10 位作者
原文摘要
Induction circuits retrieve a value by matching a query to information carried from the token before that value. How this address forms, incorporates context, and supports sequential retrieval remains less clear. We use hybrid language models as a tractable testbed and identify the historical source key as a common causal interface between local preparation and global retrieval. Probes and restoration interventions connect this interface across controlled models, pretrained recurrent and sliding-window hybrids, and released language models. We organize the analysis by address context radius and sequential retrieval depth. First, shared local parameters couple predecessor Carrying to recurrent state processing; separating these demands or biasing initialization promotes address formation. Second, adapting input representations extends an established literal circuit to category matching, making semantically compatible records active competitors. Earlier context resolves this competition through the source key, with a sharp visibility boundary in a controlled single-read model and compositional relays across local layers. Third, fixed relations support memorization shortcuts, whereas sequential generalization requires learning an output-to-query bridge between retrievals. Controlled state-tracking and compositional in-context tasks demonstrate this requirement across stacked stages, looped recurrence, and intermediate-token generation. Together, these results connect circuit formation, contextual disambiguation, and sequential use through a shared retrieval interface. Code is available at https://github.com/ckpassenger/bind-match-copy.
cs.LG