Ari Spiesberger, Juan J. Vazquez, Nicky Pochinkov, Tomáš Gavenčiak, Peli Grietzer, Gavin Leech, Nandi Schoots
原文摘要
Training on benchmark test items is a classic way to overestimate generalization to unseen problems. Standard decontamination methods miss semantic duplicates (SemDs): training items equivalent in content to a test item, but distant in string space. We call training on such items soft contamination; when the resulting gains extend to held-out items of the same benchmark (but not to related benchmarks), we call the result shallow generalization. We first embed Olmo 3's entire post-training dataset and search it for SemDs of benchmark items; we then finetune Olmo 3 and Qwen models on natural and synthetic semantic duplicates. We find (1) that soft contamination is widespread: the Olmo corpus contains SemDs of 76% of the CodeForces test set and 100% of MBPP's; (2) that finetuning on SemDs of a benchmark raises accuracy on its held-out items by up to 22pp in controlled experiments and 10-11pp at realistic contamination rates, though the effect varies across benchmarks; (3) that the gains from training on SemDs mostly do not transfer to conceptually similar benchmarks, i.e. we find shallow generalization; (4) that finetuning on items with similar embeddings that are not SemDs produces no such gain, indicating that the mechanism is not mere topical similarity; and (5) that at realistic contamination rates, soft contamination can raise held-out performance about as much as exact duplicates (or more). Benchmark gains here are therefore confounded - in a way that standard decontamination does not address.
Towards interpreting any residual stream direction by scaling activation inversion
Celeste De Schamphelaere, Juan J. Vazquez, Ari Spiesberger, Tomáš Gavenčiak, Gavin Leech, Euan Ong, Jiahai Feng
AI 速读 基于摘要
提出 Maximally-Activating Example Metamodels(MAEMs),用无标签强化学习把残差流方向映射为能强烈激活该方向的文本样例,以可验证的方式解读任意方向。
cs.LG · cs.AI
查看原文摘要
To interpret a direction in a language model's residual stream, we can look for text that strongly activates it. But when such text is found by scanning a corpus, it is limited to what that corpus contains. We develop Maximally-Activating Example Metamodels (MAEMs), language models which map residual stream directions from a subject model to text exemplars that strongly activate those directions. Our objective is to maximise the cosine similarity of a supplied residual stream direction with the activations of the text generated by the MAEM. This label-free RL objective helps to scale MAEMs on diverse sources of residual stream directions. Empirically, we find the resultant metamodel produces informative exemplars: a Qwen3.6-27B MAEM inverts held-out activations better than corpus search or a natural-language autoencoder (NLA), beats corpus search (at 10M tokens) on a majority of held-out languages/domains, and activates 89% of held-out SAE features. It also identifies concepts behind steering vectors about as well as NLAs, recovers the payloads of LoRA backdoors more often than corpus search, and can even verbalise some model workspace inner monologue. Unlike explanations produced by other metamodels (e.g. NLAs), MAEM exemplars can be verified with a forward pass on the subject model, making the MAEM objective a grounded alternative for scaling the training of metamodels.