Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care
Michal Štefánik, Timothee Mickus University of Helsinki, Marek Kadlčík Masaryk University, Bertram Højer, Michal Spiegel Kempelen Institute of Intelligent Technologies, Raúl Vázquez, Aman Sinha, Josef Kuchař Masaryk University 等 10 位作者
原文摘要
Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations.In this work, we demonstrate that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different LLM families develop equivalent sinusoidal structures, and number representations are broadly interchangeable in a large swathe of experimental setups.We show that properly factoring in this characteristic is crucial when it comes to assessing how accurately LLMs encode numeric and other ordinal information, and that mechanistically enhancing this sinusoidality can also lead to reductions of LLMs’ arithmetic errors.
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表