← 研踪 PaperTrailarXiv:2604.16656 · v2
Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion
首次提交 2026-04-18 03:21
版本更新 2026-11-02 07:31
雷达观测 2026-11-04 23:42
摘要
All languages are equal; when it comes to tokenization, some are more equal than others. Tokens are the hidden currency that dictates the cost and latency of contemporary LLM access. However, many languages written in non-Latin scripts observe a poor exchange rate: LLMs take several multiples of tokens to encode the same information in many languages as they do for English. We observe that this issue, known as token overfragmentation, persists in modern open-weight LLMs. The standard remedy is vocabulary expansion that adds target language items missing from the model's vocabulary. We focus on two core decisions in the vocabulary expansion process: What items should we add? and How should we initialize their corresponding input and output embeddings? We comprehensively study and advance the new research direction of interpretability-based vocabulary expansion along these two core decisions, making four key contributions: First, we question the conventional use of frequency-based methods to choose candidate vocabulary items to add (a decision long treated as settled), and show that interpretability-based methods offer a superior performance-token efficiency tradeoff. Second, we strengthen the case for interpretability-based embedding initialization by showing large gains (~20 pts) over baseline initialization methods for several languages written in non-Latin scripts. Third, we identify the phenomenon of "subword detokenization," where models progressively merge fragmented subword tokens into larger subwords across layers. Fourth, we propose FragMend, a method that leverages this phenomenon to push the efficiency ceiling of interpretability-based expansion, and validate its effectiveness.
作者备注：Published at COLM 2026
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-02 07:31 · 最新观测版本
Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion
All languages are equal; when it comes to tokenization, some are more equal than others. Tokens are the hidden currency that dictates the cost and latency of contemporary LLM access. However, many languages written in non-Latin scripts observe a poor exchange rate: LLMs take several multiples of tokens to encode the same information in many languages as they do for English. We observe that this issue, known as token overfragmentation, persists in modern open-weight LLMs. The standard remedy is vocabulary expansion that adds target language items missing from the model's vocabulary. We focus on two core decisions in the vocabulary expansion process: What items should we add? and How should we initialize their corresponding input and output embeddings? We comprehensively study and advance the new research direction of interpretability-based vocabulary expansion along these two core decisions, making four key contributions: First, we question the conventional use of frequency-based methods to choose candidate vocabulary items to add (a decision long treated as settled), and show that interpretability-based methods offer a superior performance-token efficiency tradeoff. Second, we strengthen the case for interpretability-based embedding initialization by showing large gains (~20 pts) over baseline initialization methods for several languages written in non-Latin scripts. Third, we identify the phenomenon of "subword detokenization," where models progressively merge fragmented subword tokens into larger subwords across layers. Fourth, we propose FragMend, a method that leverages this phenomenon to push the efficiency ceiling of interpretability-based expansion, and validate its effectiveness.
查看此版本解读 →查看此版本原文 ↗