跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.23093 · v1

What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language

首次提交 2026-11-18 01:14
版本更新 2026-11-18 01:14
雷达观测 2026-11-18 16:38
arXiv ↗PDF ↗

摘要

Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative nn-gram models that build the conditional probability tensor p(xn∣x1:n−1)p(x_n \mid x_{1:n-1}) through a hierarchical composition of order-2 and order-3 tensors. nn controls the maximal context window, whereas a rank hyperparameter dd controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing nn-gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under μμP can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate nn and dd. In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95\% of topologies without significant change in performance.

作者备注:7+13 figures, 10+30 pages

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-18 01:14 · 最新观测版本

What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language

Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative nn-gram models that build the conditional probability tensor p(xn∣x1:n−1)p(x_n \mid x_{1:n-1}) through a hierarchical composition of order-2 and order-3 tensors. nn controls the maximal context window, whereas a rank hyperparameter dd controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing nn-gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under μμP can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate nn and dd. In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95\% of topologies without significant change in performance.

查看此版本解读 →查看此版本原文 ↗