Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese
Pedro Carvalho Brom
AI 速览：一篇教程式论文，展示如何在单块消费级 GPU 上从零训练含分词器的葡萄牙语小型语言模型，并用预注册协议比较自回归与掩码扩散两种生成方式在专业法律任务上的表现。
Building a language model of one's own no longer requires a data centre. This tutorial trains one from scratch, tokenizer included, on one consumer GPU, and shows how to tell whether it is good for the tasks of a specialist domain. It proposes no new method. It releases a tested Python package and two notebooks in Brazilian Portuguese with autoregressive and masked diffusion generation on one network, a preregistered comparison protocol, legal tasks built from text alone with a Kneser-Ney floor and public yardsticks, reference numbers for models of 11 to 12M parameters, and a catalogue of defects that only measurement revealed. On Brazilian prose and federal laws, at minutes per run, the autoregressive mode peaks at 64 epochs over the same text and then memorises it, while masked diffusion improves up to 256. At their best points on test, the diffusion bound exceeds the autoregressive likelihood by 0.06 bits per byte, within the preregistered minimum effect. On held-out laws and on acts published after training, the model ties a 6-gram floor in likelihood on the same windows, and it loses to the floor at continuing those acts. It beats the floor at detecting tampered text, and fills cross-references more often than the floor and public models of 160M and 630M parameters, only in decrees, a genre absent from training, where those scorers stay near or below chance, and in filling the blank only when each group weighs the same, since per item it ties the 630M model. On laws, the public models, trained with about 5x10^2 to 10^5 times its compute, lead in tampered text and continuation. The blind human reading is preregistered and not part of this version.
Typed decision models return a probability distribution over options supplied by the caller, and software tends to act on that number as if it were a guarantee. We measure what it licenses, comparing the open-weight Laya, in five checkpoints, with Jev 1.13, available only through an API. A probability licenses a statement about groups of decisions, not about one decision, and a gate that promises an accuracy among the cases it decides alone keeps that promise only with a lower bound checked on data like its traffic. Across 190 exchangeable evaluations, a threshold chosen by point estimate breaks the promise in 147, Learn then Test keeps it in all of them, and a Beta lower-bound rule decides more items on average than Learn then Test in 128 and fewer in none, without its guarantee. Calibrated in English, Jev keeps a promise of 95% in 8 of 51 locales. Ordinary typing noise breaks a promise calibrated on clean text, and, in exploratory runs, training with noise raises the noisy accuracy with no loss on clean text. In four further domains, a fine-tuned Laya needs from 250 to 8,000 labels to be non-inferior to Jev, and does not get there on LIAR2. Under drift, the preregistered claim for a Bayesian retraining loop is not supported: the loop avoids needless retraining where the change does not degrade the model, at a cost equivalent to recalibration alone, and re-qualifying the threshold in every window keeps the promise more often where it does. On the monthly series of M3, neither decider predicts the direction of the next value better than a constant answer. Confidence separates right from wrong answers with an AUROC from 0.80 to 0.96, but it does not select faithful explanations, and exact coalition games of single decisions show what an explanation can still say through an API that rounds to two decimals. We release the frozen answers with a datasheet, a promise check and the code.