跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.26450 · v1

A CPU Inside an LLM: Zero-Gradient Compilation of Exact Circuits into Pretrained Weights

LLM中的CPU:将精确电路零梯度编译进预训练权重

首次提交 2026-11-20 01:18
版本更新 2026-11-20 01:18
雷达观测 2026-11-20 12:52
arXiv ↗PDF ↗

摘要

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid (>250×>250\times dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension (d→d+Kd \to d+K) with d/(d+K)\sqrt{d/(d+K)} norm compensation, Base-1000 canonical BPE emission, and differential (+254,−256)(+254,-256) tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed {+,−,×}\{+,-,\times\} circuits in constant TC0\mathsf{TC}^0 depth alongside 3-literal Boolean (100%, N=1,728N=1{,}728; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic (N=1,200N=1{,}200, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, ≤0.036%\le 0.036\%) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, ≤0.057%\le 0.057\%) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered >2>2K-token Tango splits (N=4,392N=4{,}392; 0/11,134 OOD triggers).

作者备注:27 pages, 3 figures, 24 tables

AI 论文解读

基于摘要 · 非全文审阅

作者提出零梯度、闭式的电路嫁接原语,将确定性算术电路直接编译进预训练语言模型权重,在无需训练样本和运行时钩子的情况下实现精确多位数算术并保持自然语言能力。

正在读取账号与研究记录…

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-20 01:18 · 最新观测版本

A CPU Inside an LLM: Zero-Gradient Compilation of Exact Circuits into Pretrained Weights

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid (>250×>250\times dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension (d→d+Kd \to d+K) with d/(d+K)\sqrt{d/(d+K)} norm compensation, Base-1000 canonical BPE emission, and differential (+254,−256)(+254,-256) tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed {+,−,×}\{+,-,\times\} circuits in constant TC0\mathsf{TC}^0 depth alongside 3-literal Boolean (100%, N=1,728N=1{,}728; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic (N=1,200N=1{,}200, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, ≤0.036%\le 0.036\%) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, ≤0.057%\le 0.057\%) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered >2>2K-token Tango splits (N=4,392N=4{,}392; 0/11,134 OOD triggers).

查看此版本解读 →查看此版本原文 ↗