Ke Cheng, Xin Xu, Zhichao Chen, Yixiao Chen, Lei Xin, Jianbo Zhao, Fanhu Zeng, Yue Liu 等 10 位作者
Induction circuits retrieve a value by matching a query to information carried from the token before that value. How this address forms, incorporates context, and supports sequential retrieval remains less clear. We use hybrid language models as a tractable testbed and identify the historical source key as a common causal interface between local preparation and global retrieval. Probes and restoration interventions connect this interface across controlled models, pretrained recurrent and sliding-window hybrids, and released language models. We organize the analysis by address context radius and sequential retrieval depth. First, shared local parameters couple predecessor Carrying to recurrent state processing; separating these demands or biasing initialization promotes address formation. Second, adapting input representations extends an established literal circuit to category matching, making semantically compatible records active competitors. Earlier context resolves this competition through the source key, with a sharp visibility boundary in a controlled single-read model and compositional relays across local layers. Third, fixed relations support memorization shortcuts, whereas sequential generalization requires learning an output-to-query bridge between retrievals. Controlled state-tracking and compositional in-context tasks demonstrate this requirement across stacked stages, looped recurrence, and intermediate-token generation. Together, these results connect circuit formation, contextual disambiguation, and sequential use through a shared retrieval interface. Code is available at https://github.com/ckpassenger/bind-match-copy.
What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use
Yixiao Chen, Ke Cheng, Yue Liu, Jun Zhang, Jie Jiang
AI 速览：作者提出一种“电路视角”的数据诊断原则，将语言模型训练中的瓶颈分为计算形成、内容可用与路径选择三类，并报告相应数据干预在350M模型实验中的效果。
What should data teach a language model at a particular point in training? A circuit view reveals three distinct bottlenecks: forming a computation, making its required content available, and selecting among available routes. A shared diagnosis-to-data principle connects them: localize the missing operation, preserve its causal relation, vary shortcut-bearing context, and re-audit the residual. Formation-sensitive selection and prerequisite ordering accelerate a binding-matching-transport path; a brief early prefix from the same training multiset retains a validation advantage through 100B tokens. Availability counterfactuals then distinguish writing content from invoking available memory, while paired supervision and context-opportunity ranking improve matched route decisions and long-context answer likelihood. A continuous 350M-model experiment connects the three interventions on the same facts: early circuit training improves subsequent learning, and the complete sequence outperforms stage-replacement controls on facts withheld from Use teaching. Independent query surfaces and opposed-source decisions expose conditional arbitration as the remaining frontier. Together, these results show why a change in the limiting operation calls for a change in supervision, not merely a new ranking of difficult examples.
The Delaunay triangulation is a classical geometric structure on a finite set of points in the plane. Its stretch factor is the largest ratio of the shortest-path distance along its edges to the Euclidean distance between the same two points. Determining the worst-case stretch factor of planar Delaunay triangulations is a long-standing open problem. We prove a uniform upper bound of 1.5949, improving the previous bound of 1.998 and narrowing the gap to the best known lower bound of 1.5932 from 0.4048 to less than 0.0017, nearly pinpointing the worst-case stretch factor. A central difficulty is that the line segment between two points may cross arbitrarily many triangles. Our progress accounting framework addresses this by converting the global bound into a family of local inequalities, each involving only a fixed number of real parameters regardless of the number of triangles crossed. Two further geometric lemmas make computer-assisted verification tractable: the pencil lemma decomposes each local transition into two geometrically independent stages, reducing verification over infinitely many possible next disks to a single differential condition; and the corner lemmas handle degenerate boundary cases where direct verification would fail. With these reductions, the proof ends with a finite verification of an explicit potential in exact rational arithmetic and outward-rounded interval arithmetic. Numerous geometric network constructions use the Delaunay stretch factor as a black box, so the tighter bound directly improves stretch-factor guarantees for a range of downstream applications, including bounded-degree plane spanners and low-weight spanners.
Fair clustering aims to prevent algorithmic bias against protected groups. We study Euclidean fair $k$-means, in which the proportion of each protected group in every cluster lies between prescribed lower and upper bounds. Because these constraints violate the nearest-center assignment property, center selection is coupled with point assignment. We introduce the drift-regularized relaxation, a new convex relaxation solved by a modified Frank-Wolfe method with linear-programming iterations, whose optimality condition controls the cost of converting its solution into $k$ fair clusters. Combined with any $ρ$-approximation algorithm for weighted $k$-means, the relaxation yields an exactly fair fractional solution of cost at most $2ρ+1+ε$ times the fractional optimum, improving the previous ratio $1+4ρ+ε$. In fixed dimension, this improves the best known ratio from $5+O(ε)$ to $3+O(ε)$. The fractional solution can be rounded to an integral one without changing the centers or increasing the cost, at a bounded additive violation of the fairness bounds. We also show that a $(3-δ)$-approximation is NP-hard for every fixed $δ>0$, even on the line with fractional assignments, so $3$ is the exact approximation threshold for fractional fair $k$-means in fixed dimension. The same guarantees hold for $k$-sparse Wasserstein barycenters, a special case of fractional fair $k$-means. More generally, they hold whenever every cluster must satisfy given homogeneous linear inequalities in the fractions of its points, such as a bound on the ratio between two groups or on the average of a numerical attribute in every cluster.
In many applications of $k$-median and $k$-means, including load balancing and redistricting, each center can serve only a bounded number of clients. This leads to capacitated clustering, in which a client may be forced to use a distant center because all nearby ones are full. This loss of locality defeats the techniques behind approximation schemes, and no constant-factor approximation is known in general metrics even for capacitated $k$-median. A long line of work therefore opens $K=k+\lfloorεk\rfloor$ centers for a fixed $ε\in(0,1]$, while respecting the capacities exactly. We give a deterministic polynomial-time algorithm for uniform soft capacities in low dimension, that is, when the clients or the center locations have bounded doubling dimension. With $K$ centers, its ratio is $3-2ε$ for $k$-median and $9-8ε$ for $k$-means, or $5-4ε$ in $\mathbb{R}^d$. Apart from $k$-median in the plane, these are the first polynomial-time ratios bounded by a constant independent of $ε$; previously, the best ratio was $O(ε^{-2}\log(1/ε))$, even for $k$-median. We also obtain ratios that stay bounded as $ε\to0$ for non-uniform and hard capacities in low dimension, and ratios polylogarithmic in $1/ε$ in general metrics. Our approach starts from a simple structural observation: a near-optimal uncapacitated solution can be split into fewer than $2k$ capacity-respecting blocks, so $2k$ centers already suffice, one per block. Our main tool, the rescue lemma, shows that once every block that sends clients to a center of the optimal solution is rescued (given a private center), that center can be closed, so each extra center effectively rescues about two blocks. This lets $\lfloorεk\rfloor$ extra centers rescue nearly an $ε$-fraction of the blocks, yielding the linear dependence on $ε$ in our ratios.
Xin Xu, Ke Cheng, Hao Wang, Yangru Huang, Yanghao Xiao, Zhichao Chen, Zhisheng Zhong, Xun Liu 等 14 位作者
Improving an agent through successive revisions requires balancing exploration and evaluation under a limited budget. A narrow search can miss useful sequences of edits, while fully evaluating every intermediate version often exhausts the budget before those sequences yield a verified improvement. Inspired by the draft--verify principle of speculative decoding, we propose Speculative Recursive Self-Improvement (SpecRSI), a two-level framework coupling inner-loop agent search with outer-loop improver learning. SpecRSI drafts successors before their predecessors are fully evaluated, while adopting only fully evaluated, admissible versions. In the inner loop, Verification-Aware Tree Search (VATS) allocates the budget between further edits and outstanding checks using the predicted verified return of a continuation. In the outer loop, Group Verification Optimization (GVO) updates the improver from completed searches while keeping the real evaluator fixed. We evaluate SpecRSI on scientific optimization (including active queue management (AQM) for network infrastructure), coding, and general agent-system benchmarks. Under matched evaluation-call budgets, SpecRSI improves scientific terminal gain over Janus by 60.0%--96.3% and area under the budget curve by 32.1%--91.6%, while achieving higher means across all 50 scientific backbone--task pairs and outperforming MetaRSI-v1 by 1.30--2.70 percentage points (pp) in system score across twenty backbones. On instance-level comparisons, it further improves PyBaMM terminal gain by 142.6% and SWE-Pro success by 6.0 pp at 7.0% and 9.2% lower token cost, respectively. These results demonstrate the efficacy of the proposed approach.