Nakyung Lee, Soobin Cho, Jiwoong Park, Gyuyeong Kim
原文摘要
We introduce NemoANN, an I/O-efficient SSD-based Approximate Nearest Neighbor Search (ANNS) system for filtered search. Existing SSD-based systems either waste I/O on post-filtering or require expensive filter-aware index rebuilds. Our key insight is that a nonmatching node can never be a result, so the search needs only its small neighbor list, not the full-precision vector on SSD. NemoANN therefore checks each predicate before the SSD read and tunnels through non-matching nodes using in-memory routing metadata, preserving graph connectivity without modifying the index. On a 100M-vector benchmark, NemoANN reduces SSD reads by about 10x and improves throughput by 8.0x over post-filtering and by 5.0x over PipeANN-Filter, a state-of-the-art filtered SSD system, at 10% selectivity and 90% recall. The gains over post-filtering grow at lower selectivity, reaching 13.1x at 5% selectivity.
We study learning rate (LR) scaling for hybrid architectures combining Transformer and State-Space Model (SSM) blocks, a class adopted by several recent production language models. In particular, we focus on the gap between the theoretical scaling rules derived for SSMs under zero-order-hold (ZOH) discretization at infinite width with growing state size, and the field-standard practical implementations using simplified-ZOH Mamba at fixed state size. Surprisingly, in this practical regime hybrid architectures achieve a near-zero LR transfer gap across widths 256-2048 and depths 4-32 up to billion-parameter scale using only the original $μ$P prescription, even though SSM operations fall outside its Tensor Programs representability conditions and every parameterization we test fails the standard coordinate-check diagnostic of $μ$P correctness. We attribute this to a two-condition decomposition of LR transfer in hybrid architectures: a global update-to-weight invariance, enforced by $μ$P's initialization and LR scaling; and a local per-component balance, provided by AdamW's per-parameter normalization. Our observations show that the optimal LR is invariant to width up to 8$\times$, that this width invariance holds across depth, sequence length, batch size, and Transformer-to-SSM ratio, and that it transfers to Nemotron-H, a production hybrid outside our custom architecture set. We hope these findings fill the gap between theoretical scaling rules and practical hybrid implementations, and stimulate further research toward bridging it.
Nakyung Lee, Sangwoo Hong Konkuk University, Jungwoo Lee Seoul National University
原文摘要
Recent research has devoted considerable effort to verifying the intermediate reasoning steps of chain-of-thought (CoT) trajectories using process reward models (PRMs) and other verifier models. However, training a PRM typically requires human annotators to assign reward scores to each reasoning step, which is both costly and time-consuming. Existing automated approaches, such as Monte Carlo (MC) estimation, also demand substantial computational resources due to repeated LLM rollouts. To overcome these limitations, we propose contrastive pointwise mutual information (CPMI), a novel automatic reward labeling method that leverages the model’s internal probability to infer step-level supervision while significantly reducing the computational burden of annotating dataset. CPMI quantifies how much a reasoning step increases the mutual information between the step and the correct target answer relative to hard-negative alternatives. This contrastive signal serves as a proxy for the step’s contribution to the final solution and yields a reliable reward. The experimental results show that CPMI-based labeling reduces dataset construction time by 84% and token generation by 98% compared to MC estimation, while achieving higher accuracy on process-level evaluations and mathematical reasoning benchmarks.
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表