跳到正文
研踪PaperTrail

论文库

检索计算机领域论文,按研究方向、作者或关键词筛选。

部分采集异常
正在读取账号与研究记录…

全站更新概况

+1295新增论文
+1026其中 AI 相关
+111已有论文修订
+95新增 AI 导读

arXiv 最近完成扫描:2026-11-17 04:58 · 全站最近入库:2026-11-17 16:10各信源进度 →

统计口径与数据新鲜度

统计 2026-11-16 16:37 — 2026-11-17 16:37(北京时间) · 点击数字查看对应简报;AI 相关包含在新增论文内。

最近完成检查:2026-11-17 04:58最近发现新增版本:2026-11-17 16:10库内论文最新更新时间:2026-11-17 02:59

每分钟刷新已保存的数据。空检查不代表采集异常;新收录可能包含历史论文。AI 导读按新增版本计数。

25,122 已收录论文20,967 AI 相关7,309 有修订112 HF 社区发现

浏览论文库

时间范围跟随排序:版本更新、本站收录或首次发表。最新收录可能包含历史论文。部分信源异常,查看状态 →

找到 18 篇论文 · 按版本更新时间排序上轮扫描起点 2026-11-17 04:57(北京时间)
新论文 · v1arXiv:2611.20180

UrbanROI: Causal Modeling for Regional ROI Forecasting under POI Entry

Zhaoqi Zhang, Miao Xie, Linyou Cai, Qianlong Xie, Xingxing Wang, Siqiang Luo, Gao Cong

原文摘要

As urban commercial environments continue to evolve rapidly, accurately modeling the economic impact of point of interest (POI) entry is essential for supporting data-driven advertising allocation, merchant onboarding, and regional business decision-making. Existing return on investment (ROI) prediction methods mainly focus on individual ads or merchants, overlooking the interaction effects among POIs and failing to capture the causal influence of structural changes in urban commercial ecosystems. In this paper, we introduce a novel research problem -- regional ROI forecasting under POI entry, which aims to predict the future ROI trajectory of a commercial region after the introduction of a new POI by modeling temporal ROI evolution and competition-driven interactions among surrounding POIs. To address these challenges, we propose UrbanROI, an intervention-aware causal forecasting framework for region-level economic outcome prediction. UrbanROI leverages an Interaction-aware Regional Graph Module to explicitly model competitive relationships among POIs based on user request contexts, spatial proximity, and request-level commercial signals, and constructs structurally aligned treatment and control graphs to simulate factual and counterfactual regional states. Extensive experiments on large-scale real-world POI and advertising datasets demonstrate that UrbanROI consistently outperforms strong baselines in post-entry regional ROI forecasting. The results demonstrate its effectiveness in temporal economic modeling, inter-POI interaction representation, and intervention-aware forecasting, providing support for regional economic analysis and business decision-making in dynamic urban commercial systems.

cs.LG

版本更新 · v2arXiv:2607.11585

Machine Learning-Based Reconstruction for Resistive Silicon Sensors

Alexander Aoki, Gaetano Barone, Leena Diehl, Gabriele Giacomini, Vagelis Gkougkousis, Hanshal Goyal, Rohan Kher, Daniel Li 等 16 位作者

原文摘要

Low-Gain Avalanche Diodes (LGADs) and AC-coupled Low-Gain Avalanche Diodes (AC-LGADs) are promising technologies for precision timing and four-dimensional tracking. In AC-LGADs, the AC pad is coupled to the resistive n$^{+}$ layer through a dielectric layer, while the gain layer remains unsegmented. This structure provides a 100\% fill factor and enables good spatial resolution with a relaxed readout pitch. The same signal-sharing mechanism that makes interpolation possible complicates the readout: charge spreads across multiple pads, the useful information can approach the electronic-noise threshold, and matrix-inversion approaches can become computationally challenging and sensitive to off-diagonal noise. In this work, we study machine-learning-based reconstruction and compression for resistive silicon sensors. We use full-waveform information from correlated pads to regularise the reconstruction and extract spatial information beyond what is available from binary readouts or reduced-amplitude summaries. We first introduce recurrent neural network models based on LSTM layers, which provide a proof-of-concept implementation for full-waveform reconstruction and have been tested for FPGA deployment using \hls. We also study routes towards bandwidth reduction with waveform rasterisation and window-selection methods, and extend the approach beyond the first model to topology-agnostic transformer-based architectures that use pad coordinates as part of the input. These models are designed to support arbitrary pad counts and geometries, mitigate edge distortions, preserve approximately $10~μ\mathrm{m}$ position resolution for $500~μ\mathrm{m}\times500~μ\mathrm{m}$ pitched sensors, and guide future resistive-silicon sensor designs

hep-ex · cs.LG · nucl-ex

版本更新 · v2arXiv:2610.37364HF 社区推荐

EmbeddingGemma 2: Unified and Efficient Multimodal Representations

Henrique Schechter Vera, Sahil Dua, Jovan Andonov, Sara Smoot, Sebastian Russo, Ivan Llanos, Max Matveev, Jae Hyeon Yoo 等 77 位作者

原文摘要

We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.

cs.LG · cs.CL · cs.CV · cs.SD

版本更新 · v3arXiv:2604.00449

Convergence of Byzantine-Resilient Gradient Tracking via Probabilistic Edge Dropout

Amirhossein Dezhboro, Fateme Maleki, Arman Adibi, Erfan Amini, Jose E. Ramirez-Marquez

原文摘要

We study distributed optimization over networks with Byzantine agents that may send arbitrary adversarial messages. We propose Gradient Tracking with Probabilistic Edge Dropout (GT-PD), which combines a universal self-centred projection that clips incoming messages to a radius-$τ$ ball around the receiver with a fully decentralized probabilistic dropout rule driven by a threshold-gated, dual-metric trust score in the decision and tracking channels. These identity-free defences bound adversarial perturbations while preserving doubly stochastic mixing. The projection preserves the honest network average exactly in every realization, even when honest messages are clipped. Under complete Byzantine isolation ($p_b=0$), GT-PD converges linearly to a neighbourhood of order $ασ^2/(μn_{\mathcal H})$ plus higher-order network terms, matching centralized stochastic gradient descent to first order in the step size. We also give a radius computable from problem data that ensures honest messages remain unclipped with high probability. Under partial isolation ($p_b>0$), a leaky variant, GT-PD-L, controls accumulated tracking perturbations and converges linearly to a bounded neighbourhood governed by stochastic variance and the clipping-to-leak ratio, under explicit conditions on the step size, leak, and aggregate Byzantine mixing weight. With two-tier dropout and $p_h=1$, Byzantine isolation adds no variance to honest consensus dynamics. MNIST experiments under Sign Flip, ALIE, and Inner Product Manipulation attacks show that GT-PD-L outperforms coordinate-wise trimmed mean by up to 4.3 percentage points under stealth attacks.

cs.LG · cs.MA · eess.SY

版本更新 · v3arXiv:2605.20555

LAPO: KL-Free Logit-Averaged Policy Optimization to Leverage RL and SFT in LLM Post-Training

Xingwei Gan, Ying Zhu

原文摘要

From a classical statistical perspective, model averaging and parameter shrinkage (e.g., Ridge regression with an $l_2$-penalty) are deeply connected mechanisms designed to reduce estimator variance and lower mean squared error. Motivated by this insight, we introduce Logit-Averaged Policy Optimization (LAPO), a KL-free and critic-free post-training framework that replaces explicit KL regularization with logit averaging inside Group Relative Policy Optimization (GRPO); in particular, averaging the logits of the trainable policy and a frozen reference policy functions as an implicit structural regularizer that stabilizes rollout trajectories. For selecting the mixing weight, we propose a compute-efficient adaptive algorithm that dynamically balances exploration and structural anchoring throughout training, evaluated against a static fixed-weight baseline. This adaptive mechanism eliminates explicit KL penalty backpropagation and expensive, fragile hyperparameter sweeps while reducing the incidence of uninformative zero-reward rollout groups. Mathematically, embedding logit averaging within the reinforcement learning loop operates as an in-training Product of Experts (PoE): the multiplicative geometric mean enables the reference anchor to act as a soft logical "AND" that vetoes non-compliant tokens, enforcing schema rigor and safety guardrails without the catastrophic forgetting typical of sequential fine-tuning. Evaluated across Qwen2.5-Instruct models (1.5B, 3B, and 7B), LAPO consistently outperforms canonical KL-regularized GRPO, achieving a 7%-15% accuracy gain on MATH and a 2%-5% gain on cn_k12 and MMLU. Finally, the resulting mixed teacher policy can be distilled back into a single student network, fully restoring standard $1\times$ inference latency.

cs.LG · cs.AI

新论文 · v1arXiv:2611.09225

Fused-Softmax for efficient quantized attention

Tianxing Wu, Juuso Korhonen, Jesus Carabano Bravo

原文摘要

Low-precision matrix throughput on modern GPUs is growing far faster than vector-arithmetic throughput. Attention therefore cannot realize the available GEMM performance because softmax has become the pipeline bottleneck. We introduce Fused-Softmax Attention, a training-free algorithm that reformulates and approximates softmax to fold its dominant operations into the attention GEMMs, moving work from vector units to matrix cores. Implemented for the AMD CDNA 4 architecture, our MXFP8, MXFP6 and MXFP4 kernels reach approximately 3.0, 4.0 and 4.6 PFLOP/s on an AMD Instinct MI355X GPU, yielding speedups of $2.1\times$, $2.8\times$ and $3.2\times$ over the BF16 attention. Importantly, the additional error introduced by Fused-Softmax remains below the error floor set by quantizing the GEMM operands. On Wan2.2-T2V-A14B, a hybrid MXFP6/4 schedule delivers up to $2.05\times$ end-to-end speedup while preserving the visual quality of the outputs. The kernels ship in AMD's open-source AITER library, and they powered AMD's MLPerf Inference v6.1 Wan2.2 submission, which ran $1.7\times$ faster than AMD's previous-round result on the same hardware. Our results show that redesigning softmax to utilize the matrix pipeline enables attention to capitalize on continued growth in low-precision matrix throughput.

cs.AI · cs.LG · cs.PF

新论文 · v1arXiv:2611.08379

面向图神经网络的模块感知优化

Module-Aware Optimization for Graph Neural Networks

Guy Hadad, Haggai Roitman, Moshe Eliasof, Bracha Shapira

AI 速读 基于摘要

作者提出图感知优化器 GraphMuon,将 Muon 的矩阵正交化更新适配到图传播特征的协方差结构,以缓解图诱导的更新放大。

cs.LG

查看原文摘要

Different optimizers impose different geometries on parameter updates. Adaptive methods such as Adam and AdamW use coordinate-wise updates, while matrix-aware optimizers such as Muon utilize weight-matrix structure through orthogonalized updates. For Graph Neural Networks (GNNs), however, the effective geometry of its weight matrices is shaped not only by its learned channel-mixing weights, but also by the graph-propagated representations it acts on. Thus, even if an update is well normalized as a matrix, its effect on the layer output can be amplified when applied to graph-propagated features. To address this gap, we introduce GraphMuon, a graph-aware optimizer that adapts Muon's matrix updates to the covariance structure of graph-propagated layer inputs. GraphMuon performs orthogonalization in a metric defined by these propagated representations and normalizes step sizes according to the update's induced effect on the layer representation. We show that theoretically, vanilla Muon can suffer from graph-induced update amplification, while our GraphMuon can mitigate this effect. As such, our GraphMuon yields a graph-metric analogue of Muon's polar update step. Empirically, GraphMuon achieves $1.32\times$-$2.42\times$ validation-loss time-to-target speedups over AdamW on three large-scale OGB benchmarks. Across 15 dataset-architecture configurations spanning node classification, inductive graph classification, and graph regression, it achieves lower time-to-target in 13 cases, while maintaining competitive or superior predictive performance. Code is available at https://github.com/guy-hadad/GraphMuon.

新论文 · v1arXiv:2611.08182

BUMP:插值中段意外的短暂峰值

BUMP: Brief Unexpected Mid-Interpolation Peak

Melinda Flóra Kiss, Márton Muntag, Adrián Csiszárik, Dániel Varga

AI 速读 基于摘要

作者在置换对齐神经网络线性插值中发现测试精度在端点附近短暂上升的 BUMP 现象,并据此提出一种权重预测混合集成方法。

cs.LG · stat.ML

查看原文摘要

We identify an intriguing phenomenon when linearly interpolating between permutation-aligned neural networks: test accuracy briefly improves near the interpolation endpoints before dropping toward the midpoint, which we term BUMP (Brief Unexpected Mid-interpolation Peak). We observe that BUMP appears consistently across datasets, architectures, training regimes, and remains robust under substantial perturbations such as early checkpoints and extreme pruning. Moreover, controlled ablations suggest that it cannot be attributed to trivial regularization effects. While similar near-endpoint gains are visible in prior interpolation plots, they have not been explicitly documented or systematically and thoroughly analyzed. Leveraging this effect, we propose a simple, novel ensembling scheme: weight-prediction mixed ensembling that combines weight mixing with prediction-space mixing. Across our experiments, the resulting ensembling method consistently outperforms standard logit ensembling in test accuracy and yields improved calibration trade-off, highlighting complementary benefits of weight- and prediction-space mixing.

新论文 · v1arXiv:2611.07666

Extracting Single-Mode Rayleigh--Taylor Instability dynamics from data via SINDy

Chuanxing Wang, Hui Luo, Guohuai Zhu, Hongbo Cai, Shaoping Zhu, Mingxing Luo

原文摘要

Rayleigh-Taylor instability (RTI) is of fundamental importance in fields such as inertial confinement fusion and supernova explosions, and its evolution is commonly described by potential--flow theory and buoyancy--drag models. However, both types of models struggle to accurately capture the overshoot behavior of bubble and spike velocities, as well as the subsequent reacceleration stage. To address these limitations, we develop a physics-constrained, data-driven framework based on the integral formulation of Sparse Identification of Nonlinear Dynamics (SINDy), which extracts low-dimensional dynamical equations for bubble and spike evolution directly from data. By introducing the nondimensional initial amplitude $A_0 = kη_0$ as an explicit parameter, we identify distinct dynamical equations for the bubble and the spike based on data spanning multiple Atwood numbers and initial amplitudes. The resulting low-dimensional dynamical equations not only reproduce the overshoot behavior of the velocity with good accuracy, but also capture the reacceleration stage well, while maintaining high predictive accuracy at unseen values of the wavenumber, acceleration, and Atwood number. This work provides a physics-constrained, data-driven framework complementary to conventional potential-flow and buoyancy--drag models, accurately describing the dynamical evolution of the bubble and spike at high Atwood numbers, and provides a new perspective for directly extracting the dynamical equations of interfacial instabilities from data.

physics.flu-dyn · cs.LG

版本更新 · v3arXiv:2611.02102

NorthCUA Technical Report: Turbocharging Computer-Use Agents through Experience at Scale

Chaoyun Zhang, Bingxuan Li, Pinxue Guo, Yangfan Gao, Zikun Cai, Tianci Xie, Hui Li, Yuqi Huang 等 30 位作者

原文摘要

Scaling computer-use agents from individual actions to complex software work requires experience that connects decisions to complete outcomes. We present NorthCUA, an integrated system for turbocharging computer-use agents through experience at scale. Its Experience Flywheel supplies diverse executable tasks to the NorthCUA Harness, which sustains long interactions, records decisions, and checks task outcomes. The Flywheel uses these records to curate verified trajectories. For unsolved tasks, recursive self-improvement evolves reusable skills and preserves verified milestones so later attempts build on earlier progress. Staged Learning combines this experience with grounding data from the Flywheel to develop GUI competence through mid-training and long-horizon supervised fine-tuning. The policy then learns from fresh attempts in the same Harness through joint desktop-web reinforcement learning, closing the loop between experience production and policy improvement. The resulting Nano, Mini, and Pro family achieves state-of-the-art results among open-source models across GUI grounding, desktop execution, and Web interaction under the reported evaluation settings. For more responsive Web workflows, NorthCUA-Ultrafast pairs direct action selection with open-ended generation on pre-RL Nano, offering a measured quality-latency tradeoff. Together, these advances establish NorthCUA as a high-performing model family and a reusable foundation for agents that carry complex software work from intent to completion.

cs.LG

新论文 · v1arXiv:2611.05984

On the Design of Hyper-Connections: Simplification, Stability, and Scalability

Xiangdong Zhang, Xiaohan Qin, Sunan Zou, Tuo Dai, Bizhao Shi, Zhuo Xia, Yebin Yang, Yizhou Chen 等 11 位作者

原文摘要

Hyper-Connections (HC) architectures are increasingly adopted in frontier language models, yet their components remain insufficiently understood and introduce non-negligible training overhead. In this work, we systematically study the HC architecture to understand its components and identify simpler, more efficient designs. We find that learned read/write gating without mixing and unconstrained mixing with fixed gates form two effective interaction groups: either alone achieves performance comparable to mHC. We also identify repeated access to the expanded residual state as a major source of training overhead, even without mixing. These findings lead to SimpleHC, which simplifies the architecture by removing residual mixing and simplifies execution through shared memory access, using the resulting additive structure and shared gate inputs. Layer-specific gates share a group-entry state, enabling batched reads and deferred writes; scalar corrections exactly preserve preceding sublayer output contributions. Our analyses support both the retained residual structure and the design of shared memory access. SimpleHC incurs only 2.4% extra end-to-end training time over a vanilla residual baseline at the largest tested scale. It maintains performance comparable to mHC when scaling from 15B to 30B MoE models, exhibits smoother gradient dynamics during standard training, and remains stable under stress. We hope SimpleHC will encourage a broader pursuit of simplicity in foundation model design, making performance gains more accessible and practical at scale.

cs.LG · cs.CL

新论文 · v1arXiv:2611.05580

Learning vs. Optimizing Players in Budgeted Games

Giannis Fikioris, Balasubramanian Sivan, Éva Tardos

原文摘要

We study repeated general games over $T$ rounds between a utility-maximizing optimizer and a learning agent, where budget constraints couple the players' decisions across rounds. Without budgets, low swap regret simultaneously guarantees the learner sublinear regret and caps the optimizer at its Stackelberg value. Budgets break this picture: adversarial no-regret becomes impossible, so the natural goals are non-manipulability---no adaptive optimizer earns more than $\mathtt{BSE} \cdot T + o(T)$, where $\mathtt{BSE}$ is its Budgeted Stackelberg Equilibrium value---and low stochastic regret when the optimizer plays a fixed mixed strategy. We give an efficient, budget-feasible learning algorithm that achieves both goals in every budgeted game: any adaptive optimizer, budgeted or not, earns at most $\mathtt{BSE} \cdot T + O(\sqrt{T})$ in expectation, and if the optimizer plays a fixed mixed strategy on each of a constant number of intervals, the learner has $O(\sqrt{T})$ regret on each such interval. Our learner best responds to forecasts of the optimizer's play that are calibrated in two novel senses, since under budgets the calibration that prevents manipulation no longer yields low stochastic regret. Response calibration, which conditions on the learner's own mixed response by embedding its continuous mixing weight as a step function in $L^2([-1,1])$, prevents manipulation without discretizing responses; dual-weighted Lagrangian calibration ensures low regret. Although low calibration error requires randomized forecasts---distributions over a continuous space that need not have a succinct representation or be directly computable---we construct such forecasts in polynomial time by simulating online gradient ascent.

cs.GT · cs.LG

新论文 · v1arXiv:2611.05184

Provably Learning Mixtures of Linear Experts

Adrian Candocia, Aamod Varma, Mark Davenport

原文摘要

Mixture-of-Experts (MoE) models are widely deployed in modern machine learning. However, few convergence guarantees exist for gradient-based training of MoEs: the objective is nonconvex, and even simple mixtures trap first-order methods in spurious local optima. We analyze a canonical gated mixture featuring linear experts and a gate induced by Gaussian clusters in the input space, and give, to our knowledge, the first global convergence guarantee for jointly training the gate and the experts of such a model by gradient EM on its likelihood. Given a logarithmic overparameterization relative to the true number of experts $m$, gradient EM converges from randomly initialized gates and least-squares-initialized experts to any prescribed accuracy in polynomially many iterations, under a separation condition on the gate centers. Our analysis establishes a gradient-dominance inequality for the coupled dynamics by decoupling the loss into per-cluster subproblems. Empirical results validate the predicted $m\log m$ phase transition and suggest that the least-squares warm start required by our analysis is not needed in practice: randomly initialized experts perform nearly as well. We also find that leaving the mixing weights unconstrained concentrates routing onto the true number of experts, whereas load-balancing penalties spread it, so that the fitted model can be pruned to a fraction of its width without loss of accuracy.

cs.LG

版本更新 · v2arXiv:2611.03260

The Cost of Mismatch: Noise Amplification in Zeroth-Order Reinforcement Learning

Lianmin Chen, Junbin Qiu, Chenxing Wei, Yao Shu, Kun He

原文摘要

Zeroth-order (ZO) optimization has emerged as a promising approach for memory-efficient large language model (LLM) post-training, yet its reliability in reinforcement learning with verifiable rewards (RLVR) remains largely unexplored. Motivated by the distinct data regimes of on-policy RL and fixed-data optimization, we develop a generic variance-theoretic framework to investigate what limits the reliability of ZO gradient estimation. Our analysis reveals that estimation reliability is structurally determined by whether the two finite-difference evaluations are performed on matched data. On-policy reinforcement learning (RL) under the mismatched-data regime produces a $1/μ^2$ oracle-noise amplification that sets an irreducible noise floor as the perturbation scale $μ$ decreases, whereas Boltzmann Targeted SFT (BOLT) under the matched-data regime eliminates this amplification via noise cancellation. We further derive convergence guarantees with explicit constants that cleanly separate the two regimes. These results motivate the ZO-BOLT algorithm, which builds on BOLT's matched-data advantage for reliable ZO optimization of an RLVR-aligned objective. Controlled perturbation-scale sweeps and matched-versus-mismatched end-to-end comparisons validate the predicted effect of data matching. Experiments across three models and two mathematical reasoning benchmarks further show that ZO-BOLT consistently improves over the pretrained baseline while retaining a near-inference memory footprint. The code is publicly available at https://github.com/lianmin-chen/the_cost_of_mismatch.

cs.LG

版本更新 · v3arXiv:2603.01563

Efficient Token-Level Policy Optimization for Masked Diffusion Models via Velocity Alignment

Chenxing Wei, Jiazhen Kang, Hong Wang, Jianqing Zhang, Hao Jiang, Xiaolong Xu, Ningyuan Sun, Ying He 等 11 位作者

原文摘要

Reinforcement Learning with Verifiable Rewards (RLVR) successfully aligns autoregressive models in rigorous domains like mathematical reasoning and code generation. However, applying RLVR to Diffusion Large Language Models (dLLMs) is hindered by intractable sequence-level likelihoods, forcing existing methods to rely on unstable, high-variance approximations. We propose Sequence Likelihood-Free Policy Optimization (LFPO), an efficient framework that circumvents this bottleneck by formulating policy alignment as discrete velocity alignment. Inspired by continuous flow matching, LFPO eschews trajectory-level estimation and directly optimizes denoising logits via contrastive token-level updates, providing a stable, low-variance surrogate. Furthermore, predicting terminal solutions from intermediate states implicitly straightens the generation trajectory, enabling high-quality sampling with significantly fewer denoising iterations. Extensive experiments on reasoning and coding benchmarks demonstrate that LFPO matches or exceeds the accuracy of likelihood-based baselines while substantially accelerating inference.

cs.LG · cs.AI

新论文 · v1arXiv:2610.36983

Advancement or Acceleration? How Data Curation Improves LLM Training

Jiaxing Wang, Zicheng Zhang, Deping Xiang, Jin Xu, Yong Li, Chao Liu, Pengzhang Liu, Qixia Jiang

原文摘要

While data curation is widely credited with improving large language model training, a basic question remains unresolved: does curation expand what a model can ultimately learn, or merely reach the same solution faster? In this paper, we formalize this distinction between advancement (improving the attainable performance) and acceleration (reaching the same performance with less data). Through an analytic characterization and real-world LLM experiments, we show that advancement requires a correctable train-target gap. When the baseline training and target objectives coincide, no curation policy improves the endpoint, although curation may still accelerate training. This reframes advancement as a property of the curation task rather than the curation method. Under a common gap-centric view of data mixture and selection, we operationalize this condition as the $Δ$-test, an observable discrepancy between the uncurated baseline's training and target losses that is computable prior to expensive curation. Across extensive experiments, an appreciably nonzero $Δ$ predicts persistent gains, whereas gains near $Δ=0$ tend to diminish as the training budget grows. Beyond individual methods, our findings recenter data curation on the task itself and point to target-set construction as a promising direction for shaping what curation can achieve.

cs.LG · cs.AI

版本更新 · v4arXiv:2608.03228

SAKI-RoPE: Score-Aware Low-Rank Key Indexing with RoPE-Aware Covariance and Random-Matrix Noise Analysis

Lin Zhang

原文摘要

Existing low-rank KV-cache indexes preserve model weights or key variance, not the attention scores that determine retrieval. We derive the expected score distortion of key-side rank-r compression, a two-sided covariance-weighted low-rank problem that uses both query and key statistics, and solve it exactly: the optimum is an optimal rank-r linear score map, computed in closed form from two d x d covariances per head, and its predicted per-head score-MSE reduction matches measurement (Pearson r = 0.997). The resulting training-free index, SAKI, beats key-PCA at every rank on LLaMA-3.1-8B, Qwen2.5-7B, Mistral-7B and Llama-3.2-3B, at every context length from 4K to 64K, and in every calibration-domain pair (0.748 -> 0.799 recall@64 at r = 32 on LLaMA-3.1-8B). Because RoPE rotates keys by their distance from the query, we make the closed form RoPE-aware by averaging the query covariance over rotations; this SAKI-RoPE, our main method, raises recall to 0.861, beats PCA on every head of all four models, and needs only 28-39% of PCA's rank to match it at r = 32 across the four models. Random-matrix theory of the covariance estimation noise shows that 512 calibration tokens suffice and that clipping the Marchenko-Pastur noise bulk helps SAKI when calibration is short. We make no perplexity claim: with the attention sink pinned, perplexity differences between indexes are small and depend on model and text, and we show that any low-rank index calibrated on a long prefix drops the attention-sink token, which collapses 32K perplexity unless the sink is pinned.

cs.LG · stat.AP

版本更新 · v2arXiv:2404.16436Google Research

Leveraging tropical reef, bird and unrelated sounds for superior transfer learning in marine bioacoustics

Ben Williams, Bart van Merriënboer, Vincent Dumoulin, Jenny Hamer, Eleni Triantafillou, Abram B. Fleishman, Matthew McKown, Jill E. Munger 等 15 位作者

原文摘要

Machine learning has the potential to revolutionize passive acoustic monitoring (PAM) for ecological assessments. However, high annotation and compute costs limit the field's efficacy. Generalizable pretrained networks can overcome these costs, but high-quality pretraining requires vast annotated libraries, limiting its current applicability primarily to bird taxa. Here, we identify the optimum pretraining strategy for a data-deficient domain using coral reef bioacoustics. We assemble ReefSet, a large annotated library of reef sounds, though modest compared to bird libraries at 2% of the sample count. Through testing few-shot transfer learning performance, we observe that pretraining on bird audio provides notably superior generalizability compared to pretraining on ReefSet or unrelated audio alone. However, our key findings show that cross-domain mixing which leverages bird, reef and unrelated audio during pretraining maximizes reef generalizability. SurfPerch, our pretrained network, provides a strong foundation for automated analysis of marine PAM data with minimal annotation and compute costs.

cs.SD · cs.AI · cs.LG · eess.AS