跳到正文
研踪PaperTrail

论文库

检索计算机领域论文，按研究方向、作者或关键词筛选。

部分采集异常
正在读取账号与研究记录…

全站更新概况

+0新增论文
+0其中 AI 相关
+0已有论文修订
+93新增 AI 导读

arXiv 最近完成扫描：2026-11-16 01:34 · 全站最近入库：2026-11-14 15:09各信源进度 →

统计口径与数据新鲜度

统计 2026-11-15 07:12 — 2026-11-16 07:12（北京时间） · 点击数字查看对应简报；AI 相关包含在新增论文内。

最近完成检查：2026-11-16 01:34最近发现新增版本：2026-11-14 15:09库内论文最新更新时间：2026-11-13 02:59

每分钟刷新已保存的数据。空检查不代表采集异常；新收录可能包含历史论文。AI 导读按新增版本计数。

23,348 已收录论文19,580 AI 相关6,729 有修订112 HF 社区发现

浏览论文库

研究日报 / 周报 →按信源浏览 →

时间范围跟随排序：版本更新、本站收录或首次发表。最新收录可能包含历史论文。部分信源异常，查看状态 →

找到 17 篇论文 · 按版本更新时间排序上轮扫描起点 2026-11-16 01:33（北京时间）
新论文 · v1arXiv:2611.17191

最大化LoRA适配器容量利用率以缓解学习-遗忘权衡

Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off

Tomohiro Sawada, Kartik Goyal

AI 速读 基于摘要

作者提出一种无需SVD的Rényi-2有效秩正则项，通过提升LoRA适配器的有效秩来改善学习与遗忘之间的权衡。

cs.LG · cs.CL

arXiv ↗PDF ↗
查看原文摘要

Low-Rank Adaptation (LoRA) often underperforms full fine-tuning, especially in heterogeneous and continual-learning settings in which substantial forgetting occurs. In general, it suffers from an undesirable learning-forgetting trade-off across diverse fine-tuning settings. We identify a potential contributor: LoRA adapters, especially high-rank ones, frequently fail to realize the available capacity afforded by the configured rank. Specifically, we find that standard LoRA updates have effective rank far below their configured rank, limiting how broadly the update energy is distributed. Therefore, we propose training mechanisms that optimize the original objective while maximizing adapter effective rank. Our main method adds a simple, stable, differentiable, and SVD-free Rényi-2 effective-rank regularizer to the language-modeling loss. We also connect capacity maximization to norm-bounded updates. In mixed-domain rank sweeps on three models evaluated with multiple random seeds, a penalty weight of $λ=0.01$ lowers mean Wiki perplexity at every tested rank of at least $16$ relative to unpenalized LoRA. Mean training perplexity improves or increases by at most $0.02$ in all but one model-rank comparison. Single-task and scheduling experiments further show that the balance between learning and retention depends on the configuration. We also observe that several LoRA variants in the literature also suffer from the identified capacity issue, and our approach provides a complementary axis of improvement to LoRA-based parameter-efficient fine-tuning. These results suggest that effective rank is not only a post-hoc diagnostic of adapter quality, but a trainable objective for improving the learning-forgetting frontier. Our code is available at https://github.com/tomohiro-sawada/lora-capacity-utilization

新论文 · v1arXiv:2611.13708

FedVP: Federated Learning across Heterogeneous Models by Pooling Locally Trained Verifiers

Yifan Li, Xi Chen

原文摘要

Conventional federated learning relies on the shared model backbone to aggregate client updates, limiting collaboration among institutions that deploy large language models (LLMs) from different families and at different scales. We propose Federated Verifier Pool (FEDVP), a new federated learning paradigm that enables heterogeneous clients to collaborate through shared verification rather than parameter aggregation. Each client trains a verifier using local data and responses generated by its chosen LLM backbone. The server pools these independently trained verifiers and combines their judgments to select answers without merging model parameters. This allows both LLM backbones and verifiers to differ in architecture and size, while keeping the LLM backbones frozen and deriving collaborative gains from improved answer selection. We evaluate FEDVP across five public medical question-answering datasets treated as federated clients, using Qwen and LLaMA backbones ranging from 3B to 14B parameters. FEDVP consistently improves performance across all five clients and outperforms the strongest conventional federated baselines by 1.46-5.79 percentage points in mean accuracy.

cs.LG

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2603.15507

Federated Learning of Binary Neural Networks: Enabling Low-Cost Inference

Nitin Priyadarshini Shankar, Soham Lahiri, Sheetal Kalyani, Saurav Prakash

原文摘要

Federated Learning (FL) enables distributed clients to jointly train a global model without sharing their raw data. However, deploying the resulting deep neural networks on resource-constrained edge devices remains challenging due to their inference-time computational and memory requirements. Binary Neural Networks (BNNs) offer substantial efficiency gains by representing weights and activations using a single bit, but achieving competitive accuracy under such aggressive quantization remains difficult. Rotation-aware binarization reduces quantization errors by aligning real-valued weights with their binary representations. Despite its effectiveness in centralized learning, applying rotation-aware binarization in FL is non-trivial due to discrepancies between local and global models, particularly under heterogeneous client data distributions. To address this challenge, we propose FedBNN, a federated rotation-aware binarization framework that leverages global model information to guide local binarization while incorporating locally learned rotation information into server-side construction of the global binary model. Experiments across multiple benchmark datasets, model architectures, and degrees of data heterogeneity demonstrate that the proposed integrated methodology achieves accuracy competitive with full-precision federated models while generally outperforming existing federated binary learning approaches. Overall, FedBNN enables accurate binary inference in heterogeneous federated settings, with savings of up to $58\times$ in FLOPs and $32\times$ in memory over full-precision models, offering a practical balance between predictive performance and deployment efficiency on resource-constrained edge devices.

cs.LG · cs.CV

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2603.13389

Logit-Conditioned Diffusion Decoding for Frozen Discrete-Token VLMs

Ji Woo Hong, Gwanhyeong Koo, Hee Suk Yoon, Eunseop Yoon, SooHwan Eom, Qi Dai, Chong Luo, Chang D. Yoo

原文摘要

Pre-trained discrete-token VLMs generate images by sampling sequences of codebook indices, but their visual fidelity is bounded by the VQ-VAE decoding stage. Prior work addresses this bottleneck by replacing the discrete tokenizer with continuous or hybrid visual token representations, or by integrating diffusion into the generation pipeline; both require foundation-scale VLM retraining or realignment. We show this is unnecessary: before each token is sampled, discrete-token VLMs already compute a full logit distribution over the codebook, but standard decoding discards it after selecting a single index. Under Otsu's threshold, the pre-sampling logits of capable VLMs assign statistically meaningful probability to at least two codebook entries per token, and recovering this distribution can substantially close the fidelity gap. We propose DCDD, a post-hoc framework that conditions a diffusion decoder on this discarded distribution while leaving the VLM frozen. At inference, DCDD converts the VLM's pre-sampling logits into distribution-weighted code vectors; during training, it uses VQ-VAE encoder-derived proxy logits calibrated to match the distributional statistics of the VLM's inference-time logits. The decoder maps these representations to high-fidelity images as an alternative to the native VQ-VAE decoder. Trained on ImageNet-1K for 50K steps, DCDD reduces reconstruction FID by 74% and Janus-Pro 7B's generation FID on MJHQ-30K by 34%, while semantic alignment across GenEval, DPG-Bench, and WISE remain consistent. Controlled comparisons with diffusion models from the same backbone family, applied either as an image-to-image refiner or as an identically trained decoder on the sampled codes, confirm that the gain stems from conditioning on the VLM's pre-sampling logit distribution rather than from the diffusion model itself.

cs.CV · cs.LG

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.11973

Beyond Prediction Residuals: Conformal Regret Calibration for Robust Contextual Linear Optimization

Lingjie Zhao, Hansheng Jiang, Wei Qi

原文摘要

Predict-then-optimize systems convert contextual forecasts into decisions, yet conformal robust optimization typically calibrates prediction residuals rather than downstream loss. For linear optimization, this parameter-space geometry can be badly misaligned with decision risk: large objective perturbations may preserve the optimizer, while small boundary-crossing perturbations may cause substantial regret. We introduce Conformal Regret Sets, a post-hoc, model-agnostic framework that conformalizes an optimization-derived regret surrogate and uses its calibrated level set for minimax-regret refinement. We first show that directly conformalizing exact regret, although sufficient to certify the nominal predict-then-optimize decision, yields a degenerate robust problem: absent an exogenous bounded support, every alternative decision has unbounded worst-case regret. We therefore use SPO+, a pointwise upper bound on exact regret, as the conformal score, obtaining distribution-free finite-sample marginal regret guarantees under exchangeability. For contextual linear programs, the resulting set decomposes into polyhedral pieces aligned with the normal fan of the feasible polytope. A vertex is active if and only if its predicted suboptimality gap is at most half the calibrated threshold $η$, enabling local, output-sensitive generation of all relevant pieces. Moreover, if $Γ$ is the predicted gap to the closest competing vertex, the nominal and refined decisions admit the sharper worst-case-regret certificate $(η-2Γ)_+$, with an exact early exit when $η\leq 2Γ$. Experiments on contextual fractional knapsack validate the geometry and certificates: continuous refinement reduces the $0.95$-quantile and $\operatorname{CVaR}_{0.95}$ of regret by $9.1\%$ and $13.1\%$ relative to nominal PTO, while vertex-restricted refinement changes only $10.0\%$ of nominal decisions.

stat.ML · cs.LG

查看论文
arXiv ↗PDF ↗
版本更新 · v4arXiv:2602.17739

GeneZip: Region-Aware Compression for Long Context DNA Modeling

Jianan Zhao, Xixian Liu, Zhihao Zhan, Xinyu Yuan, Hongyu Guo, Jian Tang

原文摘要

Long-context DNA modeling is computationally expensive, motivating existing approaches to reduce sequence length through uniform downsampling. However, functional information and sequence redundancy are unevenly distributed across the genome, calling for finer resolution in functionally dense regions and stronger compression of redundant sequence. We introduce GeneZip, a region-aware DNA compression framework that learns where to allocate a limited token budget. Building on H-Net-style dynamic routing, GeneZip combines a Region-Aware Ratio objective, which uses regional labels derived from GENCODE gene annotations to specify compression targets during training, with bounded routing for stable token counts. At inference, it compresses raw DNA without annotations. GeneZip provides three main benefits. First, it preserves predictive utility under high compression: GeneZip-70M achieves 137.6-fold compression with a validation perplexity increase of 0.31 relative to uncompressed Mamba2-84M, and the best average rank across four DNALongBench tasks among compression models with leakage-controlled pretraining. Second, it learns region-aware allocation with emergent repeat-aware compression. Alternative regional priors yield task-dependent performance trade-offs. Without repeat-specific supervision, GeneZip compresses transposable-element-derived and tandem repeats more strongly than matched non-repeat sequence; its tandem-repeat compression preference transfers to a nine-species evaluation panel without further training. Third, it enables efficient long-context modeling: on 450-kb inputs, GeneZip-70M achieves 36.1-fold higher inference throughput than JanusDNA-7.7M on a single H100 GPU.

q-bio.GN · cs.AI · cs.LG

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2605.27769

Smoothed Score Queries and the Complexity of Sampling

Jingbo Liu

原文摘要

We study the query complexity of sampling from high-dimensional Gaussian distributions using gradient information. In the standard oracle model, exact gradients expose only matrix-vector products with the precision matrix, leading to polynomial approximation barriers and a characteristic $\sqrtκ$ dependence on the known spectral range. We show that this barrier disappears when the sampler is allowed to query smoothed scores, namely gradients of the logarithms of the Gaussian-convolved densities. For a centered Gaussian target with precision matrix $Λ$, a smoothed-score query at noise level $τ$ gives access to the resolvent $(Λ+τ^{-1}I)^{-1}$. Combining geometrically spaced noise levels with sinc-quadrature rational approximation, we obtain a sampler with $q=O((\logκ+\log(e\sqrt d/δ_{\rm TV}))\log(e\sqrt d/δ_{\rm TV}))$ smoothed-score queries for total variation error $δ_{\rm TV}$, improving the condition-number dependence from $\sqrtκ$ to logarithmic. We also study finite-bit gradient oracles. Using coordinatewise quantization of the transformed smoothed-score answers and a final dithering step, we obtain a sampling scheme whose total communicated gradient information is polylogarithmic in $κ$; in particular, for fixed dimension and accuracy, the bit complexity is $O(\log^2κ)$. To complement these upper bounds, we introduce a channel-synthesis, or reverse-Shannon, converse technique for sampling lower bounds. This converts total-variation simulation guarantees into communication requirements and yields an $Ω(d\logκ)$ bit lower bound for $d\ge2$ and fixed total variation error below one, as $κ\to\infty$. Together, these results identify smoothed scores as a provably more informative oracle for sampling and give nearly matching upper and lower bounds for its finite-bit complexity.

cs.DS · cs.IT · cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v3arXiv:2507.12412

Contrastive Nonmyopic Objective Cost-Tradeoff Acquisition for Longitudinal Data

Dzung Dinh, Boqi Chen, Yunni Qu, Marc Niethammer, Junier Oliva

原文摘要

In many critical domains, features are not freely available at inference time: each measurement may come with a cost of time, money, and risk. Longitudinal prediction further complicates this setting because both features and labels evolve over time, and measurements missed at earlier timepoints may become permanently unavailable. We propose CAPE (Contrastive Acquisition-Plan Embeddings), a partitioning acquisition policy for longitudinal active feature acquisition. CAPE first defines NOCT, an oracle-based objective that scores a set of future feature-time acquisitions by its expected predictive loss together with its acquisition cost. Based on NOCT, CAPE learns a contrastive acquisition-utility manifold of masked partial observations, where proximity reflects the expected utility of future acquisitions. Offline, training states are embedded and partitioned at each rollout timepoint; candidate future plans are scored within each partition using plug-in NOCT, and the best partition-level acquisition plans are cached. During inference, CAPE retrieves the cached plan via nearest-centroid lookup, enabling low-latency adaptive acquisitions. Experiments on synthetic and real-world healthcare datasets demonstrate that CAPE outperforms nonparametric, greedy, and RL-based alternatives, achieving higher accuracy at lower acquisition costs. Code is publicly available at https://github.com/dzungdinhh/CAPE.

cs.LG · cs.AI

查看论文
arXiv ↗PDF ↗
版本更新 · v3arXiv:2605.06332

LINC: Decoupling Local Consequence Scoring from Hidden Matching in Constructive Neural Routing

Shaofeng Qin, Li Wang

原文摘要

Constructive neural routing solvers usually score the next action by matching a decoder context against candidate embeddings computed once per instance from static node features such as coordinates, demands, and time windows; deterministic one-step consequences such as travel, waiting, slack, and capacity changes are not provided as inputs and must be inferred implicitly by the model. We propose LINC, a decoder-side candidate decision architecture that decouples computing these consequences from learning how much they matter. The environment computes the consequences explicitly at each step; candidate-level consequences are scored by a state-conditioned shared linear comparator, while feasible-set summaries modulate the decoder context. This keeps the cached embeddings and standard global matching while giving the scoring function direct access to deterministic transition quantities. The Capacitated Vehicle Routing Problem with Time Windows (CVRPTW) serves as the main constrained-routing testbed, and the same interface extends to the Capacitated Vehicle Routing Problem (CVRP) and Traveling Salesman Problem (TSP). Across external benchmarks and no-retraining scale-transfer settings, LINC consistently improves strong neural baselines, with the advantage becoming more pronounced as test size moves further beyond the training scale, especially on constrained routing problems.

cs.LG

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2511.18571

SAMBA: Toward an EEG Foundation Model via Differential Mamba and Coordinate-Driven Spatial Embedding

Jiazhen Hong, Geoffrey Mackellar, Soheila Ghane

原文摘要

Modeling long-context electroencephalography (EEG) signals is critical for learning generalizable EEG representations, as EEG recordings are acquired at high sampling rates and often span extended durations that capture evolving neural states beyond short task- specific windows. Transformer-based models have shown promise in modeling short sequences of a few seconds; however, their quadratic complexity limits scalability to longer contexts. Moreover, substantial variability in electrode montages across datasets and recording devices poses a major challenge for building EEG models that generalize across domains. We propose SAMBA, a self-supervised learning framework with a Mamba- based U-shaped encoder-decoder architecture, which effectively captures long-range temporal dependencies and montage variability in EEG data. Leveraging the inherent ability of Mamba in processing long context sizes, we introduce: (1) Temporal Semantic Random Masking for semantic-level sequence reconstruction, (2) a Multi-Head Differential Mamba module to suppress redundancy and emphasize salient temporal structures, and (3) a Coordinate-Driven Spatial Embedding that learns unified embeddings in a three- dimensional Euclidean space, enabling robustness across devices. Experiments on thirteen EEG datasets across diverse tasks, electrode configurations, and sequence durations demonstrate that SAMBA achieves competitive or improved performance compared to recent state-of-the-art methods, while maintaining low memory consumption and efficient inference. The learned spatial weight maps exhibit close alignment with task-relevant neurophysiological regions, highlighting the interpretability of SAMBA. The code is available at: https://github.com/Jiazhen-Hong/SAMBA

cs.LG

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2610.12358

Subspace Uncertainty and Sharp Sampling Thresholds on the Boolean Cube

Thomas Weinberger

原文摘要

For a known finite-dimensional model subspace, classical Gaussian regression theory guarantees that least squares eventually attains the parametric rate, with prediction error scaling as the subspace dimension divided by the sample size. Yet subspaces with the same number of unknown coefficients can require very different sample sizes to reach this regime. For subspaces of polynomials on the Boolean cube, a fundamental question is how restrictions on degree and ambient dimension jointly determine this threshold. Our main result sharply characterizes the worst-subspace minimax sample threshold over a substantial parameter range, revealing an exponential dependence on degree and precisely quantifying the joint effects of degree, ambient dimension, and subspace dimension. A small modification of our lower-bound construction reveals a strong separation: a single subspace can admit noiseless recovery with optimal sample complexity independent of degree, yet require essentially the worst-case sample size, exponential in degree, to attain the noisy parametric rate. To establish the sampling lower bound, we sharpen the Polyanskiy-Samorodnitsky uncertainty principle (J. Funct. Anal., '19) to uniformly optimal precision and extend its concentration guarantee to large subspaces in which every function concentrates most of its energy on one shared Hamming ball. The sampling upper bound follows from the sharp moment bound of Kirshner and Samorodnitsky (IEEE Trans. Inf. Theory, '21) combined with empirical-process bounds.

math.PR · cs.IT · cs.LG · math.CO

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2610.38582

Robust Active Learning for More Efficient and Reliable Paths to Optimal Design

Hayriye Pehlivan Solak, Francesco Di Fiore, Jeroen Wackers, Riccardo Pellegrini, Andrea Serani, Matteo Diez, Laura Mainini

原文摘要

Simulation-driven design optimization requires active learning strategies that efficiently combine models with different computational costs and levels of fidelity. We introduce Multifidelity Cost-Aware Screening (MF-CAS), which enables the joint selection of sampling location and fidelity using a surrogate that provides decomposed uncertainty intervals rather than a complete predictive probability distribution. MF-CAS exploits the contribution of each fidelity to the high-fidelity uncertainty together with its computational cost to determine whether a candidate should first be evaluated at low fidelity or directly progressed towards high fidelity. The method is assessed on the constrained 14-dimensional hydrodynamic optimization of the DTMB 5415 hull under a deliberately limited computational budget. MF-CAS achieves an approximately 9.0% resistance reduction, compared with 6.7% for the best baseline optimisation method, demonstrating that incorporating fidelity-resolved uncertainty directly into the joint location-fidelity decision can substantially improve multifidelity active learning efficiency.

cs.CE · cs.LG

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2610.37393

Learning Quality-Controllable Robot Policies with Iterative Hindsight Relabeling

Sinjae Kang, Dong Kyu Shin, Jaehong Kim, Kamin Lee, Hyoeun Kim, Kaixin Wang, Li Zhao, Seonghyeon Ye 等 9 位作者

原文摘要

Robot policy evaluation often relies on a binary success signal for task completion. In real-world deployment, however, desired execution behaviors are also defined by quality attributes such as execution speed and motion smoothness, which need to be dynamically controlled according to task requirements. To address this, we introduce Quality-Controllable Policy (QCP), a robot policy that enables inference-time control over both individual quality attributes and their combinations. To extend the controllable quality range beyond initial demonstrations, we further propose an iterative hindsight relabeling loop: QCP collects online rollouts under target quality labels, adds successful trajectories to the dataset, relabels the accumulated trajectories in hindsight according to their measured quality, and fine-tunes the policy. Experiments on Robomimic and RoboEval demonstrate that QCP consistently improves both task execution quality and success rates across iterations, producing successful rollouts that exceed the quality of the initial demonstrations. Finally, we validate QCP on real-world manipulation tasks, where controlling speed increases throughput in conveyor operations and controlling transport stability improves success when carrying wine glasses, demonstrating that quality-controllable policies yield practical gains in physical deployment.

cs.RO · cs.AI · cs.LG

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2610.37005

Compote: Collapse-Resistant Multiple Hypothesis Learning For Sequential Recommendation

Louis Castres Saint Martin, Clement Calauzenes, Alain Rakotomamonjy, Patrick Gallinari

原文摘要

Next-item prediction is inherently ambiguous: a single interaction history often admits multiple plausible continuations across distinct catalogue regions. Standard sequential recommenders compress these alternatives into a single prediction, conflating user intents. We introduce COllapse-resistant Multiple hyPOThesis lEarning (COMPOTE), a multi-hypothesis transformer trained with an annealed objective that transitions from uniform supervision to a winner-takes-all assignment. This mechanism forces hypotheses to specialize into distinct catalogue regions, while inference relies on an efficient, normalizer-free max-similarity rule. Theoretically, we show that in the population case, the optimal solutions partition the conditional next-item distribution according to a max-entropy quantization principle, while pooled scores preserve the Bayes-optimal ranking. We also prove a generalization bound scaling logarithmically with the number of hypotheses. Across three Amazon Reviews 2023 benchmarks, COMPOTE outperforms strong ID-based, generative, and multi-interest baselines using item IDs alone. Matched-capacity ablations confirm that performance gains stem directly from the annealed objective, preventing the head collapse observed in conventional multi-head architectures, rather than added capacity, ensembling, or routing.

cs.LG · cs.IR

查看论文
arXiv ↗PDF ↗
版本更新 · v4arXiv:2605.24042

Hidden-State Privacy Has an Empty Middle

Alexander Okezue Bell

原文摘要

Of $1{,}536$ Gaussian release covariances we tested for single-layer hidden-state privacy, zero achieve both moderate utility and moderate privacy against an adaptive retrieval attacker. We prove a complementary Fisher-ball lower bound: every full-rank Gaussian release at $O(1)$ Fisher utility admits a direction whose Mahalanobis signal grows linearly in hidden width, ruling out uniform Gaussian safety in the class and matching the empirical empty middle. The diagonal inverse-Fisher release $Σ^\star_{\mathrm{diag}}(\mathcal{K}) = (2\mathcal{K}/d)\,\mathrm{diag}(1/F_{ii})$ is the unique minimax-optimal diagonal mechanism at first-order KL budget $\mathcal{K}$ and the only release with worst-attacker top-1 $\le 0.001$ at every point of a 32 model-layer grid, but it sits on a privacy/utility edge rather than filling the middle. A generalized-eigen mechanism reaching $13\times$ Pareto reduction under Euclidean retrieval collapses to $100\%$ top-1 under the adaptive Mahalanobis attacker, and a full-trajectory sequence inverter recovers $94\%$ of clean GPT-2 prefixes but $0\%$ under $Σ_{\mathrm{diag}}$. A split-memory transformer trained from scratch reaches $G_{\mathrm{Mah}} \in [20, 33]$ at 90M and maintains a $6$--$24\times$ advantage over same-budget GPT baselines from 30M to 1B at a fixed-token language-modeling loss penalty; pretrained models top out at 9.3. These results reframe hidden-state release from mechanism-design within the Gaussian class to architecture or release co-design.

cs.LG · cs.AI

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2502.02617Google Research

PolarQuant: Quantizing KV Caches with Polar Transformation

Insu Han, Praneeth Kacham, Amin Karbasi, Vahab Mirrokni, Amir Zandieh

原文摘要

Large language models (LLMs) require significant memory to store Key-Value (KV) embeddings in their KV cache, especially when handling long-range contexts. Quantization of these KV embeddings is a common technique to reduce memory consumption. This work introduces PolarQuant, a novel quantization method employing random preconditioning and polar transformation. Our method transforms the KV embeddings into polar coordinates using an efficient recursive algorithm and then quantizes resulting angles. Our key insight is that, after random preconditioning, the angles in the polar representation exhibit a tightly bounded and highly concentrated distribution with an analytically computable form. This nice distribution eliminates the need for explicit normalization, a step required by traditional quantization methods which introduces significant memory overhead because quantization parameters (e.g., zero point and scale) must be stored in full precision per each data block. PolarQuant bypasses this normalization step, enabling substantial memory savings. The long-context evaluation demonstrates that PolarQuant compresses the KV cache by over x4.2 while achieving the best quality scores compared to the state-of-the-art methods.

cs.LG · cs.AI

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2003.04974Google Research

Transformer++

Prakhar Thapak, Prodip Hore

原文摘要

Recent advancements in attention mechanisms have replaced recurrent neural networks and its variants for machine translation tasks. Transformer using attention mechanism solely achieved state-of-the-art results in sequence modeling. Neural machine translation based on the attention mechanism is parallelizable and addresses the problem of handling long-range dependencies among words in sentences more effectively than recurrent neural networks. One of the key concepts in attention is to learn three matrices, query, key, and value, where global dependencies among words are learned through linearly projecting word embeddings through these matrices. Multiple query, key, value matrices can be learned simultaneously focusing on a different subspace of the embedded dimension, which is called multi-head in Transformer. We argue that certain dependencies among words could be learned better through an intermediate context than directly modeling word-word dependencies. This could happen due to the nature of certain dependencies or lack of patterns that lend them difficult to be modeled globally using multi-head self-attention. In this work, we propose a new way of learning dependencies through a context in multi-head using convolution. This new form of multi-head attention along with the traditional form achieves better results than Transformer on the WMT 2014 English-to-German and English-to-French translation tasks. We also introduce a framework to learn POS tagging and NER information during the training of encoder which further improves results achieving a new state-of-the-art of 32.1 BLEU, better than existing best by 1.4 BLEU, on the WMT 2014 English-to-German and 44.6 BLEU, better than existing best by 1.1 BLEU, on the WMT 2014 English-to-French translation tasks. We call this Transformer++.

cs.CL · cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗