← 研踪 PaperTrailarXiv:2602.18002 · v2
Asynchronous Heavy-Tailed Optimization
首次提交 2026-02-20 13:28
版本更新 2026-10-29 04:02
雷达观测 2026-10-30 10:22
摘要
Heavy-tailed stochastic gradient noise, commonly observed in transformer models, can destabilize the optimization process. Recent works mainly focus on developing and understanding approaches to address heavy-tailed noise in the centralized or distributed, synchronous setting, while asynchronous optimization under such noise remains underexplored, especially in the practical settings with local updates. In this work, we first investigate two asynchronous communication schemes (server- and client-centric) that handle stragglers under heavy-tailed gradient noise. We further propose and theoretically analyze algorithmic modifications based on delay-aware learning rate scheduling and delay compensation to enhance the performance of asynchronous algorithms. Our convergence guarantees achieve the optimal rate under heavy-tailed noise and improve delay tolerance compared with existing asynchronous approaches. Empirically, our approaches outperform prior synchronous and asynchronous methods in terms of accuracy/runtime trade-offs and are more robust to hyperparameters in both image and language tasks.
作者备注：9-page main body, 25-page appendix, 5 figures
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-29 04:02 · 最新观测版本
Asynchronous Heavy-Tailed Optimization
Heavy-tailed stochastic gradient noise, commonly observed in transformer models, can destabilize the optimization process. Recent works mainly focus on developing and understanding approaches to address heavy-tailed noise in the centralized or distributed, synchronous setting, while asynchronous optimization under such noise remains underexplored, especially in the practical settings with local updates. In this work, we first investigate two asynchronous communication schemes (server- and client-centric) that handle stragglers under heavy-tailed gradient noise. We further propose and theoretically analyze algorithmic modifications based on delay-aware learning rate scheduling and delay compensation to enhance the performance of asynchronous algorithms. Our convergence guarantees achieve the optimal rate under heavy-tailed noise and improve delay tolerance compared with existing asynchronous approaches. Empirically, our approaches outperform prior synchronous and asynchronous methods in terms of accuracy/runtime trade-offs and are more robust to hyperparameters in both image and language tasks.
查看此版本解读 →查看此版本原文 ↗