跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.18511 · v1

On numerical issues of divergences between nearly equal distributions

近乎相等分布间散度的数值问题

首次提交 2026-11-14 02:26
版本更新 2026-11-14 02:26
雷达观测 2026-11-16 13:44
arXiv ↗PDF ↗

摘要

Barron et al. (2026) recently identified numerical issues in the softmax and soft cross-entropy loss. We show that similar issues are present in standard divergence operations between categorical distributions. This includes the forward and reverse Kullback-Leibler (KL) divergence and generalized Jensen-Shannon divergence (JSD), as well as Monte Carlo KL estimates and entropy. When the distributions are nearly equal, rounding and cancellation issues affect the forward and gradient computations. These issues can affect settings using divergences in optimization objectives, such as knowledge distillation, reinforcement learning, and proximal methods. We propose numerically stable reformulations and provide example implementations for each setting, validated by property-based tests.

AI 论文解读

基于摘要 · 非全文审阅

作者指出分类分布间常用散度(前向/反向KL、广义JSD、蒙特卡洛KL估计与熵)在分布近乎相等时存在舍入与相消的数值问题,并提出数值稳定的重写形式。

正在读取账号与研究记录…

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-14 02:26 · 最新观测版本

On numerical issues of divergences between nearly equal distributions

Barron et al. (2026) recently identified numerical issues in the softmax and soft cross-entropy loss. We show that similar issues are present in standard divergence operations between categorical distributions. This includes the forward and reverse Kullback-Leibler (KL) divergence and generalized Jensen-Shannon divergence (JSD), as well as Monte Carlo KL estimates and entropy. When the distributions are nearly equal, rounding and cancellation issues affect the forward and gradient computations. These issues can affect settings using divergences in optimization objectives, such as knowledge distillation, reinforcement learning, and proximal methods. We propose numerically stable reformulations and provide example implementations for each setting, validated by property-based tests.

查看此版本解读 →查看此版本原文 ↗