← 研踪 PaperTrailarXiv:2603.03405 · v2
Surprisal-Rényi Free Energy
首次提交 2026-03-04 00:42
版本更新 2026-11-03 23:40
雷达观测 2026-11-04 23:39
摘要
The forward and reverse Kullback-Leibler (KL) divergences arise as limiting objectives in learning and inference, yet they induce markedly different inductive biases that cannot be explained by expectation terms alone. We introduce the Surprisal-Rényi Free Energy (SRFE), a normalized Rényi-Chernoff log-moment functional of the likelihood ratio. Although SRFE is algebraically related to Rényi divergence for fixed values of the order parameter, its endpoint-symmetric normalization yields a distinct free-energy viewpoint for analyzing how forward- and reverse KL behavior emerges near singular limits. We show that SRFE recovers the forward and reverse KL divergences as endpoint limits and derive local expansions around both regimes in which the variance of the log-likelihood ratio appears as a first-order correction away from the KL objective. These expansions reveal an explicit mean-variance tradeoff governing departures from KL-dominated behavior. We further establish a Gibbs-type variational characterization in terms of weighted KL projections and prove that SRFE controls large deviations of excess code length through Chernoff-type bounds, giving a Minimum Description Length interpretation of the functional. We evaluate SRFE on two synthetic settings (two-dimensional Gaussian-mixture variational inference and multi-class classification) and the long-tailed CIFAR-10 and CIFAR-100 (CIFAR-LT) datasets. On CIFAR-LT, retraining the best configuration of each loss over five seeds, SRFE matches cross-entropy accuracy on CIFAR-10-LT and gives mixed results on CIFAR-100-LT, improving few-shot accuracy at one of three imbalance factors. Together, these results position SRFE as a variance- and tail-sensitive, normalized Rényi-Chernoff free energy that clarifies the geometric, statistical, and large-deviation structure underlying the transition between forward- and reverse-KL limits.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-03 23:40 · 最新观测版本
Surprisal-Rényi Free Energy
The forward and reverse Kullback-Leibler (KL) divergences arise as limiting objectives in learning and inference, yet they induce markedly different inductive biases that cannot be explained by expectation terms alone. We introduce the Surprisal-Rényi Free Energy (SRFE), a normalized Rényi-Chernoff log-moment functional of the likelihood ratio. Although SRFE is algebraically related to Rényi divergence for fixed values of the order parameter, its endpoint-symmetric normalization yields a distinct free-energy viewpoint for analyzing how forward- and reverse KL behavior emerges near singular limits. We show that SRFE recovers the forward and reverse KL divergences as endpoint limits and derive local expansions around both regimes in which the variance of the log-likelihood ratio appears as a first-order correction away from the KL objective. These expansions reveal an explicit mean-variance tradeoff governing departures from KL-dominated behavior. We further establish a Gibbs-type variational characterization in terms of weighted KL projections and prove that SRFE controls large deviations of excess code length through Chernoff-type bounds, giving a Minimum Description Length interpretation of the functional. We evaluate SRFE on two synthetic settings (two-dimensional Gaussian-mixture variational inference and multi-class classification) and the long-tailed CIFAR-10 and CIFAR-100 (CIFAR-LT) datasets. On CIFAR-LT, retraining the best configuration of each loss over five seeds, SRFE matches cross-entropy accuracy on CIFAR-10-LT and gives mixed results on CIFAR-100-LT, improving few-shot accuracy at one of three imbalance factors. Together, these results position SRFE as a variance- and tail-sensitive, normalized Rényi-Chernoff free energy that clarifies the geometric, statistical, and large-deviation structure underlying the transition between forward- and reverse-KL limits.
查看此版本解读 →查看此版本原文 ↗