← 研踪 PaperTrailarXiv:2408.02697 · v6
Why Rectified Power Unit Networks Fail and How to Improve It: An Effective Field Theory Perspective
首次提交 2024-08-04 21:05
版本更新 2026-11-05 23:35
雷达观测 2026-11-06 13:11
摘要
The Rectified Power Unit (RePU) activation function, a differentiable generalization of the Rectified Linear Unit (ReLU), has shown promise in constructing neural networks due to its smoothness properties. However, deep RePU networks often suffer from critical issues such as vanishing or exploding values during training, rendering them unstable regardless of hyperparameter initialization. Leveraging the perspective of effective field theory, we identify the root causes of these failures and propose the Modified Rectified Power Unit (MRePU) activation function. MRePU addresses RePU's limitations while preserving its advantages, such as differentiability and universal approximation properties. Theoretical analysis demonstrates that MRePU satisfies the criticality conditions in the limit of a small kernel, and a finite-kernel analysis yields an initialization rule for deep MRePU networks. Experiments on polynomial regression, physics-informed neural networks (PINNs) and real-world vision tasks show that MRePU networks initialized by this rule remain trainable at depths at which RePU networks fail, while their accuracy relative to standard activations depends on the task.
作者备注：57 pages, 21 figures
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v6 · 2026-11-05 23:35 · 最新观测版本
Why Rectified Power Unit Networks Fail and How to Improve It: An Effective Field Theory Perspective
The Rectified Power Unit (RePU) activation function, a differentiable generalization of the Rectified Linear Unit (ReLU), has shown promise in constructing neural networks due to its smoothness properties. However, deep RePU networks often suffer from critical issues such as vanishing or exploding values during training, rendering them unstable regardless of hyperparameter initialization. Leveraging the perspective of effective field theory, we identify the root causes of these failures and propose the Modified Rectified Power Unit (MRePU) activation function. MRePU addresses RePU's limitations while preserving its advantages, such as differentiability and universal approximation properties. Theoretical analysis demonstrates that MRePU satisfies the criticality conditions in the limit of a small kernel, and a finite-kernel analysis yields an initialization rule for deep MRePU networks. Experiments on polynomial regression, physics-informed neural networks (PINNs) and real-world vision tasks show that MRePU networks initialized by this rule remain trainable at depths at which RePU networks fail, while their accuracy relative to standard activations depends on the task.
查看此版本解读 →查看此版本原文 ↗