跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.25624 · v1

Deviations from the NTK Regime in CNNs under Joint Depth-Width Scaling

首次提交 2026-11-19 15:09
版本更新 2026-11-19 15:09
雷达观测 2026-11-20 19:58
arXiv ↗PDF ↗

摘要

The concentration and training-time constancy of the neural tangent kernel (NTK) depend on the interplay between network width, depth, and initialization. While joint depth-width scaling has been studied in fully-connected, residual, and recurrent networks, its effects in convolutional neural networks (CNNs) remain less understood. We investigate these effects numerically in CNNs, examining NTK dispersion at initialization and kernel rotation during training. Let depth LL be the number of convolutional layers, and width CC the number of channels per layer. We find that NTK dispersion increases with L/CL/C in the critical and chaotic initialization regimes, but remains small in the ordered regime. At critical initialization, dispersion curves approximately collapse as a function of L/CL/C, suggesting a nontrivial joint scaling regime. During training, kernel rotation generally increases with L/CL/C, even in the ordered regime where initial dispersion is small. Together, these results suggest that L/CL/C may govern NTK deviations from the infinite-width limit in certain regimes, and that NTK concentration at initialization and stability during training exhibit different scaling behaviors.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-19 15:09 · 最新观测版本

Deviations from the NTK Regime in CNNs under Joint Depth-Width Scaling

The concentration and training-time constancy of the neural tangent kernel (NTK) depend on the interplay between network width, depth, and initialization. While joint depth-width scaling has been studied in fully-connected, residual, and recurrent networks, its effects in convolutional neural networks (CNNs) remain less understood. We investigate these effects numerically in CNNs, examining NTK dispersion at initialization and kernel rotation during training. Let depth LL be the number of convolutional layers, and width CC the number of channels per layer. We find that NTK dispersion increases with L/CL/C in the critical and chaotic initialization regimes, but remains small in the ordered regime. At critical initialization, dispersion curves approximately collapse as a function of L/CL/C, suggesting a nontrivial joint scaling regime. During training, kernel rotation generally increases with L/CL/C, even in the ordered regime where initial dispersion is small. Together, these results suggest that L/CL/C may govern NTK deviations from the infinite-width limit in certain regimes, and that NTK concentration at initialization and stability during training exhibit different scaling behaviors.

查看此版本解读 →查看此版本原文 ↗