Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networks
Xuan Qi, Yi Wei, Fanqi Yu, Furao Shen, Vittorio Murino, Cigdem Beyan
原文摘要
The effect of batch normalization (BN) on the function realized during training is less understood than its effect on optimization. We study training-time BN in continuous piecewise-affine (CPA) networks through the affine regions they induce in local $\ell_\infty$ windows around the data. Conditioned on a mini-batch, every switching hyperplane of a BN layer with learned parameters and a general CPA activation is a translate of a hyperplane through the batch centroid, at an offset given by a standardized quantity times the batch spread, independent of the raw bias. We prove an exact formula, valid under general position, for the number of regions in a window in terms of window cuts and interior intersections. For a BN layer it gives a lower bound on the local count in terms of the standardized offsets; the bound depends on the batch only through its spread and not on the biases or the batch location. Without BN the best such bound is $1$: for any weights and batch, some biases leave the window as a single region. We prove that the formula holds in deeper layers, inside parent affine regions of any rank. By the exact formula, BN increases the expected local count whenever it increases the expected numbers of window cuts and of interior intersections, which we examine empirically. Offset distributions of networks trained on synthetic data, MNIST, CIFAR-10 and TinyImageNet show higher window-cut rates under BN, and exact enumeration in two dimensions attributes most of the additional BN regions to interior intersections.
cs.LG · stat.ML