Sigurd Holmsen, Berit Omli Øksnes, Ingrid Hobæk Haff, Sylvia Richardson, Ali Ramezani-Kebrya
原文摘要
We introduce VINE: a non-parametric mutual information (MI) estimator based on simplified D-vine copulas. Approximating the joint density with a simplified D-vine allows the density ratio to simplify to a product of pair copulas, which we can estimate non-parametrically. This structural simplification yields VINE, a flexible and computationally efficient MI estimator. We compare VINE against SotA neural MI estimators on a range of datasets, both under settings where the simplified D-vine assumption holds and where it is violated, demonstrating competitive accuracy at a lower computational cost for low- to medium-dimensional datasets.
Johan Mylius-Kroken, Elisabeth Wetzer, Ali Ramezani-Kebrya, Robert Jenssen, Kristoffer Knutsen Wickstrøm
原文摘要
Mutual information (MI) is fundamental to representation-learning analysis in deep networks, but in high dimensions MI estimators depend on hyperparameters whose bias often dominates the signal. We show that continuous piecewise-linear (CPWL) networks, including ReLU networks, admit an intrinsic information-theoretic analysis that avoids this pitfall: their inherent partition of the input space into convex linear regions assigns to every input a discrete region label $Π$ alongside the continuous representation $T$ and label $Y$. We introduce a decomposition of the mutual information $I(Y;T)$ into interpretable geometric terms. We show that the dominant term, the *routing information* $I(Y; Π)$, admits an estimable plug-in estimator, that in the well-sampled regime, approximates a population lower bound on the joint information $I(Y; Π, T)$. When the partition saturates, a functional-equivalence quotient collapses regions implementing the same linear operator and restores informativeness; this extension has a single bounded, dimensionless parameter, which is reported through the quotient's compression ratio. The routing estimator agrees strongly with three established MI baselines applied to the same networks and data. The functional quotient recovers informative estimates in the regime bordering the well-sampled and under-sampled regime. We provide a framework for analysing the internal information geometry of CPWL networks, grounded in their inherent piecewise-linear structure rather than in external discretisation.