知识蒸馏的两个数学模型
Two mathematical models of knowledge distillation
Audrey Xie, Ludwig Schmidt, John Duchi
AI 速读 基于摘要
作者提出并分析知识蒸馏的数学模型，主张其收益主要来自更易优化的损失景观而非更好的模型。
cs.LG
查看原文摘要
Many hypotheses compete to explain the successes of knowledge distillation. To help address this, we propose and analyze a mathematical model of distillation, which suggests that distillation's performance comes not from obtaining better models but from easier to optimize landscapes. For generalized linear models trained with stochastic gradient descent, we prove that distillation fits performant student models asymptotically more quickly than non-distilled models. In rank-1 matrix approximation, we characterize conditions on the target matrix under which gradient descent with distillation converges strictly faster than training on the supervised objective. The theory helps delineate the ways distillation provides benefits (i.e., in optimization speed, not in generalization), and experiments on real datasets corroborate the theoretical predictions.