Specificity-Guided Knowledge Diversion for Decomposable Generative Models
Yucheng Xie, Fu Feng, Jing Wang, Xin Geng, Yong Rui
原文摘要
Generative model pre-training typically optimizes a model as a whole without explicitly organizing its parameters for selective reuse. Decomposable pre-training instead learns reusable components for downstream adaptation while retaining strong generative performance. However, existing knowledge diversion methods incur substantial parameter overhead through weight factorization and rely on fixed component assignments to induce specialization, potentially compromising in-distribution generation quality. We therefore propose Specificity-Guided Knowledge Diversion (SGKDiv), which learns how generative knowledge is shared and specialized for effective generation and selective downstream reuse. SGKDiv constructs Knowledge Units from paired feed-forward channels, preserving native computational structure without SVD-based reparameterization. Calibrated condition-wise routing combines a shared usage prior with conditional adjustments, while complementary regularization shapes their allocation, diversity, and magnitude. Together, these designs replace predefined component assignments with adaptively learned and explicitly regulated knowledge specialization. Experiments across diverse model and training settings demonstrate better in-distribution generation than dense and MoE baselines, alongside stronger downstream adaptation than representative PEFT methods across eight datasets.
cs.LG · cs.CV