Domain-Routed Delta Optimization for Multi-Teacher On-Policy Distillation

摘要变化

Multi-teacher on-policy distillation assigns eachcan sampleyield auneven domain teacher but typically updates shared student parameterscapabilities. We introducepropose Domain-Routed Delta Optimization (DRD-MOPD), which specifies which parameter components each domain's samples update. The student is the sum ofcomposing a frozen base and trainable components initialized fromwith expert-minus-base weights. All components participate in everyinitialized forwardtrainable pass;deltas. fourDomains update modes define the correspondence between sample domains and components,their eachdeltas with independent optimizer states. We analyze these update rules and measure retention of the original expert geometry through angles, projection coefficients, and energyclipping. Prior workAfter suchjoint astraining, OpenPost-MOPD distills directlyseparate fromexports the base model; 97.02% of its released final checkpoint's increment energy lies outside the span of the original expert directions. In comparison, our Own checkpoint after 20 updates retains coefficients nearplus one along each component's original direction and pairwise angles ranging from 88.04trained todelta 92.41per degrees;domain, itsselected exportedby BF16known student'stask incrementdomain haswithout 4retraining.90% outsideUsing energyAdamW, versus 10.52% for the sum of saved FP32 components. The sameunweighted checkpointfull achievesstudent areaches 32.7283% macro scoreMacro across six benchmarks, comparedafter with20 updates, versus 32.4421% for itsa initialmatched task-arithmeticshared mergecontrol. These measurements describe geometry and capability at thePost-separate evaluatedraises trainingMacro stageswith andOpen-MOPD settingsweighting.