面向多教师在线策略蒸馏的领域路由增量优化
Domain-Routed Delta Optimization for Multi-Teacher On-Policy Distillation
Lin Yao
AI 速读 基于摘要
提出 DRD-MOPD:在冻结基座模型上为各领域挂载专家初始化的可训练增量,并以独立优化器状态与裁剪分别更新,训练后按已知任务领域导出对应增量。
cs.LG · cs.CL
查看原文摘要
Multi-teacher on-policy distillation can yield uneven domain capabilities. We propose Domain-Routed Delta Optimization (DRD-MOPD), composing a frozen base with expert-initialized trainable deltas. Domains update their deltas with independent optimizer states and clipping. After joint training, Post-separate exports the base plus one trained delta per domain, selected by known task domain without retraining. Using AdamW, the unweighted full student reaches 32.83% Macro across six benchmarks after 20 updates, versus 32.21% for a matched shared control. Post-separate raises Macro with Open-MOPD weighting.