On-Policy Self-Distillation (OPSD) improves reasoning by training a student on its own reasoning, using the same model conditioned on reference solutions as a teacher. Applying OPSD to multilingual reasoning requires the teacher to provide useful feedback across languages, even when the student makes mistakes in intermediate steps. We observe a phenomenon we call Teacher Drift: as these mistakes accumulate, the teacher may follow the student's incorrect steps instead of correcting them. Our analysis shows that this shift tends to occur earlier in lower-resource languages. To account for these differences across languages, we propose Teachable On-Policy Self-Distillation (TOPSD), which determines where and whether teacher supervision should be applied to each reasoning trajectory. TOPSD uses teacher-student likelihood to estimate where drift begins and select a prefix for supervision. It then verifies whether the teacher can recover the correct answer from that prefix, applying distillation only to prefixes that pass verification. Experiments on multilingual mathematical reasoning across ten languages and three model scales show that TOPSD consistently outperforms conventional OPSD, particularly in low- and medium-resource languages.
Jinyoung Park, Sanghyeok Lee, Omar Zia Khan, Hyunwoo J. Kim, Joo-Kyung Kim
原文摘要
Multi-hop question answering requires retrieval agents to construct a connected evidence path across reasoning turns rather than retrieve isolated relevant evidence for a single query. However, in existing GraphRAG methods, retrieval and reinforcement learning are often optimized with mismatched signals: retrieval is driven by local semantic relevance, while outcome-level reinforcement learning assigns the reward by assessing the final answer without identifying which retrieval actions progress the reasoning path. We introduce HyperGraphPro, a progress-aware hypergraph RAG framework that focuses on evidence-path progress as the shared objective for retrieval and policy optimization. For retrieval, HyperGraphPro reranks candidate hyperedges using query-conditioned structural distinctiveness, combining entity-query relevance with a local-vs-global hypergraph incidence signal to favor entities that are informative in the current graph neighborhood. For policy optimization, HyperGraphPro augments uniform trajectory-level reward assignment with stepwise progress-aware policy optimization, which assigns step-level rewards according to the coverage of answer-guided target hyperedges. Experiments on multi-hop question answering benchmarks demonstrate that HyperGraphPro consistently improves reasoning accuracy over existing GraphRAG methods.