BiCAA: Bidirectional Credit Assignment for Search-Augmented Agent
Yibin Huang, Bin Xu, Hailong Cao, Conghui Zhu
By iteratively retrieving and integrating external evidence, multi-step search enables search-augmented agents to tackle complex question answering. However, vanilla GRPO assigns rewards based solely on final answers, providing limited guidance for intermediate search steps. To provide fine-grained process feedback, we evaluate the value of each search step: it should acquire evidence that advances problem solving while avoiding redundant retrieval. Building on these complementary criteria, we introduce BiCAA, which combines Forward Solvability Gain (FSG) and Subsequent Reasoning Gain (SRG) for bidirectional credit assignment. To reward progress in evidence acquisition, FSG quantifies changes in ground-truth answer likelihood after each interaction. SRG compares the subsequent action's likelihood with and without the current interaction to approximately measure its support for subsequent reasoning, using this support as a proxy signal for non-redundant search to encourage effective and efficient search. BiCAA calibrates local feedback through asymmetric fusion; for policy optimization, it integrates accumulated process rewards with terminal outcome supervision. Across seven open-domain QA benchmarks and two Qwen backbones, experiments show improvements in average F1, particularly on multi-hop tasks.
尚无 AI 解读，展示原文摘要