← 研踪 PaperTrailarXiv:2604.17693 · v3
COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams
首次提交 2026-04-20 09:14
版本更新 2026-11-12 00:57
雷达观测 2026-11-13 14:16
摘要
In cooperative teams where agents act in a fixed order and share a single team-level reward (multi-agent language systems, sequential robotic tasks), per-agent credit assignment is under-determined. Critic-based approaches scale poorly as the number of agents grows owing to the costly maintenance of joint/factored critic(s), whereas the existing critic-free alternatives have other issues: common credit across agents that couples every agent's signal to teammate noise, importance-sampling corrections for upstream-update staleness that incur variance exponential in team size, or per-agent counterfactual replay that isolates each agent's effect at the price of extra environment or reward calls. We propose COSAC, a critic-free policy gradient framework for sequential cooperative teams. COSAC fits an additive per-agent decomposition of the team reward by a single ridge regression on the rollout batch (giving each agent a learning signal decoupled from teammate noise), and computes each agent's counterfactual advantage from fictitious continuations of the current policy (policy forward passes that replace both importance-sampling reweighting and per-agent environment replay, at no extra environment or reward cost). The estimator instantiates the Sequential Aristocrat Utility (SeqAU), our extension of Wolpert and Tumer's (2001) aristocrat utility to sequential teams. We derive bias and conditional variance bounds for SeqAU credits: the direct-effect bound has no explicit team-size factor, while the indirect effect's reward-noise bound scales with the number of downstream agents. Sequential bandits test the analysis across team sizes up to K=16. In sequential multi-agent LLM fine-tuning, COSAC reduces training regret by 17% on consensus-based science reasoning relative to the best critic-free multi-agent baseline and by 61% on Python coding relative to MA-GRPO.
作者备注：To appear at NeurIPS 2026; 10 main text pages
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-12 00:57 · 最新观测版本
COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams
In cooperative teams where agents act in a fixed order and share a single team-level reward (multi-agent language systems, sequential robotic tasks), per-agent credit assignment is under-determined. Critic-based approaches scale poorly as the number of agents grows owing to the costly maintenance of joint/factored critic(s), whereas the existing critic-free alternatives have other issues: common credit across agents that couples every agent's signal to teammate noise, importance-sampling corrections for upstream-update staleness that incur variance exponential in team size, or per-agent counterfactual replay that isolates each agent's effect at the price of extra environment or reward calls. We propose COSAC, a critic-free policy gradient framework for sequential cooperative teams. COSAC fits an additive per-agent decomposition of the team reward by a single ridge regression on the rollout batch (giving each agent a learning signal decoupled from teammate noise), and computes each agent's counterfactual advantage from fictitious continuations of the current policy (policy forward passes that replace both importance-sampling reweighting and per-agent environment replay, at no extra environment or reward cost). The estimator instantiates the Sequential Aristocrat Utility (SeqAU), our extension of Wolpert and Tumer's (2001) aristocrat utility to sequential teams. We derive bias and conditional variance bounds for SeqAU credits: the direct-effect bound has no explicit team-size factor, while the indirect effect's reward-noise bound scales with the number of downstream agents. Sequential bandits test the analysis across team sizes up to K=16. In sequential multi-agent LLM fine-tuning, COSAC reduces training regret by 17% on consensus-based science reasoning relative to the best critic-free multi-agent baseline and by 61% on Python coding relative to MA-GRPO.
查看此版本解读 →查看此版本原文 ↗