arXiv:2610.35013 · v1
Multiscale Policy Optimization for Long-Horizon LLM Agents
Tao RenHui YangJia DengTingyang ChenQiyao WangShuo TangZiyang ZengYuhua JiangGuanghao LiHuatong SongYining LiYifan LuoKunbao DingFeng ChangChuan HaoRan TaoBryan DaiJiafeng LiangJinyang Jiang
首次提交 2026-10-26 16:22
版本更新 2026-10-26 16:22
雷达观测 2026-10-29 00:36
摘要
Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We propose Multiscale Policy Optimization (MPO), combining Multiscale Advantage Estimation (MAE) and turn-level importance sampling to align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.