← 研踪 PaperTrailarXiv:2605.30749 · v2
FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance
首次提交 2026-05-29 10:25
版本更新 2026-11-04 20:08
雷达观测 2026-11-05 11:10
摘要
Maximum entropy reinforcement learning (MaxEnt-RL) enables robust exploration, yet practical implementations often restrict policies to simple Gaussians. While recent approaches incorporate expressive generative policies via importance-weighted supervised learning, they are prone to importance weight collapse, which limits their scalability in high-dimensional action spaces. Our key insight is to mitigate this limitation by localizing the sampling region, avoiding the weight degeneracy induced by importance sampling over the entire action space. To instantiate this insight, we introduce \textbf{FLAG} (\textbf{F}low policy with \textbf{L}atent-\textbf{A}ugmented \textbf{G}uidance). FLAG augments the state space with a flow latent variable and optimizes a provably consistent proxy MaxEnt-RL objective. We empirically demonstrate that FLAG enables expressive policy optimization with limited importance samples and scales to state-based high-dimensional control tasks. Furthermore, FLAG achieves state-of-the-art performance among expressive-policy MaxEnt online RL methods on state-based high-dimensional control benchmarks.
作者备注：NeurIPS 2026, Project page: https://flag-rl.github.io/
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-04 20:08 · 最新观测版本
FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance
Maximum entropy reinforcement learning (MaxEnt-RL) enables robust exploration, yet practical implementations often restrict policies to simple Gaussians. While recent approaches incorporate expressive generative policies via importance-weighted supervised learning, they are prone to importance weight collapse, which limits their scalability in high-dimensional action spaces. Our key insight is to mitigate this limitation by localizing the sampling region, avoiding the weight degeneracy induced by importance sampling over the entire action space. To instantiate this insight, we introduce \textbf{FLAG} (\textbf{F}low policy with \textbf{L}atent-\textbf{A}ugmented \textbf{G}uidance). FLAG augments the state space with a flow latent variable and optimizes a provably consistent proxy MaxEnt-RL objective. We empirically demonstrate that FLAG enables expressive policy optimization with limited importance samples and scales to state-based high-dimensional control tasks. Furthermore, FLAG achieves state-of-the-art performance among expressive-policy MaxEnt online RL methods on state-based high-dimensional control benchmarks.
查看此版本解读 →查看此版本原文 ↗