← 研踪 PaperTrailarXiv:2604.00686 · v2
Full-Gradient Successor Feature Learning for Multi-Task Transfer in Reinforcement Learning
首次提交 2026-04-01 17:44
版本更新 2026-11-08 00:15
雷达观测 2026-11-10 11:14
摘要
Successor Features (SFs) provide a principled framework for multi-task transfer in Reinforcement Learning (RL) by decoupling environment dynamics from reward functions. By learning representations of expected future feature occupancies, SFs enable knowledge acquired in one task to be transferred to other tasks with different reward functions. However, conventional SF learning relies on semi-gradient Temporal Difference (TD) updates, which, when combined with nonlinear function approximation, generally lack convergence guarantees and can exhibit instability. This motivates exploring alternative learning objectives that offer stronger theoretical properties. Inspired by Full-Gradient DQN, we propose Full-Gradient Successor Feature Representations Q-Learning (FG-SFRQL), which extends full-gradient optimization to successor feature learning by minimizing the squared SF Bellman residual and differentiating through both the prediction and the bootstrapped target. We establish almost-sure convergence of FG-SFRQL under randomized task sampling, characterize the objective it minimizes, and show that averaging the bootstrapped target over $N$ next-state transitions reduces double-sampling bias in proportion to $1/N$. Through experiments in both discrete- and continuous-state domains, we demonstrate that full-gradient learning achieves lower Bellman residuals and improved transfer performance compared to semi-gradient baselines, with FG-SFRQL attaining the highest final return across all five evaluated domains.
作者备注：Submitted to ECC 2027
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-08 00:15 · 最新观测版本
Full-Gradient Successor Feature Learning for Multi-Task Transfer in Reinforcement Learning
Successor Features (SFs) provide a principled framework for multi-task transfer in Reinforcement Learning (RL) by decoupling environment dynamics from reward functions. By learning representations of expected future feature occupancies, SFs enable knowledge acquired in one task to be transferred to other tasks with different reward functions. However, conventional SF learning relies on semi-gradient Temporal Difference (TD) updates, which, when combined with nonlinear function approximation, generally lack convergence guarantees and can exhibit instability. This motivates exploring alternative learning objectives that offer stronger theoretical properties. Inspired by Full-Gradient DQN, we propose Full-Gradient Successor Feature Representations Q-Learning (FG-SFRQL), which extends full-gradient optimization to successor feature learning by minimizing the squared SF Bellman residual and differentiating through both the prediction and the bootstrapped target. We establish almost-sure convergence of FG-SFRQL under randomized task sampling, characterize the objective it minimizes, and show that averaging the bootstrapped target over $N$ next-state transitions reduces double-sampling bias in proportion to $1/N$. Through experiments in both discrete- and continuous-state domains, we demonstrate that full-gradient learning achieves lower Bellman residuals and improved transfer performance compared to semi-gradient baselines, with FG-SFRQL attaining the highest final return across all five evaluated domains.
查看此版本解读 →查看此版本原文 ↗