Full-Gradient Successor Feature Learning for Multi-Task Transfer in Reinforcement Learning
Ritish Shrirao, Aditya Priyadarshi, Raghuram Bharadwaj Diddigi
原文摘要
Successor Features (SFs) provide a principled framework for multi-task transfer in Reinforcement Learning (RL) by decoupling environment dynamics from reward functions. By learning representations of expected future feature occupancies, SFs enable knowledge acquired in one task to be transferred to other tasks with different reward functions. However, conventional SF learning relies on semi-gradient Temporal Difference (TD) updates, which, when combined with nonlinear function approximation, generally lack convergence guarantees and can exhibit instability. This motivates exploring alternative learning objectives that offer stronger theoretical properties. Inspired by Full-Gradient DQN, we propose Full-Gradient Successor Feature Representations Q-Learning (FG-SFRQL), which extends full-gradient optimization to successor feature learning by minimizing the squared SF Bellman residual and differentiating through both the prediction and the bootstrapped target. We establish almost-sure convergence of FG-SFRQL under randomized task sampling, characterize the objective it minimizes, and show that averaging the bootstrapped target over $N$ next-state transitions reduces double-sampling bias in proportion to $1/N$. Through experiments in both discrete- and continuous-state domains, we demonstrate that full-gradient learning achieves lower Bellman residuals and improved transfer performance compared to semi-gradient baselines, with FG-SFRQL attaining the highest final return across all five evaluated domains.
cs.LG