跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.26504 · v1

Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning

网格细胞作为谱后继特征:一种可复用的持续强化学习基础

首次提交 2026-11-20 01:54
版本更新 2026-11-20 01:54
雷达观测 2026-11-20 12:52
arXiv ↗PDF ↗

摘要

Continual reinforcement learning requires agents to adapt when the task changes while reusing what they know about the environment. Successor features separate dynamics from task-specific reward, but their effectiveness depends on the choice of base features. Taking the identification of entorhinal grid cells with eigenvectors of the successor representation as a design principle, we build this basis from eigenvectors of the transition matrix of a fixed reference policy. With such a basis, the successor features at a state-action pair are a linear combination of the base features at that same pair, which is not true in general, and the coefficients are known in closed form: each successor feature is its base feature multiplied by θi=1/(1−γλi)θ_i = 1/(1-γλ_i), where λiλ_i is its eigenvalue. The long-term return is thus predicted from the same features as the immediate reward, no successor-feature network or replay buffer is needed, and adapting to a new task reduces to a linear regression on the reward. The agent acts greedily on the action values of the reference policy, which is one step of policy improvement over it and suffices to reach a single goal. In a two-goal gridworld, with the uniform random walk as reference policy, whose eigenvectors are the low-frequency Laplacian eigenvectors, agents built this way maintain near-maximal return across every change of goal, whereas SimpleSF and DQN lose most of their return at each change.

作者备注:15 pages, 3 figures. Accepted at the NeurReps and WiML workshops at NeurIPS 2026

AI 论文解读

基于摘要 · 非全文审阅

作者提出以参考策略转移矩阵的特征向量作为后继特征基,使后继特征与即时奖励共享同一组特征,从而无需后继特征网络或回放缓冲区即可在任务变化时通过线性回归快速适应。

正在读取账号与研究记录…

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-20 01:54 · 最新观测版本

Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning

Continual reinforcement learning requires agents to adapt when the task changes while reusing what they know about the environment. Successor features separate dynamics from task-specific reward, but their effectiveness depends on the choice of base features. Taking the identification of entorhinal grid cells with eigenvectors of the successor representation as a design principle, we build this basis from eigenvectors of the transition matrix of a fixed reference policy. With such a basis, the successor features at a state-action pair are a linear combination of the base features at that same pair, which is not true in general, and the coefficients are known in closed form: each successor feature is its base feature multiplied by θi=1/(1−γλi)θ_i = 1/(1-γλ_i), where λiλ_i is its eigenvalue. The long-term return is thus predicted from the same features as the immediate reward, no successor-feature network or replay buffer is needed, and adapting to a new task reduces to a linear regression on the reward. The agent acts greedily on the action values of the reference policy, which is one step of policy improvement over it and suffices to reach a single goal. In a two-goal gridworld, with the uniform random walk as reference policy, whose eigenvectors are the low-frequency Laplacian eigenvectors, agents built this way maintain near-maximal return across every change of goal, whereas SimpleSF and DQN lose most of their return at each change.

查看此版本解读 →查看此版本原文 ↗