← 研踪 PaperTrailarXiv:2603.26464 · v2
Automatic feature identification in least-squares policy iteration using the Koopman operator framework
首次提交 2026-03-27 22:31
版本更新 2026-11-07 00:38
雷达观测 2026-11-09 13:22
摘要
In this paper, we present a Koopman autoencoder-based least-squares policy iteration (KAE-LSPI) algorithm for reinforcement learning (RL). The KAE-LSPI algorithm is based on reformulating the so-called least-squares fixed-point approximation method in terms of extended dynamic mode decomposition (EDMD), thereby enabling automatic feature learning via the Koopman autoencoder (KAE) framework. The approach is motivated by the lack of a systematic procedure for selecting features or kernels in linear RL techniques. Unlike previous approaches, no features or kernels need to be fixed a priori in our method. We discuss the theoretical convergence of KAE-LSPI under suitable assumptions, similar to kernel-based least-squares policy iteration (KLSPI). We empirically compare our method with two existing algorithms: least-squares policy iteration (LSPI) and KLSPI. In particular, for the chainwalk problem, the resulting policies are either optimal or nearly optimal for both three approaches. For the cart-pole system, KAE-LSPI outperforms the other two approaches in terms of policy performance.
作者备注：15 pages, 4 figures
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 00:38 · 最新观测版本
Automatic feature identification in least-squares policy iteration using the Koopman operator framework
In this paper, we present a Koopman autoencoder-based least-squares policy iteration (KAE-LSPI) algorithm for reinforcement learning (RL). The KAE-LSPI algorithm is based on reformulating the so-called least-squares fixed-point approximation method in terms of extended dynamic mode decomposition (EDMD), thereby enabling automatic feature learning via the Koopman autoencoder (KAE) framework. The approach is motivated by the lack of a systematic procedure for selecting features or kernels in linear RL techniques. Unlike previous approaches, no features or kernels need to be fixed a priori in our method. We discuss the theoretical convergence of KAE-LSPI under suitable assumptions, similar to kernel-based least-squares policy iteration (KLSPI). We empirically compare our method with two existing algorithms: least-squares policy iteration (LSPI) and KLSPI. In particular, for the chainwalk problem, the resulting policies are either optimal or nearly optimal for both three approaches. For the cart-pole system, KAE-LSPI outperforms the other two approaches in terms of policy performance.
查看此版本解读 →查看此版本原文 ↗