← 研踪 PaperTrailarXiv:2509.21172 · v3
Inverse Reinforcement Learning with Just Classification and a Few Regressions
首次提交 2025-09-25 21:53
版本更新 2026-11-12 09:58
雷达观测 2026-11-13 14:16
摘要
Inverse reinforcement learning (IRL) seeks to recover a reward function from observed behavior, but rewards are typically only partially identified: many reward--value pairs can induce the same behavior policy. We study this problem in the maximum-entropy, or Gumbel-shock, model under a broad class of statewise affine normalization constraints, with anchor-action constraints as a special case. This leads to Generalized Policy-to--to-Reward (GenPQR), a modular approach to normalized reward recovery based on policy estimation and -evaluation via the Bellman equation; both components can be instantiated with off-the-shelf classification and regression methods. We establish modular finite-sample guarantees under general function approximation, with separate terms for policy-estimation and -function estimation error. We also give explicit bounds for GenPQR with fitted -evaluation. In experiments, GenPQR improves on DeepPQR in reward recovery. Our guarantees extend beyond anchor actions, accommodate large action spaces, and are not tied to a specific neural-network architecture or training procedure.
作者备注：Accepted to NeurIPS 2026
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-12 09:58 · 最新观测版本
Inverse Reinforcement Learning with Just Classification and a Few Regressions
Inverse reinforcement learning (IRL) seeks to recover a reward function from observed behavior, but rewards are typically only partially identified: many reward--value pairs can induce the same behavior policy. We study this problem in the maximum-entropy, or Gumbel-shock, model under a broad class of statewise affine normalization constraints, with anchor-action constraints as a special case. This leads to Generalized Policy-to--to-Reward (GenPQR), a modular approach to normalized reward recovery based on policy estimation and -evaluation via the Bellman equation; both components can be instantiated with off-the-shelf classification and regression methods. We establish modular finite-sample guarantees under general function approximation, with separate terms for policy-estimation and -function estimation error. We also give explicit bounds for GenPQR with fitted -evaluation. In experiments, GenPQR improves on DeepPQR in reward recovery. Our guarantees extend beyond anchor actions, accommodate large action spaces, and are not tied to a specific neural-network architecture or training procedure.
查看此版本解读 →查看此版本原文 ↗