← 研踪 PaperTrailarXiv:2605.00654 · v2
Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation
首次提交 2026-05-01 21:36
版本更新 2026-11-01 19:36
雷达观测 2026-11-04 23:43
摘要
For a risk-averse finite-horizon Markov Decision Process, we introduce a special class of Markov coherent risk measures, called mini-batch measures. We also define the class of multipattern risk-averse problems that reduce to linear MDPs in the risk-neutral case. We use both concepts in a feature-based $Q$-learning method with multipattern $Q$-factor approximation and we prove a high-probability regret bound of $\widetilde{\mathcal{O}}\big(d^{3/2}HN^{H-1}\sqrt{K}\big)$, where $d$ is the number of features, $H$ is the horizon, $N\ge2$ is the mini-batch size, $K$ is the number of episodes, and $\widetilde{\mathcal{O}}$ hides logarithmic factors. We also propose an economical version of the $Q$-learning method that streamlines the policy evaluation (backward) step. The theoretical results are illustrated on a stochastic assignment problem and a short-horizon multi-armed bandit problem.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-01 19:36 · 最新观测版本
Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation
For a risk-averse finite-horizon Markov Decision Process, we introduce a special class of Markov coherent risk measures, called mini-batch measures. We also define the class of multipattern risk-averse problems that reduce to linear MDPs in the risk-neutral case. We use both concepts in a feature-based $Q$-learning method with multipattern $Q$-factor approximation and we prove a high-probability regret bound of $\widetilde{\mathcal{O}}\big(d^{3/2}HN^{H-1}\sqrt{K}\big)$, where $d$ is the number of features, $H$ is the horizon, $N\ge2$ is the mini-batch size, $K$ is the number of episodes, and $\widetilde{\mathcal{O}}$ hides logarithmic factors. We also propose an economical version of the $Q$-learning method that streamlines the policy evaluation (backward) step. The theoretical results are illustrated on a stochastic assignment problem and a short-horizon multi-armed bandit problem.
查看此版本解读 →查看此版本原文 ↗