arXiv:2611.10981 · v1
An Analysis of Reinforcement Learning with Function Approximation
摘要
We address the problem of computing the optimal Q-function in Markov decision problems with infinite state-space. We analyze the convergence properties of several variations of Q-learning when combined with function approximation, extending the analysis of TD-learning in Tsitsiklis and Van Roy (1996) to stochastic control settings. We identify conditions under which such approximate methods converge with probability 1. We conclude with a brief discussion on the general applicability of our results and compare them with several related works.
作者备注：This version of the paper corrects the one that was presented at the 25th International Conference on Machine Learning. We corrected condition (7) of Theorem 1 and its proof, as well as the importance-sampling update (11). We also made the assumptions of Theorems 1 and 2 more precise and revised the proof of Theorem 2, adding an appendix with the auxiliary results it uses
期刊信息：Francisco S. Melo, Sean P. Meyn, M. Isabel Ribeiro. "An analysis of reinforcement learning with function approximation." Proc. 25th Int. Conf. Machine Learning, pp. 664-671, 2008