Sample-Efficient Ensemble Reinforcement Learning at Scale
Yoav Feigenbaum, Dmitrii Krylov, Roy Fox
原文摘要
Ensemble methods such as MeanQ have been shown to improve the sample efficiency of relatively small model-free deep reinforcement learning (RL) architectures, including DQN and similarly sized variants such as Rainbow. However, to our knowledge, the effects of ensembling on large-scale architectures such as Bigger Better Faster (BBF) have not been studied. In this work, we present MeanQ-BBF, a hybrid model-free algorithm that combines the principles of MeanQ and BBF. Through statistically rigorous evaluation on the Atari 100K benchmark, we show that in contrast to naive scaling of the network size of BBF, which degrades performance, MeanQ-BBF is able to leverage the increase in model capacity to improve performance over BBF. Moreover, MeanQ-BBF does this in a computationally efficient way by organizing the added parameters into largely independent learners. Finally, we discuss the performance-cost tradeoff in the context of practical sample-efficient RL.
cs.LG