Comparing exploration strategies for Q-learning in random stochastic mazes

Arryon D. Tijsma, Madalina M. Drugan, Marco A. Wiering

Onderzoeksoutput: Conference contributionAcademicpeer review

34 Citaten (Scopus)
135 Downloads (Pure)

Samenvatting

Balancing the ratio between exploration and exploitation is an important problem in reinforcement learning. This paper evaluates four different exploration strategies combined with Q-learning using random stochastic mazes to investigate their performances. We will compare: UCB-1, softmax, epsilon-greedy, and pursuit. For this purpose we adapted the UCB-1 and pursuit strategies to be used in the Q-learning algorithm. The mazes consist of a single optimal goal state and two suboptimal goal states that lie closer to the starting position of the agent, which makes efficient exploration an important part of the learning agent. Furthermore, we evaluate two different kinds of reward functions, a normalized one with rewards between 0 and 1, and an unnormalized reward function that penalizes the agent for each step with a negative reward. We have performed an extensive grid-search to find the best parameters for each method and used the best parameters on novel randomly generated maze problems of different sizes.
The results show that softmax exploration outperforms the other strategies, although it is harder to tune its temperature parameter. The worst performing exploration strategy is epsilon-greedy.
Originele taal-2English
Titel2016 IEEE Symposium Series on Computational Intelligence, SSCI 2016
UitgeverijInstitute of Electrical and Electronics Engineers Inc.
Aantal pagina's8
ISBN van elektronische versie978-1-5090-4240-1
ISBN van geprinte versie978-1-5090-4241-8
DOI's
StatusPublished - 9-feb-2017
Evenement2016 IEEE Symposium Series on Computational Intelligence, SSCI 2016 - Athens, Greece
Duur: 6-dec-20169-dec-2016

Publicatie series

Naam2016 IEEE Symposium Series on Computational Intelligence, SSCI 2016

Conference

Conference2016 IEEE Symposium Series on Computational Intelligence, SSCI 2016
LandGreece
StadAthens
Periode06/12/201609/12/2016

Citeer dit