Research Repository

See what's under the surface

Deep reward shaping from demonstrations.

Hussein, Ahmed; Elyan, Eyad; Gaber, Mohamed Medhat; Jayne, Chrisina

Authors

Ahmed Hussein

Eyad Elyan

Mohamed Medhat Gaber

Chrisina Jayne

Abstract

Deep reinforcement learning is rapidly gaining attention due to recent successes in a variety of problems. The combination of deep learning and reinforcement learning allows for a generic learning process that does not consider specific knowledge of the task. However, learning from scratch becomes more difficult when tasks involve long trajectories with delayed rewards. The chances of finding the rewards using trial and error become much smaller compared to tasks where the agent continuously interacts with the environment. This is the case in many real life applications which poses a limitation to current methods. In this paper we propose a novel method for combining learning from demonstrations and experience to expedite and improve deep reinforcement learning. Demonstrations from a teacher are used to shape a potential reward function by training a deep supervised convolutional neural network. The shaped function is added to the reward function used in deep-Q-learning (DQN) to perform off-policy training through trial and error. The proposed method is demonstrated on navigation tasks that are learned from raw pixels without utilizing any knowledge of the problem. Navigation tasks represent a typical AI problem that is relevant to many real applications and where only delayed rewards (usually terminal) are available to the agent. The results show that using the proposed shaped rewards significantly improves the performance of the agent over standard DQN. This improvement is more pronounced the sparser the rewards are.

Start Date May 14, 2017
Publication Date Jul 3, 2017
Print ISSN 2161-4393
Electronic ISSN 2161-4407
Publisher Institute of Electrical and Electronics Engineers
Article Number 7965896
Pages 510-517
Series ISSN 2161-4407
Institution Citation HUSSEIN, A., ELYAN, E., GABER, M.M. and JAYNE, C. 2017. Deep reward shaping from demonstrations. In Proceedings of the 2017 International joint conference on neural networks (IJCNN 2017), 14-19 May 2017, Anchorage, USA. Piscataway: IEEE [online], article number 7965896, pages 510-517. Available from: https://doi.org/10.1109/IJCNN.2017.7965896
DOI https://doi.org/10.1109/IJCNN.2017.7965896
Keywords Deep reinforcement; Reinforcement learning; Generic learning process; DeepQlearning (DQN)

Files




Downloadable Citations