Deep imitation learning with memory for robocup soccer simulation.

Hussein, Ahmed; Elyan, Eyad; Jayne, Chrisina

Abstract

Imitation learning is a field that is rapidly gaining attention due to its relevance to many autonomous agent applications. Providing demonstrations of effective behaviour to teach the agent is useful in real world challenges such as sparse rewards and dynamic environments. However, most imitation learning approaches don't retain a memory of previous actions and treat the demonstrations as independent and identically distributed samples. This neglects the temporal dependency between low-level actions that are performed in sequence to achieve the desired behaviour. This paper proposes an imitation learning method to learn sequences of actions by utilizing memory in deep neural networks. Long short-term memory networks are utilized to capture the temporal dependencies in a teacher's demonstrations. This way, past states and actions provide context for performing following actions. The network is trained using raw low-level features and directly maps the input to low-level parametrized actions in real-time. This minimizes the need for task specific knowledge to be manually employed in the learning process compared to related approaches. The proposed methods are evaluated on a benchmark soccer simulator and compared to supervised learning and data-aggregation approaches. The results show that utilizing memory while learning significantly improves the performance and generalization of the agent and can provide a stationary policy than can produce robust predictions at any point in the sequence.

Citation

HUSSEIN, A., ELYAN, E. and JAYNE, C. 2018. Deep imitation learning with memory for robocup soccer simulation. In Pimenidis, E. and Jayne, C. (eds.) Proceedings of the 19th International conference on engineering applications of neural networks (EANN 2018), 3-5 September 2018, Bristol, UK. Communications in computer and information science, 893. Cham: Springer [online], pages 31-43. Available from: https://doi.org/10.1007/978-3-319-98204-5_3

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 19th International conference on engineering applications of neural networks (EANN 2018)
• Start Date: Sep 3, 2018
• End Date: Sep 5, 2018
• Acceptance Date: May 31, 2018
• Online Publication Date: Jul 27, 2018
• Publication Date: Jul 27, 2018
• Deposit Date: Oct 16, 2018
• Publicly Available Date: Jul 28, 2019
• Print ISSN: 1865-0929
• Electronic ISSN: 1865-0937
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Pages: 31-43
• Series Title: Communications in computer and information science
• Series Number: 893
• Series ISSN: 1865-0937
• ISBN: 9783319982038
• DOI: https://doi.org/10.1007/978-3-319-98204-5_3
• Keywords: Imitation learning; Sequences; Deep neural networks
• Public URL: http://hdl.handle.net/10059/3174
• Contract Date: Oct 16, 2018