Evolutionary Reinforcement Learning
Description
Many significant and headline breakthroughs in AI over the past decade have been powered in part through Reinforcement Learning (RL). This includes beating human-level performance in video games of various complexity from Atari (Mnih et al. 2013; Nature) to StarCraft (Vinyals et al. 2019; Nature) and strategy games like Chess and Go (Silver et al. 2018; Science). It also demonstrates state-of-the-art performance in real-world applications such as control for quadruped robots (Lee et al., 2020; Nature), high-speed drones (Kaufman et al. 2023; Nature) and plasma reactors (Degrave et al. 2022; Nature). Even modern instruction following language models and chatbots like ChatGPT (Christiano et al. 2017) and more recent reasoning-centric models like o1 are possible in part due to RL, which has re-ignited interest in the field.
Interestingly, elements of Evolutionary Computation (EC) such as population-based training (Jaderberg et al. 2017) and Quality-Diversity algorithms (Pugh et al. 2016) represent key elements in some of these breakthroughs. The combination of EC and RL methods is not new and has gained more popularity and interest recently as researchers discover the limitations of the individual approaches, opening up many exciting new research avenues and opportunities.
This tutorial will give a modern overview of the various synergies and questions addressed when combining EC and RL methods, relying on examples from the literature. Past achievements and major contributions, as well as specific challenges and future opportunities at the intersection of EC and RL, will be presented. The tutorial will in particular focus on:
- What is RL?
- Deep learning for RL
- Neuroevolution for RL
- Meta-learning RL with Evolution
- Quality-Diversity for RL
- Curriculum Learning and Environment Generation using EC and RL
- Other modern combinations of EC and RL
- Open questions and future challenges
The tutorial will effectively complement the Complex Systems, Neuroevolution, Evolutionary Machine Learning, Genetic Algorithms and Real-World Applications tracks, each of which respectively contains several papers about RL. For instance, 14/178 of the papers in the proceedings of GECCO in 2024, 13/180 papers in GECCO 2023, and 9/158 papers in GECCO 2022 contained RL elements in their title, keywords or abstract.
Organizers
Antoine Cully is Lecturer (Assistant Professor) at Imperial College London (United Kingdom). His research is at the intersection between artificial intelligence and robotics. He applies machine learning approaches, like evolutionary algorithms, on robots to increase their versatility and their adaptation capabilities. In particular, he has recently developed Quality-Diversity optimization algorithms to enable robots to autonomously learn large behavioural repertoires. For instance, this approach enabled legged robots to autonomously learn how to walk in every direction or to adapt to damage situations. Antoine Cully received the M.Sc. and the Ph.D. degrees in robotics and artificial intelligence from the Sorbonne Université in Paris, France, in 2012 and 2015, respectively, and the engineer degree from the School of Engineering Polytech’Sorbonne, in 2012. His Ph.D. dissertation has received three Best-Thesis awards. He has published several journal papers in prestigious journals including Nature, IEEE Transaction in Evolutionary Computation, and the International Journal of Robotics Research. His work was featured on the cover of Nature (Cully et al., 2015), received the "Outstanding Paper of 2015" award from the Society for Artificial Life (2016), the French "La Recherche" award (2016), and two Best-Paper awards from GECCO (2021, 2022).
Bryan is a Senior AI Research Scientist at Autodesk. He focuses on open-ended learning systems which can continuously generate a diversity of interesting problems and corresponding novel solutions, with the potential to lead to increasingly intelligent, creative and general-purpose AI systems. To enable such open-ended systems, he works on increasing the efficiency and scalability of Quality-Diversity algorithms, which encourages novelty and diversity to enable more creative search processes. Bryan’s work is at the intersection of reinforcement learning, robotics and evolutionary computation and he has co-authored papers in venues such as ICLR, ICRA, GECCO, ALIFE and TMLR. Bryan obtained his PhD from Imperial College London in 2024. Before that, he completed his MEng year abroad at MIT and has an undergraduate degree in Mechanical Engineering from Imperial College London.
Manon Flageat is a final-year PhD candidate at Imperial College London (United Kingdom). Her research focuses on Quality-Diversity algorithms, in particular applied to uncertain environments, as well as Deep Reinforcement Learning and synergies between these two types of learning algorithms. She is also a Teaching Scholar in the Department of Computing, Imperial College London and has been Course organizer of the Reinforcement Learning lecture for three years in a row. She has co-authored papers in journals such as IEEE Transaction on Evolutionary Computation and ACM Transactions on Evolutionary Learning and Optimization, and in venues such as GECCO and ALife. She received the best paper award from GECCO 2023 for work on mixing Quality-Diversity with Reinforcement Learning. She was also keynote speaker in the EvoRL workshop in GECCO 2023.
Paul is a Research Associate at Imperial College London (United Kingdom). He obtained his PhD from ISAE-SUPAERO (Toulouse, France) in 2024, during which he researched methods to leverage structures of evolutionary reinforcement learning to improve learning mechanisms. His work focused on Neural Network representation and on Evolution Strategies for policy search, including their combination with Deep Reinforcement Learning through actor injection, and with Quality-Diversity algorithms. He has co-authored papers in venues such as GECCO and IEEE CEC. He received the best paper awkward from GECCO 2024 for work on Quality with Just Enough Diversity, combining the strengths of Evolution Strategies and Quality-Diversity to learn more effective stepping stones.