Benchmarking single- and multi-objective optimization algorithms: how to make your experimental data more valuable
Description
Comparing and evaluating optimization algorithms by empirical means is an important – and probably the most commonly applied – approach to gaining insight into evolutionary computation methods. However, while our community tends to agree that generating and analyzing sound benchmarking data is far from trivial, we treat the process in a rather wasteful manner, giving little importance to a standardization of data records, data sharing, and similar.
With this tutorial, we will share our experience on how to boost the efficacy of our benchmarking efforts at almost no cost using the IOHprofiler software framework and its recent extensions to anytime performance measures and multi-objective optimization. A strong focus will be put on demonstrating the ease by which IOHprofiler modules can be combined with other benchmarking and optimization toolboxes such as COCO, Nevergrad, and Pymoo.
We will also discuss how benchmarking data can be more easily shared within the community and the benefits that this brings, in terms of core research contributions, but also towards more sustainable research practices in evolutionary computation.
Organizers
Carola Doerr, formerly Winzen, is a CNRS research director at Sorbonne Université in Paris, France. Carola's main research activities are in the analysis of black-box optimization algorithms, both by mathematical and by empirical means. Carola is associate editor of IEEE Transactions on Evolutionary Computation, ACM Transactions on Evolutionary Learning and Optimization (TELO), and the Evolutionary Computation journal. She is/was program chair for the BBSR track at GECCO 2025 and 2024, the GECH track at GECCO 2023, for PPSN 2020, FOGA 2019, and for the theory tracks of GECCO 2015 and 2017. She has organized Dagstuhl seminars and Lorentz Center workshops. Together with Pascal Kerschke, Carola leads the 'Algorithm selection and configuration' working group of COST action CA22137. Carola's works have been distinguished by several awards, among them the CNRS bronze medal, the Otto Hahn Medal of the Max Planck Society, and best paper awards at GECCO, CEC, and EvoApplications.
Diederick Vermetten is a PhD student at LIACS. He is part of the core development team of IOHprofiler, with a focus on the IOHanalyzer. His research interests include benchmarking of optimization heuristics, dynamic algorithm selection and configuration as well as hyperparameter optimization.
Jacob de Nobel is a PhD student at LIACS, and is currently one of the core developers for the IOHexperimenter. His research concerns the real world application of optimization algorithms for finding better speech encoding strategies for cochlear implants, which are neuroprosthesis for people with profound hearing loss.
Thomas Bäck is Full Professor of Computer Science at the Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands, where he is head of the Natural Computing group since 2002. He received his PhD (adviser: Hans-Paul Schwefel) in computer science from Dortmund University, Germany, in 1994, and then worked for the Informatik Centrum Dortmund (ICD) as department leader of the Center for Applied Systems Analysis. From 2000 - 2009, Thomas was Managing Director of NuTech Solutions GmbH and CTO of NuTech Solutions, Inc. He gained ample experience in solving real-life problems in optimization and data mining through working with global enterprises such as BMW, Beiersdorf, Daimler, Ford, Honda, and many others. Thomas Bäck has more than 350 publications on natural computing, as well as two books on evolutionary algorithms: Evolutionary Algorithms in Theory and Practice (1996), Contemporary Evolution Strategies (2013). He is co-editor of the Handbook of Evolutionary Computation, and most recently, the Handbook of Natural Computing. He is also editorial board member and associate editor of a number of journals on evolutionary and natural computing. Thomas received the best dissertation award from the German Society of Computer Science (Gesellschaft für Informatik, GI) in 1995 and the IEEE Computational Intelligence Society Evolutionary Computation Pioneer Award in 2015.