BENCH@GECCO25 - Good Benchmarking Practices for Evolutionary Computation
Webpage: https://sites.google.com/view/benchmarking-network/home
Description
Benchmarking plays a vital role in understanding the performance and search behaviour of sampling-based optimization techniques such as evolutionary algorithms. This workshop will continue our workshop series on good benchmarking practices at different conferences in the context of EC that we started in 2020. The core theme is on benchmarking evolutionary computation methods and related sampling-based optimization heuristics, but each year, the focus is changed.
For GECCO 2025, our focus will be on **“Benchmarking for humans and machines - Differences and Similarities”**.
Many currently popular benchmarks are designed to be interpretable by humans with specific questions in mind. For example, if the algorithm can exploit separability or how it handles disconnected Pareto fronts. As a result, they do not attempt to cover the full space of interesting problems. However, when used in the context of automated algorithm selection, algorithm configuration, and similar machine learning tasks, data requirements may change, as the ability for manual interpretation is no longer a restriction.
At the same time, benchmarking results in publications are often presented as aggregates without heeding the original intent of the benchmark designer. So even without the involvement of machines, the benchmarking data is typically suboptimally presented and interpreted.
In this workshop, we will be addressing the following questions:
- What are key similarities and differences between benchmarks designed for human vs. machine interpretation?
- Are there inherent differences between human vs. machine interpretable benchmarking pipelines that require the experimental setup, apart from the size of the generated data sets, to be different?
- How can we best support the analysis of benchmarking data, for manual interpretation and machine-based learning.
Organizers
Vanessa Volz is currently a tenure track researcher in the Evolutionary Intelligence (EI) group at Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands. She received her PhD in 2019 from TU Dortmund University, Germany, for her work on surrogate-assisted evolutionary algorithms applied to game optimisation. Her current research focus is on transfer learning in the context of evolutionary computation, especially in the context of recurring or otherwise dynamic problems.
Carola Doerr, formerly Winzen, is a CNRS research director at Sorbonne Université in Paris, France. Carola's main research activities are in the analysis of black-box optimization algorithms, both by mathematical and by empirical means. Carola is associate editor of IEEE Transactions on Evolutionary Computation, ACM Transactions on Evolutionary Learning and Optimization (TELO), and the Evolutionary Computation journal. She is/was program chair for the BBSR track at GECCO 2025 and 2024, the GECH track at GECCO 2023, for PPSN 2020, FOGA 2019, and for the theory tracks of GECCO 2015 and 2017. She has organized Dagstuhl seminars and Lorentz Center workshops. Together with Pascal Kerschke, Carola leads the 'Algorithm selection and configuration' working group of COST action CA22137. Carola's works have been distinguished by several awards, among them the CNRS bronze medal, the Otto Hahn Medal of the Max Planck Society, and best paper awards at GECCO, CEC, and EvoApplications.
Boris Naujoks is a professor for Applied Mathematics at TH Köln - Cologne University of Applied Sciences (CUAS). He joint CUAs directly after he received his PhD from Dortmund Technical University in 2011. During his time in Dortmund, Boris worked as a research assistant in different projects and gained industrial experience working for different SMEs. Meanwhile, he enjoys the combination of teaching mathematics as well as computer science and exploring EC and CI techniques at the Campus Gummersbach of CUAS. He focuses on multiobjective (evolutionary) optimization, in particular hypervolume based algorithms, and the (industrial) applicability of the explored methods.
Mike Preuss is assistant professor at LIACS, the Computer Science department of Leiden University. He works in AI, namely game AI, natural computing, and social media computing. Mike received his PhD in 2013 from the Chair of Algorithm Engineering at TU Dortmund, Germany, and was with ERCIS at the WWU Muenster, Germany, from 2013 to 2018. His research interests focus on the field of evolutionary algorithms for real-valued problems, namely on multi-modal and multi-objective optimization, and on computational intelligence and machine learning methods for computer games. Recently, he is also involved in Social Media Computing, and he is publications chair of the upcoming multi-disciplinary MISDOOM conference 2019. He is associate editor of the IEEE ToG journal and has been member of the organizational team of several conferences in the last years, in various functions, as general co-chair, proceedings chair, competition chair, workshops chair.
Olaf Mersmann is a Professor for Data Science at TH Köln - University of Applied Sciences. He received his BSc, MSc and PhD in Statistics from TU Dortmund. His research interests include using statistical and machine learning methods on large benchmark databases to gain insight into the structure of the algorithm choice problem.