Benchmarking heuristic algorithms is vital for understanding under which conditions and on what kind of problems certain algorithms perform well. Most benchmarks are performance-based, to test algorithm performance under a wide set of conditions. There are also resource- and behaviour-based benchmarks to test resource consumption and behaviour of algorithms. In this Tutorial, we focus on behaviour benchmarking of algorithms and more specifically we focus on Structural Bias (SB).
SB is a form of bias inherent to the iterative heuristic optimiser in the search space that also affects the performance of the optimisation algorithm. Detecting whether, when and what type of SB occurs in a heuristic optimisation algorithm can provide guidance on what needs to be improved in these algorithms, besides helping to identify conditions under which such bias would not occur.
In the tutorial, we start by defining the problem of detecting and identifying different types of structural bias, including many visual examples. We then introduce state-of-the-art methods for bias detection. We follow up with SB results for several well-known and popular optimisation heuristics, give insights and show best practices to avoid SB in algorithm development. We conclude with a live demo of the Python-based BIAS toolkit which analyses a few well-known optimisation heuristics. Participants will be provided with links to live tools, necessary code and data.

Aims and learning objectives of this tutorial:
- convey the importance of benchmarking heuristic algorithms to comprehend their performance across different problem scenarios;
- concentrate on behaviour benchmarking, specifically delving into the concept of Structural Bias in iterative heuristic optimisers;
- enable participants to detect, analyse, and understand the occurrence and impact of Structural Bias in heuristic optimisation algorithms;
- provide insights into how detecting SB can lead to improved algorithm development and refinement;
- showcase the functionality and usage of developed toolkits as practical tools for detecting and addressing Structural Bias;
- share findings from analysing structural bias across well-known optimisation heuristics, offering insights and patterns.

Prerequisite Knowledge of Audience:
- familiarity with standard heuristic optimisation algorithms and benchmarking practices accepted in the community
- ability to follow programming examples in Python

Main references:
1 B. van Stein, D. Vermetten, F. Caraffini, A.V. Kononova, “Deep BIAS: detecting structural bias using explainable AI”, GECCO '23 Companion: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 455–458, Lisbon, Portugal, 15-19 July 2023, doi: 10.1145/3583133.3590551, preprint: arXiv:abs/2304.01869
2 D. Vermetten, B. van Stein, F. Caraffini, L.L. Minku, A.V. Kononova, “BIAS: A Toolbox for Benchmarking Structural Bias in the Continuous Domain", IEEE Transactions on Evolutionary Computation, vol. 26 (6), pp. 1380–1393, 2022, doi: 10.1109/TEVC.2022.3189848, preprint: 10.36227/techrxiv.16594880.v1
3 A.V. Kononova, D.W. Corne, P. De Wilde, V. Shneer, F. Caraffini, “Structural bias in population-based algorithms”, Information Sciences, Elsevier, vol. 298, pp. 468–490, Elsevier, 2015, doi: 10.1016/j.ins.2014.11.035, arXiv:abs/1408.5350