Skip to main content

Research Repository

Advanced Search

Investigating benchmark correlations when comparing algorithms with parameter tuning: detailed experiments and results.

Christie, Lee A.; Brownlee, Alexander E.I.; Woodward, John R.


Alexander E.I. Brownlee

John R. Woodward


Benchmarks are important to demonstrate the utility of optimisation algorithms, but there is controversy about the practice of benchmarking; we could select instances that present our algorithm favourably, and dismiss those on which our algorithm underperforms. Several papers highlight the pitfalls concerned with benchmarking, some of which concern the context of the automated design of algorithms, where we use a set of problem instances (benchmarks) to train our algorithm. As with machine learning, if the training set does not reflect the test set, the algorithm will not generalize. This raises some open questions concerning the use of test instances to automatically design algorithms. We use differential evolution and sweep the parameter settings to investigate the practice of benchmarking using the BBOB benchmarks. We make three key findings. Firstly, several benchmark functions are highly correlated. This may lead to the false conclusion that an algorithm performs well in general, when it performs poorly on a few key instances, possibly introducing unwanted bias to a resulting automatically designed algorithm. Secondly, the number of evaluations can have a large effect on the conclusion. Finally, a systematic sweep of the parameters shows how performance varies with time across the space of algorithm configurations. The datasets, including all computed features, the evolved policies and their performances, and the visualisations for all feature sets are available from the University of Stirling Data Repository (


CHRISTIE, L.A., BROWNLEE, A.E.I. and WOODWARD, J.R. 2018. Investigating benchmark correlations when comparing algorithms with parameter tuning: detailed experiments and results. Stirling: University of Stirling [online]. Available from:

Report Type Technical Report
Online Publication Date Apr 30, 2018
Publication Date Apr 30, 2018
Deposit Date Mar 9, 2020
Publicly Available Date Mar 25, 2020
Publisher University of Stirling
Keywords Benchmarking; Benchmarks; Black-Box Optimization Benchmarking (BBOB); Differential evolution; Continuous optimisation; Algorithm automation
Public URL
Publisher URL
Related Public URLs


Related Outputs

You might also like

Downloadable Citations