COCO: The Bi-objective Black-Box Optimization Benchmarking (`bbob-biobj`) Test Suite¶

See also: ArXiv e-prints, arXiv:1604.00359, 2016.

Several test function suites for numerical benchmarking of multiobjective optimization algorithms have been proposed in recent years. While having desirable properties like well-understood Pareto sets and Pareto fronts with shapes of various kinds, most of the currently used functions posess properties which are arguably under-represented in real-world problems. Those properties mainly stem from the easier construction of such problems—overrepresenting properties such as no dependencies between variables, Pareto sets exactly located at the bound constraints, or the differentiation between position and distance variables. Here, we propose an alternative way and define the bbob-biobj test suite with 55 bi-objective functions and its extended bbob-biobj-ext version with 92 bi-objective functions in continuous domain which are both derived from combining functions of the well-known single-objective noiseless bbob test suite. Besides giving the actual function definitions and presenting their (known) properties, this documentation also aims at giving the rationale behind our approach in terms of function groups, instances, and potential objective space normalization.

Example plots of the Pareto front approximation, found by NSGA-II on selected bbob-biobj functions. In blue the non-dominated points at the end of different independent runs, in red the points that are non-dominated over all runs.

Preliminaries, Definitions, and Scope¶

In the following, we consider bi-objective, unconstrained minimization problems of the form

$\min_{x \in \mathbb{R}^n} f(x)=(f_\alpha(x),f_\beta(x)),$

where $n$ is the number of variables of the problem (also called the problem dimension), $f_\alpha: \mathbb{R}^n \rightarrow \mathbb{R}$ and $f_\beta: \mathbb{R}^n \rightarrow \mathbb{R}$ are the two objective functions, and the $\min$ operator is related to the standard dominance relation. A solution $x\in\mathbb{R}^n$ is thereby said to dominate another solution $y\in\mathbb{R}^n$ if $f_\alpha(x) \leq f_\alpha(y)$ and $f_\beta(x) \leq f_\beta(y)$ hold and at least one of the inequalities is strict.

Solutions which are not dominated by any other solution in the search space are called Pareto-optimal or efficient solutions. All Pareto-optimal solutions constitute the Pareto set of which an approximation is sought. The Pareto set’s image in the objective space $f(\mathbb{R}^n)$ is called Pareto front.

The objective of the minimization problem is to find, with as few evaluations of $f$ as possible, a set of non-dominated solutions which is (i) as large as possible and (ii) has $f$ -values as close to the Pareto front as possible. [1]

[1]	Distance in $f$ -space is defined here such that nadir and ideal point have in each coordinate distance one. Neither of these points is however freely accessible to the optimization algorithm.

In the following, we remind useful definitions.

function instance, problem

Each function $f^\theta: \mathbb{R}^n \to \mathbb{R}^m$ within COCO is parametrized with parameter values $\theta \in \Theta$ . A parameter value determines a so-called function instance. For example, $\theta$ encodes the location of the optimum of single-objective functions, which means that different instances have shifted optima. In the bbob-biobj test suite, $m=2$ and the function instances are determined by the instances of the underlying single-objective functions.

A problem is a function instance of a specific dimension $n$ .

ideal point

The ideal point is defined as the vector in objective space that contains the optimal $f$ -value for each objective independently. More precisely let $f_\alpha^{\rm opt}:= \inf_{x\in \mathbb{R}^n} f_\alpha(x)$ and $f_\beta^{\rm opt}:= \inf_{x\in \mathbb{R}^n} f_\beta(x)$ , the ideal point is given by

$\begin{equation*} z_{\rm ideal} = (f_\alpha^{\rm opt},f_\beta^{\rm opt}). \end{equation*}$

nadir point

The nadir point (in objective space) consists in each objective of the worst value obtained by a Pareto-optimal solution. More precisely, let $\mathcal{PO}$ be the set of Pareto optimal points. Then the nadir point satisfies

$\begin{equation*} z_{\rm nadir} = \left( \sup_{x \in \mathcal{PO}} f_\alpha(x), \sup_{x \in \mathcal{PO}} f_\beta(x) \right). \end{equation*}$

In the case of two objectives with a unique global minimum each (that is, a single point in the search space maps to the global minimum)

$\begin{equation*} z_{\rm nadir} = \left( f_\alpha(x_{\rm opt,\beta}), f_\beta(x_{\rm opt,\alpha}) \right), \end{equation*}$

where $x_{\rm opt,\alpha}= \arg \min f_\alpha(x)$ and $x_{\rm opt,\beta}= \arg \min f_\beta(x)$ .

Overview of the Proposed `bbob-biobj` Test Suite¶

The bbob-biobj test suite provides 55 bi-objective functions in six dimensions (2, 3, 5, 10, 20, and 40) with a large number of possible instances. The 55 functions are derived from combining a subset of the 24 well-known single-objective functions of the bbob test suite [HAN2009] which has been used since 2009 in the BBOB workshop series. While concrete details on each of the 55 bbob-biobj functions are given in Section The bbob-biobj Test Functions and Their Properties, we will detail here the main rationale behind them together with their common properties.

The Single-objective `bbob` Functions¶

The bbob-biobj test suite is designed to be able to assess performance of algorithms with respect to well-identified difficulties in optimization typically occurring in real-world problems. A multi-objective problem being a combination of single-objective problems, one can obtain multi-objective problems with representative difficulties by simply combining single objective functions with representative difficulties observed in real-world problems. For this purpose we naturally use the single-objective bbob suite [HAN2009].

Combining all 24 bbob functions in pairs thereby results in $24^2=576$ bi-objective functions overall. We however assume that multi-objective optimization algorithms are not sensitive to permutations of the objective functions such that combining the 24 bbob functions and taking out the function $(g_2,g_1)$ if the function $(g_1,g_2)$ is present results in $24 + {24 \choose 2} = 24 + (24\times 23)/2 = (24\times 25)/2 = 300$ functions.

Some first tests, e.g. in [BRO2015], showed that having 300 functions is impracticable in terms of the overall running time of the benchmarking experiment. We then decided to exploit the organization of the bbob functions into classes to choose a subset of functions. More precisely, the 24 original bbob functions are grouped into five function classes where each class gathers functions with similar properties, namely

separable functions

functions with low or moderate conditioning

functions with high conditioning and unimodal

multi-modal functions with adequate global structure,

multi-modal functions with weak global structure.

To create the bbob-biobj suite, we choose two functions within each class. This way we do not introduce any bias towards a specific class. In addition within each class, the functions are chosen to be the most representative without repeating similar functions. For example, only one Ellipsoid, one Rastrigin, and one Gallagher function are included in the bbob-biobj suite although they appear in separate versions in the bbob suite. Finally our choice of 10 bbob functions for creating the bbob-biobj test suite is the following:

Separable functions
- Sphere (function 1 in bbob suite)
- Ellipsoid separable (function 2 in bbob suite)
Functions with low or moderate conditioning
- Attractive sector (function 6 in bbob suite)
- Rosenbrock original (function 8 in bbob suite)
Functions with high conditioning and unimodal
- Sharp ridge (function 13 in bbob suite)
- Sum of different powers (function 14 in bbob suite)
Multi-modal functions with adequate global structure
- Rastrigin (function 15 in bbob suite)
- Schaffer F7, condition 10 (function 17 in bbob suite)
Multi-modal functions with weak global structure
- Schwefel x*sin(x) (function 20 in bbob suite)
- Gallagher 101 peaks (function 21 in bbob suite)

Using the above described pairwise combinations, this results in having $10+{10 \choose 2} = 55$ bi-objective functions in the final bbob-biobj suite. These functions are denoted $f_1$ to $f_{55}$ in the sequel.

Function Groups¶

From combining the original bbob function classes, we obtain 15 function classes to structure the 55 bi-objective functions of the bbob-biobj test suite. Each function class contains three or four functions. We are listing below the function classes and in parenthesis the functions that belong to the respective class:

separable - separable (functions $f_1$ , $f_2$ , $f_{11}$ )

separable - moderate ( $f_3$ , $f_4$ , $f_{12}$ , $f_{13}$ )

separable - ill-conditioned ( $f_5$ , $f_6$ , $f_{14}$ , $f_{15}$ )

separable - multi-modal ( $f_7$ , $f_8$ , $f_{16}$ , $f_{17}$ )

separable - weakly-structured ( $f_9$ , $f_{10}$ , $f_{18}$ , $f_{19}$ )

moderate - moderate ( $f_{20}$ , $f_{21}$ , $f_{28}$ )

moderate - ill-conditioned ( $f_{22}$ , $f_{23}$ , $f_{29}$ , $f_{30}$ )

moderate - multi-modal ( $f_{24}$ , $f_{25}$ , $f_{31}$ , $f_{32}$ )

moderate - weakly-structured ( $f_{26}$ , $f_{27}$ , $f_{33}$ , $f_{34}$ )

ill-conditioned - ill-conditioned ( $f_{35}$ , $f_{36}$ , $f_{41}$ )

ill-conditioned - multi-modal ( $f_{37}$ , $f_{38}$ , $f_{42}$ , $f_{43}$ )

ill-conditioned - weakly-structured ( $f_{39}$ , $f_{40}$ , $f_{44}$ , $f_{45}$ )

multi-modal - multi-modal ( $f_{46}$ , $f_{47}$ , $f_{50}$ )

multi-modal - weakly structured ( $f_{48}$ , $f_{49}$ , $f_{51}$ , $f_{52}$ )

weakly structured - weakly structured ( $f_{53}$ , $f_{54}$ , $f_{55}$ )

More details about the single functions can be found in Section The bbob-biobj Test Functions and Their Properties. We however first describe their common properties in the coming sections.

Normalization of Objectives¶

None of the 55 bbob-biobj functions is explicitly normalized and the optimization algorithms therefore have to cope with objective values in different ranges. Typically, different orders of magnitude between the objective values can be observed.

However, to facilitate comparison of algorithm performance over different functions, we normalize the objectives based on the ideal and nadir points before calculating the hypervolume indicator [BRO2016biperf]. Both points can be computed, because the global optimum is known and is unique for the 10 bbob base functions. In the black-box optimization benchmarking setup, however, the values of the ideal and nadir points are not accessible to the optimization algorithm [HAN2016ex].

Instances¶

Our test functions are parametrized and instances are instantiations of the underlying parameters (see [HAN2016co]). The instances for the bi-objective functions are obtained using instances of each single objective function composing the bi-objective one. In addition, we assert two conditions:

1. The Euclidean distance between the two single-objective optima (also called the extreme optimal points) in the search space is at least $10^{-4}$ .

2. The Euclidean distance between the ideal and the nadir point in the non-normalized objective space is at least $10^{-1}$ .

We associate to an instance, an instance-id which is an integer. The relation between the instance-id, $K^{f}_{\rm id}$ , of a bi-objective function $f = (f_\alpha, f_\beta)$ and the instance-ids, $K_{\rm id}^{f_\alpha}$ and $K_{\rm id}^{f_\beta}$ , of its underlying single-objective functions $f_\alpha$ and $f_\beta$ is the following:

$K_{\rm id}^{f_\alpha} = 2 K^{f}_{\rm id} + 1$ and

$K_{\rm id}^{f_\beta} = K_{\rm id}^{f_\alpha} + 1$

If we find that above conditions are not satisfied for all dimensions and functions in the bbob-biobj suite, we increase the instance-id of the second objective successively until both properties are fulfilled. For example, the bbob-biobj instance-id 8 corresponds to the instance-id 17 for the first objective and instance-id 18 for the second objective while for the bbob-biobj instance-id 9, the first instance-id is 19 but for the second objective, instance-id 21 is chosen instead of instance-id 20.

Exceptions to the above rule are, for historical reasons, the bbob-biobj instance-ids 1 and 2 in order to match the instance-ids 1 to 5 with the ones proposed in [BRO2015]. The bbob-biobj instance-id 1 contains the single-objective instance-ids 2 and 4 and the bbob-biobj instance-id 2 contains the two instance-ids 3 and 5.

For each bi-objective function and given dimension, the bbob-biobj suite contains 10 instances. [2]

[2] In principle, as for the instance generation for the bbob suite, the number of possible instances for the bbob-biobj suite is unlimited [HAN2016co]. However, running some tests with too few instances will render the potential statistics and their interpretation problematic while even the tiniest observed difference can be made statistically significant with a high enough number of instances. A good compromise to avoid either pitfall seems to lie between, say, 9 and 19 instances.

The `bbob-biobj` Test Functions and Their Properties¶

In the following, we detail all 55 bbob-biobj functions and their properties.

The following table gives an overview and quick access to the functions, inner cell IDs refer to the bbob-biobj functions, outer column and row annotations refer to the single-objective bbob functions.

	$f_1$	$f_2$	$f_6$	$f_8$	$f_{13}$	$f_{14}$	$f_{15}$	$f_{17}$	$f_{20}$	$f_{21}$
$f_1$	f1	f2	f3	f4	f5	f6	f7	f8	f9	f10
$f_2$		f11	f12	f13	f14	f15	f16	f17	f18	f19
$f_6$			f20	f21	f22	f23	f24	f25	f26	f27
$f_8$				f28	f29	f30	f31	f32	f33	f34
$f_{13}$					f35	f36	f37	f38	f39	f40
$f_{14}$						f41	f42	f43	f44	f45
$f_{15}$							f46	f47	f48	f49
$f_{17}$								f50	f51	f52
$f_{20}$									f53	f54
$f_{21}$										f55

Some Function Properties¶

In the description of the 55 bbob-biobj functions below, several general properties of objective functions will be mentioned that are defined here in short. It depends on these properties whether the optimization problem is easy or hard to solve.

A separable function does not show any dependencies between the variables and can therefore be solved by applying $n$ consecutive one-dimensional optimizations along the coordinate axes while keeping the other variables fixed. Consequently, non-separable problems must be considered. They are much more difficult to solve. The typical well-established technique to generate non-separable functions from separable ones is the application of a rotation matrix $\mathbf R$ to $x$ , that is $x \in \mathbb{R}^n \mapsto g(\mathbf R x)$ , where $g$ is a separable function.

A unimodal function has only one local minimum which is at the same time also its global one. A multimodal function has at least two local minima which is highly common in practical optimization problems.

Ill-conditioning is another typical challenge in real-parameter optimization and, besides multimodality, probably the most common one. In a general case, we can consider a function as ill-conditioned if for solution points from the same level-set “the minimal displacement [...] that produces a given function value improvement differs by orders of magnitude” [HAN2011]. Conditioning can be rigorously formalized in the case of convex quadratic functions, $f(x) = \frac{1}{2} x^THx$ where $H$ is a symmetric positive definite matrix, as the condition number of the Hessian matrix $H$ . Since contour lines associated to a convex quadratic function are ellipsoids, the condition number corresponds to the square root of the ratio between the largest axis of the ellipsoid and the shortest axis.

The proposed bbob-biobj testbed contains ill-conditioned functions with a typical conditioning of $10^6$ . We believe this is a realistic requirement, while we have seen practical problems with conditioning as large as $10^{10}$ .

Domain Bounds¶

All bi-objective functions provided in the bbob-biobj suite are unbounded, i.e., defined on the entire real-valued space $\mathbb{R}^n$ . The search domain of interest is defined as $[-100,100]^n$ , outside of which non-dominated solutions are quite unlikely to be found. [3] The majority of non-dominated solutions are likely to lie even within $[-5,5]^n$ .

While we believe that the domain of interest contains the Pareto set, due to the nature of the bbob-biobj function definitions, there is no guarantee that this is always the case. However, the extremal solutions and their neighborhood ball of radius one are guaranteed to lie within $[-5,5]^n$ .

[3]	The functions `coco_problem_get_smallest_value_of_interest` and `coco_problem_get_largest_value_of_interest` of the COCO platform allow the optimizer to retrieve the search domain of interest from the `coco_problem_t`, for example to generate the initial search points.

Provided Search Space and Objective Space Plots¶

In order to better understand the properties of the 55 bbob-biobj functions, we display for each of them plots of the best known Pareto front approximation in objective space in original scaling (as seen by the algorithm) and in log-scale, normalized such that the ideal point is at $[0,0]$ and the nadir point is at $[1,1]$ . We also provide plots illustrating the best known Pareto set approximation in search space (all depicted in black). For the latter, two different plots are provided: a plot showing the projection onto a coordinate-axes-parallel cut defined by two variables and a plot that projects all points onto a random cutting plane which contains both single-objective optima and that also shows the contour lines of both objective functions on this plane.

In addition to the best Pareto set/Pareto front approximations, cuts through the search space are shown along (i) random lines through each optimum (in blue), (ii) lines along each coordinate axis through each optimum (blue dotted lines), (iii) the line through both optima (in red), (iv) two fully random lines [4] (in yellow), and (v) a random line in the random projection plane going through both optima [5] (in green).

All lines are normalized (of length 10 with the support vector in the middle). Ticks along the lines in the objective space plots indicate the ends of line segments of the same length in search space. Thicker points on the lines depict solutions that are non-dominated with respect to all points on the same line. Furthermore, the search space plots highlight the projected region $[-5,5]^n$ as gray-shaded area while the gray-shaded area in the objective space plots highlight the region of interest between ideal ( $+$ ) and nadir point ( $\times$ ). Note that, to keep the plots to a manageable size, the Pareto set and Pareto front approximations are carefully downsampled such that only one solution per grid point is shown—with the precision of 2 decimals for the search space plots and 3 decimals for the objective space plots to define the grid. The number of considered and actually displayed solutions is indicated in the search space plots’ legends. All plots are provided for one instance here only and for dimension 5 for the moment.

[4]	of random direction and with a support vector, drawn uniformly at random in $[-4,4]^n$

[5]	with a random direction within the plane and a support vector, drawn uniformly at random in $[-4,4]$ in the coordinate system of the cutting plane`

The 55 `bbob-biobj` Functions¶

$f_1$ : Sphere/Sphere¶

Combination of two sphere functions ( $f_1$ in the bbob suite).

Both objectives are unimodal, highly symmetric, rotational and scale invariant. The Pareto set is known to be a straight line and the Pareto front is convex. Furthermore, the normalized hypervolume value of the entire Pareto front with respect to the nadir point as reference point can be computed analytically as the integral $1-\int_{0}^{1} (1-\sqrt{x})^2dx = -\frac{1}{2}+\frac{4}{3}=0.833333\ldots$ .

Considered as the simplest bi-objective problem in continuous domain.

Contained in the separable - separable function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 1 in dimension 5 for the first instance.

$f_2$ : Sphere/Ellipsoid separable¶

Combination of the sphere function ( $f_1$ in the bbob suite) and the separable ellipsoid function ( $f_2$ in the bbob suite).

Both objectives are unimodal and separable. While the first objective is truly convex-quadratic with a condition number of 1, the second objective is only globally quadratic with smooth local irregularities and highly ill-conditioned with a condition number of about $10^6$ .

Contained in the separable - separable function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 2 in dimension 5 for the first instance.

$f_3$ : Sphere/Attractive sector¶

Combination of the sphere function ( $f_1$ in the bbob suite) and the attractive sector function ( $f_6$ in the bbob suite).

Both objective functions are unimodal, but only the first objective is separable and truly convex quadratic. The attractive sector function is highly asymmetric, where only one hypercone (with angular base area) with a volume of roughly $(1/2)^n$ yields low function values. The optimum of it is located at the tip of this cone.

Contained in the separable - moderate function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 3 in dimension 5 for the first instance.

$f_4$ : Sphere/Rosenbrock original¶

Combination of the sphere function ( $f_1$ in the bbob suite) and the original, i.e., unrotated Rosenbrock function ( $f_8$ in the bbob suite).

The first objective is separable and truly convex, the second objective is partially separable (tri-band structure). The first objective is unimodal while the second objective has a local optimum with an attraction volume of about 25%.

Contained in the separable - moderate function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 4 in dimension 5 for the first instance.

$f_5$ : Sphere/Sharp ridge¶

Combination of the sphere function ( $f_1$ in the bbob suite) and the sharp ridge function ( $f_{13}$ in the bbob suite).

Both objective functions are unimodal. In addition to the simple, separable, and differentiable first objective, a sharp, i.e., non-differentiable ridge has to be followed for optimizing the (non-separable) second objective. The gradient towards the ridge remains constant, when the ridge is approached from a given point. Approaching the ridge is initially effective, but becomes ineffective close to the ridge when the rigde needs to be followed in direction to its optimum. The necessary change in search behavior close to the ridge is difficult to diagnose, because the gradient towards the ridge does not flatten out.

Contained in the separable - ill-conditioned function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 5 in dimension 5 for the first instance.

$f_6$ : Sphere/Sum of different powers¶

Combination of the sphere function ( $f_1$ in the bbob suite) and the sum of different powers function ( $f_{14}$ in the bbob suite).

Both objective functions are unimodal. The first objective is separable, the second non-separable. When approaching the second objective’s optimum, the difference in sensitivity between different directions in search space increases unboundedly.

Contained in the separable - ill-conditioned function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 6 in dimension 5 for the first instance.

$f_7$ : Sphere/Rastrigin¶

Combination of the sphere function ( $f_1$ in the bbob suite) and the Rastrigin function ( $f_{15}$ in the bbob suite).

In addition to the simple sphere function, the prototypical highly multimodal Rastrigin function needs to be solved which has originally a very regular and symmetric structure for the placement of the optima. Here, however, transformations are performed to alleviate the original symmetry and regularity in the second objective.

The properties of the second objective contain non-separabilty, multimodality (roughly $10^n$ local optima), a conditioning of about 10, and a large global amplitude compared to the local amplitudes.

Contained in the separable - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 7 in dimension 5 for the first instance.

$f_8$ : Sphere/Schaffer F7, condition 10¶

Combination of the sphere function ( $f_1$ in the bbob suite) and the Schaffer F7 function with condition number 10 ( $f_{17}$ in the bbob suite).

In addition to the simple sphere function, an asymmetric, non-separable, and highly multimodal function needs to be solved to approach the Pareto front/Pareto set where the frequency and amplitude of the modulation in the second objective vary. The conditioning of the second objective and thus the entire bi-objective function is low.

Contained in the separable - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 8 in dimension 5 for the first instance.

$f_9$ : Sphere/Schwefel x*sin(x)¶

Combination of the sphere function ( $f_1$ in the bbob suite) and the Schwefel function ( $f_{20}$ in the bbob suite).

While the first objective function is separable and unimodal, the second objective function is partially separable and highly multimodal—having the most prominent $2^n$ minima located comparatively close to the corners of the unpenalized search area.

Contained in the separable - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 9 in dimension 5 for the first instance.

$f_{10}$ : Sphere/Gallagher 101 peaks¶

Combination of the sphere function ( $f_1$ in the bbob suite) and the Gallagher function with 101 peaks ( $f_{21}$ in the bbob suite).

While the first objective function is separable and unimodal, the second objective function is non-separable and consists of 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the separable - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 10 in dimension 5 for the first instance.

$f_{11}$ : Ellipsoid separable/Ellipsoid separable¶

Combination of two separable ellipsoid functions ( $f_2$ in the bbob suite).

Both objectives are unimodal, separable, only globally quadratic with smooth local irregularities, and highly ill-conditioned with a condition number of about $10^6$ .

Contained in the separable - separable function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 11 in dimension 5 for the first instance.

$f_{12}$ : Ellipsoid separable/Attractive sector¶

Combination of the separable ellipsoid function ( $f_2$ in the bbob suite) and the attractive sector function ( $f_6$ in the bbob suite).

Both objective functions are unimodal but only the first one is separable. The first objective function, in addition, is globally quadratic with smooth local irregularities, and highly ill-conditioned with a condition number of about $10^6$ . The second objective function is highly asymmetric, where only one hypercone (with angular base area) with a volume of roughly $(1/2)^n$ yields low function values. The optimum of it is located at the tip of this cone.

Contained in the separable - moderate function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 12 in dimension 5 for the first instance.

$f_{13}$ : Ellipsoid separable/Rosenbrock original¶

Combination of the separable ellipsoid function ( $f_2$ in the bbob suite) and the original, i.e., unrotated Rosenbrock function ( $f_8$ in the bbob suite).

Only the first objective is separable and unimodal. The second objective is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. In addition, the first objective function shows smooth local irregularities from a globally convex quadratic function and is highly ill-conditioned with a condition number of about $10^6$ .

Contained in the separable - moderate function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 13 in dimension 5 for the first instance.

$f_{14}$ : Ellipsoid separable/Sharp ridge¶

Combination of the separable ellipsoid function ( $f_2$ in the bbob suite) and the sharp ridge function ( $f_{13}$ in the bbob suite).

Both objective functions are unimodal but only the first one is separable.

The first objective is globally quadratic but with smooth local irregularities and highly ill-conditioned with a condition number of about $10^6$ . For optimizing the second objective, a sharp, i.e., non-differentiable ridge has to be followed.

Contained in the separable - ill-conditioned function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 14 in dimension 5 for the first instance.

$f_{15}$ : Ellipsoid separable/Sum of different powers¶

Combination of the separable ellipsoid function ( $f_2$ in the bbob suite) and the sum of different powers function ( $f_{14}$ in the bbob suite).

Both objective functions are unimodal but only the first one is separable.

The first objective is globally quadratic but with smooth local irregularities and highly ill-conditioned with a condition number of about $10^6$ . When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.

Contained in the separable - ill-conditioned function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 15 in dimension 5 for the first instance.

$f_{16}$ : Ellipsoid separable/Rastrigin¶

Combination of the separable ellipsoid function ( $f_2$ in the bbob suite) and the Rastrigin function ( $f_{15}$ in the bbob suite).

The objective functions show rather opposite properties. The first one is separable, the second not. The first one is unimodal, the second highly multimodal (roughly $10^n$ local optima). The first one is highly ill-conditioning (condition number of $10^6$ ), the second one has a conditioning of about 10. Local non-linear transformations are performed in both objective functions to alleviate the original symmetry and regularity of the two baseline functions.

Contained in the separable - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 16 in dimension 5 for the first instance.

$f_{17}$ : Ellipsoid separable/Schaffer F7, condition 10¶

Combination of the separable ellipsoid function ( $f_2$ in the bbob suite) and the Schaffer F7 function with condition number 10 ( $f_{17}$ in the bbob suite).

Also here, both single objectives possess opposing properties. The first objective is unimodal, besides small local non-linearities symmetric, separable and highly ill-conditioned while the second objective is highly multi-modal, asymmetric, and non-separable, with only a low conditioning.

Contained in the separable - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 17 in dimension 5 for the first instance.

$f_{18}$ : Ellipsoid separable/Schwefel x*sin(x)¶

Combination of the separable ellipsoid function ( $f_2$ in the bbob suite) and the Schwefel function ( $f_{20}$ in the bbob suite).

The first objective is unimodal, separable and highly ill-conditioned. The second objective is partially separable and highly multimodal—having the most prominent $2^n$ minima located comparatively close to the corners of the unpenalized search area.

Contained in the separable - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 18 in dimension 5 for the first instance.

$f_{19}$ : Ellipsoid separable/Gallagher 101 peaks¶

Combination of the separable ellipsoid function ( $f_2$ in the bbob suite) and the Gallagher function with 101 peaks ( $f_{21}$ in the bbob suite).

While the first objective function is separable, unimodal, and highly ill-conditioned (condition number of about $10^6$ ), the second objective function is non-separable and consists of 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the separable - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 19 in dimension 5 for the first instance.

$f_{20}$ : Attractive sector/Attractive sector¶

Combination of two attractive sector functions ( $f_6$ in the bbob suite). Both functions are unimodal and highly asymmetric, where only one hypercone (with angular base area) per objective with a volume of roughly $(1/2)^n$ yields low function values. The objective functions’ optima are located at the tips of those two cones.

Contained in the moderate - moderate function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 20 in dimension 5 for the first instance.

$f_{21}$ : Attractive sector/Rosenbrock original¶

Combination of the attractive sector function ( $f_6$ in the bbob suite) and the Rosenbrock function ( $f_8$ in the bbob suite).

The first function is unimodal but highly asymmetric, where only one hypercone (with angular base area) with a volume of roughly $(1/2)^n$ yields low function values (with the optimum at the tip of the cone). The second objective is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%.

Contained in the moderate - moderate function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 21 in dimension 5 for the first instance.

$f_{22}$ : Attractive sector/Sharp ridge¶

Combination of the attractive sector function ( $f_6$ in the bbob suite) and the sharp ridge function ( $f_{13}$ in the bbob suite).

Both objective functions are unimodal and non-separable. The first objective is highly asymmetric in the sense that only one hypercone (with angular base area) with a volume of roughly $(1/2)^n$ yields low function values (with the optimum at the tip of the cone). For optimizing the second objective, a sharp, i.e., non-differentiable ridge has to be followed.

Contained in the moderate - ill-conditioned function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 22 in dimension 5 for the first instance.

$f_{23}$ : Attractive sector/Sum of different powers¶

Combination of the attractive sector function ( $f_6$ in the bbob suite) and the sum of different powers function ( $f_{14}$ in the bbob suite).

Both objective functions are unimodal and non-separable. The first objective is highly asymmetric in the sense that only one hypercone (with angular base area) with a volume of roughly $(1/2)^n$ yields low function values (with the optimum at the tip of the cone). When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.

Contained in the moderate - ill-conditioned function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 23 in dimension 5 for the first instance.

$f_{24}$ : Attractive sector/Rastrigin¶

Combination of the attractive sector function ( $f_6$ in the bbob suite) and the Rastrigin function ( $f_{15}$ in the bbob suite).

Both objectives are non-separable, and the second one is highly multi-modal (roughly $10^n$ local optima) while the first one is unimodal. Further properties are that the first objective is highly assymetric and the second has a conditioning of about 10.

Contained in the moderate - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 24 in dimension 5 for the first instance.

$f_{25}$ : Attractive sector/Schaffer F7, condition 10¶

Combination of the attractive sector function ( $f_6$ in the bbob suite) and the Schaffer F7 function with condition number 10 ( $f_{17}$ in the bbob suite).

Both objectives are non-separable and asymmetric. While the first objective is unimodal, the second one is a highly multi-modal function with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the moderate - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 25 in dimension 5 for the first instance.

$f_{26}$ : Attractive sector/Schwefel x*sin(x)¶

Combination of the attractive sector function ( $f_6$ in the bbob suite) and the Schwefel function ( $f_{20}$ in the bbob suite).

The first objective is non-separable, unimodal, and asymmetric. The second objective is partially separable and highly multimodal—having the most prominent $2^n$ minima located comparatively close to the corners of the unpenalized search area.

Contained in the moderate - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 26 in dimension 5 for the first instance.

$f_{27}$ : Attractive sector/Gallagher 101 peaks¶

Combination of the attractive sector function ( $f_6$ in the bbob suite) and the Gallagher function with 101 peaks ( $f_{21}$ in the bbob suite).

Both objective functions are non-separable but only the first is unimodal. The first objective function is furthermore asymmetric. The second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the moderate - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 27 in dimension 5 for the first instance.

$f_{28}$ : Rosenbrock original/Rosenbrock original¶

Combination of two Rosenbrock functions ( $f_8$ in the bbob suite).

Both objectives are partially separable (tri-band structure) and have a local optimum with an attraction volume of about 25%.

Contained in the moderate - moderate function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 28 in dimension 5 for the first instance.

$f_{29}$ : Rosenbrock original/Sharp ridge¶

Combination of the Rosenbrock function ( $f_8$ in the bbob suite) and the sharp ridge function ( $f_{13}$ in the bbob suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective is unimodal and non-separable and, for optimizing it, a sharp, i.e., non-differentiable ridge has to be followed.

Contained in the moderate - ill-conditioned function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 29 in dimension 5 for the first instance.

$f_{30}$ : Rosenbrock original/Sum of different powers¶

Combination of the Rosenbrock function ( $f_8$ in the bbob suite) and the sum of different powers function ( $f_{14}$ in the bbob suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective function is unimodal and non-separable. When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.

Contained in the moderate - ill-conditioned function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 30 in dimension 5 for the first instance.

$f_{31}$ : Rosenbrock original/Rastrigin¶

Combination of the Rosenbrock function ( $f_8$ in the bbob suite) and the Rastrigin function ( $f_{15}$ in the bbob suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective function is non-separable and highly multi-modal (roughly $10^n$ local optima).

Contained in the moderate - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 31 in dimension 5 for the first instance.

$f_{32}$ : Rosenbrock original/Schaffer F7, condition 10¶

Combination of the Rosenbrock function ( $f_8$ in the bbob suite) and the Schaffer F7 function with condition number 10 ( $f_{17}$ in the bbob suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective function is non-separable, asymmetric, and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the moderate - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 32 in dimension 5 for the first instance.

$f_{33}$ : Rosenbrock original/Schwefel x*sin(x)¶

Combination of the Rosenbrock function ( $f_8$ in the bbob suite) and the Schwefel function ( $f_{20}$ in the bbob suite).

Both objective functions are partially separable. While the first objective function has a local optimum with an attraction volume of about 25%, the second objective function is highly multimodal—having the most prominent $2^n$ minima located comparatively close to the corners of its unpenalized search area.

Contained in the moderate - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 33 in dimension 5 for the first instance.

$f_{34}$ : Rosenbrock original/Gallagher 101 peaks¶

Combination of the Rosenbrock function ( $f_8$ in the bbob suite) and the Gallagher function with 101 peaks ( $f_{21}$ in the bbob suite).

The first objective function is partially separable, the second one non-separable. While the first objective function has a local optimum with an attraction volume of about 25%, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the moderate - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 34 in dimension 5 for the first instance.

$f_{35}$ : Sharp ridge/Sharp ridge¶

Combination of two sharp ridge functions ( $f_{13}$ in the bbob suite).

Both objective functions are unimodal and non-separable and, for optimizing them, two sharp, i.e., non-differentiable ridges have to be followed.

Contained in the ill-conditioned - ill-conditioned function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 35 in dimension 5 for the first instance.

$f_{36}$ : Sharp ridge/Sum of different powers¶

Combination of the sharp ridge function ( $f_{13}$ in the bbob suite) and the sum of different powers function ( $f_{14}$ in the bbob suite).

Both functions are uni-modal and non-separable. For optimizing the first objective, a sharp, i.e., non-differentiable ridge has to be followed. When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.

Contained in the ill-conditioned - ill-conditioned function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 36 in dimension 5 for the first instance.

$f_{37}$ : Sharp ridge/Rastrigin¶

Combination of the sharp ridge function ( $f_{13}$ in the bbob suite) and the Rastrigin function ( $f_{15}$ in the bbob suite).

Both functions are non-separable. While the first one is unimodal and non-differentiable at its ridge, the second objective function is highly multi-modal (roughly $10^n$ local optima).

Contained in the ill-conditioned - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 37 in dimension 5 for the first instance.

$f_{38}$ : Sharp ridge/Schaffer F7, condition 10¶

Combination of the sharp ridge function ( $f_{13}$ in the bbob suite) and the Schaffer F7 function with condition number 10 ( $f_{17}$ in the bbob suite).

Both functions are non-separable. While the first one is unimodal and non-differentiable at its ridge, the second objective function is asymmetric and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the ill-conditioned - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 38 in dimension 5 for the first instance.

$f_{39}$ : Sharp ridge/Schwefel x*sin(x)¶

Combination of the sharp ridge function ( $f_{13}$ in the bbob suite) and the Schwefel function ( $f_{20}$ in the bbob suite).

While the first objective function is unimodal, non-separable, and non-differentiable at its ridge, the second objective function is highly multimodal—having the most prominent $2^n$ minima located comparatively close to the corners of its unpenalized search area.

Contained in the ill-conditioned - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 39 in dimension 5 for the first instance.

$f_{40}$ : Sharp ridge/Gallagher 101 peaks¶

Combination of the sharp ridge function ( $f_{13}$ in the bbob suite) and the Gallagher function with 101 peaks ( $f_{21}$ in the bbob suite).

Both objective functions are non-separable. While the first objective function is unimodal and non-differentiable at its ridge, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the ill-conditioned - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 40 in dimension 5 for the first instance.

$f_{41}$ : Sum of different powers/Sum of different powers¶

Combination of two sum of different powers functions ( $f_{14}$ in the bbob suite).

Both functions are uni-modal and non-separable where the sensitivies of the variables in the rotated search space become more and more different when approaching the objectives’ optima.

Contained in the ill-conditioned - ill-conditioned function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 41 in dimension 5 for the first instance.

$f_{42}$ : Sum of different powers/Rastrigin¶

Combination of the sum of different powers functions ( $f_{14}$ in the bbob suite) and the Rastrigin function ( $f_{15}$ in the bbob suite).

Both objective functions are non-separable. While the first one is unimodal, the second objective function is highly multi-modal (roughly $10^n$ local optima).

Contained in the ill-conditioned - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 42 in dimension 5 for the first instance.

$f_{43}$ : Sum of different powers/Schaffer F7, condition 10¶

Combination of the sum of different powers functions ( $f_{14}$ in the bbob suite) and the Schaffer F7 function with condition number 10 ( $f_{17}$ in the bbob suite).

Both objective functions are non-separable. While the first one is unimodal with an increasing conditioning once the optimum is approached, the second objective function is asymmetric and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the ill-conditioned - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 43 in dimension 5 for the first instance.

$f_{44}$ : Sum of different powers/Schwefel x*sin(x)¶

Combination of the sum of different powers functions ( $f_{14}$ in the bbob suite) and the Schwefel function ( $f_{20}$ in the bbob suite).

Both objectives are non-separable. While the first objective function is unimodal, the second objective function is highly multimodal—having the most prominent $2^n$ minima located comparatively close to the corners of its unpenalized search area.

Contained in the ill-conditioned - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 44 in dimension 5 for the first instance.

$f_{45}$ : Sum of different powers/Gallagher 101 peaks¶

Combination of the sum of different powers functions ( $f_{14}$ in the bbob suite) and the Gallagher function with 101 peaks ( $f_{21}$ in the bbob suite).

Both objective functions are non-separable. While the first objective function is unimodal, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the ill-conditioned - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 45 in dimension 5 for the first instance.

$f_{46}$ : Rastrigin/Rastrigin¶

Combination of two Rastrigin functions ( $f_{15}$ in the bbob suite).

Both objective functions are non-separable and highly multi-modal (roughly $10^n$ local optima).

Contained in the multi-modal - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 46 in dimension 5 for the first instance.

$f_{47}$ : Rastrigin/Schaffer F7, condition 10¶

Combination of the Rastrigin function ( $f_{15}$ in the bbob suite) and the Schaffer F7 function with condition number 10 ( $f_{17}$ in the bbob suite).

Both objective functions are non-separable and highly multi-modal.

Contained in the multi-modal - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 47 in dimension 5 for the first instance.

$f_{48}$ : Rastrigin/Schwefel x*sin(x)¶

Combination of the Rastrigin function ( $f_{15}$ in the bbob suite) and the Schwefel function ( $f_{20}$ in the bbob suite).

Both objective functions are non-separable and highly multi-modal where the first has roughly $10^n$ local optima and the most prominent $2^n$ minima of the second objective function are located comparatively close to the corners of its unpenalized search area.

Contained in the multi-modal - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 48 in dimension 5 for the first instance.

$f_{49}$ : Rastrigin/Gallagher 101 peaks¶

Combination of the Rastrigin function ( $f_{15}$ in the bbob suite) and the Gallagher function with 101 peaks ( $f_{21}$ in the bbob suite).

Both objective functions are non-separable and highly multi-modal where the first has roughly $10^n$ local optima and the second has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function).

Contained in the multi-modal - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 49 in dimension 5 for the first instance.

$f_{50}$ : Schaffer F7, condition 10/Schaffer F7, condition 10¶

Combination of two Schaffer F7 functions with condition number 10 ( $f_{17}$ in the bbob suite).

Both objective functions are non-separable and highly multi-modal.

Contained in the multi-modal - multi-modal function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 50 in dimension 5 for the first instance.

$f_{51}$ : Schaffer F7, condition 10/Schwefel x*sin(x)¶

Combination of the Schaffer F7 function with condition number 10 ( $f_{17}$ in the bbob suite) and the Schwefel function ( $f_{20}$ in the bbob suite).

Both objective functions are non-separable and highly multi-modal. While frequency and amplitude of the modulation vary in an almost regular fashion in the first objective function, the second objective function posseses less global structure.

Contained in the multi-modal - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 51 in dimension 5 for the first instance.

$f_{52}$ : Schaffer F7, condition 10/Gallagher 101 peaks¶

Combination of the Schaffer F7 function with condition number 10 ( $f_{17}$ in the bbob suite) and the Gallagher function with 101 peaks ( $f_{21}$ in the bbob suite).

Both objective functions are non-separable and highly multi-modal. While frequency and amplitude of the modulation vary in an almost regular fashion in the first objective function, the second has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function).

Contained in the multi-modal - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 52 in dimension 5 for the first instance.

$f_{53}$ : Schwefel xsin(x)/Schwefel xsin(x)¶

Combination of two Schwefel functions ( $f_{20}$ in the bbob suite).

Both objective functions are non-separable and highly multi-modal where the most prominent $2^n$ minima of each objective function are located comparatively close to the corners of its unpenalized search area. Due to the combinatorial nature of the Schwefel function, it is likely in low dimensions that the Pareto set goes through the origin of the search space.

Contained in the weakly-structured - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 53 in dimension 5 for the first instance.

$f_{54}$ : Schwefel x*sin(x)/Gallagher 101 peaks¶

Combination of the Schwefel function ( $f_{20}$ in the bbob suite) and the Gallagher function with 101 peaks ( $f_{21}$ in the bbob suite).

Both objective functions are non-separable and highly multi-modal. For the first objective function, the most prominent $2^n$ minima are located comparatively close to the corners of its unpenalized search area. For the second objective, position and height of all 101 optima are unrelated and randomly chosen (different for each instantiation of the function).

Contained in the weakly-structured - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 54 in dimension 5 for the first instance.

$f_{55}$ : Gallagher 101 peaks/Gallagher 101 peaks¶

Combination of two Gallagher functions with 101 peaks ( $f_{21}$ in the bbob suite).

Both objective functions are non-separable and highly multi-modal. Position and height of all 101 optima in each objective function are unrelated and randomly chosen and thus, no global structure is present.

Contained in the weakly-structured - weakly-structured function class.

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 55 in dimension 5 for the first instance.

The Extended `bbob-biobj-ext` Test Suite and Its Functions¶

Having all combinations of only a subset of the single-objective bbob functions in a test suite like the above bbob-biobj one has advantages but also a few disadvantages. Using only a subet of the 24 bbob functions introduces a bias towards the chosen functions and reduces the amount of different difficulties, a bi-objective algorithm is exposed to in the benchmarking exercise. Allowing all combinations of bbob functions increases the percentage of problems for which both objectives are from different bbob function groups while, in practice, it can often be assumed that both objective functions come from a similar “function domain”.

The rationale behind the following extended bbob-biobj test suite, denoted as bbob-biobj-ext, is to reduce the mentioned effects. To this end, we add all within-group combinations of bbob functions which are not already in the bbob-biobj suite and which do not combine a function with itself. For technical reasons, we also remove the Weierstrass functions ( $f_{16}$ in the bbob suite) because the optimum is not necessarily unique and computing the nadir point therefore technically more challenging than for the other functions. This extension adds $3*(4+3+2+1-1) + 2*(3+2+1-1) = 3*9+2*5=37$ functions, resulting in 92 functions overall.

The following table details which single-objective bbob functions are contained in the 92 bbob-biobj-ext functions (outer column and row annotations) and indicates their IDs. Note that the IDs of the first 55 bbob0biobj-ext functions are the same than for the bbob-biobj test suite for compatibility reasons.

$f_1$

$f_2$

$f_3$

$f_4$

$f_5$

$f_6$

$f_7$

$f_8$

$f_9$

$f_{10}$

$f_{11}$

$f_{12}$

$f_{13}$

$f_{14}$

$f_{15}$

$f_{16}$

$f_{17}$

$f_{18}$

$f_{19}$

$f_{20}$

$f_{21}$

$f_{22}$

$f_{23}$

$f_{24}$

$f_1$

f1

f2

f56

f57

f58

$f_2$

f59

f60

f61

$f_3$

f62

f63

$f_4$

f64

$f_5$

$f_6$

f65

f66

$f_7$

f67

f68

$f_8$

f69

$f_9$

$f_{10}$

f70

f71

f72

f73

$f_{11}$

f74

f75

f76

$f_{12}$

f77

f78

$f_{13}$

$f_{14}$

$f_{15}$

f79

f80

f48

f49

$f_{16}$

$f_{17}$

f50

f81

f82

f51

f52

$f_{18}$

f83

$f_{19}$

$f_{20}$

f53

f54

f84

f85

f86

$f_{21}$

f55

f87

f88

f89

$f_{22}$

f90

f91

$f_{23}$

f92

$f_{24}$

The 92 functions of the bbob-biobj-ext test suite and their IDs (in the table cells) together with the information about which single-objective bbob functions are used to define them (outer column and row annotations).

Function Groups¶

Like for the bbob-biobj test suite, we obtain 15 function classes to structure the 92 bi-objective functions of the bbob-biobj-ext test suite. Depending on whether a function class combines functions from the same or from different bbob function classes, each function class contains 8, 12 or just four functions. We are listing below the function classes and in parenthesis the functions that belong to the respective class:

separable - separable (12 functions: f1, f2, f11, f56-64)

separable - moderate (f3, f4, f12, f13)

separable - ill-conditioned (f5, f6, f14, f15)

separable - multi-modal (f7, f8, f16, f17)

separable - weakly-structured (f9, f10, f18, f19)

moderate - moderate (8 functions: f20, f21, f28, f65-f69)

moderate - ill-conditioned (f22, f23, f29, f30)

moderate - multi-modal (f24, f25, f31, f32)

moderate - weakly-structured (f26, f27, f33, f34)

ill-conditioned - ill-conditioned (12 functions: f35, f36, f41, f70-78)

ill-conditioned - multi-modal (f37, f38, f42, f43)

ill-conditioned - weakly-structured (f39, f40, f44, f45)

multi-modal - multi-modal (8 functions: f46, f47, f50, f79-83)

multi-modal - weakly structured (f48, f49, f51, f52)

weakly structured - weakly structured (12 functions: f53-55, f84-92)

Normalization and Instances¶

Normalization of the objectives and instances are handled for the bbob-biobj-ext in the same manner as for the bbob-biobj suite, i.e., no normalization of the objective functions is taking place for the algorithm benchmarking and 15 instances are prescribed for a typical experiment.

Acknowledgments

This work was supported by the grant ANR-12-MONU-0009 (NumBBO) of the French National Research Agency. We also thank Ilya Loshchilov and Oswin Krause for their initial suggestions on how to extend the bbob-biobj test suite.

References

[BRO2016biperf]

D. Brockhoff, T. Tušar, D. Tušar, T. Wagner, N. Hansen, A. Auger, (2016). Biobjective Performance Assessment with the COCO Platform. ArXiv e-prints, arXiv:1605.01746.

[BRO2015]

(1, 2) D. Brockhoff, T.-D. Tran, and N. Hansen (2015). Benchmarking Numerical Multiobjective Optimizers Revisited. In Proceedings of the 2015 GECCO Genetic and Evolutionary Computation Conference, pp. 639-646, ACM.

[HAN2016co]

(1, 2) N. Hansen, A. Auger, O. Mersmann, T. Tušar, D. Brockhoff (2016). COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting, ArXiv e-prints, arXiv:1603.08785.

[HAN2009]

(1, 2) N. Hansen, S. Finck, R. Ros, and A. Auger (2009). Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions. Research Report RR-6829, Inria, updated February 2010.

[HAN2011]

N. Hansen, R. Ros, N. Mauny, M. Schoenauer, and A. Auger (2011). Impacts of Invariance in Search: When CMA-ES and PSO Face Ill-Conditioned and Non-Separable Problems. Applied Soft Computing. Vol. 11, pp. 5755-5769. Elsevier.

[HAN2016ex]

N. Hansen, T. Tušar, A. Auger, D. Brockhoff, O. Mersmann (2016). COCO: The Experimental Procedure, ArXiv e-prints, arXiv:1603.08776.

COCO: The Bi-objective Black-Box Optimization Benchmarking (bbob-biobj) Test Suite¶

Preliminaries, Definitions, and Scope¶

Overview of the Proposed bbob-biobj Test Suite¶

The Single-objective bbob Functions¶

Function Groups¶

Normalization of Objectives¶

Instances¶

The bbob-biobj Test Functions and Their Properties¶

Some Function Properties¶

Domain Bounds¶

Provided Search Space and Objective Space Plots¶

The 55 bbob-biobj Functions¶

: Sphere/Sphere¶

: Sphere/Ellipsoid separable¶

: Sphere/Attractive sector¶

: Sphere/Rosenbrock original¶

: Sphere/Sharp ridge¶

: Sphere/Sum of different powers¶

: Sphere/Rastrigin¶

: Sphere/Schaffer F7, condition 10¶

: Sphere/Schwefel x*sin(x)¶

: Sphere/Gallagher 101 peaks¶

: Ellipsoid separable/Ellipsoid separable¶

: Ellipsoid separable/Attractive sector¶

: Ellipsoid separable/Rosenbrock original¶

: Ellipsoid separable/Sharp ridge¶

: Ellipsoid separable/Sum of different powers¶

: Ellipsoid separable/Rastrigin¶

: Ellipsoid separable/Schaffer F7, condition 10¶

: Ellipsoid separable/Schwefel x*sin(x)¶

: Ellipsoid separable/Gallagher 101 peaks¶

: Attractive sector/Attractive sector¶

: Attractive sector/Rosenbrock original¶

: Attractive sector/Sharp ridge¶

: Attractive sector/Sum of different powers¶

: Attractive sector/Rastrigin¶

: Attractive sector/Schaffer F7, condition 10¶

: Attractive sector/Schwefel x*sin(x)¶

: Attractive sector/Gallagher 101 peaks¶

: Rosenbrock original/Rosenbrock original¶

: Rosenbrock original/Sharp ridge¶

: Rosenbrock original/Sum of different powers¶

: Rosenbrock original/Rastrigin¶

: Rosenbrock original/Schaffer F7, condition 10¶

: Rosenbrock original/Schwefel x*sin(x)¶

: Rosenbrock original/Gallagher 101 peaks¶

: Sharp ridge/Sharp ridge¶

: Sharp ridge/Sum of different powers¶

: Sharp ridge/Rastrigin¶

: Sharp ridge/Schaffer F7, condition 10¶

: Sharp ridge/Schwefel x*sin(x)¶

: Sharp ridge/Gallagher 101 peaks¶

: Sum of different powers/Sum of different powers¶

: Sum of different powers/Rastrigin¶

: Sum of different powers/Schaffer F7, condition 10¶

: Sum of different powers/Schwefel x*sin(x)¶

: Sum of different powers/Gallagher 101 peaks¶

: Rastrigin/Rastrigin¶

: Rastrigin/Schaffer F7, condition 10¶

: Rastrigin/Schwefel x*sin(x)¶

: Rastrigin/Gallagher 101 peaks¶

: Schaffer F7, condition 10/Schaffer F7, condition 10¶

: Schaffer F7, condition 10/Schwefel x*sin(x)¶

: Schaffer F7, condition 10/Gallagher 101 peaks¶

: Schwefel x*sin(x)/Schwefel x*sin(x)¶

: Schwefel x*sin(x)/Gallagher 101 peaks¶

: Gallagher 101 peaks/Gallagher 101 peaks¶

The Extended bbob-biobj-ext Test Suite and Its Functions¶

Function Groups¶

Normalization and Instances¶

Acknowledgments

References

COCO: The Bi-objective Black-Box Optimization Benchmarking (`bbob-biobj`) Test Suite¶

Overview of the Proposed `bbob-biobj` Test Suite¶

The Single-objective `bbob` Functions¶

The `bbob-biobj` Test Functions and Their Properties¶

The 55 `bbob-biobj` Functions¶

$f_1$ : Sphere/Sphere¶

$f_2$ : Sphere/Ellipsoid separable¶

$f_3$ : Sphere/Attractive sector¶

$f_4$ : Sphere/Rosenbrock original¶

$f_5$ : Sphere/Sharp ridge¶

$f_6$ : Sphere/Sum of different powers¶

$f_7$ : Sphere/Rastrigin¶

$f_8$ : Sphere/Schaffer F7, condition 10¶

$f_9$ : Sphere/Schwefel x*sin(x)¶

$f_{10}$ : Sphere/Gallagher 101 peaks¶

$f_{11}$ : Ellipsoid separable/Ellipsoid separable¶

$f_{12}$ : Ellipsoid separable/Attractive sector¶

$f_{13}$ : Ellipsoid separable/Rosenbrock original¶

$f_{14}$ : Ellipsoid separable/Sharp ridge¶

$f_{15}$ : Ellipsoid separable/Sum of different powers¶

$f_{16}$ : Ellipsoid separable/Rastrigin¶

$f_{17}$ : Ellipsoid separable/Schaffer F7, condition 10¶

$f_{18}$ : Ellipsoid separable/Schwefel x*sin(x)¶

$f_{19}$ : Ellipsoid separable/Gallagher 101 peaks¶

$f_{20}$ : Attractive sector/Attractive sector¶

$f_{21}$ : Attractive sector/Rosenbrock original¶

$f_{22}$ : Attractive sector/Sharp ridge¶

$f_{23}$ : Attractive sector/Sum of different powers¶

$f_{24}$ : Attractive sector/Rastrigin¶

$f_{25}$ : Attractive sector/Schaffer F7, condition 10¶

$f_{26}$ : Attractive sector/Schwefel x*sin(x)¶

$f_{27}$ : Attractive sector/Gallagher 101 peaks¶

$f_{28}$ : Rosenbrock original/Rosenbrock original¶

$f_{29}$ : Rosenbrock original/Sharp ridge¶

$f_{30}$ : Rosenbrock original/Sum of different powers¶

$f_{31}$ : Rosenbrock original/Rastrigin¶

$f_{32}$ : Rosenbrock original/Schaffer F7, condition 10¶

$f_{33}$ : Rosenbrock original/Schwefel x*sin(x)¶

$f_{34}$ : Rosenbrock original/Gallagher 101 peaks¶

$f_{35}$ : Sharp ridge/Sharp ridge¶

$f_{36}$ : Sharp ridge/Sum of different powers¶

$f_{37}$ : Sharp ridge/Rastrigin¶

$f_{38}$ : Sharp ridge/Schaffer F7, condition 10¶

$f_{39}$ : Sharp ridge/Schwefel x*sin(x)¶

$f_{40}$ : Sharp ridge/Gallagher 101 peaks¶

$f_{41}$ : Sum of different powers/Sum of different powers¶

$f_{42}$ : Sum of different powers/Rastrigin¶

$f_{43}$ : Sum of different powers/Schaffer F7, condition 10¶

$f_{44}$ : Sum of different powers/Schwefel x*sin(x)¶

$f_{45}$ : Sum of different powers/Gallagher 101 peaks¶

$f_{46}$ : Rastrigin/Rastrigin¶

$f_{47}$ : Rastrigin/Schaffer F7, condition 10¶

$f_{48}$ : Rastrigin/Schwefel x*sin(x)¶

$f_{49}$ : Rastrigin/Gallagher 101 peaks¶

$f_{50}$ : Schaffer F7, condition 10/Schaffer F7, condition 10¶

$f_{51}$ : Schaffer F7, condition 10/Schwefel x*sin(x)¶

$f_{52}$ : Schaffer F7, condition 10/Gallagher 101 peaks¶

$f_{53}$ : Schwefel xsin(x)/Schwefel xsin(x)¶

$f_{54}$ : Schwefel x*sin(x)/Gallagher 101 peaks¶

$f_{55}$ : Gallagher 101 peaks/Gallagher 101 peaks¶

The Extended `bbob-biobj-ext` Test Suite and Its Functions¶