COCO: The Bi-objective Black-Box Optimization Benchmarking (bbob-biobj) Test Suite

See also: ArXiv e-prints, arXiv:1604.00359, 2016.

Several test function suites for numerical benchmarking of multiobjective optimization algorithms have been proposed in recent years. While having desirable properties like well-understood Pareto sets and Pareto fronts with shapes of various kinds, most of the currently used functions posess properties which are arguably under-represented in real-world problems. Those properties mainly stem from the easier construction of such problems—overrepresenting properties such as no dependencies between variables, Pareto sets exactly located at the bound constraints, or the differentiation between position and distance variables. Here, we propose an alternative way and define the bbob-biobj test suite with 55 bi-objective functions and its extended bbob-biobj-ext version with 92 bi-objective functions in continuous domain which are both derived from combining functions of the well-known single-objective noiseless bbob test suite. Besides giving the actual function definitions and presenting their (known) properties, this documentation also aims at giving the rationale behind our approach in terms of function groups, instances, and potential objective space normalization.

_images/examples-bbob-biobj.png

Example plots of the Pareto front approximation, found by NSGA-II on selected bbob-biobj functions. In blue the non-dominated points at the end of different independent runs, in red the points that are non-dominated over all runs.

Preliminaries, Definitions, and Scope

In the following, we consider bi-objective, unconstrained minimization problems of the form

\min_{x \in \mathbb{R}^n} f(x)=(f_\alpha(x),f_\beta(x)),

where n is the number of variables of the problem (also called the problem dimension), f_\alpha: \mathbb{R}^n \rightarrow \mathbb{R} and f_\beta: \mathbb{R}^n \rightarrow \mathbb{R} are the two objective functions, and the \min operator is related to the standard dominance relation. A solution x\in\mathbb{R}^n is thereby said to dominate another solution y\in\mathbb{R}^n if f_\alpha(x) \leq f_\alpha(y) and f_\beta(x) \leq f_\beta(y) hold and at least one of the inequalities is strict.

Solutions which are not dominated by any other solution in the search space are called Pareto-optimal or efficient solutions. All Pareto-optimal solutions constitute the Pareto set of which an approximation is sought. The Pareto set’s image in the objective space f(\mathbb{R}^n) is called Pareto front.

The objective of the minimization problem is to find, with as few evaluations of f as possible, a set of non-dominated solutions which is (i) as large as possible and (ii) has f-values as close to the Pareto front as possible. [1]

[1]Distance in f-space is defined here such that nadir and ideal point have in each coordinate distance one. Neither of these points is however freely accessible to the optimization algorithm.

In the following, we remind useful definitions.

function instance, problem

Each function f^\theta: \mathbb{R}^n \to \mathbb{R}^m within COCO is parametrized with parameter values \theta \in \Theta. A parameter value determines a so-called function instance. For example, \theta encodes the location of the optimum of single-objective functions, which means that different instances have shifted optima. In the bbob-biobj test suite, m=2 and the function instances are determined by the instances of the underlying single-objective functions.

A problem is a function instance of a specific dimension n.

ideal point

The ideal point is defined as the vector in objective space that contains the optimal f-value for each objective independently. More precisely let f_\alpha^{\rm opt}:= \inf_{x\in \mathbb{R}^n} f_\alpha(x) and f_\beta^{\rm opt}:= \inf_{x\in \mathbb{R}^n} f_\beta(x), the ideal point is given by

\begin{equation*}
    z_{\rm ideal}  =  (f_\alpha^{\rm opt},f_\beta^{\rm opt}).
\end{equation*}

nadir point

The nadir point (in objective space) consists in each objective of the worst value obtained by a Pareto-optimal solution. More precisely, let \mathcal{PO} be the set of Pareto optimal points. Then the nadir point satisfies

\begin{equation*}
    z_{\rm nadir}  =   \left( \sup_{x \in \mathcal{PO}} f_\alpha(x),
 \sup_{x \in \mathcal{PO}} f_\beta(x)  \right).
\end{equation*}

In the case of two objectives with a unique global minimum each (that is, a single point in the search space maps to the global minimum)

\begin{equation*}
    z_{\rm nadir}  =   \left( f_\alpha(x_{\rm opt,\beta}),
  f_\beta(x_{\rm opt,\alpha})  \right),
\end{equation*}

where x_{\rm opt,\alpha}= \arg \min f_\alpha(x) and x_{\rm opt,\beta}= \arg \min f_\beta(x).

Overview of the Proposed bbob-biobj Test Suite

The bbob-biobj test suite provides 55 bi-objective functions in six dimensions (2, 3, 5, 10, 20, and 40) with a large number of possible instances. The 55 functions are derived from combining a subset of the 24 well-known single-objective functions of the bbob test suite [HAN2009] which has been used since 2009 in the BBOB workshop series. While concrete details on each of the 55 bbob-biobj functions are given in Section The bbob-biobj Test Functions and Their Properties, we will detail here the main rationale behind them together with their common properties.

The Single-objective bbob Functions

The bbob-biobj test suite is designed to be able to assess performance of algorithms with respect to well-identified difficulties in optimization typically occurring in real-world problems. A multi-objective problem being a combination of single-objective problems, one can obtain multi-objective problems with representative difficulties by simply combining single objective functions with representative difficulties observed in real-world problems. For this purpose we naturally use the single-objective bbob suite [HAN2009].

Combining all 24 bbob functions in pairs thereby results in 24^2=576 bi-objective functions overall. We however assume that multi-objective optimization algorithms are not sensitive to permutations of the objective functions such that combining the 24 bbob functions and taking out the function (g_2,g_1) if the function (g_1,g_2) is present results in 24 + {24 \choose 2} = 24 + (24\times 23)/2 = (24\times 25)/2 = 300 functions.

Some first tests, e.g. in [BRO2015], showed that having 300 functions is impracticable in terms of the overall running time of the benchmarking experiment. We then decided to exploit the organization of the bbob functions into classes to choose a subset of functions. More precisely, the 24 original bbob functions are grouped into five function classes where each class gathers functions with similar properties, namely

  1. separable functions
  2. functions with low or moderate conditioning
  3. functions with high conditioning and unimodal
  4. multi-modal functions with adequate global structure,
  5. multi-modal functions with weak global structure.

To create the bbob-biobj suite, we choose two functions within each class. This way we do not introduce any bias towards a specific class. In addition within each class, the functions are chosen to be the most representative without repeating similar functions. For example, only one Ellipsoid, one Rastrigin, and one Gallagher function are included in the bbob-biobj suite although they appear in separate versions in the bbob suite. Finally our choice of 10 bbob functions for creating the bbob-biobj test suite is the following:

  • Separable functions
  • Functions with low or moderate conditioning
  • Functions with high conditioning and unimodal
  • Multi-modal functions with adequate global structure
  • Multi-modal functions with weak global structure

Using the above described pairwise combinations, this results in having 10+{10 \choose 2} = 55 bi-objective functions in the final bbob-biobj suite. These functions are denoted f_1 to f_{55} in the sequel.

Function Groups

From combining the original bbob function classes, we obtain 15 function classes to structure the 55 bi-objective functions of the bbob-biobj test suite. Each function class contains three or four functions. We are listing below the function classes and in parenthesis the functions that belong to the respective class:

  1. separable - separable (functions f_1, f_2, f_{11})
  2. separable - moderate (f_3, f_4, f_{12}, f_{13})
  3. separable - ill-conditioned (f_5, f_6, f_{14}, f_{15})
  4. separable - multi-modal (f_7, f_8, f_{16}, f_{17})
  5. separable - weakly-structured (f_9, f_{10}, f_{18}, f_{19})
  6. moderate - moderate (f_{20}, f_{21}, f_{28})
  7. moderate - ill-conditioned (f_{22}, f_{23}, f_{29}, f_{30})
  8. moderate - multi-modal (f_{24}, f_{25}, f_{31}, f_{32})
  9. moderate - weakly-structured (f_{26}, f_{27}, f_{33}, f_{34})
  10. ill-conditioned - ill-conditioned (f_{35}, f_{36}, f_{41})
  11. ill-conditioned - multi-modal (f_{37}, f_{38}, f_{42}, f_{43})
  12. ill-conditioned - weakly-structured (f_{39}, f_{40}, f_{44}, f_{45})
  13. multi-modal - multi-modal (f_{46}, f_{47}, f_{50})
  14. multi-modal - weakly structured (f_{48}, f_{49}, f_{51}, f_{52})
  15. weakly structured - weakly structured (f_{53}, f_{54}, f_{55})

More details about the single functions can be found in Section The bbob-biobj Test Functions and Their Properties. We however first describe their common properties in the coming sections.

Normalization of Objectives

None of the 55 bbob-biobj functions is explicitly normalized and the optimization algorithms therefore have to cope with objective values in different ranges. Typically, different orders of magnitude between the objective values can be observed.

However, to facilitate comparison of algorithm performance over different functions, we normalize the objectives based on the ideal and nadir points before calculating the hypervolume indicator [BRO2016biperf]. Both points can be computed, because the global optimum is known and is unique for the 10 bbob base functions. In the black-box optimization benchmarking setup, however, the values of the ideal and nadir points are not accessible to the optimization algorithm [HAN2016ex].

Instances

Our test functions are parametrized and instances are instantiations of the underlying parameters (see [HAN2016co]). The instances for the bi-objective functions are obtained using instances of each single objective function composing the bi-objective one. In addition, we assert two conditions:

1. The Euclidean distance between the two single-objective optima (also called the extreme optimal points) in the search space is at least 10^{-4}.

2. The Euclidean distance between the ideal and the nadir point in the non-normalized objective space is at least 10^{-1}.

We associate to an instance, an instance-id which is an integer. The relation between the instance-id, K^{f}_{\rm id}, of a bi-objective function f = (f_\alpha, f_\beta) and the instance-ids, K_{\rm id}^{f_\alpha} and K_{\rm id}^{f_\beta}, of its underlying single-objective functions f_\alpha and f_\beta is the following:

  • K_{\rm id}^{f_\alpha} = 2 K^{f}_{\rm id} + 1 and
  • K_{\rm id}^{f_\beta} = K_{\rm id}^{f_\alpha} + 1

If we find that above conditions are not satisfied for all dimensions and functions in the bbob-biobj suite, we increase the instance-id of the second objective successively until both properties are fulfilled. For example, the bbob-biobj instance-id 8 corresponds to the instance-id 17 for the first objective and instance-id 18 for the second objective while for the bbob-biobj instance-id 9, the first instance-id is 19 but for the second objective, instance-id 21 is chosen instead of instance-id 20.

Exceptions to the above rule are, for historical reasons, the bbob-biobj instance-ids 1 and 2 in order to match the instance-ids 1 to 5 with the ones proposed in [BRO2015]. The bbob-biobj instance-id 1 contains the single-objective instance-ids 2 and 4 and the bbob-biobj instance-id 2 contains the two instance-ids 3 and 5.

For each bi-objective function and given dimension, the bbob-biobj suite contains 10 instances. [2]

[2]In principle, as for the instance generation for the bbob suite, the number of possible instances for the bbob-biobj suite is unlimited [HAN2016co]. However, running some tests with too few instances will render the potential statistics and their interpretation problematic while even the tiniest observed difference can be made statistically significant with a high enough number of instances. A good compromise to avoid either pitfall seems to lie between, say, 9 and 19 instances.

The bbob-biobj Test Functions and Their Properties

In the following, we detail all 55 bbob-biobj functions and their properties.

The following table gives an overview and quick access to the functions, inner cell IDs refer to the bbob-biobj functions, outer column and row annotations refer to the single-objective bbob functions.

  f_1 f_2 f_6 f_8 f_{13} f_{14} f_{15} f_{17} f_{20} f_{21}
f_1 f1 f2 f3 f4 f5 f6 f7 f8 f9 f10
f_2   f11 f12 f13 f14 f15 f16 f17 f18 f19
f_6     f20 f21 f22 f23 f24 f25 f26 f27
f_8       f28 f29 f30 f31 f32 f33 f34
f_{13}         f35 f36 f37 f38 f39 f40
f_{14}           f41 f42 f43 f44 f45
f_{15}             f46 f47 f48 f49
f_{17}               f50 f51 f52
f_{20}                 f53 f54
f_{21}                   f55

Some Function Properties

In the description of the 55 bbob-biobj functions below, several general properties of objective functions will be mentioned that are defined here in short. It depends on these properties whether the optimization problem is easy or hard to solve.

A separable function does not show any dependencies between the variables and can therefore be solved by applying n consecutive one-dimensional optimizations along the coordinate axes while keeping the other variables fixed. Consequently, non-separable problems must be considered. They are much more difficult to solve. The typical well-established technique to generate non-separable functions from separable ones is the application of a rotation matrix \mathbf R to x, that is x \in \mathbb{R}^n \mapsto g(\mathbf R x), where g is a separable function.

A unimodal function has only one local minimum which is at the same time also its global one. A multimodal function has at least two local minima which is highly common in practical optimization problems.

Ill-conditioning is another typical challenge in real-parameter optimization and, besides multimodality, probably the most common one. In a general case, we can consider a function as ill-conditioned if for solution points from the same level-set “the minimal displacement [...] that produces a given function value improvement differs by orders of magnitude” [HAN2011]. Conditioning can be rigorously formalized in the case of convex quadratic functions, f(x) = \frac{1}{2} x^THx where H is a symmetric positive definite matrix, as the condition number of the Hessian matrix H. Since contour lines associated to a convex quadratic function are ellipsoids, the condition number corresponds to the square root of the ratio between the largest axis of the ellipsoid and the shortest axis.

The proposed bbob-biobj testbed contains ill-conditioned functions with a typical conditioning of 10^6. We believe this is a realistic requirement, while we have seen practical problems with conditioning as large as 10^{10}.

Domain Bounds

All bi-objective functions provided in the bbob-biobj suite are unbounded, i.e., defined on the entire real-valued space \mathbb{R}^n. The search domain of interest is defined as [-100,100]^n, outside of which non-dominated solutions are quite unlikely to be found. [3] The majority of non-dominated solutions are likely to lie even within [-5,5]^n.

While we believe that the domain of interest contains the Pareto set, due to the nature of the bbob-biobj function definitions, there is no guarantee that this is always the case. However, the extremal solutions and their neighborhood ball of radius one are guaranteed to lie within [-5,5]^n.

[3]The functions coco_problem_get_smallest_value_of_interest and coco_problem_get_largest_value_of_interest of the COCO platform allow the optimizer to retrieve the search domain of interest from the coco_problem_t, for example to generate the initial search points.

Provided Search Space and Objective Space Plots

In order to better understand the properties of the 55 bbob-biobj functions, we display for each of them plots of the best known Pareto front approximation in objective space in original scaling (as seen by the algorithm) and in log-scale, normalized such that the ideal point is at [0,0] and the nadir point is at [1,1]. We also provide plots illustrating the best known Pareto set approximation in search space (all depicted in black). For the latter, two different plots are provided: a plot showing the projection onto a coordinate-axes-parallel cut defined by two variables and a plot that projects all points onto a random cutting plane which contains both single-objective optima and that also shows the contour lines of both objective functions on this plane.

In addition to the best Pareto set/Pareto front approximations, cuts through the search space are shown along (i) random lines through each optimum (in blue), (ii) lines along each coordinate axis through each optimum (blue dotted lines), (iii) the line through both optima (in red), (iv) two fully random lines [4] (in yellow), and (v) a random line in the random projection plane going through both optima [5] (in green).

All lines are normalized (of length 10 with the support vector in the middle). Ticks along the lines in the objective space plots indicate the ends of line segments of the same length in search space. Thicker points on the lines depict solutions that are non-dominated with respect to all points on the same line. Furthermore, the search space plots highlight the projected region [-5,5]^n as gray-shaded area while the gray-shaded area in the objective space plots highlight the region of interest between ideal (+) and nadir point (\times). Note that, to keep the plots to a manageable size, the Pareto set and Pareto front approximations are carefully downsampled such that only one solution per grid point is shown—with the precision of 2 decimals for the search space plots and 3 decimals for the objective space plots to define the grid. The number of considered and actually displayed solutions is indicated in the search space plots’ legends. All plots are provided for one instance here only and for dimension 5 for the moment.

[4]of random direction and with a support vector, drawn uniformly at random in [-4,4]^n
[5]with a random direction within the plane and a support vector, drawn uniformly at random in [-4,4] in the coordinate system of the cutting plane`

The 55 bbob-biobj Functions

f_1: Sphere/Sphere

Combination of two sphere functions (f_1 in the bbob suite).

Both objectives are unimodal, highly symmetric, rotational and scale invariant. The Pareto set is known to be a straight line and the Pareto front is convex. Furthermore, the normalized hypervolume value of the entire Pareto front with respect to the nadir point as reference point can be computed analytically as the integral 1-\int_{0}^{1} (1-\sqrt{x})^2dx = -\frac{1}{2}+\frac{4}{3}=0.833333\ldots.

Considered as the simplest bi-objective problem in continuous domain.

Contained in the separable - separable function class.

f01-i01-d05-searchspace f01-i01-d05-searchspace-projected

f01-i01-d05-logobjspace f01-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 1 in dimension 5 for the first instance.

f_2: Sphere/Ellipsoid separable

Combination of the sphere function (f_1 in the bbob suite) and the separable ellipsoid function (f_2 in the bbob suite).

Both objectives are unimodal and separable. While the first objective is truly convex-quadratic with a condition number of 1, the second objective is only globally quadratic with smooth local irregularities and highly ill-conditioned with a condition number of about 10^6.

Contained in the separable - separable function class.

f02-i01-d05-searchspace f02-i01-d05-searchspace-projected

f02-i01-d05-logobjspace f02-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 2 in dimension 5 for the first instance.

f_3: Sphere/Attractive sector

Combination of the sphere function (f_1 in the bbob suite) and the attractive sector function (f_6 in the bbob suite).

Both objective functions are unimodal, but only the first objective is separable and truly convex quadratic. The attractive sector function is highly asymmetric, where only one hypercone (with angular base area) with a volume of roughly (1/2)^n yields low function values. The optimum of it is located at the tip of this cone.

Contained in the separable - moderate function class.

f03-i01-d05-searchspace f03-i01-d05-searchspace-projected

f03-i01-d05-logobjspace f03-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 3 in dimension 5 for the first instance.

f_4: Sphere/Rosenbrock original

Combination of the sphere function (f_1 in the bbob suite) and the original, i.e., unrotated Rosenbrock function (f_8 in the bbob suite).

The first objective is separable and truly convex, the second objective is partially separable (tri-band structure). The first objective is unimodal while the second objective has a local optimum with an attraction volume of about 25%.

Contained in the separable - moderate function class.

f04-i01-d05-searchspace f04-i01-d05-searchspace-projected

f04-i01-d05-logobjspace f04-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 4 in dimension 5 for the first instance.

f_5: Sphere/Sharp ridge

Combination of the sphere function (f_1 in the bbob suite) and the sharp ridge function (f_{13} in the bbob suite).

Both objective functions are unimodal. In addition to the simple, separable, and differentiable first objective, a sharp, i.e., non-differentiable ridge has to be followed for optimizing the (non-separable) second objective. The gradient towards the ridge remains constant, when the ridge is approached from a given point. Approaching the ridge is initially effective, but becomes ineffective close to the ridge when the rigde needs to be followed in direction to its optimum. The necessary change in search behavior close to the ridge is difficult to diagnose, because the gradient towards the ridge does not flatten out.

Contained in the separable - ill-conditioned function class.

f05-i01-d05-searchspace f05-i01-d05-searchspace-projected

f05-i01-d05-logobjspace f05-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 5 in dimension 5 for the first instance.

f_6: Sphere/Sum of different powers

Combination of the sphere function (f_1 in the bbob suite) and the sum of different powers function (f_{14} in the bbob suite).

Both objective functions are unimodal. The first objective is separable, the second non-separable. When approaching the second objective’s optimum, the difference in sensitivity between different directions in search space increases unboundedly.

Contained in the separable - ill-conditioned function class.

f06-i01-d05-searchspace f06-i01-d05-searchspace-projected

f06-i01-d05-logobjspace f06-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 6 in dimension 5 for the first instance.

f_7: Sphere/Rastrigin

Combination of the sphere function (f_1 in the bbob suite) and the Rastrigin function (f_{15} in the bbob suite).

In addition to the simple sphere function, the prototypical highly multimodal Rastrigin function needs to be solved which has originally a very regular and symmetric structure for the placement of the optima. Here, however, transformations are performed to alleviate the original symmetry and regularity in the second objective.

The properties of the second objective contain non-separabilty, multimodality (roughly 10^n local optima), a conditioning of about 10, and a large global amplitude compared to the local amplitudes.

Contained in the separable - multi-modal function class.

f07-i01-d05-searchspace f07-i01-d05-searchspace-projected

f07-i01-d05-logobjspace f07-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 7 in dimension 5 for the first instance.

f_8: Sphere/Schaffer F7, condition 10

Combination of the sphere function (f_1 in the bbob suite) and the Schaffer F7 function with condition number 10 (f_{17} in the bbob suite).

In addition to the simple sphere function, an asymmetric, non-separable, and highly multimodal function needs to be solved to approach the Pareto front/Pareto set where the frequency and amplitude of the modulation in the second objective vary. The conditioning of the second objective and thus the entire bi-objective function is low.

Contained in the separable - multi-modal function class.

f08-i01-d05-searchspace f08-i01-d05-searchspace-projected

f08-i01-d05-logobjspace f08-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 8 in dimension 5 for the first instance.

f_9: Sphere/Schwefel x*sin(x)

Combination of the sphere function (f_1 in the bbob suite) and the Schwefel function (f_{20} in the bbob suite).

While the first objective function is separable and unimodal, the second objective function is partially separable and highly multimodal—having the most prominent 2^n minima located comparatively close to the corners of the unpenalized search area.

Contained in the separable - weakly-structured function class.

f09-i01-d05-searchspace f09-i01-d05-searchspace-projected

f09-i01-d05-logobjspace f09-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 9 in dimension 5 for the first instance.

f_{10}: Sphere/Gallagher 101 peaks

Combination of the sphere function (f_1 in the bbob suite) and the Gallagher function with 101 peaks (f_{21} in the bbob suite).

While the first objective function is separable and unimodal, the second objective function is non-separable and consists of 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the separable - weakly-structured function class.

f10-i01-d05-searchspace f10-i01-d05-searchspace-projected

f10-i01-d05-logobjspace f10-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 10 in dimension 5 for the first instance.

f_{11}: Ellipsoid separable/Ellipsoid separable

Combination of two separable ellipsoid functions (f_2 in the bbob suite).

Both objectives are unimodal, separable, only globally quadratic with smooth local irregularities, and highly ill-conditioned with a condition number of about 10^6.

Contained in the separable - separable function class.

f11-i01-d05-searchspace f11-i01-d05-searchspace-projected

f11-i01-d05-logobjspace f11-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 11 in dimension 5 for the first instance.

f_{12}: Ellipsoid separable/Attractive sector

Combination of the separable ellipsoid function (f_2 in the bbob suite) and the attractive sector function (f_6 in the bbob suite).

Both objective functions are unimodal but only the first one is separable. The first objective function, in addition, is globally quadratic with smooth local irregularities, and highly ill-conditioned with a condition number of about 10^6. The second objective function is highly asymmetric, where only one hypercone (with angular base area) with a volume of roughly (1/2)^n yields low function values. The optimum of it is located at the tip of this cone.

Contained in the separable - moderate function class.

f12-i01-d05-searchspace f12-i01-d05-searchspace-projected

f12-i01-d05-logobjspace f12-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 12 in dimension 5 for the first instance.

f_{13}: Ellipsoid separable/Rosenbrock original

Combination of the separable ellipsoid function (f_2 in the bbob suite) and the original, i.e., unrotated Rosenbrock function (f_8 in the bbob suite).

Only the first objective is separable and unimodal. The second objective is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. In addition, the first objective function shows smooth local irregularities from a globally convex quadratic function and is highly ill-conditioned with a condition number of about 10^6.

Contained in the separable - moderate function class.

f13-i01-d05-searchspace f13-i01-d05-searchspace-projected

f13-i01-d05-logobjspace f13-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 13 in dimension 5 for the first instance.

f_{14}: Ellipsoid separable/Sharp ridge

Combination of the separable ellipsoid function (f_2 in the bbob suite) and the sharp ridge function (f_{13} in the bbob suite).

Both objective functions are unimodal but only the first one is separable.

The first objective is globally quadratic but with smooth local irregularities and highly ill-conditioned with a condition number of about 10^6. For optimizing the second objective, a sharp, i.e., non-differentiable ridge has to be followed.

Contained in the separable - ill-conditioned function class.

f14-i01-d05-searchspace f14-i01-d05-searchspace-projected

f14-i01-d05-logobjspace f14-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 14 in dimension 5 for the first instance.

f_{15}: Ellipsoid separable/Sum of different powers

Combination of the separable ellipsoid function (f_2 in the bbob suite) and the sum of different powers function (f_{14} in the bbob suite).

Both objective functions are unimodal but only the first one is separable.

The first objective is globally quadratic but with smooth local irregularities and highly ill-conditioned with a condition number of about 10^6. When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.

Contained in the separable - ill-conditioned function class.

f15-i01-d05-searchspace f15-i01-d05-searchspace-projected

f15-i01-d05-logobjspace f15-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 15 in dimension 5 for the first instance.

f_{16}: Ellipsoid separable/Rastrigin

Combination of the separable ellipsoid function (f_2 in the bbob suite) and the Rastrigin function (f_{15} in the bbob suite).

The objective functions show rather opposite properties. The first one is separable, the second not. The first one is unimodal, the second highly multimodal (roughly 10^n local optima). The first one is highly ill-conditioning (condition number of 10^6), the second one has a conditioning of about 10. Local non-linear transformations are performed in both objective functions to alleviate the original symmetry and regularity of the two baseline functions.

Contained in the separable - multi-modal function class.

f16-i01-d05-searchspace f16-i01-d05-searchspace-projected

f16-i01-d05-logobjspace f16-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 16 in dimension 5 for the first instance.

f_{17}: Ellipsoid separable/Schaffer F7, condition 10

Combination of the separable ellipsoid function (f_2 in the bbob suite) and the Schaffer F7 function with condition number 10 (f_{17} in the bbob suite).

Also here, both single objectives possess opposing properties. The first objective is unimodal, besides small local non-linearities symmetric, separable and highly ill-conditioned while the second objective is highly multi-modal, asymmetric, and non-separable, with only a low conditioning.

Contained in the separable - multi-modal function class.

f17-i01-d05-searchspace f17-i01-d05-searchspace-projected

f17-i01-d05-logobjspace f17-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 17 in dimension 5 for the first instance.

f_{18}: Ellipsoid separable/Schwefel x*sin(x)

Combination of the separable ellipsoid function (f_2 in the bbob suite) and the Schwefel function (f_{20} in the bbob suite).

The first objective is unimodal, separable and highly ill-conditioned. The second objective is partially separable and highly multimodal—having the most prominent 2^n minima located comparatively close to the corners of the unpenalized search area.

Contained in the separable - weakly-structured function class.

f18-i01-d05-logobjspace f18-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 18 in dimension 5 for the first instance.

f_{19}: Ellipsoid separable/Gallagher 101 peaks

Combination of the separable ellipsoid function (f_2 in the bbob suite) and the Gallagher function with 101 peaks (f_{21} in the bbob suite).

While the first objective function is separable, unimodal, and highly ill-conditioned (condition number of about 10^6), the second objective function is non-separable and consists of 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the separable - weakly-structured function class.

f19-i01-d05-searchspace f19-i01-d05-searchspace-projected

f19-i01-d05-logobjspace f19-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 19 in dimension 5 for the first instance.

f_{20}: Attractive sector/Attractive sector

Combination of two attractive sector functions (f_6 in the bbob suite). Both functions are unimodal and highly asymmetric, where only one hypercone (with angular base area) per objective with a volume of roughly (1/2)^n yields low function values. The objective functions’ optima are located at the tips of those two cones.

Contained in the moderate - moderate function class.

f20-i01-d05-searchspace f20-i01-d05-searchspace-projected

f20-i01-d05-logobjspace f20-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 20 in dimension 5 for the first instance.

f_{21}: Attractive sector/Rosenbrock original

Combination of the attractive sector function (f_6 in the bbob suite) and the Rosenbrock function (f_8 in the bbob suite).

The first function is unimodal but highly asymmetric, where only one hypercone (with angular base area) with a volume of roughly (1/2)^n yields low function values (with the optimum at the tip of the cone). The second objective is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%.

Contained in the moderate - moderate function class.

f21-i01-d05-searchspace f21-i01-d05-searchspace-projected

f21-i01-d05-logobjspace f21-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 21 in dimension 5 for the first instance.

f_{22}: Attractive sector/Sharp ridge

Combination of the attractive sector function (f_6 in the bbob suite) and the sharp ridge function (f_{13} in the bbob suite).

Both objective functions are unimodal and non-separable. The first objective is highly asymmetric in the sense that only one hypercone (with angular base area) with a volume of roughly (1/2)^n yields low function values (with the optimum at the tip of the cone). For optimizing the second objective, a sharp, i.e., non-differentiable ridge has to be followed.

Contained in the moderate - ill-conditioned function class.

f22-i01-d05-searchspace f22-i01-d05-searchspace-projected

f22-i01-d05-logobjspace f22-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 22 in dimension 5 for the first instance.

f_{23}: Attractive sector/Sum of different powers

Combination of the attractive sector function (f_6 in the bbob suite) and the sum of different powers function (f_{14} in the bbob suite).

Both objective functions are unimodal and non-separable. The first objective is highly asymmetric in the sense that only one hypercone (with angular base area) with a volume of roughly (1/2)^n yields low function values (with the optimum at the tip of the cone). When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.

Contained in the moderate - ill-conditioned function class.

f23-i01-d05-searchspace f23-i01-d05-searchspace-projected

f23-i01-d05-logobjspace f23-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 23 in dimension 5 for the first instance.

f_{24}: Attractive sector/Rastrigin

Combination of the attractive sector function (f_6 in the bbob suite) and the Rastrigin function (f_{15} in the bbob suite).

Both objectives are non-separable, and the second one is highly multi-modal (roughly 10^n local optima) while the first one is unimodal. Further properties are that the first objective is highly assymetric and the second has a conditioning of about 10.

Contained in the moderate - multi-modal function class.

f24-i01-d05-searchspace f24-i01-d05-searchspace-projected

f24-i01-d05-logobjspace f24-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 24 in dimension 5 for the first instance.

f_{25}: Attractive sector/Schaffer F7, condition 10

Combination of the attractive sector function (f_6 in the bbob suite) and the Schaffer F7 function with condition number 10 (f_{17} in the bbob suite).

Both objectives are non-separable and asymmetric. While the first objective is unimodal, the second one is a highly multi-modal function with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the moderate - multi-modal function class.

f25-i01-d05-searchspace f25-i01-d05-searchspace-projected

f25-i01-d05-logobjspace f25-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 25 in dimension 5 for the first instance.

f_{26}: Attractive sector/Schwefel x*sin(x)

Combination of the attractive sector function (f_6 in the bbob suite) and the Schwefel function (f_{20} in the bbob suite).

The first objective is non-separable, unimodal, and asymmetric. The second objective is partially separable and highly multimodal—having the most prominent 2^n minima located comparatively close to the corners of the unpenalized search area.

Contained in the moderate - weakly-structured function class.

f26-i01-d05-searchspace f26-i01-d05-searchspace-projected

f26-i01-d05-logobjspace f26-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 26 in dimension 5 for the first instance.

f_{27}: Attractive sector/Gallagher 101 peaks

Combination of the attractive sector function (f_6 in the bbob suite) and the Gallagher function with 101 peaks (f_{21} in the bbob suite).

Both objective functions are non-separable but only the first is unimodal. The first objective function is furthermore asymmetric. The second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the moderate - weakly-structured function class.

f27-i01-d05-searchspace f27-i01-d05-searchspace-projected

f27-i01-d05-logobjspace f27-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 27 in dimension 5 for the first instance.

f_{28}: Rosenbrock original/Rosenbrock original

Combination of two Rosenbrock functions (f_8 in the bbob suite).

Both objectives are partially separable (tri-band structure) and have a local optimum with an attraction volume of about 25%.

Contained in the moderate - moderate function class.

f28-i01-d05-searchspace f28-i01-d05-searchspace-projected

f28-i01-d05-logobjspace f28-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 28 in dimension 5 for the first instance.

f_{29}: Rosenbrock original/Sharp ridge

Combination of the Rosenbrock function (f_8 in the bbob suite) and the sharp ridge function (f_{13} in the bbob suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective is unimodal and non-separable and, for optimizing it, a sharp, i.e., non-differentiable ridge has to be followed.

Contained in the moderate - ill-conditioned function class.

f29-i01-d05-searchspace f29-i01-d05-searchspace-projected

f29-i01-d05-logobjspace f29-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 29 in dimension 5 for the first instance.

f_{30}: Rosenbrock original/Sum of different powers

Combination of the Rosenbrock function (f_8 in the bbob suite) and the sum of different powers function (f_{14} in the bbob suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective function is unimodal and non-separable. When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.

Contained in the moderate - ill-conditioned function class.

f30-i01-d05-searchspace f30-i01-d05-searchspace-projected

f30-i01-d05-logobjspace f30-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 30 in dimension 5 for the first instance.

f_{31}: Rosenbrock original/Rastrigin

Combination of the Rosenbrock function (f_8 in the bbob suite) and the Rastrigin function (f_{15} in the bbob suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective function is non-separable and highly multi-modal (roughly 10^n local optima).

Contained in the moderate - multi-modal function class.

f31-i01-d05-searchspace f31-i01-d05-searchspace-projected

f31-i01-d05-logobjspace f31-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 31 in dimension 5 for the first instance.

f_{32}: Rosenbrock original/Schaffer F7, condition 10

Combination of the Rosenbrock function (f_8 in the bbob suite) and the Schaffer F7 function with condition number 10 (f_{17} in the bbob suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective function is non-separable, asymmetric, and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the moderate - multi-modal function class.

f32-i01-d05-searchspace f32-i01-d05-searchspace-projected

f32-i01-d05-logobjspace f32-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 32 in dimension 5 for the first instance.

f_{33}: Rosenbrock original/Schwefel x*sin(x)

Combination of the Rosenbrock function (f_8 in the bbob suite) and the Schwefel function (f_{20} in the bbob suite).

Both objective functions are partially separable. While the first objective function has a local optimum with an attraction volume of about 25%, the second objective function is highly multimodal—having the most prominent 2^n minima located comparatively close to the corners of its unpenalized search area.

Contained in the moderate - weakly-structured function class.

f33-i01-d05-searchspace f33-i01-d05-searchspace-projected

f33-i01-d05-logobjspace f33-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 33 in dimension 5 for the first instance.

f_{34}: Rosenbrock original/Gallagher 101 peaks

Combination of the Rosenbrock function (f_8 in the bbob suite) and the Gallagher function with 101 peaks (f_{21} in the bbob suite).

The first objective function is partially separable, the second one non-separable. While the first objective function has a local optimum with an attraction volume of about 25%, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the moderate - weakly-structured function class.

f34-i01-d05-searchspace f34-i01-d05-searchspace-projected

f34-i01-d05-logobjspace f34-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 34 in dimension 5 for the first instance.

f_{35}: Sharp ridge/Sharp ridge

Combination of two sharp ridge functions (f_{13} in the bbob suite).

Both objective functions are unimodal and non-separable and, for optimizing them, two sharp, i.e., non-differentiable ridges have to be followed.

Contained in the ill-conditioned - ill-conditioned function class.

f35-i01-d05-searchspace f35-i01-d05-searchspace-projected

f35-i01-d05-logobjspace f35-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 35 in dimension 5 for the first instance.

f_{36}: Sharp ridge/Sum of different powers

Combination of the sharp ridge function (f_{13} in the bbob suite) and the sum of different powers function (f_{14} in the bbob suite).

Both functions are uni-modal and non-separable. For optimizing the first objective, a sharp, i.e., non-differentiable ridge has to be followed. When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.

Contained in the ill-conditioned - ill-conditioned function class.

f36-i01-d05-searchspace f36-i01-d05-searchspace-projected

f36-i01-d05-logobjspace f36-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 36 in dimension 5 for the first instance.

f_{37}: Sharp ridge/Rastrigin

Combination of the sharp ridge function (f_{13} in the bbob suite) and the Rastrigin function (f_{15} in the bbob suite).

Both functions are non-separable. While the first one is unimodal and non-differentiable at its ridge, the second objective function is highly multi-modal (roughly 10^n local optima).

Contained in the ill-conditioned - multi-modal function class.

f37-i01-d05-searchspace f37-i01-d05-searchspace-projected

f37-i01-d05-logobjspace f37-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 37 in dimension 5 for the first instance.

f_{38}: Sharp ridge/Schaffer F7, condition 10

Combination of the sharp ridge function (f_{13} in the bbob suite) and the Schaffer F7 function with condition number 10 (f_{17} in the bbob suite).

Both functions are non-separable. While the first one is unimodal and non-differentiable at its ridge, the second objective function is asymmetric and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the ill-conditioned - multi-modal function class.

f38-i01-d05-searchspace f38-i01-d05-searchspace-projected

f38-i01-d05-logobjspace f38-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 38 in dimension 5 for the first instance.

f_{39}: Sharp ridge/Schwefel x*sin(x)

Combination of the sharp ridge function (f_{13} in the bbob suite) and the Schwefel function (f_{20} in the bbob suite).

While the first objective function is unimodal, non-separable, and non-differentiable at its ridge, the second objective function is highly multimodal—having the most prominent 2^n minima located comparatively close to the corners of its unpenalized search area.

Contained in the ill-conditioned - weakly-structured function class.

f39-i01-d05-searchspace f39-i01-d05-searchspace-projected

f39-i01-d05-logobjspace f39-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 39 in dimension 5 for the first instance.

f_{40}: Sharp ridge/Gallagher 101 peaks

Combination of the sharp ridge function (f_{13} in the bbob suite) and the Gallagher function with 101 peaks (f_{21} in the bbob suite).

Both objective functions are non-separable. While the first objective function is unimodal and non-differentiable at its ridge, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the ill-conditioned - weakly-structured function class.

f40-i01-d05-searchspace f40-i01-d05-searchspace-projected

f40-i01-d05-logobjspace f40-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 40 in dimension 5 for the first instance.

f_{41}: Sum of different powers/Sum of different powers

Combination of two sum of different powers functions (f_{14} in the bbob suite).

Both functions are uni-modal and non-separable where the sensitivies of the variables in the rotated search space become more and more different when approaching the objectives’ optima.

Contained in the ill-conditioned - ill-conditioned function class.

f41-i01-d05-searchspace f41-i01-d05-searchspace-projected

f41-i01-d05-logobjspace f41-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 41 in dimension 5 for the first instance.

f_{42}: Sum of different powers/Rastrigin

Combination of the sum of different powers functions (f_{14} in the bbob suite) and the Rastrigin function (f_{15} in the bbob suite).

Both objective functions are non-separable. While the first one is unimodal, the second objective function is highly multi-modal (roughly 10^n local optima).

Contained in the ill-conditioned - multi-modal function class.

f42-i01-d05-searchspace f42-i01-d05-searchspace-projected

f42-i01-d05-logobjspace f42-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 42 in dimension 5 for the first instance.

f_{43}: Sum of different powers/Schaffer F7, condition 10

Combination of the sum of different powers functions (f_{14} in the bbob suite) and the Schaffer F7 function with condition number 10 (f_{17} in the bbob suite).

Both objective functions are non-separable. While the first one is unimodal with an increasing conditioning once the optimum is approached, the second objective function is asymmetric and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the ill-conditioned - multi-modal function class.

f43-i01-d05-searchspace f43-i01-d05-searchspace-projected

f43-i01-d05-logobjspace f43-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 43 in dimension 5 for the first instance.

f_{44}: Sum of different powers/Schwefel x*sin(x)

Combination of the sum of different powers functions (f_{14} in the bbob suite) and the Schwefel function (f_{20} in the bbob suite).

Both objectives are non-separable. While the first objective function is unimodal, the second objective function is highly multimodal—having the most prominent 2^n minima located comparatively close to the corners of its unpenalized search area.

Contained in the ill-conditioned - weakly-structured function class.

f44-i01-d05-searchspace f44-i01-d05-searchspace-projected

f44-i01-d05-logobjspace f44-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 44 in dimension 5 for the first instance.

f_{45}: Sum of different powers/Gallagher 101 peaks

Combination of the sum of different powers functions (f_{14} in the bbob suite) and the Gallagher function with 101 peaks (f_{21} in the bbob suite).

Both objective functions are non-separable. While the first objective function is unimodal, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the ill-conditioned - weakly-structured function class.

f45-i01-d05-searchspace f45-i01-d05-searchspace-projected

f45-i01-d05-logobjspace f45-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 45 in dimension 5 for the first instance.

f_{46}: Rastrigin/Rastrigin

Combination of two Rastrigin functions (f_{15} in the bbob suite).

Both objective functions are non-separable and highly multi-modal (roughly 10^n local optima).

Contained in the multi-modal - multi-modal function class.

f46-i01-d05-searchspace f46-i01-d05-searchspace-projected

f46-i01-d05-logobjspace f46-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 46 in dimension 5 for the first instance.

f_{47}: Rastrigin/Schaffer F7, condition 10

Combination of the Rastrigin function (f_{15} in the bbob suite) and the Schaffer F7 function with condition number 10 (f_{17} in the bbob suite).

Both objective functions are non-separable and highly multi-modal.

Contained in the multi-modal - multi-modal function class.

f47-i01-d05-searchspace f47-i01-d05-searchspace-projected

f47-i01-d05-logobjspace f47-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 47 in dimension 5 for the first instance.

f_{48}: Rastrigin/Schwefel x*sin(x)

Combination of the Rastrigin function (f_{15} in the bbob suite) and the Schwefel function (f_{20} in the bbob suite).

Both objective functions are non-separable and highly multi-modal where the first has roughly 10^n local optima and the most prominent 2^n minima of the second objective function are located comparatively close to the corners of its unpenalized search area.

Contained in the multi-modal - weakly-structured function class.

f48-i01-d05-searchspace f48-i01-d05-searchspace-projected

f48-i01-d05-logobjspace f48-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 48 in dimension 5 for the first instance.

f_{49}: Rastrigin/Gallagher 101 peaks

Combination of the Rastrigin function (f_{15} in the bbob suite) and the Gallagher function with 101 peaks (f_{21} in the bbob suite).

Both objective functions are non-separable and highly multi-modal where the first has roughly 10^n local optima and the second has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function).

Contained in the multi-modal - weakly-structured function class.

f49-i01-d05-searchspace f49-i01-d05-searchspace-projected

f49-i01-d05-logobjspace f49-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 49 in dimension 5 for the first instance.

f_{50}: Schaffer F7, condition 10/Schaffer F7, condition 10

Combination of two Schaffer F7 functions with condition number 10 (f_{17} in the bbob suite).

Both objective functions are non-separable and highly multi-modal.

Contained in the multi-modal - multi-modal function class.

f50-i01-d05-searchspace f50-i01-d05-searchspace-projected

f50-i01-d05-logobjspace f50-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 50 in dimension 5 for the first instance.

f_{51}: Schaffer F7, condition 10/Schwefel x*sin(x)

Combination of the Schaffer F7 function with condition number 10 (f_{17} in the bbob suite) and the Schwefel function (f_{20} in the bbob suite).

Both objective functions are non-separable and highly multi-modal. While frequency and amplitude of the modulation vary in an almost regular fashion in the first objective function, the second objective function posseses less global structure.

Contained in the multi-modal - weakly-structured function class.

f51-i01-d05-searchspace f51-i01-d05-searchspace-projected

f51-i01-d05-logobjspace f51-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 51 in dimension 5 for the first instance.

f_{52}: Schaffer F7, condition 10/Gallagher 101 peaks

Combination of the Schaffer F7 function with condition number 10 (f_{17} in the bbob suite) and the Gallagher function with 101 peaks (f_{21} in the bbob suite).

Both objective functions are non-separable and highly multi-modal. While frequency and amplitude of the modulation vary in an almost regular fashion in the first objective function, the second has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function).

Contained in the multi-modal - weakly-structured function class.

f52-i01-d05-searchspace f52-i01-d05-searchspace-projected

f52-i01-d05-logobjspace f52-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 52 in dimension 5 for the first instance.

f_{53}: Schwefel x*sin(x)/Schwefel x*sin(x)

Combination of two Schwefel functions (f_{20} in the bbob suite).

Both objective functions are non-separable and highly multi-modal where the most prominent 2^n minima of each objective function are located comparatively close to the corners of its unpenalized search area. Due to the combinatorial nature of the Schwefel function, it is likely in low dimensions that the Pareto set goes through the origin of the search space.

Contained in the weakly-structured - weakly-structured function class.

f53-i01-d05-searchspace f53-i01-d05-searchspace-projected

f53-i01-d05-logobjspace f53-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 53 in dimension 5 for the first instance.

f_{54}: Schwefel x*sin(x)/Gallagher 101 peaks

Combination of the Schwefel function (f_{20} in the bbob suite) and the Gallagher function with 101 peaks (f_{21} in the bbob suite).

Both objective functions are non-separable and highly multi-modal. For the first objective function, the most prominent 2^n minima are located comparatively close to the corners of its unpenalized search area. For the second objective, position and height of all 101 optima are unrelated and randomly chosen (different for each instantiation of the function).

Contained in the weakly-structured - weakly-structured function class.

f54-i01-d05-searchspace f54-i01-d05-searchspace-projected

f54-i01-d05-logobjspace f54-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 54 in dimension 5 for the first instance.

f_{55}: Gallagher 101 peaks/Gallagher 101 peaks

Combination of two Gallagher functions with 101 peaks (f_{21} in the bbob suite).

Both objective functions are non-separable and highly multi-modal. Position and height of all 101 optima in each objective function are unrelated and randomly chosen and thus, no global structure is present.

Contained in the weakly-structured - weakly-structured function class.

f55-i01-d05-searchspace f55-i01-d05-searchspace-projected

f55-i01-d05-logobjspace f55-i01-d05-objspace

Illustration of search space (first row) and objective space (second row) for bbob-biobj function 55 in dimension 5 for the first instance.

The Extended bbob-biobj-ext Test Suite and Its Functions

Having all combinations of only a subset of the single-objective bbob functions in a test suite like the above bbob-biobj one has advantages but also a few disadvantages. Using only a subet of the 24 bbob functions introduces a bias towards the chosen functions and reduces the amount of different difficulties, a bi-objective algorithm is exposed to in the benchmarking exercise. Allowing all combinations of bbob functions increases the percentage of problems for which both objectives are from different bbob function groups while, in practice, it can often be assumed that both objective functions come from a similar “function domain”.

The rationale behind the following extended bbob-biobj test suite, denoted as bbob-biobj-ext, is to reduce the mentioned effects. To this end, we add all within-group combinations of bbob functions which are not already in the bbob-biobj suite and which do not combine a function with itself. For technical reasons, we also remove the Weierstrass functions (f_{16} in the bbob suite) because the optimum is not necessarily unique and computing the nadir point therefore technically more challenging than for the other functions. This extension adds 3*(4+3+2+1-1) + 2*(3+2+1-1) = 3*9+2*5=37 functions, resulting in 92 functions overall.

The following table details which single-objective bbob functions are contained in the 92 bbob-biobj-ext functions (outer column and row annotations) and indicates their IDs. Note that the IDs of the first 55 bbob0biobj-ext functions are the same than for the bbob-biobj test suite for compatibility reasons.

  f_1 f_2 f_3 f_4 f_5 f_6 f_7 f_8 f_9 f_{10} f_{11} f_{12} f_{13} f_{14} f_{15} f_{16} f_{17} f_{18} f_{19} f_{20} f_{21} f_{22} f_{23} f_{24}
f_1 f1 f2 f56 f57 f58 f3   f4         f5 f6 f7   f8     f9 f10      
f_2   f11 f59 f60 f61 f12   f13         f14 f15 f16   f17     f18 f19      
f_3       f62 f63                                      
f_4         f64                                      
f_5                                                
f_6           f20 f65 f21 f66       f22 f23 f24   f25     f26 f27      
f_7               f67 f68                              
f_8               f28 f69       f29 f30 f31   f32     f33 f34      
f_9                                                
f_{10}                     f70 f71 f72 f73                    
f_{11}                       f74 f75 f76                    
f_{12}                         f77 f78                    
f_{13}                         f35 f36 f37   f38     f39 f40      
f_{14}                           f41 f42   f43     f44 f45      
f_{15}                             f46   f47 f79 f80 f48 f49      
f_{16}                                                
f_{17}                                 f50 f81 f82 f51 f52      
f_{18}                                     f83          
f_{19}                                                
f_{20}                                       f53 f54 f84 f85 f86
f_{21}                                         f55 f87 f88 f89
f_{22}                                             f90 f91
f_{23}                                               f92
f_{24}                                                
The 92 functions of the bbob-biobj-ext test suite and their IDs (in the table cells) together with the information about which single-objective bbob functions are used to define them (outer column and row annotations).

Function Groups

Like for the bbob-biobj test suite, we obtain 15 function classes to structure the 92 bi-objective functions of the bbob-biobj-ext test suite. Depending on whether a function class combines functions from the same or from different bbob function classes, each function class contains 8, 12 or just four functions. We are listing below the function classes and in parenthesis the functions that belong to the respective class:

  1. separable - separable (12 functions: f1, f2, f11, f56-64)
  2. separable - moderate (f3, f4, f12, f13)
  3. separable - ill-conditioned (f5, f6, f14, f15)
  4. separable - multi-modal (f7, f8, f16, f17)
  5. separable - weakly-structured (f9, f10, f18, f19)
  6. moderate - moderate (8 functions: f20, f21, f28, f65-f69)
  7. moderate - ill-conditioned (f22, f23, f29, f30)
  8. moderate - multi-modal (f24, f25, f31, f32)
  9. moderate - weakly-structured (f26, f27, f33, f34)
  10. ill-conditioned - ill-conditioned (12 functions: f35, f36, f41, f70-78)
  11. ill-conditioned - multi-modal (f37, f38, f42, f43)
  12. ill-conditioned - weakly-structured (f39, f40, f44, f45)
  13. multi-modal - multi-modal (8 functions: f46, f47, f50, f79-83)
  14. multi-modal - weakly structured (f48, f49, f51, f52)
  15. weakly structured - weakly structured (12 functions: f53-55, f84-92)

Normalization and Instances

Normalization of the objectives and instances are handled for the bbob-biobj-ext in the same manner as for the bbob-biobj suite, i.e., no normalization of the objective functions is taking place for the algorithm benchmarking and 15 instances are prescribed for a typical experiment.

Acknowledgments

This work was supported by the grant ANR-12-MONU-0009 (NumBBO) of the French National Research Agency. We also thank Ilya Loshchilov and Oswin Krause for their initial suggestions on how to extend the bbob-biobj test suite.

References

[BRO2016biperf]D. Brockhoff, T. Tušar, D. Tušar, T. Wagner, N. Hansen, A. Auger, (2016). Biobjective Performance Assessment with the COCO Platform. ArXiv e-prints, arXiv:1605.01746.
[BRO2015](1, 2) D. Brockhoff, T.-D. Tran, and N. Hansen (2015). Benchmarking Numerical Multiobjective Optimizers Revisited. In Proceedings of the 2015 GECCO Genetic and Evolutionary Computation Conference, pp. 639-646, ACM.
[HAN2016co](1, 2) N. Hansen, A. Auger, O. Mersmann, T. Tušar, D. Brockhoff (2016). COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting, ArXiv e-prints, arXiv:1603.08785.
[HAN2009](1, 2) N. Hansen, S. Finck, R. Ros, and A. Auger (2009). Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions. Research Report RR-6829, Inria, updated February 2010.
[HAN2011]N. Hansen, R. Ros, N. Mauny, M. Schoenauer, and A. Auger (2011). Impacts of Invariance in Search: When CMA-ES and PSO Face Ill-Conditioned and Non-Separable Problems. Applied Soft Computing. Vol. 11, pp. 5755-5769. Elsevier.
[HAN2016ex]N. Hansen, T. Tušar, A. Auger, D. Brockhoff, O. Mersmann (2016). COCO: The Experimental Procedure, ArXiv e-prints, arXiv:1603.08776.