# COCO: The Bi-objective Black-Box Optimization Benchmarking (`bbob-biobj`

) Test Suite¶

See also: *ArXiv e-prints*, arXiv:1604.00359, 2016.

Several test function suites for numerical benchmarking of multiobjective
optimization algorithms have been proposed in recent years. While having
desirable properties like well-understood Pareto sets and Pareto fronts with shapes
of various kinds, most of the currently used functions posess properties
which are arguably under-represented in real-world problems. Those properties
mainly stem from the easier construction of such problems—overrepresenting
properties such as no dependencies between variables, Pareto sets exactly located at
the bound constraints, or the differentiation between position and distance
variables. Here, we propose an alternative way and define the `bbob-biobj`

test suite
with 55 bi-objective functions and its extended `bbob-biobj-ext`

version with
92 bi-objective functions in continuous domain which are both derived from
combining functions of the well-known single-objective noiseless `bbob`

test suite.
Besides giving the actual function definitions and presenting their (known) properties, this
documentation also aims at giving the rationale behind our approach in terms
of function groups, instances, and potential objective space normalization.

## Preliminaries, Definitions, and Scope¶

In the following, we consider bi-objective, unconstrained
**minimization** problems of the form

where is the number of variables of the problem (also called
the problem dimension),
and are the two
objective functions, and the operator is related to the
standard *dominance relation*. A solution
is thereby said to *dominate* another solution if
and hold and at
least one of the inequalities is strict.

Solutions which are not dominated by any other solution in the search
space are called *Pareto-optimal* or *efficient solutions*. All
Pareto-optimal solutions constitute the *Pareto set* of which an
approximation is sought. The Pareto set’s image in the
objective space is called *Pareto front*.

The objective of the minimization problem is to find, with as few evaluations of as possible, a set of non-dominated solutions which is (i) as large as possible and (ii) has -values as close to the Pareto front as possible. [1]

[1] | Distance in -space is defined here such that nadir and ideal point have in each coordinate distance one. Neither of these points is however freely accessible to the optimization algorithm. |

In the following, we remind useful definitions.

*function instance, problem*Each function within COCO is parametrized with parameter values . A parameter value determines a so-called

*function instance*. For example, encodes the location of the optimum of single-objective functions, which means that different instances have shifted optima. In the`bbob-biobj`

test suite, and the function instances are determined by the instances of the underlying single-objective functions.A

*problem*is a function instance of a specific dimension .*ideal point*The ideal point is defined as the vector in objective space that contains the optimal -value for each objective

*independently*. More precisely let and , the ideal point is given by*nadir point*The

*nadir point*(in objective space) consists in each objective of the worst value obtained by a Pareto-optimal solution. More precisely, let be the set of Pareto optimal points. Then the nadir point satisfiesIn the case of two objectives with a unique global minimum each (that is, a single point in the search space maps to the global minimum)

where and .

## Overview of the Proposed `bbob-biobj`

Test Suite¶

The `bbob-biobj`

test suite provides 55 bi-objective functions in six
dimensions (2, 3, 5, 10, 20, and 40) with a large number of possible instances.
The 55 functions are derived from combining a subset of the 24 well-known
single-objective functions of the `bbob`

test suite [HAN2009] which
has been used since 2009 in the BBOB workshop series.
While concrete details on each of
the 55 `bbob-biobj`

functions are given in Section
The bbob-biobj Test Functions and Their Properties, we will detail here the main rationale behind
them together with their common properties.

### The Single-objective `bbob`

Functions¶

The `bbob-biobj`

test suite is designed to be able to assess performance of algorithms with respect to well-identified difficulties in optimization typically occurring in real-world problems. A multi-objective problem being a combination of single-objective problems, one can obtain multi-objective problems with representative difficulties by simply combining single objective functions with representative difficulties observed in real-world problems. For this purpose we naturally use the single-objective `bbob`

suite [HAN2009].

Combining all 24 `bbob`

functions in pairs thereby results in
bi-objective functions overall. We however assume that
multi-objective optimization algorithms are not sensitive to permutations of
the objective functions such that combining the 24 `bbob`

functions and
taking out the function if the function
is present results in functions.

Some first tests, e.g. in [BRO2015], showed that having 300 functions is
impracticable in terms of the overall running time of the benchmarking
experiment. We then decided to exploit the organization of the `bbob`

functions into classes to choose a subset of functions. More precisely, the 24
original `bbob`

functions are grouped into five function classes where each
class gathers functions with similar properties, namely

- separable functions
- functions with low or moderate conditioning
- functions with high conditioning and unimodal
- multi-modal functions with adequate global structure,
- multi-modal functions with weak global structure.

To create the `bbob-biobj`

suite, we choose two functions within each class. This way we do not introduce any bias towards a specific class. In addition within each class, the functions are chosen to be the most
representative without repeating similar functions. For example,
only one Ellipsoid, one Rastrigin, and one Gallagher function are
included in the `bbob-biobj`

suite although they appear in
separate versions in the `bbob`

suite. Finally our choice of 10 `bbob`

functions for creating the `bbob-biobj`

test suite is the following:

- Separable functions
- Sphere (function 1 in
`bbob`

suite) - Ellipsoid separable (function 2 in
`bbob`

suite)

- Sphere (function 1 in
- Functions with low or moderate conditioning
- Attractive sector (function 6 in
`bbob`

suite) - Rosenbrock original (function 8 in
`bbob`

suite)

- Attractive sector (function 6 in
- Functions with high conditioning and unimodal
- Sharp ridge (function 13 in
`bbob`

suite) - Sum of different powers (function 14 in
`bbob`

suite)

- Sharp ridge (function 13 in
- Multi-modal functions with adequate global structure
- Rastrigin (function 15 in
`bbob`

suite) - Schaffer F7, condition 10 (function 17 in
`bbob`

suite)

- Rastrigin (function 15 in
- Multi-modal functions with weak global structure
- Schwefel x*sin(x) (function 20 in
`bbob`

suite) - Gallagher 101 peaks (function 21 in
`bbob`

suite)

- Schwefel x*sin(x) (function 20 in

Using the above described pairwise combinations, this results in having bi-objective functions in the final bbob-biobj suite. These functions are denoted to in the sequel.

### Function Groups¶

From combining the original `bbob`

function classes, we obtain 15 function
classes to structure the 55 bi-objective functions of the `bbob-biobj`

test
suite. Each function class contains three or four functions. We are listing
below the function classes and in parenthesis the functions that belong to
the respective class:

- separable - separable (functions , , )
- separable - moderate (, , , )
- separable - ill-conditioned (, , , )
- separable - multi-modal (, , , )
- separable - weakly-structured (, , , )
- moderate - moderate (, , )
- moderate - ill-conditioned (, , , )
- moderate - multi-modal (, , , )
- moderate - weakly-structured (, , , )
- ill-conditioned - ill-conditioned (, , )
- ill-conditioned - multi-modal (, , , )
- ill-conditioned - weakly-structured (, , , )
- multi-modal - multi-modal (, , )
- multi-modal - weakly structured (, , , )
- weakly structured - weakly structured (, , )

More details about the single functions can be found in Section The bbob-biobj Test Functions and Their Properties. We however first describe their common properties in the coming sections.

### Normalization of Objectives¶

None of the 55 `bbob-biobj`

functions is explicitly normalized and the
optimization algorithms therefore have to cope with objective values in
different ranges. Typically, different orders of magnitude
between the objective values can be observed.

However, to facilitate comparison of algorithm performance over different functions,
we normalize the objectives based on the ideal and nadir points
before calculating the hypervolume indicator [BRO2016biperf].
Both points can be computed, because the global
optimum is known and is unique for the 10 `bbob`

base functions.
In the black-box optimization benchmarking setup, however, the values of the
ideal and nadir points are not accessible to the optimization algorithm
[HAN2016ex].

### Instances¶

Our test functions are parametrized and instances are instantiations of the underlying parameters (see [HAN2016co]). The instances for the bi-objective functions are obtained using instances of each single objective function composing the bi-objective one. In addition, we assert two conditions:

1. The Euclidean distance between the two single-objective optima (also called the extreme optimal points) in the search space is at least .

2. The Euclidean distance between the ideal and the nadir point in the non-normalized objective space is at least .

We associate to an instance, an instance-id which is an integer. The relation between the instance-id, , of a bi-objective function and the instance-ids, and , of its underlying single-objective functions and is the following:

- and

If we find that above conditions are not satisfied for all dimensions and
functions in the `bbob-biobj`

suite, we increase the instance-id of the
second objective successively until both properties are fulfilled.
For example, the `bbob-biobj`

instance-id
8 corresponds to the instance-id 17 for the first objective and instance-id 18 for
the second objective while for the `bbob-biobj`

instance-id 9, the
first instance-id is 19 but for the second objective, instance-id 21 is chosen
instead of instance-id 20.

Exceptions to the above rule are, for historical reasons, the
`bbob-biobj`

instance-ids 1 and 2 in order to match the instance-ids
1 to 5 with the ones proposed in [BRO2015]. The `bbob-biobj`

instance-id 1 contains the single-objective instance-ids 2 and 4 and
the `bbob-biobj`

instance-id 2 contains the two instance-ids 3 and 5.

For each bi-objective function and given dimension, the `bbob-biobj`

suite
contains 10 instances. [2]

[2] | In principle, as for the instance generation for the `bbob` suite,
the number of possible instances for the `bbob-biobj` suite is unlimited
[HAN2016co].
However, running some tests with too few instances will render the
potential statistics and their interpretation problematic while even the
tiniest observed difference can be made statistically significant with a
high enough number of instances. A good compromise to avoid either pitfall
seems to lie between, say, 9 and 19 instances. |

## The `bbob-biobj`

Test Functions and Their Properties¶

In the following, we detail all 55 `bbob-biobj`

functions
and their properties.

The following table gives an overview and quick access to the functions,
inner cell IDs refer to the `bbob-biobj`

functions, outer column and row
annotations refer to the single-objective `bbob`

functions.

f1 | f2 | f3 | f4 | f5 | f6 | f7 | f8 | f9 | f10 | |

f11 | f12 | f13 | f14 | f15 | f16 | f17 | f18 | f19 | ||

f20 | f21 | f22 | f23 | f24 | f25 | f26 | f27 | |||

f28 | f29 | f30 | f31 | f32 | f33 | f34 | ||||

f35 | f36 | f37 | f38 | f39 | f40 | |||||

f41 | f42 | f43 | f44 | f45 | ||||||

f46 | f47 | f48 | f49 | |||||||

f50 | f51 | f52 | ||||||||

f53 | f54 | |||||||||

f55 |

### Some Function Properties¶

In the description of the 55 `bbob-biobj`

functions below, several
general properties of objective functions will be mentioned that
are defined here in short. It depends on these properties whether the optimization problem
is easy or hard to solve.

A *separable* function does not show any dependencies between the
variables and can therefore be solved by applying consecutive
one-dimensional optimizations along the coordinate axes while
keeping the other variables fixed. Consequently, *non-separable*
problems must be considered. They are much more difficult to solve. The
typical well-established technique to generate non-separable
functions from separable ones is the application of a rotation matrix
to , that is ,
where is a separable function.

A *unimodal* function has only one local minimum which is at the same
time also its global one.
A *multimodal* function has at least two local minima which is highly common
in practical optimization problems.

*Ill-conditioning* is another typical challenge in real-parameter
optimization and, besides multimodality, probably the most common one.
In a general case, we can consider a function as ill-conditioned if for
solution points from the same level-set “the minimal displacement [...] that
produces a given function value improvement differs by
orders of magnitude” [HAN2011].
Conditioning can be rigorously formalized in the
case of convex quadratic functions,
where is a symmetric
positive definite matrix, as the condition number of the Hessian matrix
. Since contour lines associated to a convex quadratic function
are ellipsoids, the condition number corresponds to the square root of
the ratio between the largest axis of the ellipsoid and the shortest axis.

The proposed `bbob-biobj`

testbed contains ill-conditioned functions
with a typical conditioning of . We believe this is a realistic
requirement, while we have seen practical problems with conditioning
as large as .

### Domain Bounds¶

All bi-objective functions provided in the `bbob-biobj`

suite are unbounded,
i.e., defined on the entire real-valued space .
The search domain of interest is defined as , outside of
which non-dominated solutions are quite unlikely to be found. [3] The majority of non-dominated solutions are likely to lie even within .

While we believe that the domain of interest contains the Pareto set,
due to the nature of the `bbob-biobj`

function definitions, there is no
guarantee that this is always the case.
However, the extremal solutions and their neighborhood ball of radius one
are guaranteed to lie within .

[3] | The functions `coco_problem_get_smallest_value_of_interest` and
`coco_problem_get_largest_value_of_interest`
of the COCO platform allow the optimizer
to retrieve the search domain of interest from the `coco_problem_t` ,
for example to generate the initial search points. |

### Provided Search Space and Objective Space Plots¶

In order to better understand the properties of the 55 `bbob-biobj`

functions, we display for each of them plots
of the best known Pareto front approximation in objective space in original scaling (as seen by the algorithm) and in
log-scale, normalized such that the ideal point is at and the nadir point is at . We
also provide plots illustrating the best known Pareto set approximation in search space (all depicted in black).
For the latter, two different plots are provided: a plot showing the projection onto a coordinate-axes-parallel cut
defined by two variables and a plot that projects all points onto a random cutting plane which contains both
single-objective optima and that also shows the contour lines of both objective functions on this plane.

In addition to the best Pareto set/Pareto front approximations, cuts through the search space are shown along (i) random lines through each optimum (in blue), (ii) lines along each coordinate axis through each optimum (blue dotted lines), (iii) the line through both optima (in red), (iv) two fully random lines [4] (in yellow), and (v) a random line in the random projection plane going through both optima [5] (in green).

All lines are normalized (of length 10 with the support vector in the middle). Ticks along the lines in the objective space plots indicate the ends of line segments of the same length in search space. Thicker points on the lines depict solutions that are non-dominated with respect to all points on the same line. Furthermore, the search space plots highlight the projected region as gray-shaded area while the gray-shaded area in the objective space plots highlight the region of interest between ideal () and nadir point (). Note that, to keep the plots to a manageable size, the Pareto set and Pareto front approximations are carefully downsampled such that only one solution per grid point is shown—with the precision of 2 decimals for the search space plots and 3 decimals for the objective space plots to define the grid. The number of considered and actually displayed solutions is indicated in the search space plots’ legends. All plots are provided for one instance here only and for dimension 5 for the moment.

[4] | of random direction and with a support vector, drawn uniformly at random in |

[5] | with a random direction within the plane and a support vector, drawn uniformly at random in in the coordinate system of the cutting plane` |

### The 55 `bbob-biobj`

Functions¶

#### : Sphere/Sphere¶

Combination of two sphere functions ( in the `bbob`

suite).

Both objectives are unimodal, highly symmetric, rotational and scale invariant. The Pareto set is known to be a straight line and the Pareto front is convex. Furthermore, the normalized hypervolume value of the entire Pareto front with respect to the nadir point as reference point can be computed analytically as the integral .

Considered as the simplest bi-objective problem in continuous domain.

Contained in the *separable - separable* function class.

`bbob-biobj`

function 1 in dimension 5 for the first instance.#### : Sphere/Ellipsoid separable¶

Combination of the sphere function ( in the `bbob`

suite)
and the separable ellipsoid function ( in the `bbob`

suite).

Both objectives are unimodal and separable. While the first objective is truly convex-quadratic with a condition number of 1, the second objective is only globally quadratic with smooth local irregularities and highly ill-conditioned with a condition number of about .

Contained in the *separable - separable* function class.

`bbob-biobj`

function 2 in dimension 5 for the first instance.#### : Sphere/Attractive sector¶

Combination of the sphere function ( in the `bbob`

suite)
and the attractive sector function ( in the `bbob`

suite).

Both objective functions are unimodal, but only the first objective is
separable and truly convex quadratic. The attractive sector
function is highly asymmetric, where only one *hypercone* (with
angular base area) with a volume of roughly
yields low function values. The optimum of it is located at the tip
of this cone.

Contained in the *separable - moderate* function class.

`bbob-biobj`

function 3 in dimension 5 for the first instance.#### : Sphere/Rosenbrock original¶

Combination of the sphere function ( in the `bbob`

suite)
and the original, i.e., unrotated Rosenbrock function ( in the `bbob`

suite).

The first objective is separable and truly convex, the second objective is partially separable (tri-band structure). The first objective is unimodal while the second objective has a local optimum with an attraction volume of about 25%.

Contained in the *separable - moderate* function class.

`bbob-biobj`

function 4 in dimension 5 for the first instance.#### : Sphere/Sharp ridge¶

Combination of the sphere function ( in the `bbob`

suite)
and the sharp ridge function ( in the `bbob`

suite).

Both objective functions are unimodal.
In addition to the simple, separable, and differentiable first
objective, a sharp, i.e., non-differentiable ridge has to be
followed for optimizing the (non-separable) second objective. The
gradient towards the ridge remains constant, when the ridge is
approached from a given point.
Approaching the ridge is initially effective, but becomes ineffective
close to the ridge when the rigde needs to be followed in direction
to its optimum. The necessary change in *search behavior* close to
the ridge is difficult to diagnose, because the gradient
towards the ridge does not flatten out.

Contained in the *separable - ill-conditioned* function class.

`bbob-biobj`

function 5 in dimension 5 for the first instance.#### : Sphere/Sum of different powers¶

Combination of the sphere function ( in the `bbob`

suite)
and the sum of different powers function ( in the `bbob`

suite).

Both objective functions are unimodal. The first objective is separable, the second non-separable. When approaching the second objective’s optimum, the difference in sensitivity between different directions in search space increases unboundedly.

Contained in the *separable - ill-conditioned* function class.

`bbob-biobj`

function 6 in dimension 5 for the first instance.#### : Sphere/Rastrigin¶

Combination of the sphere function ( in the `bbob`

suite)
and the Rastrigin function ( in the `bbob`

suite).

In addition to the simple sphere function, the prototypical highly multimodal Rastrigin function needs to be solved which has originally a very regular and symmetric structure for the placement of the optima. Here, however, transformations are performed to alleviate the original symmetry and regularity in the second objective.

The properties of the second objective contain non-separabilty, multimodality (roughly local optima), a conditioning of about 10, and a large global amplitude compared to the local amplitudes.

Contained in the *separable - multi-modal* function class.

`bbob-biobj`

function 7 in dimension 5 for the first instance.#### : Sphere/Schaffer F7, condition 10¶

Combination of the sphere function ( in the `bbob`

suite)
and the Schaffer F7 function with condition number 10 ( in the `bbob`

suite).

In addition to the simple sphere function, an asymmetric, non-separable, and highly multimodal function needs to be solved to approach the Pareto front/Pareto set where the frequency and amplitude of the modulation in the second objective vary. The conditioning of the second objective and thus the entire bi-objective function is low.

Contained in the *separable - multi-modal* function class.

`bbob-biobj`

function 8 in dimension 5 for the first instance.#### : Sphere/Schwefel x*sin(x)¶

Combination of the sphere function ( in the `bbob`

suite)
and the Schwefel function ( in the `bbob`

suite).

While the first objective function is separable and unimodal, the second objective function is partially separable and highly multimodal—having the most prominent minima located comparatively close to the corners of the unpenalized search area.

Contained in the *separable - weakly-structured* function class.

`bbob-biobj`

function 9 in dimension 5 for the first instance.#### : Sphere/Gallagher 101 peaks¶

Combination of the sphere function ( in the `bbob`

suite)
and the Gallagher function with 101 peaks ( in the `bbob`

suite).

While the first objective function is separable and unimodal, the second objective function is non-separable and consists of 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the *separable - weakly-structured* function class.

`bbob-biobj`

function 10 in dimension 5 for the first instance.#### : Ellipsoid separable/Ellipsoid separable¶

Combination of two separable ellipsoid functions ( in the `bbob`

suite).

Both objectives are unimodal, separable, only globally quadratic with smooth local irregularities, and highly ill-conditioned with a condition number of about .

Contained in the *separable - separable* function class.

`bbob-biobj`

function 11 in dimension 5 for the first instance.#### : Ellipsoid separable/Attractive sector¶

Combination of the separable ellipsoid function ( in the `bbob`

suite)
and the attractive sector function ( in the `bbob`

suite).

Both objective functions are unimodal but only the first
one is separable. The first objective function, in addition,
is globally quadratic with smooth local irregularities, and
highly ill-conditioned with a condition number of about
. The second objective function is highly
asymmetric, where only one *hypercone* (with
angular base area) with a volume of roughly
yields low function values. The optimum of it is located at
the tip of this cone.

Contained in the *separable - moderate* function class.

`bbob-biobj`

function 12 in dimension 5 for the first instance.#### : Ellipsoid separable/Rosenbrock original¶

Combination of the separable ellipsoid function ( in the `bbob`

suite) and the original, i.e., unrotated Rosenbrock function
( in the `bbob`

suite).

Only the first objective is separable and unimodal. The second objective is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. In addition, the first objective function shows smooth local irregularities from a globally convex quadratic function and is highly ill-conditioned with a condition number of about .

Contained in the *separable - moderate* function class.

`bbob-biobj`

function 13 in dimension 5 for the first instance.#### : Ellipsoid separable/Sharp ridge¶

Combination of the separable ellipsoid function ( in the `bbob`

suite) and the sharp ridge function ( in the `bbob`

suite).

Both objective functions are unimodal but only the first one is separable.

The first objective is globally quadratic but with smooth local irregularities and highly ill-conditioned with a condition number of about . For optimizing the second objective, a sharp, i.e., non-differentiable ridge has to be followed.

Contained in the *separable - ill-conditioned* function class.

`bbob-biobj`

function 14 in dimension 5 for the first instance.#### : Ellipsoid separable/Sum of different powers¶

Combination of the separable ellipsoid function ( in the `bbob`

suite) and the sum of different powers function
( in the `bbob`

suite).

Both objective functions are unimodal but only the first one is separable.

The first objective is globally quadratic but with smooth local irregularities and highly ill-conditioned with a condition number of about . When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.

Contained in the *separable - ill-conditioned* function class.

`bbob-biobj`

function 15 in dimension 5 for the first instance.#### : Ellipsoid separable/Rastrigin¶

Combination of the separable ellipsoid function ( in the `bbob`

suite) and the Rastrigin function ( in the `bbob`

suite).

The objective functions show rather opposite properties. The first one is separable, the second not. The first one is unimodal, the second highly multimodal (roughly local optima). The first one is highly ill-conditioning (condition number of ), the second one has a conditioning of about 10. Local non-linear transformations are performed in both objective functions to alleviate the original symmetry and regularity of the two baseline functions.

Contained in the *separable - multi-modal* function class.

`bbob-biobj`

function 16 in dimension 5 for the first instance.#### : Ellipsoid separable/Schaffer F7, condition 10¶

Combination of the separable ellipsoid function ( in the `bbob`

suite) and the Schaffer F7 function with condition number 10
( in the `bbob`

suite).

Also here, both single objectives possess opposing properties. The first objective is unimodal, besides small local non-linearities symmetric, separable and highly ill-conditioned while the second objective is highly multi-modal, asymmetric, and non-separable, with only a low conditioning.

Contained in the *separable - multi-modal* function class.

`bbob-biobj`

function 17 in dimension 5 for the first instance.#### : Ellipsoid separable/Schwefel x*sin(x)¶

Combination of the separable ellipsoid function ( in the `bbob`

suite) and the Schwefel function ( in the `bbob`

suite).

The first objective is unimodal, separable and highly ill-conditioned. The second objective is partially separable and highly multimodal—having the most prominent minima located comparatively close to the corners of the unpenalized search area.

Contained in the *separable - weakly-structured* function class.

`bbob-biobj`

function 18 in dimension 5 for the first instance.#### : Ellipsoid separable/Gallagher 101 peaks¶

Combination of the separable ellipsoid function ( in the `bbob`

suite) and the Gallagher function with 101 peaks ( in the `bbob`

suite).

While the first objective function is separable, unimodal, and highly ill-conditioned (condition number of about ), the second objective function is non-separable and consists of 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the *separable - weakly-structured* function class.

`bbob-biobj`

function 19 in dimension 5 for the first instance.#### : Attractive sector/Attractive sector¶

Combination of two attractive sector functions ( in the `bbob`

suite).
Both functions are unimodal and highly asymmetric, where only one
*hypercone* (with angular base area) per objective with a volume of
roughly yields low function values. The objective
functions’ optima are located at the tips of those two cones.

Contained in the *moderate - moderate* function class.

`bbob-biobj`

function 20 in dimension 5 for the first instance.#### : Attractive sector/Rosenbrock original¶

Combination of the attractive sector function ( in the `bbob`

suite) and the Rosenbrock function ( in the `bbob`

suite).

The first function is unimodal but highly asymmetric, where only one
*hypercone* (with angular base area) with a volume of
roughly yields low function values (with the
optimum at the tip of the cone). The second
objective is partially separable (tri-band structure) and has a local
optimum with an attraction volume of about 25%.

Contained in the *moderate - moderate* function class.

`bbob-biobj`

function 21 in dimension 5 for the first instance.#### : Attractive sector/Sharp ridge¶

Combination of the attractive sector function ( in the `bbob`

suite) and the sharp ridge function ( in the `bbob`

suite).

Both objective functions are unimodal and non-separable. The
first objective is highly asymmetric in the sense that only one
*hypercone* (with angular base area) with a volume of
roughly yields low function values (with the
optimum at the tip of the cone). For optimizing the second
objective, a sharp, i.e., non-differentiable ridge has to be followed.

Contained in the *moderate - ill-conditioned* function class.

`bbob-biobj`

function 22 in dimension 5 for the first instance.#### : Attractive sector/Sum of different powers¶

Combination of the attractive sector function ( in the `bbob`

suite) and the sum of different powers function
( in the `bbob`

suite).

Both objective functions are unimodal and non-separable. The
first objective is highly asymmetric in the sense that only one
*hypercone* (with angular base area) with a volume of
roughly yields low function values (with the
optimum at the tip of the cone). When approaching the second
objective’s optimum, the sensitivies of the variables in the
rotated search space become more and more different.

Contained in the *moderate - ill-conditioned* function class.

`bbob-biobj`

function 23 in dimension 5 for the first instance.#### : Attractive sector/Rastrigin¶

Combination of the attractive sector function ( in the `bbob`

suite) and the Rastrigin function
( in the `bbob`

suite).

Both objectives are non-separable, and the second one is highly multi-modal (roughly local optima) while the first one is unimodal. Further properties are that the first objective is highly assymetric and the second has a conditioning of about 10.

Contained in the *moderate - multi-modal* function class.

`bbob-biobj`

function 24 in dimension 5 for the first instance.#### : Attractive sector/Schaffer F7, condition 10¶

Combination of the attractive sector function ( in the `bbob`

suite) and the Schaffer F7 function with condition number 10
( in the `bbob`

suite).

Both objectives are non-separable and asymmetric. While the first objective is unimodal, the second one is a highly multi-modal function with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the *moderate - multi-modal* function class.

`bbob-biobj`

function 25 in dimension 5 for the first instance.#### : Attractive sector/Schwefel x*sin(x)¶

Combination of the attractive sector function ( in the `bbob`

suite) and the Schwefel function ( in the `bbob`

suite).

The first objective is non-separable, unimodal, and asymmetric. The second objective is partially separable and highly multimodal—having the most prominent minima located comparatively close to the corners of the unpenalized search area.

Contained in the *moderate - weakly-structured* function class.

`bbob-biobj`

function 26 in dimension 5 for the first instance.#### : Attractive sector/Gallagher 101 peaks¶

Combination of the attractive sector function ( in the `bbob`

suite) and the Gallagher function with 101 peaks ( in the `bbob`

suite).

Both objective functions are non-separable but only the first is unimodal. The first objective function is furthermore asymmetric. The second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the *moderate - weakly-structured* function class.

`bbob-biobj`

function 27 in dimension 5 for the first instance.#### : Rosenbrock original/Rosenbrock original¶

Combination of two Rosenbrock functions ( in the `bbob`

suite).

Both objectives are partially separable (tri-band structure) and have a local optimum with an attraction volume of about 25%.

Contained in the *moderate - moderate* function class.

`bbob-biobj`

function 28 in dimension 5 for the first instance.#### : Rosenbrock original/Sharp ridge¶

Combination of the Rosenbrock function ( in the `bbob`

suite) and the
sharp ridge function ( in the `bbob`

suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective is unimodal and non-separable and, for optimizing it, a sharp, i.e., non-differentiable ridge has to be followed.

Contained in the *moderate - ill-conditioned* function class.

`bbob-biobj`

function 29 in dimension 5 for the first instance.#### : Rosenbrock original/Sum of different powers¶

Combination of the Rosenbrock function ( in the `bbob`

suite) and the sum of different powers function
( in the `bbob`

suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective function is unimodal and non-separable. When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.

Contained in the *moderate - ill-conditioned* function class.

`bbob-biobj`

function 30 in dimension 5 for the first instance.#### : Rosenbrock original/Rastrigin¶

Combination of the Rosenbrock function ( in the `bbob`

suite) and the Rastrigin function
( in the `bbob`

suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective function is non-separable and highly multi-modal (roughly local optima).

Contained in the *moderate - multi-modal* function class.

`bbob-biobj`

function 31 in dimension 5 for the first instance.#### : Rosenbrock original/Schaffer F7, condition 10¶

Combination of the Rosenbrock function ( in the `bbob`

suite) and the
Schaffer F7 function with condition number 10
( in the `bbob`

suite).

The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective function is non-separable, asymmetric, and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the *moderate - multi-modal* function class.

`bbob-biobj`

function 32 in dimension 5 for the first instance.#### : Rosenbrock original/Schwefel x*sin(x)¶

Combination of the Rosenbrock function ( in the `bbob`

suite) and the
Schwefel function ( in the `bbob`

suite).

Both objective functions are partially separable. While the first objective function has a local optimum with an attraction volume of about 25%, the second objective function is highly multimodal—having the most prominent minima located comparatively close to the corners of its unpenalized search area.

Contained in the *moderate - weakly-structured* function class.

`bbob-biobj`

function 33 in dimension 5 for the first instance.#### : Rosenbrock original/Gallagher 101 peaks¶

Combination of the Rosenbrock function ( in the `bbob`

suite) and
the Gallagher function with 101 peaks ( in the `bbob`

suite).

The first objective function is partially separable, the second one non-separable. While the first objective function has a local optimum with an attraction volume of about 25%, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the *moderate - weakly-structured* function class.

`bbob-biobj`

function 34 in dimension 5 for the first instance.#### : Sharp ridge/Sharp ridge¶

Combination of two sharp ridge functions ( in the `bbob`

suite).

Both objective functions are unimodal and non-separable and, for optimizing them, two sharp, i.e., non-differentiable ridges have to be followed.

Contained in the *ill-conditioned - ill-conditioned* function class.

`bbob-biobj`

function 35 in dimension 5 for the first instance.#### : Sharp ridge/Sum of different powers¶

Combination of the sharp ridge function ( in the `bbob`

suite) and the
sum of different powers function
( in the `bbob`

suite).

Both functions are uni-modal and non-separable. For optimizing the first objective, a sharp, i.e., non-differentiable ridge has to be followed. When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.

Contained in the *ill-conditioned - ill-conditioned* function class.

`bbob-biobj`

function 36 in dimension 5 for the first instance.#### : Sharp ridge/Rastrigin¶

Combination of the sharp ridge function ( in the `bbob`

suite) and the Rastrigin function
( in the `bbob`

suite).

Both functions are non-separable. While the first one is unimodal and non-differentiable at its ridge, the second objective function is highly multi-modal (roughly local optima).

Contained in the *ill-conditioned - multi-modal* function class.

`bbob-biobj`

function 37 in dimension 5 for the first instance.#### : Sharp ridge/Schaffer F7, condition 10¶

Combination of the sharp ridge function ( in the `bbob`

suite) and the
Schaffer F7 function with condition number 10
( in the `bbob`

suite).

Both functions are non-separable. While the first one is unimodal and non-differentiable at its ridge, the second objective function is asymmetric and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the *ill-conditioned - multi-modal* function class.

`bbob-biobj`

function 38 in dimension 5 for the first instance.#### : Sharp ridge/Schwefel x*sin(x)¶

Combination of the sharp ridge function ( in the `bbob`

suite) and the
Schwefel function ( in the `bbob`

suite).

While the first objective function is unimodal, non-separable, and non-differentiable at its ridge, the second objective function is highly multimodal—having the most prominent minima located comparatively close to the corners of its unpenalized search area.

Contained in the *ill-conditioned - weakly-structured* function class.

`bbob-biobj`

function 39 in dimension 5 for the first instance.#### : Sharp ridge/Gallagher 101 peaks¶

Combination of the sharp ridge function ( in the `bbob`

suite) and the
Gallagher function with 101 peaks ( in the `bbob`

suite).

Both objective functions are non-separable. While the first objective function is unimodal and non-differentiable at its ridge, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the *ill-conditioned - weakly-structured* function class.

`bbob-biobj`

function 40 in dimension 5 for the first instance.#### : Sum of different powers/Sum of different powers¶

Combination of two sum of different powers functions
( in the `bbob`

suite).

Both functions are uni-modal and non-separable where the sensitivies of the variables in the rotated search space become more and more different when approaching the objectives’ optima.

Contained in the *ill-conditioned - ill-conditioned* function class.

`bbob-biobj`

function 41 in dimension 5 for the first instance.#### : Sum of different powers/Rastrigin¶

Combination of the sum of different powers functions
( in the `bbob`

suite) and the Rastrigin function
( in the `bbob`

suite).

Both objective functions are non-separable. While the first one is unimodal, the second objective function is highly multi-modal (roughly local optima).

Contained in the *ill-conditioned - multi-modal* function class.

`bbob-biobj`

function 42 in dimension 5 for the first instance.#### : Sum of different powers/Schaffer F7, condition 10¶

Combination of the sum of different powers functions
( in the `bbob`

suite) and the Schaffer F7 function with
condition number 10 ( in the `bbob`

suite).

Both objective functions are non-separable. While the first one is unimodal with an increasing conditioning once the optimum is approached, the second objective function is asymmetric and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.

Contained in the *ill-conditioned - multi-modal* function class.

`bbob-biobj`

function 43 in dimension 5 for the first instance.#### : Sum of different powers/Schwefel x*sin(x)¶

Combination of the sum of different powers functions
( in the `bbob`

suite) and the Schwefel function ( in the `bbob`

suite).

Both objectives are non-separable. While the first objective function is unimodal, the second objective function is highly multimodal—having the most prominent minima located comparatively close to the corners of its unpenalized search area.

Contained in the *ill-conditioned - weakly-structured* function class.

`bbob-biobj`

function 44 in dimension 5 for the first instance.#### : Sum of different powers/Gallagher 101 peaks¶

Combination of the sum of different powers functions
( in the `bbob`

suite) and the Gallagher function with
101 peaks ( in the `bbob`

suite).

Both objective functions are non-separable. While the first objective function is unimodal, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.

Contained in the *ill-conditioned - weakly-structured* function class.

`bbob-biobj`

function 45 in dimension 5 for the first instance.#### : Rastrigin/Rastrigin¶

Combination of two Rastrigin functions
( in the `bbob`

suite).

Both objective functions are non-separable and highly multi-modal (roughly local optima).

Contained in the *multi-modal - multi-modal* function class.

`bbob-biobj`

function 46 in dimension 5 for the first instance.#### : Rastrigin/Schaffer F7, condition 10¶

Combination of the Rastrigin function
( in the `bbob`

suite) and the Schaffer F7 function with
condition number 10 ( in the `bbob`

suite).

Both objective functions are non-separable and highly multi-modal.

Contained in the *multi-modal - multi-modal* function class.

`bbob-biobj`

function 47 in dimension 5 for the first instance.#### : Rastrigin/Schwefel x*sin(x)¶

Combination of the Rastrigin function
( in the `bbob`

suite) and the Schwefel function ( in the `bbob`

suite).

Both objective functions are non-separable and highly multi-modal where the first has roughly local optima and the most prominent minima of the second objective function are located comparatively close to the corners of its unpenalized search area.

Contained in the *multi-modal - weakly-structured* function class.

`bbob-biobj`

function 48 in dimension 5 for the first instance.#### : Rastrigin/Gallagher 101 peaks¶

Combination of the Rastrigin function
( in the `bbob`

suite) and the Gallagher function with
101 peaks ( in the `bbob`

suite).

Both objective functions are non-separable and highly multi-modal where the first has roughly local optima and the second has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function).

Contained in the *multi-modal - weakly-structured* function class.

`bbob-biobj`

function 49 in dimension 5 for the first instance.#### : Schaffer F7, condition 10/Schaffer F7, condition 10¶

Combination of two Schaffer F7 functions with
condition number 10 ( in the `bbob`

suite).

Both objective functions are non-separable and highly multi-modal.

Contained in the *multi-modal - multi-modal* function class.

`bbob-biobj`

function 50 in dimension 5 for the first instance.#### : Schaffer F7, condition 10/Schwefel x*sin(x)¶

Combination of the Schaffer F7 function with
condition number 10 ( in the `bbob`

suite)
and the Schwefel function ( in the `bbob`

suite).

Both objective functions are non-separable and highly multi-modal. While frequency and amplitude of the modulation vary in an almost regular fashion in the first objective function, the second objective function posseses less global structure.

Contained in the *multi-modal - weakly-structured* function class.

`bbob-biobj`

function 51 in dimension 5 for the first instance.#### : Schaffer F7, condition 10/Gallagher 101 peaks¶

Combination of the Schaffer F7 function with
condition number 10 ( in the `bbob`

suite)
and the Gallagher function with
101 peaks ( in the `bbob`

suite).

Both objective functions are non-separable and highly multi-modal. While frequency and amplitude of the modulation vary in an almost regular fashion in the first objective function, the second has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function).

Contained in the *multi-modal - weakly-structured* function class.

`bbob-biobj`

function 52 in dimension 5 for the first instance.#### : Schwefel x*sin(x)/Schwefel x*sin(x)¶

Combination of two Schwefel functions ( in the `bbob`

suite).

Both objective functions are non-separable and highly multi-modal where the most prominent minima of each objective function are located comparatively close to the corners of its unpenalized search area. Due to the combinatorial nature of the Schwefel function, it is likely in low dimensions that the Pareto set goes through the origin of the search space.

Contained in the *weakly-structured - weakly-structured* function class.

`bbob-biobj`

function 53 in dimension 5 for the first instance.#### : Schwefel x*sin(x)/Gallagher 101 peaks¶

Combination of the Schwefel function ( in the `bbob`

suite) and the Gallagher function with
101 peaks ( in the `bbob`

suite).

Both objective functions are non-separable and highly multi-modal. For the first objective function, the most prominent minima are located comparatively close to the corners of its unpenalized search area. For the second objective, position and height of all 101 optima are unrelated and randomly chosen (different for each instantiation of the function).

Contained in the *weakly-structured - weakly-structured* function class.

`bbob-biobj`

function 54 in dimension 5 for the first instance.#### : Gallagher 101 peaks/Gallagher 101 peaks¶

Combination of two Gallagher functions with
101 peaks ( in the `bbob`

suite).

Both objective functions are non-separable and highly multi-modal. Position and height of all 101 optima in each objective function are unrelated and randomly chosen and thus, no global structure is present.

Contained in the *weakly-structured - weakly-structured* function class.

`bbob-biobj`

function 55 in dimension 5 for the first instance.## The Extended `bbob-biobj-ext`

Test Suite and Its Functions¶

Having all combinations of only a subset of the single-objective `bbob`

functions in a test suite
like the above `bbob-biobj`

one has
advantages but also a few disadvantages. Using only a subet of the 24 `bbob`

functions
introduces a bias towards the chosen functions and reduces the amount of different difficulties,
a bi-objective algorithm is exposed to in the benchmarking exercise. Allowing all combinations of
`bbob`

functions increases the percentage of problems for which both objectives are from different
`bbob`

function groups while, in practice, it can often be assumed that both objective functions
come from a similar “function domain”.

The rationale behind the following extended `bbob-biobj`

test suite, denoted as `bbob-biobj-ext`

,
is to reduce the mentioned effects. To this end, we add all within-group combinations of `bbob`

functions which are not already in the `bbob-biobj`

suite and which do not combine a function
with itself. For technical reasons, we also remove the Weierstrass functions ( in the
`bbob`

suite) because the optimum is not necessarily unique and computing the nadir point
therefore technically more challenging than for the other functions.
This extension adds functions, resulting in
92 functions overall.

The following table details which single-objective `bbob`

functions are contained in the
92 `bbob-biobj-ext`

functions (outer column and row annotations) and indicates their IDs.
Note that the IDs of the first 55 `bbob0biobj-ext`

functions are the same than for the
`bbob-biobj`

test suite for compatibility reasons.

f1 | f2 | f56 | f57 | f58 | f3 | f4 | f5 | f6 | f7 | f8 | f9 | f10 | ||||||||||||

f11 | f59 | f60 | f61 | f12 | f13 | f14 | f15 | f16 | f17 | f18 | f19 | |||||||||||||

f62 | f63 | |||||||||||||||||||||||

f64 | ||||||||||||||||||||||||

f20 | f65 | f21 | f66 | f22 | f23 | f24 | f25 | f26 | f27 | |||||||||||||||

f67 | f68 | |||||||||||||||||||||||

f28 | f69 | f29 | f30 | f31 | f32 | f33 | f34 | |||||||||||||||||

f70 | f71 | f72 | f73 | |||||||||||||||||||||

f74 | f75 | f76 | ||||||||||||||||||||||

f77 | f78 | |||||||||||||||||||||||

f35 | f36 | f37 | f38 | f39 | f40 | |||||||||||||||||||

f41 | f42 | f43 | f44 | f45 | ||||||||||||||||||||

f46 | f47 | f79 | f80 | f48 | f49 | |||||||||||||||||||

f50 | f81 | f82 | f51 | f52 | ||||||||||||||||||||

f83 | ||||||||||||||||||||||||

f53 | f54 | f84 | f85 | f86 | ||||||||||||||||||||

f55 | f87 | f88 | f89 | |||||||||||||||||||||

f90 | f91 | |||||||||||||||||||||||

f92 | ||||||||||||||||||||||||

The 92 functions of the`bbob-biobj-ext`

test suite and their IDs (in the table cells) together with the information about which single-objective`bbob`

functions are used to define them (outer column and row annotations).

### Function Groups¶

Like for the `bbob-biobj`

test suite, we obtain 15 function
classes to structure the 92 bi-objective functions of the `bbob-biobj-ext`

test
suite. Depending on whether a function class combines functions from the same or
from different `bbob`

function classes, each function class contains
8, 12 or just four functions. We are listing
below the function classes and in parenthesis the functions that belong to
the respective class:

- separable - separable (12 functions: f1, f2, f11, f56-64)
- separable - moderate (f3, f4, f12, f13)
- separable - ill-conditioned (f5, f6, f14, f15)
- separable - multi-modal (f7, f8, f16, f17)
- separable - weakly-structured (f9, f10, f18, f19)
- moderate - moderate (8 functions: f20, f21, f28, f65-f69)
- moderate - ill-conditioned (f22, f23, f29, f30)
- moderate - multi-modal (f24, f25, f31, f32)
- moderate - weakly-structured (f26, f27, f33, f34)
- ill-conditioned - ill-conditioned (12 functions: f35, f36, f41, f70-78)
- ill-conditioned - multi-modal (f37, f38, f42, f43)
- ill-conditioned - weakly-structured (f39, f40, f44, f45)
- multi-modal - multi-modal (8 functions: f46, f47, f50, f79-83)
- multi-modal - weakly structured (f48, f49, f51, f52)
- weakly structured - weakly structured (12 functions: f53-55, f84-92)

### Normalization and Instances¶

Normalization of the objectives and instances are handled for the `bbob-biobj-ext`

in the
same manner as for the `bbob-biobj`

suite, i.e., no normalization of the objective
functions is taking place for the algorithm benchmarking and 15 instances are prescribed for
a typical experiment.

## Acknowledgments

This work was supported by the grant ANR-12-MONU-0009 (NumBBO)
of the French National Research Agency. We also thank Ilya Loshchilov and Oswin Krause for their
initial suggestions on how to extend the `bbob-biobj`

test suite.

## References

[BRO2016biperf] | D. Brockhoff, T. Tušar, D. Tušar, T. Wagner, N. Hansen, A. Auger, (2016).
Biobjective Performance Assessment with the COCO Platform. ArXiv e-prints, arXiv:1605.01746. |

[BRO2015] | (1, 2) D. Brockhoff, T.-D. Tran, and N. Hansen (2015).
Benchmarking Numerical Multiobjective Optimizers Revisited. In
Proceedings of the 2015 GECCO Genetic and Evolutionary Computation Conference,
pp. 639-646, ACM. |

[HAN2016co] | (1, 2) N. Hansen, A. Auger, O. Mersmann, T. Tušar, D. Brockhoff (2016).
COCO: A Platform for Comparing Continuous Optimizers in a Black-Box
Setting, ArXiv e-prints, arXiv:1603.08785. |

[HAN2009] | (1, 2) N. Hansen, S. Finck, R. Ros, and A. Auger (2009).
Real-parameter black-box optimization benchmarking 2009: Noiseless
functions definitions. Research Report RR-6829, Inria, updated
February 2010. |

[HAN2011] | N. Hansen, R. Ros, N. Mauny, M. Schoenauer, and A. Auger (2011). Impacts
of Invariance in Search: When CMA-ES and PSO Face Ill-Conditioned and
Non-Separable Problems. Applied Soft Computing. Vol. 11, pp. 5755-5769.
Elsevier. |

[HAN2016ex] | N. Hansen, T. Tušar, A. Auger, D. Brockhoff, O. Mersmann (2016).
COCO: The Experimental Procedure, ArXiv e-prints, arXiv:1603.08776. |