COCO: The Bi-objective Black-Box Optimization Benchmarking (bbob-biobj
) Test Suite¶
See also: ArXiv e-prints,
arXiv:1604.00359, 2016.Several test function suites for numerical benchmarking of multiobjective
optimization algorithms have been proposed in recent years. While having
desirable properties like well-understood Pareto sets and Pareto fronts with shapes
of various kinds, most of the currently used functions posess properties
which are arguably under-represented in real-world problems. Those properties
mainly stem from the easier construction of such problems—overrepresenting
properties such as no dependencies between variables, Pareto sets exactly located at
the bound constraints, or the differentiation between position and distance
variables. Here, we propose an alternative way and define the bbob-biobj
test suite
with 55 bi-objective functions and its extended bbob-biobj-ext
version with
92 bi-objective functions in continuous domain which are both derived from
combining functions of the well-known single-objective noiseless bbob
test suite.
Besides giving the actual function definitions and presenting their (known) properties, this
documentation also aims at giving the rationale behind our approach in terms
of function groups, instances, and potential objective space normalization.

Example plots of the Pareto front approximation, found by NSGA-II on
selected bbob-biobj
functions. In blue the non-dominated points at the
end of different independent runs, in red the points that are
non-dominated over all runs.
Preliminaries, Definitions, and Scope¶
In the following, we consider bi-objective, unconstrained minimization problems of the form
where is the number of variables of the problem (also called
the problem dimension),
and
are the two
objective functions, and the
operator is related to the
standard dominance relation. A solution
is thereby said to dominate another solution
if
and
hold and at
least one of the inequalities is strict.
Solutions which are not dominated by any other solution in the search
space are called Pareto-optimal or efficient solutions. All
Pareto-optimal solutions constitute the Pareto set of which an
approximation is sought. The Pareto set’s image in the
objective space is called Pareto front.
The objective of the minimization problem is to find, with as few evaluations
of as possible, a set of non-dominated solutions which is (i) as large
as possible and (ii) has
-values as close to the Pareto front as possible. [1]
[1] | Distance in ![]() |
In the following, we remind useful definitions.
- function instance, problem
Each function
within COCO is parametrized with parameter values
. A parameter value determines a so-called function instance. For example,
encodes the location of the optimum of single-objective functions, which means that different instances have shifted optima. In the
bbob-biobj
test suite,and the function instances are determined by the instances of the underlying single-objective functions.
A problem is a function instance of a specific dimension
.
- ideal point
The ideal point is defined as the vector in objective space that contains the optimal
-value for each objective independently. More precisely let
and
, the ideal point is given by
- nadir point
The nadir point (in objective space) consists in each objective of the worst value obtained by a Pareto-optimal solution. More precisely, let
be the set of Pareto optimal points. Then the nadir point satisfies
In the case of two objectives with a unique global minimum each (that is, a single point in the search space maps to the global minimum)
where
and
.
Overview of the Proposed bbob-biobj
Test Suite¶
The bbob-biobj
test suite provides 55 bi-objective functions in six
dimensions (2, 3, 5, 10, 20, and 40) with a large number of possible instances.
The 55 functions are derived from combining a subset of the 24 well-known
single-objective functions of the bbob
test suite [HAN2009] which
has been used since 2009 in the BBOB workshop series.
While concrete details on each of
the 55 bbob-biobj
functions are given in Section
The bbob-biobj Test Functions and Their Properties, we will detail here the main rationale behind
them together with their common properties.
The Single-objective bbob
Functions¶
The bbob-biobj
test suite is designed to be able to assess performance of algorithms with respect to well-identified difficulties in optimization typically occurring in real-world problems. A multi-objective problem being a combination of single-objective problems, one can obtain multi-objective problems with representative difficulties by simply combining single objective functions with representative difficulties observed in real-world problems. For this purpose we naturally use the single-objective bbob
suite [HAN2009].
Combining all 24 bbob
functions in pairs thereby results in
bi-objective functions overall. We however assume that
multi-objective optimization algorithms are not sensitive to permutations of
the objective functions such that combining the 24
bbob
functions and
taking out the function if the function
is present results in
functions.
Some first tests, e.g. in [BRO2015], showed that having 300 functions is
impracticable in terms of the overall running time of the benchmarking
experiment. We then decided to exploit the organization of the bbob
functions into classes to choose a subset of functions. More precisely, the 24
original bbob
functions are grouped into five function classes where each
class gathers functions with similar properties, namely
- separable functions
- functions with low or moderate conditioning
- functions with high conditioning and unimodal
- multi-modal functions with adequate global structure,
- multi-modal functions with weak global structure.
To create the bbob-biobj
suite, we choose two functions within each class. This way we do not introduce any bias towards a specific class. In addition within each class, the functions are chosen to be the most
representative without repeating similar functions. For example,
only one Ellipsoid, one Rastrigin, and one Gallagher function are
included in the bbob-biobj
suite although they appear in
separate versions in the bbob
suite. Finally our choice of 10 bbob
functions for creating the bbob-biobj
test suite is the following:
- Separable functions
- Sphere (function 1 in
bbob
suite) - Ellipsoid separable (function 2 in
bbob
suite)
- Sphere (function 1 in
- Functions with low or moderate conditioning
- Attractive sector (function 6 in
bbob
suite) - Rosenbrock original (function 8 in
bbob
suite)
- Attractive sector (function 6 in
- Functions with high conditioning and unimodal
- Sharp ridge (function 13 in
bbob
suite) - Sum of different powers (function 14 in
bbob
suite)
- Sharp ridge (function 13 in
- Multi-modal functions with adequate global structure
- Rastrigin (function 15 in
bbob
suite) - Schaffer F7, condition 10 (function 17 in
bbob
suite)
- Rastrigin (function 15 in
- Multi-modal functions with weak global structure
- Schwefel x*sin(x) (function 20 in
bbob
suite) - Gallagher 101 peaks (function 21 in
bbob
suite)
- Schwefel x*sin(x) (function 20 in
Using the above described pairwise combinations, this results in
having bi-objective functions in
the final bbob-biobj suite. These functions are denoted
to
in the sequel.
Function Groups¶
From combining the original bbob
function classes, we obtain 15 function
classes to structure the 55 bi-objective functions of the bbob-biobj
test
suite. Each function class contains three or four functions. We are listing
below the function classes and in parenthesis the functions that belong to
the respective class:
- separable - separable (functions
,
,
)
- separable - moderate (
,
,
,
)
- separable - ill-conditioned (
,
,
,
)
- separable - multi-modal (
,
,
,
)
- separable - weakly-structured (
,
,
,
)
- moderate - moderate (
,
,
)
- moderate - ill-conditioned (
,
,
,
)
- moderate - multi-modal (
,
,
,
)
- moderate - weakly-structured (
,
,
,
)
- ill-conditioned - ill-conditioned (
,
,
)
- ill-conditioned - multi-modal (
,
,
,
)
- ill-conditioned - weakly-structured (
,
,
,
)
- multi-modal - multi-modal (
,
,
)
- multi-modal - weakly structured (
,
,
,
)
- weakly structured - weakly structured (
,
,
)
More details about the single functions can be found in Section The bbob-biobj Test Functions and Their Properties. We however first describe their common properties in the coming sections.
Normalization of Objectives¶
None of the 55 bbob-biobj
functions is explicitly normalized and the
optimization algorithms therefore have to cope with objective values in
different ranges. Typically, different orders of magnitude
between the objective values can be observed.
However, to facilitate comparison of algorithm performance over different functions,
we normalize the objectives based on the ideal and nadir points
before calculating the hypervolume indicator [BRO2016biperf].
Both points can be computed, because the global
optimum is known and is unique for the 10 bbob
base functions.
In the black-box optimization benchmarking setup, however, the values of the
ideal and nadir points are not accessible to the optimization algorithm
[HAN2016ex].
Instances¶
Our test functions are parametrized and instances are instantiations of the underlying parameters (see [HAN2016co]). The instances for the bi-objective functions are obtained using instances of each single objective function composing the bi-objective one. In addition, we assert two conditions:
1. The Euclidean distance between the two single-objective optima (also called the extreme optimal points) in the search space is at least
.
2. The Euclidean distance between the ideal and the nadir point in the non-normalized objective space is at least
.
We associate to an instance, an instance-id which is an integer. The relation between the
instance-id, , of a bi-objective function
and the instance-ids,
and
, of its
underlying single-objective functions
and
is the following:
and
If we find that above conditions are not satisfied for all dimensions and
functions in the bbob-biobj
suite, we increase the instance-id of the
second objective successively until both properties are fulfilled.
For example, the bbob-biobj
instance-id
8 corresponds to the instance-id 17 for the first objective and instance-id 18 for
the second objective while for the bbob-biobj
instance-id 9, the
first instance-id is 19 but for the second objective, instance-id 21 is chosen
instead of instance-id 20.
Exceptions to the above rule are, for historical reasons, the
bbob-biobj
instance-ids 1 and 2 in order to match the instance-ids
1 to 5 with the ones proposed in [BRO2015]. The bbob-biobj
instance-id 1 contains the single-objective instance-ids 2 and 4 and
the bbob-biobj
instance-id 2 contains the two instance-ids 3 and 5.
For each bi-objective function and given dimension, the bbob-biobj
suite
contains 10 instances. [2]
[2] | In principle, as for the instance generation for the bbob suite,
the number of possible instances for the bbob-biobj suite is unlimited
[HAN2016co].
However, running some tests with too few instances will render the
potential statistics and their interpretation problematic while even the
tiniest observed difference can be made statistically significant with a
high enough number of instances. A good compromise to avoid either pitfall
seems to lie between, say, 9 and 19 instances. |
The bbob-biobj
Test Functions and Their Properties¶
In the following, we detail all 55 bbob-biobj
functions
and their properties.
The following table gives an overview and quick access to the functions,
inner cell IDs refer to the bbob-biobj
functions, outer column and row
annotations refer to the single-objective bbob
functions.
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
|
![]() |
f1 | f2 | f3 | f4 | f5 | f6 | f7 | f8 | f9 | f10 |
![]() |
f11 | f12 | f13 | f14 | f15 | f16 | f17 | f18 | f19 | |
![]() |
f20 | f21 | f22 | f23 | f24 | f25 | f26 | f27 | ||
![]() |
f28 | f29 | f30 | f31 | f32 | f33 | f34 | |||
![]() |
f35 | f36 | f37 | f38 | f39 | f40 | ||||
![]() |
f41 | f42 | f43 | f44 | f45 | |||||
![]() |
f46 | f47 | f48 | f49 | ||||||
![]() |
f50 | f51 | f52 | |||||||
![]() |
f53 | f54 | ||||||||
![]() |
f55 |
Some Function Properties¶
In the description of the 55 bbob-biobj
functions below, several
general properties of objective functions will be mentioned that
are defined here in short. It depends on these properties whether the optimization problem
is easy or hard to solve.
A separable function does not show any dependencies between the
variables and can therefore be solved by applying consecutive
one-dimensional optimizations along the coordinate axes while
keeping the other variables fixed. Consequently, non-separable
problems must be considered. They are much more difficult to solve. The
typical well-established technique to generate non-separable
functions from separable ones is the application of a rotation matrix
to
, that is
,
where
is a separable function.
A unimodal function has only one local minimum which is at the same time also its global one. A multimodal function has at least two local minima which is highly common in practical optimization problems.
Ill-conditioning is another typical challenge in real-parameter
optimization and, besides multimodality, probably the most common one.
In a general case, we can consider a function as ill-conditioned if for
solution points from the same level-set “the minimal displacement [...] that
produces a given function value improvement differs by
orders of magnitude” [HAN2011].
Conditioning can be rigorously formalized in the
case of convex quadratic functions,
where
is a symmetric
positive definite matrix, as the condition number of the Hessian matrix
. Since contour lines associated to a convex quadratic function
are ellipsoids, the condition number corresponds to the square root of
the ratio between the largest axis of the ellipsoid and the shortest axis.
The proposed bbob-biobj
testbed contains ill-conditioned functions
with a typical conditioning of . We believe this is a realistic
requirement, while we have seen practical problems with conditioning
as large as
.
Domain Bounds¶
All bi-objective functions provided in the bbob-biobj
suite are unbounded,
i.e., defined on the entire real-valued space .
The search domain of interest is defined as
, outside of
which non-dominated solutions are quite unlikely to be found. [3] The majority of non-dominated solutions are likely to lie even within
.
While we believe that the domain of interest contains the Pareto set,
due to the nature of the bbob-biobj
function definitions, there is no
guarantee that this is always the case.
However, the extremal solutions and their neighborhood ball of radius one
are guaranteed to lie within .
[3] | The functions coco_problem_get_smallest_value_of_interest and
coco_problem_get_largest_value_of_interest
of the COCO platform allow the optimizer
to retrieve the search domain of interest from the coco_problem_t ,
for example to generate the initial search points. |
Provided Search Space and Objective Space Plots¶
In order to better understand the properties of the 55 bbob-biobj
functions, we display for each of them plots
of the best known Pareto front approximation in objective space in original scaling (as seen by the algorithm) and in
log-scale, normalized such that the ideal point is at and the nadir point is at
. We
also provide plots illustrating the best known Pareto set approximation in search space (all depicted in black).
For the latter, two different plots are provided: a plot showing the projection onto a coordinate-axes-parallel cut
defined by two variables and a plot that projects all points onto a random cutting plane which contains both
single-objective optima and that also shows the contour lines of both objective functions on this plane.
In addition to the best Pareto set/Pareto front approximations, cuts through the search space are shown along (i) random lines through each optimum (in blue), (ii) lines along each coordinate axis through each optimum (blue dotted lines), (iii) the line through both optima (in red), (iv) two fully random lines [4] (in yellow), and (v) a random line in the random projection plane going through both optima [5] (in green).
All lines are normalized (of length 10 with the support vector in the middle). Ticks along the lines in the
objective space plots indicate the ends of line segments of the same length in search space. Thicker points on the lines
depict solutions that are non-dominated with respect to all points on the same line.
Furthermore, the search space plots highlight the projected region as gray-shaded area while
the gray-shaded area in the objective space plots highlight the region of interest between ideal (
) and
nadir point (
). Note that, to keep the plots to a manageable size, the Pareto set and Pareto front
approximations are carefully downsampled such that only one solution per grid point is shown—with the
precision of 2 decimals for the search space plots and 3 decimals for the objective space plots to define
the grid. The number of considered and actually displayed solutions is indicated in the search space plots’
legends. All plots are provided for one instance here only and for dimension 5 for the moment.
[4] | of random direction and with a support vector, drawn uniformly at random in ![]() |
[5] | with a random direction within the plane and a support vector, drawn uniformly at random in ![]() |
The 55 bbob-biobj
Functions¶
: Sphere/Sphere¶
Combination of two sphere functions ( in the
bbob
suite).
Both objectives are unimodal, highly symmetric, rotational and scale
invariant. The Pareto set is known to be a straight line and the Pareto
front is convex. Furthermore, the normalized hypervolume value of the
entire Pareto front with respect to the nadir point as reference point
can be computed analytically as the integral
.
Considered as the simplest bi-objective problem in continuous domain.
Contained in the separable - separable function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 1 in dimension 5 for the first instance.
: Sphere/Ellipsoid separable¶
Combination of the sphere function ( in the
bbob
suite)
and the separable ellipsoid function ( in the
bbob
suite).
Both objectives are unimodal and separable. While the first objective is
truly convex-quadratic with a condition number of 1, the second
objective is only globally quadratic with smooth local
irregularities and highly ill-conditioned with a condition number of
about .
Contained in the separable - separable function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 2 in dimension 5 for the first instance.
: Sphere/Attractive sector¶
Combination of the sphere function ( in the
bbob
suite)
and the attractive sector function ( in the
bbob
suite).
Both objective functions are unimodal, but only the first objective is
separable and truly convex quadratic. The attractive sector
function is highly asymmetric, where only one hypercone (with
angular base area) with a volume of roughly
yields low function values. The optimum of it is located at the tip
of this cone.
Contained in the separable - moderate function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 3 in dimension 5 for the first instance.
: Sphere/Rosenbrock original¶
Combination of the sphere function ( in the
bbob
suite)
and the original, i.e., unrotated Rosenbrock function ( in the
bbob
suite).
The first objective is separable and truly convex, the second objective is partially separable (tri-band structure). The first objective is unimodal while the second objective has a local optimum with an attraction volume of about 25%.
Contained in the separable - moderate function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 4 in dimension 5 for the first instance.
: Sphere/Sharp ridge¶
Combination of the sphere function ( in the
bbob
suite)
and the sharp ridge function ( in the
bbob
suite).
Both objective functions are unimodal. In addition to the simple, separable, and differentiable first objective, a sharp, i.e., non-differentiable ridge has to be followed for optimizing the (non-separable) second objective. The gradient towards the ridge remains constant, when the ridge is approached from a given point. Approaching the ridge is initially effective, but becomes ineffective close to the ridge when the rigde needs to be followed in direction to its optimum. The necessary change in search behavior close to the ridge is difficult to diagnose, because the gradient towards the ridge does not flatten out.
Contained in the separable - ill-conditioned function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 5 in dimension 5 for the first instance.
: Sphere/Sum of different powers¶
Combination of the sphere function ( in the
bbob
suite)
and the sum of different powers function ( in the
bbob
suite).
Both objective functions are unimodal. The first objective is separable, the second non-separable. When approaching the second objective’s optimum, the difference in sensitivity between different directions in search space increases unboundedly.
Contained in the separable - ill-conditioned function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 6 in dimension 5 for the first instance.
: Sphere/Rastrigin¶
Combination of the sphere function ( in the
bbob
suite)
and the Rastrigin function ( in the
bbob
suite).
In addition to the simple sphere function, the prototypical highly multimodal Rastrigin function needs to be solved which has originally a very regular and symmetric structure for the placement of the optima. Here, however, transformations are performed to alleviate the original symmetry and regularity in the second objective.
The properties of the second objective contain non-separabilty,
multimodality (roughly local optima), a conditioning of
about 10, and a large global amplitude compared to the local amplitudes.
Contained in the separable - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 7 in dimension 5 for the first instance.
: Sphere/Schaffer F7, condition 10¶
Combination of the sphere function ( in the
bbob
suite)
and the Schaffer F7 function with condition number 10 ( in the
bbob
suite).
In addition to the simple sphere function, an asymmetric, non-separable, and highly multimodal function needs to be solved to approach the Pareto front/Pareto set where the frequency and amplitude of the modulation in the second objective vary. The conditioning of the second objective and thus the entire bi-objective function is low.
Contained in the separable - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 8 in dimension 5 for the first instance.
: Sphere/Schwefel x*sin(x)¶
Combination of the sphere function ( in the
bbob
suite)
and the Schwefel function ( in the
bbob
suite).
While the first objective function is separable and unimodal,
the second objective function is partially separable and highly
multimodal—having the most prominent minima located
comparatively close to the corners of the unpenalized search area.
Contained in the separable - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 9 in dimension 5 for the first instance.
: Sphere/Gallagher 101 peaks¶
Combination of the sphere function ( in the
bbob
suite)
and the Gallagher function with 101 peaks ( in the
bbob
suite).
While the first objective function is separable and unimodal, the second objective function is non-separable and consists of 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.
Contained in the separable - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 10 in dimension 5 for the first instance.
: Ellipsoid separable/Ellipsoid separable¶
Combination of two separable ellipsoid functions ( in the
bbob
suite).
Both objectives are unimodal, separable, only globally
quadratic with smooth local irregularities, and highly
ill-conditioned with a condition number of
about .
Contained in the separable - separable function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 11 in dimension 5 for the first instance.
: Ellipsoid separable/Attractive sector¶
Combination of the separable ellipsoid function ( in the
bbob
suite)
and the attractive sector function ( in the
bbob
suite).
Both objective functions are unimodal but only the first
one is separable. The first objective function, in addition,
is globally quadratic with smooth local irregularities, and
highly ill-conditioned with a condition number of about
. The second objective function is highly
asymmetric, where only one hypercone (with
angular base area) with a volume of roughly
yields low function values. The optimum of it is located at
the tip of this cone.
Contained in the separable - moderate function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 12 in dimension 5 for the first instance.
: Ellipsoid separable/Rosenbrock original¶
Combination of the separable ellipsoid function ( in the
bbob
suite) and the original, i.e., unrotated Rosenbrock function
( in the
bbob
suite).
Only the first objective is separable and unimodal. The second
objective is partially separable (tri-band structure) and has a local
optimum with an attraction volume of about 25%.
In addition, the first objective function shows smooth local
irregularities from a globally convex quadratic function and is
highly ill-conditioned with a condition number of about
.
Contained in the separable - moderate function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 13 in dimension 5 for the first instance.
: Ellipsoid separable/Sharp ridge¶
Combination of the separable ellipsoid function ( in the
bbob
suite) and the sharp ridge function ( in the
bbob
suite).
Both objective functions are unimodal but only the first one is separable.
The first objective is globally quadratic but with smooth local
irregularities and highly ill-conditioned with a condition number of
about . For optimizing the second objective, a sharp,
i.e., non-differentiable ridge has to be followed.
Contained in the separable - ill-conditioned function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 14 in dimension 5 for the first instance.
: Ellipsoid separable/Sum of different powers¶
Combination of the separable ellipsoid function ( in the
bbob
suite) and the sum of different powers function
( in the
bbob
suite).
Both objective functions are unimodal but only the first one is separable.
The first objective is globally quadratic but with smooth local
irregularities and highly ill-conditioned with a condition number of
about . When approaching the second objective’s optimum,
the sensitivies of the variables in the rotated search space become
more and more different.
Contained in the separable - ill-conditioned function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 15 in dimension 5 for the first instance.
: Ellipsoid separable/Rastrigin¶
Combination of the separable ellipsoid function ( in the
bbob
suite) and the Rastrigin function ( in the
bbob
suite).
The objective functions show rather opposite properties.
The first one is separable, the second not. The first one
is unimodal, the second highly multimodal (roughly local
optima). The first one is highly ill-conditioning (condition number of
), the second one has a conditioning of about 10. Local
non-linear transformations are performed in both objective functions
to alleviate the original symmetry and regularity of the two
baseline functions.
Contained in the separable - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 16 in dimension 5 for the first instance.
: Ellipsoid separable/Schaffer F7, condition 10¶
Combination of the separable ellipsoid function ( in the
bbob
suite) and the Schaffer F7 function with condition number 10
( in the
bbob
suite).
Also here, both single objectives possess opposing properties. The first objective is unimodal, besides small local non-linearities symmetric, separable and highly ill-conditioned while the second objective is highly multi-modal, asymmetric, and non-separable, with only a low conditioning.
Contained in the separable - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 17 in dimension 5 for the first instance.
: Ellipsoid separable/Schwefel x*sin(x)¶
Combination of the separable ellipsoid function ( in the
bbob
suite) and the Schwefel function ( in the
bbob
suite).
The first objective is unimodal, separable and highly ill-conditioned.
The second objective is partially separable and highly multimodal—having
the most prominent minima located comparatively close to the
corners of the unpenalized search area.
Contained in the separable - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 18 in dimension 5 for the first instance.
: Ellipsoid separable/Gallagher 101 peaks¶
Combination of the separable ellipsoid function ( in the
bbob
suite) and the Gallagher function with 101 peaks ( in the
bbob
suite).
While the first objective function is separable, unimodal, and
highly ill-conditioned (condition number of about ),
the second objective function is non-separable and consists
of 101 optima with position and height being unrelated and
randomly chosen (different for each instantiation of the function).
The conditioning around the global optimum of the second
objective function is about 30.
Contained in the separable - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 19 in dimension 5 for the first instance.
: Attractive sector/Attractive sector¶
Combination of two attractive sector functions ( in the
bbob
suite).
Both functions are unimodal and highly asymmetric, where only one
hypercone (with angular base area) per objective with a volume of
roughly yields low function values. The objective
functions’ optima are located at the tips of those two cones.
Contained in the moderate - moderate function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 20 in dimension 5 for the first instance.
: Attractive sector/Rosenbrock original¶
Combination of the attractive sector function ( in the
bbob
suite) and the Rosenbrock function ( in the
bbob
suite).
The first function is unimodal but highly asymmetric, where only one
hypercone (with angular base area) with a volume of
roughly yields low function values (with the
optimum at the tip of the cone). The second
objective is partially separable (tri-band structure) and has a local
optimum with an attraction volume of about 25%.
Contained in the moderate - moderate function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 21 in dimension 5 for the first instance.
: Attractive sector/Sharp ridge¶
Combination of the attractive sector function ( in the
bbob
suite) and the sharp ridge function ( in the
bbob
suite).
Both objective functions are unimodal and non-separable. The
first objective is highly asymmetric in the sense that only one
hypercone (with angular base area) with a volume of
roughly yields low function values (with the
optimum at the tip of the cone). For optimizing the second
objective, a sharp, i.e., non-differentiable ridge has to be followed.
Contained in the moderate - ill-conditioned function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 22 in dimension 5 for the first instance.
: Attractive sector/Sum of different powers¶
Combination of the attractive sector function ( in the
bbob
suite) and the sum of different powers function
( in the
bbob
suite).
Both objective functions are unimodal and non-separable. The
first objective is highly asymmetric in the sense that only one
hypercone (with angular base area) with a volume of
roughly yields low function values (with the
optimum at the tip of the cone). When approaching the second
objective’s optimum, the sensitivies of the variables in the
rotated search space become more and more different.
Contained in the moderate - ill-conditioned function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 23 in dimension 5 for the first instance.
: Attractive sector/Rastrigin¶
Combination of the attractive sector function ( in the
bbob
suite) and the Rastrigin function
( in the
bbob
suite).
Both objectives are non-separable, and the second one
is highly multi-modal (roughly local
optima) while the first one is unimodal. Further
properties are that the first objective is highly
assymetric and the second has a conditioning of about 10.
Contained in the moderate - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 24 in dimension 5 for the first instance.
: Attractive sector/Schaffer F7, condition 10¶
Combination of the attractive sector function ( in the
bbob
suite) and the Schaffer F7 function with condition number 10
( in the
bbob
suite).
Both objectives are non-separable and asymmetric. While the first objective is unimodal, the second one is a highly multi-modal function with a low conditioning where frequency and amplitude of the modulation vary.
Contained in the moderate - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 25 in dimension 5 for the first instance.
: Attractive sector/Schwefel x*sin(x)¶
Combination of the attractive sector function ( in the
bbob
suite) and the Schwefel function ( in the
bbob
suite).
The first objective is non-separable, unimodal, and asymmetric.
The second objective is partially separable and highly multimodal—having
the most prominent minima located comparatively close to the
corners of the unpenalized search area.
Contained in the moderate - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 26 in dimension 5 for the first instance.
: Attractive sector/Gallagher 101 peaks¶
Combination of the attractive sector function ( in the
bbob
suite) and the Gallagher function with 101 peaks ( in the
bbob
suite).
Both objective functions are non-separable but only the first is unimodal. The first objective function is furthermore asymmetric. The second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.
Contained in the moderate - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 27 in dimension 5 for the first instance.
: Rosenbrock original/Rosenbrock original¶
Combination of two Rosenbrock functions ( in the
bbob
suite).
Both objectives are partially separable (tri-band structure) and have a local optimum with an attraction volume of about 25%.
Contained in the moderate - moderate function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 28 in dimension 5 for the first instance.
: Rosenbrock original/Sharp ridge¶
Combination of the Rosenbrock function ( in the
bbob
suite) and the
sharp ridge function ( in the
bbob
suite).
The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective is unimodal and non-separable and, for optimizing it, a sharp, i.e., non-differentiable ridge has to be followed.
Contained in the moderate - ill-conditioned function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 29 in dimension 5 for the first instance.
: Rosenbrock original/Sum of different powers¶
Combination of the Rosenbrock function ( in the
bbob
suite) and the sum of different powers function
( in the
bbob
suite).
The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective function is unimodal and non-separable. When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.
Contained in the moderate - ill-conditioned function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 30 in dimension 5 for the first instance.
: Rosenbrock original/Rastrigin¶
Combination of the Rosenbrock function ( in the
bbob
suite) and the Rastrigin function
( in the
bbob
suite).
The first objective function is partially separable (tri-band structure)
and has a local optimum with an attraction volume of about 25%.
The second objective function is non-separable and
highly multi-modal (roughly local
optima).
Contained in the moderate - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 31 in dimension 5 for the first instance.
: Rosenbrock original/Schaffer F7, condition 10¶
Combination of the Rosenbrock function ( in the
bbob
suite) and the
Schaffer F7 function with condition number 10
( in the
bbob
suite).
The first objective function is partially separable (tri-band structure) and has a local optimum with an attraction volume of about 25%. The second objective function is non-separable, asymmetric, and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.
Contained in the moderate - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 32 in dimension 5 for the first instance.
: Rosenbrock original/Schwefel x*sin(x)¶
Combination of the Rosenbrock function ( in the
bbob
suite) and the
Schwefel function ( in the
bbob
suite).
Both objective functions are partially separable.
While the first objective function has a local optimum with an attraction
volume of about 25%, the second objective function is highly
multimodal—having the most prominent minima located
comparatively close to the corners of its unpenalized search area.
Contained in the moderate - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 33 in dimension 5 for the first instance.
: Rosenbrock original/Gallagher 101 peaks¶
Combination of the Rosenbrock function ( in the
bbob
suite) and
the Gallagher function with 101 peaks ( in the
bbob
suite).
The first objective function is partially separable, the second one non-separable. While the first objective function has a local optimum with an attraction volume of about 25%, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.
Contained in the moderate - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 34 in dimension 5 for the first instance.
: Sharp ridge/Sharp ridge¶
Combination of two sharp ridge functions ( in the
bbob
suite).
Both objective functions are unimodal and non-separable and, for optimizing them, two sharp, i.e., non-differentiable ridges have to be followed.
Contained in the ill-conditioned - ill-conditioned function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 35 in dimension 5 for the first instance.
: Sharp ridge/Sum of different powers¶
Combination of the sharp ridge function ( in the
bbob
suite) and the
sum of different powers function
( in the
bbob
suite).
Both functions are uni-modal and non-separable. For optimizing the first objective, a sharp, i.e., non-differentiable ridge has to be followed. When approaching the second objective’s optimum, the sensitivies of the variables in the rotated search space become more and more different.
Contained in the ill-conditioned - ill-conditioned function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 36 in dimension 5 for the first instance.
: Sharp ridge/Rastrigin¶
Combination of the sharp ridge function ( in the
bbob
suite) and the Rastrigin function
( in the
bbob
suite).
Both functions are non-separable. While the first one
is unimodal and non-differentiable at its ridge, the second objective
function is highly multi-modal (roughly local optima).
Contained in the ill-conditioned - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 37 in dimension 5 for the first instance.
: Sharp ridge/Schaffer F7, condition 10¶
Combination of the sharp ridge function ( in the
bbob
suite) and the
Schaffer F7 function with condition number 10
( in the
bbob
suite).
Both functions are non-separable. While the first one is unimodal and non-differentiable at its ridge, the second objective function is asymmetric and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.
Contained in the ill-conditioned - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 38 in dimension 5 for the first instance.
: Sharp ridge/Schwefel x*sin(x)¶
Combination of the sharp ridge function ( in the
bbob
suite) and the
Schwefel function ( in the
bbob
suite).
While the first objective function is unimodal, non-separable, and
non-differentiable at its ridge, the second objective function is highly
multimodal—having the most prominent minima located
comparatively close to the corners of its unpenalized search area.
Contained in the ill-conditioned - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 39 in dimension 5 for the first instance.
: Sharp ridge/Gallagher 101 peaks¶
Combination of the sharp ridge function ( in the
bbob
suite) and the
Gallagher function with 101 peaks ( in the
bbob
suite).
Both objective functions are non-separable. While the first objective function is unimodal and non-differentiable at its ridge, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.
Contained in the ill-conditioned - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 40 in dimension 5 for the first instance.
: Sum of different powers/Sum of different powers¶
Combination of two sum of different powers functions
( in the
bbob
suite).
Both functions are uni-modal and non-separable where the sensitivies of the variables in the rotated search space become more and more different when approaching the objectives’ optima.
Contained in the ill-conditioned - ill-conditioned function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 41 in dimension 5 for the first instance.
: Sum of different powers/Rastrigin¶
Combination of the sum of different powers functions
( in the
bbob
suite) and the Rastrigin function
( in the
bbob
suite).
Both objective functions are non-separable. While the first one
is unimodal, the second objective
function is highly multi-modal (roughly local optima).
Contained in the ill-conditioned - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 42 in dimension 5 for the first instance.
: Sum of different powers/Schaffer F7, condition 10¶
Combination of the sum of different powers functions
( in the
bbob
suite) and the Schaffer F7 function with
condition number 10 ( in the
bbob
suite).
Both objective functions are non-separable. While the first one is unimodal with an increasing conditioning once the optimum is approached, the second objective function is asymmetric and highly multi-modal with a low conditioning where frequency and amplitude of the modulation vary.
Contained in the ill-conditioned - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 43 in dimension 5 for the first instance.
: Sum of different powers/Schwefel x*sin(x)¶
Combination of the sum of different powers functions
( in the
bbob
suite) and the Schwefel function ( in the
bbob
suite).
Both objectives are non-separable.
While the first objective function is unimodal,
the second objective function is highly multimodal—having the most
prominent minima located comparatively close to the corners
of its unpenalized search area.
Contained in the ill-conditioned - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 44 in dimension 5 for the first instance.
: Sum of different powers/Gallagher 101 peaks¶
Combination of the sum of different powers functions
( in the
bbob
suite) and the Gallagher function with
101 peaks ( in the
bbob
suite).
Both objective functions are non-separable. While the first objective function is unimodal, the second objective function has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function). The conditioning around the global optimum of the second objective function is about 30.
Contained in the ill-conditioned - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 45 in dimension 5 for the first instance.
: Rastrigin/Rastrigin¶
Combination of two Rastrigin functions
( in the
bbob
suite).
Both objective functions are non-separable and highly multi-modal
(roughly local optima).
Contained in the multi-modal - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 46 in dimension 5 for the first instance.
: Rastrigin/Schaffer F7, condition 10¶
Combination of the Rastrigin function
( in the
bbob
suite) and the Schaffer F7 function with
condition number 10 ( in the
bbob
suite).
Both objective functions are non-separable and highly multi-modal.
Contained in the multi-modal - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 47 in dimension 5 for the first instance.
: Rastrigin/Schwefel x*sin(x)¶
Combination of the Rastrigin function
( in the
bbob
suite) and the Schwefel function ( in the
bbob
suite).
Both objective functions are non-separable and highly multi-modal where
the first has roughly local optima and the most prominent
minima of the second objective function are located
comparatively close to the corners of its unpenalized search area.
Contained in the multi-modal - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 48 in dimension 5 for the first instance.
: Rastrigin/Gallagher 101 peaks¶
Combination of the Rastrigin function
( in the
bbob
suite) and the Gallagher function with
101 peaks ( in the
bbob
suite).
Both objective functions are non-separable and highly multi-modal where
the first has roughly local optima and the second has
101 optima with position and height being unrelated and randomly
chosen (different for each instantiation of the function).
Contained in the multi-modal - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 49 in dimension 5 for the first instance.
: Schaffer F7, condition 10/Schaffer F7, condition 10¶
Combination of two Schaffer F7 functions with
condition number 10 ( in the
bbob
suite).
Both objective functions are non-separable and highly multi-modal.
Contained in the multi-modal - multi-modal function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 50 in dimension 5 for the first instance.
: Schaffer F7, condition 10/Schwefel x*sin(x)¶
Combination of the Schaffer F7 function with
condition number 10 ( in the
bbob
suite)
and the Schwefel function ( in the
bbob
suite).
Both objective functions are non-separable and highly multi-modal. While frequency and amplitude of the modulation vary in an almost regular fashion in the first objective function, the second objective function posseses less global structure.
Contained in the multi-modal - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 51 in dimension 5 for the first instance.
: Schaffer F7, condition 10/Gallagher 101 peaks¶
Combination of the Schaffer F7 function with
condition number 10 ( in the
bbob
suite)
and the Gallagher function with
101 peaks ( in the
bbob
suite).
Both objective functions are non-separable and highly multi-modal. While frequency and amplitude of the modulation vary in an almost regular fashion in the first objective function, the second has 101 optima with position and height being unrelated and randomly chosen (different for each instantiation of the function).
Contained in the multi-modal - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 52 in dimension 5 for the first instance.
: Schwefel x*sin(x)/Schwefel x*sin(x)¶
Combination of two Schwefel functions ( in the
bbob
suite).
Both objective functions are non-separable and highly multi-modal where
the most prominent minima of each objective function are
located comparatively close to the corners of its unpenalized search area.
Due to the combinatorial nature of the Schwefel function, it is likely
in low dimensions that the Pareto set goes through the origin of the
search space.
Contained in the weakly-structured - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 53 in dimension 5 for the first instance.
: Schwefel x*sin(x)/Gallagher 101 peaks¶
Combination of the Schwefel function ( in the
bbob
suite) and the Gallagher function with
101 peaks ( in the
bbob
suite).
Both objective functions are non-separable and highly multi-modal.
For the first objective function, the most prominent minima
are located comparatively close to the corners of its unpenalized search
area. For the second objective, position and height of all
101 optima are unrelated and randomly
chosen (different for each instantiation of the function).
Contained in the weakly-structured - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 54 in dimension 5 for the first instance.
: Gallagher 101 peaks/Gallagher 101 peaks¶
Combination of two Gallagher functions with
101 peaks ( in the
bbob
suite).
Both objective functions are non-separable and highly multi-modal. Position and height of all 101 optima in each objective function are unrelated and randomly chosen and thus, no global structure is present.
Contained in the weakly-structured - weakly-structured function class.
Illustration of search space (first row) and objective space (second row) forbbob-biobj
function 55 in dimension 5 for the first instance.The Extended bbob-biobj-ext
Test Suite and Its Functions¶
Having all combinations of only a subset of the single-objective bbob
functions in a test suite
like the above bbob-biobj
one has
advantages but also a few disadvantages. Using only a subet of the 24 bbob
functions
introduces a bias towards the chosen functions and reduces the amount of different difficulties,
a bi-objective algorithm is exposed to in the benchmarking exercise. Allowing all combinations of
bbob
functions increases the percentage of problems for which both objectives are from different
bbob
function groups while, in practice, it can often be assumed that both objective functions
come from a similar “function domain”.
The rationale behind the following extended bbob-biobj
test suite, denoted as bbob-biobj-ext
,
is to reduce the mentioned effects. To this end, we add all within-group combinations of bbob
functions which are not already in the bbob-biobj
suite and which do not combine a function
with itself. For technical reasons, we also remove the Weierstrass functions ( in the
bbob
suite) because the optimum is not necessarily unique and computing the nadir point
therefore technically more challenging than for the other functions.
This extension adds functions, resulting in
92 functions overall.
The following table details which single-objective bbob
functions are contained in the
92 bbob-biobj-ext
functions (outer column and row annotations) and indicates their IDs.
Note that the IDs of the first 55 bbob0biobj-ext
functions are the same than for the
bbob-biobj
test suite for compatibility reasons.
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
|
![]() |
f1 | f2 | f56 | f57 | f58 | f3 | f4 | f5 | f6 | f7 | f8 | f9 | f10 | |||||||||||
![]() |
f11 | f59 | f60 | f61 | f12 | f13 | f14 | f15 | f16 | f17 | f18 | f19 | ||||||||||||
![]() |
f62 | f63 | ||||||||||||||||||||||
![]() |
f64 | |||||||||||||||||||||||
![]() |
||||||||||||||||||||||||
![]() |
f20 | f65 | f21 | f66 | f22 | f23 | f24 | f25 | f26 | f27 | ||||||||||||||
![]() |
f67 | f68 | ||||||||||||||||||||||
![]() |
f28 | f69 | f29 | f30 | f31 | f32 | f33 | f34 | ||||||||||||||||
![]() |
||||||||||||||||||||||||
![]() |
f70 | f71 | f72 | f73 | ||||||||||||||||||||
![]() |
f74 | f75 | f76 | |||||||||||||||||||||
![]() |
f77 | f78 | ||||||||||||||||||||||
![]() |
f35 | f36 | f37 | f38 | f39 | f40 | ||||||||||||||||||
![]() |
f41 | f42 | f43 | f44 | f45 | |||||||||||||||||||
![]() |
f46 | f47 | f79 | f80 | f48 | f49 | ||||||||||||||||||
![]() |
||||||||||||||||||||||||
![]() |
f50 | f81 | f82 | f51 | f52 | |||||||||||||||||||
![]() |
f83 | |||||||||||||||||||||||
![]() |
||||||||||||||||||||||||
![]() |
f53 | f54 | f84 | f85 | f86 | |||||||||||||||||||
![]() |
f55 | f87 | f88 | f89 | ||||||||||||||||||||
![]() |
f90 | f91 | ||||||||||||||||||||||
![]() |
f92 | |||||||||||||||||||||||
![]() |
The 92 functions of thebbob-biobj-ext
test suite and their IDs (in the table cells) together with the information about which single-objectivebbob
functions are used to define them (outer column and row annotations).
Function Groups¶
Like for the bbob-biobj
test suite, we obtain 15 function
classes to structure the 92 bi-objective functions of the bbob-biobj-ext
test
suite. Depending on whether a function class combines functions from the same or
from different bbob
function classes, each function class contains
8, 12 or just four functions. We are listing
below the function classes and in parenthesis the functions that belong to
the respective class:
- separable - separable (12 functions: f1, f2, f11, f56-64)
- separable - moderate (f3, f4, f12, f13)
- separable - ill-conditioned (f5, f6, f14, f15)
- separable - multi-modal (f7, f8, f16, f17)
- separable - weakly-structured (f9, f10, f18, f19)
- moderate - moderate (8 functions: f20, f21, f28, f65-f69)
- moderate - ill-conditioned (f22, f23, f29, f30)
- moderate - multi-modal (f24, f25, f31, f32)
- moderate - weakly-structured (f26, f27, f33, f34)
- ill-conditioned - ill-conditioned (12 functions: f35, f36, f41, f70-78)
- ill-conditioned - multi-modal (f37, f38, f42, f43)
- ill-conditioned - weakly-structured (f39, f40, f44, f45)
- multi-modal - multi-modal (8 functions: f46, f47, f50, f79-83)
- multi-modal - weakly structured (f48, f49, f51, f52)
- weakly structured - weakly structured (12 functions: f53-55, f84-92)
Normalization and Instances¶
Normalization of the objectives and instances are handled for the bbob-biobj-ext
in the
same manner as for the bbob-biobj
suite, i.e., no normalization of the objective
functions is taking place for the algorithm benchmarking and 15 instances are prescribed for
a typical experiment.
Acknowledgments
This work was supported by the grant ANR-12-MONU-0009 (NumBBO)
of the French National Research Agency. We also thank Ilya Loshchilov and Oswin Krause for their
initial suggestions on how to extend the bbob-biobj
test suite.
References
[BRO2016biperf] | D. Brockhoff, T. Tušar, D. Tušar, T. Wagner, N. Hansen, A. Auger, (2016). Biobjective Performance Assessment with the COCO Platform. ArXiv e-prints, arXiv:1605.01746. |
[BRO2015] | (1, 2) D. Brockhoff, T.-D. Tran, and N. Hansen (2015). Benchmarking Numerical Multiobjective Optimizers Revisited. In Proceedings of the 2015 GECCO Genetic and Evolutionary Computation Conference, pp. 639-646, ACM. |
[HAN2016co] | (1, 2) N. Hansen, A. Auger, O. Mersmann, T. Tušar, D. Brockhoff (2016). COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting, ArXiv e-prints, arXiv:1603.08785. |
[HAN2009] | (1, 2) N. Hansen, S. Finck, R. Ros, and A. Auger (2009). Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions. Research Report RR-6829, Inria, updated February 2010. |
[HAN2011] | N. Hansen, R. Ros, N. Mauny, M. Schoenauer, and A. Auger (2011). Impacts of Invariance in Search: When CMA-ES and PSO Face Ill-Conditioned and Non-Separable Problems. Applied Soft Computing. Vol. 11, pp. 5755-5769. Elsevier. |
[HAN2016ex] | N. Hansen, T. Tušar, A. Auger, D. Brockhoff, O. Mersmann (2016). COCO: The Experimental Procedure, ArXiv e-prints, arXiv:1603.08776. |