GECCO Workshop on Real-Parameter Black-Box Optimization Benchmarking (BBOB 2016) - focus on multi-objective problems

Welcome to the web page of the 6th GECCO Workshop on Real-Parameter Black-Box Optimization Benchmarking (BBOB 2016) with focus on multi-objective problems with two objective functions which took place during GECCO 2016.

WORKSHOP ON REAL-PARAMETER BLACK-BOX OPTIMIZATION BENCHMARKING (BBOB 2016) - with a new focus on multi-objective problems

hold as part of the

2016 Genetic and Evolutionary Computation Conference (GECCO-2016)
July 20-24, Denver, CO, USA
http://gecco-2016.sigevo.org/
Submission Deadline: extended to Sunday, April 17, 2016 (from Saturday, April 3, 2016)
register for news Coco quick start (scroll down a bit) latest Coco release


Quantifying and comparing the performance of optimization algorithms is a difficult and tedious task to achieve. Previously, the Coco platform has provided tools to ease this process for single-objective problems by: (1) an implemented, well-motivated benchmark function testbed, (2) a simple and sound experimental set-up, (3) the generation of output data and (4) the post-processing and presentation of the results in graphs and tables. For the first time, this year, we provide(d) an extension of the Coco platform towards a multi-objective testbed (with two objective functions) and with nearly the same procedure as in previous BBOB workshops.

The remaining tasks for participants were therefore: run your favorite multi-objective black-box optimizer (old or new) by using the wrappers provided and run the post-processing procedure (provided as well) that will generate automatically all the material for a workshop paper (LaTeX templates are provided). A description of the algorithm and the discussion of the results completes the paper writing.

We encourage(d) particularly submissions related to multi-objective algorithms for expensive optimization (with a limited budget) and also algorithms from outside the evolutionary computation community. Please note that submissions related to the existing single-objective BBOB testbeds (noiseless and noisy) were still welcome although the focus was on the new bi-objective testbed.

During the workshop, algorithms and results were presented by the participants. An overall analysis and comparison was accomplished by the organizers and the overall process was critically reviewed.

Updates and News

Get updated about the latest news regarding the workshop and releases and bugfixes of the supporting NumBBO/Coco plateform, by registering at http://numbbo.github.io/register.

Supporting material

Most likely, you want to read the Coco quick start (scroll down a bit). This page also provides the code for the benchmark functions, for running the experiments in C, Java, Matlab, Octave, and Python, and for postprocessing the experiment data into plots, tables, html pages, and publisher-conform PDFs via provided LaTeX templates. Please refer to http://numbbo.github.io/coco-doc/experimental-setup/ for more details on the general experimental set-up for black-box optimization benchmarking.

The latest (hopefully) stable release of the Coco software can be downloaded as a whole here. Please use at least version v1.0 for running your benchmarking experiments.

Documentation of the functions used in the bbob-biobj suite for BBOB 2016 are provided at http://numbbo.github.io/coco-doc/bbob-biobj/functions/ .

Note that the current release of the new Coco platform does not contain the original noisy BBOB testbed, such that you must use the old code at https://numbbo.github.io/coco/oldcode/bboball15.03.tar.gz for the time being if you want to compare your algorithm on the noisy testbed.

Submissions

Submissions of benchmarking results of new or existing numerical optimization algorithms in terms of bi-objective or single-objective optimization were welcome and should have been done through the following form at http://numbbo.github.io/submit.

Eventually, the following papers have been accepted:

  • Ilya Loshchilov and Tobias Glasmachers: Anytime Bi-Objective Optimization with a Hybrid Multi-Objective CMA-ES (HMO-CMA-ES)
  • Oswin Krause, Tobias Glasmachers, Nikolaus Hansen, and Christian Igel: Unbounded Population MO-CMA-ES for the Bi-Objective BBOB Test Suite
  • Kouhei Nishida and Youhei Akimoto: Evaluating the Population Size Adaptation Mechanism for CMA-ES on the BBOB Noiseless Testbed
  • Kouhei Nishida and Youhei Akimoto: Evaluating the Population Size Adaptation Mechanism for CMA-ES on the BBOB Noisy Testbed
  • Cheryl Wong, Abdullah Al-Dujaili, and Suresh Sundaram: Hypervolume-based DIRECT for Multi-Objective Optimisation
  • Abdullah Al-Dujaili and Suresh Sundaram: A MATLAB Toolbox for Surrogate-Assisted Multi-Objective Optimization: A Preliminary Study
  • Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar, and Tobias Wagner: Benchmarking the Pure Random Search on the Bi-objective BBOB-2016 Testbed
  • Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar, and Tobias Wagner: The Impact of Variation Operators on the Performance of SMS-EMOA on the Bi-objective BBOB-2016 Test Suite
  • Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar, and Tobias Wagner: Benchmarking MATLAB's gamultiobj (NSGA-II) on the Bi-objective BBOB-2016 Test Suite
  • Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar, and Tobias Wagner: Benchmarking RM-MEDA on the Bi-objective BBOB-2016 Test Suite
  • Tea Tušar and Bogdan Filipič: Performance of the DEMO algorithm on the bi-objective BBOB test suite
  • Anne Auger, Dimo Brockhoff, Nikolaus Hansen, Dejan Tušar, Tea Tušar, and Tobias Wagner: The Impact of Search Volume on the Performance of RANDOMSEARCH on the Bi-objective BBOB-2016 Test Suite

Data

The data sets of all submitted algorithms can be found at https://numbbo.github.io/data-archive/bbob-biobj/ for the bbob-biobj test suite and at https://numbbo.github.io/data-archive/bbob and https://numbbo.github.io/data-archive/bbob-noisy/ for the bbob and bbob-noisy test suites respectively.

Workshop Schedule

All BBOB-2016 sessions took place on the first day of GECCO (July 20, 2016) in the Wind Star B room. Speakers are highlighted with a star behind the name. Please click on the provided links to download the slides.

Session I
08:30 - 09:30 The BBOBies: Introduction to Blackbox Optimization Benchmarking (slides)
09:30 - 09:55 Tea Tušar*, Bogdan Filipič: Performance of the DEMO algorithm on the bi-objective BBOB test suite (slides)
09:55 - 10:20 Ilya Loshchilov, Tobias Glasmachers*: Anytime Bi-Objective Optimization with a Hybrid Multi-Objective CMA-ES (HMO-CMA-ES) (slides)
Session II
10:40 - 10:55 The BBOBies: Session Introduction (slides)
10:55 - 11:20 Cheryl Wong*, Abdullah Al-Dujaili, and Suresh Sundaram: Hypervolume-based DIRECT for Multi-Objective Optimisation (slides)
11:20 - 11:45 Abdullah Al-Dujaili and Suresh Sundaram (speaker: Cheryl Wong): A MATLAB Toolbox for Surrogate-Assisted Multi-Objective Optimization: A Preliminary Study (slides)
11:45 - 12:10 Oswin Krause*, Tobias Glasmachers, Nikolaus Hansen, and Christian Igel: Unbounded Population MO-CMA-ES for the Bi-Objective BBOB Test Suite (slides)
12:10 - 12:30 The BBOBies: Session Wrap-up (slides)
Session III
14:00 - 14:15 The BBOBies: Session Introduction (slides)
14:15 - 14:40 Kouhei Nishida* and Youhei Akimoto: Evaluating the Population Size Adaptation Mechanism for CMA-ES (slides)
14:40 - 15:05 The BBOBies: Wrap-up of all BBOB-2016 Results (slides)
15:05 - 15:30 Thomas Weise*: optimizationBenchmarking.org: An Introduction
15:30 - 15:50 Open Discussion

Important Dates

  • 01/20/2016 first version of the new Coco platform released as 0.5-beta
  • 01/30/2016 (planned: 01/29/2016) release 0.7-beta of the Coco software with the main functionality to run experiments
  • (planned: 02/12/2016, replaced by 7 intermediate releases) first complete release 0.9 of the software
  • 03/29/2016 (planned: 03/18/2016) final release 1.0 for producing the papers
  • 04/17/2016 new paper and data submission deadline (extended from 04/02/2016)
  • 04/20/2016 decision notification
  • 05/04/2016 deadline camera-ready papers
  • 07/20/2016 workshop

Organizers

  • Anne Auger, Inria Saclay - Ile-de-France
  • Dimo Brockhoff, Inria Lille - Nord Euruope
  • Nikolaus Hansen, Inria Saclay - Ile-de-France
  • Dejan Tušar, Inria Lille - Nord Europe
  • Tea Tušar, Inria Lille - Nord Europe
  • Tobias Wagner, TU Dortmund University