cocoex.ExperimentRepeater

class documentation

class ExperimentRepeater:

Constructor: ExperimentRepeater(budget_multiplier, min_successes)

Allow to automatically repeat an experiment based on budget and successes.

The class tracks evaluations and the final_target_hit status ("successes") of the problem instances of a cocoex.Suite. Based on the stored information, the done method allows to run sub-experiment sweeps until some given budget is exhaused, namely, budget_multiplier * dimension.

Example code snippet:

[...]
repeater = cocoex.ExperimentRepeater(budget_multiplier)
[...]
while not repeater.done():
    for problem in suite:
        if repeater.done(problem):
            continue
        problem.observe_with(observer)  # generates the data for cocopp postprocessing
        fmin(problem, repeater.initial_solution_proposal(problem))
        repeater.track(problem)  # record evaluations and success

The call repeater.done() checks whether the budget is exhausted or enough trials were successful on all problems, or an instance lacks a trial on some problems, it is akin to, but not exactly the same as, all(repeater.done(p) for p in suite). ExperimentRepeater(0) does exactly one sweep and no repetitions.

The stored data can be queried, in which case problem can be a (id_function, dimension, id_instance) tuple. In particular, the methods evaluations, successes and trials return a list of the respective data for each instance of instances. The last entry of the argument tuple is ignored in these calls and can be None or omitted. The all method allows to query all (id_function, dimension) entries at once:

>> evals = repeater.all(repeater.evaluations)  # returns a `dict`
>> fun, dim = 1, 10
>> evals[(fun, dim)] == repeater.evaluations((fun, dim, None))
True

Terminology: a "problem instance" is defined by the triple (id_function, dimension, id_instance) and "problem" may refer to any or all problem instances with the same (id_function, dimension).

Details

Only the track method relies on passing a cocoex.Problem instance, otherwise a (id_function, dimension, id_instance) tuple is eligible. The tracked information is stored in the ._data dictionary with (id_function, dimension) as keys. The data method implements safe access to this dictionary.

When problem instances are repeated in a single suite, they may be partially skipped after the first full sweep. That is, the configuration 1-5,1-5,1-5 can also lead to four trials of each instance 1-5, because all instances have been repeated the same number of times.

>>> import cocoex  # some not very meaningful testing
>>> repeater = cocoex.ExperimentRepeater(2)
>>> assert not repeater.done((1, 2))
>>> assert repeater._sweeps == 0 and not repeater.done() and repeater._sweeps == 1
>>> assert len(repeater.data((1, 10))) == 0
>>> assert len(repeater.evaluations((1, 2))) == 0

Method	`__init__`	`min_successes=11` for instances 1-5 provokes at least three
Method	`all`	return `dict` with `method(key)` as values and
Method	`budget_exhausted`	compare average evaluations per instance with `budget_from_dimension`
Method	`budget_from_dimension`	return budget_multiplier x ((problem.dimension or dimension) + self.offset).
Method	`data`	return reference to `dict` with instance as keys,
Method	`data_of_instance`	return `list` of `(evaluations, success)` of problem instance
Method	`done`	return `False` iff this/any `problem` instance remains to be (re-)run.
Method	`evaluations`	return `list` of summed evaluations per problem instance
Method	`initial_solution_proposal`	return allzeros in the first of any `nonzero_odds + 1` trials
Method	`instances`	return `list` of instance IDs tracked for (id_function, dimension) of `problem`.
Method	`message_sweep`	return status message and '' when nothing was tracked yet
Method	`missing_trials`	return number of missing trials for this problem instance
Method	`n_problem_instances`	number of problem instances (of `problem`) with at least one trial
Method	`n_trials`	current number of tracked trials from all instances (of `problem`)
Method	`problem_to_ids`	return `(id_function, dimension, id_instance)` of `problem`
Method	`remaining_problems`	return `list` of (probably) remaining problem instances to run.
Method	`succeeded`	return `True` iff problem had at least `min_success` successes
Method	`successes`	return `list` of successes per instance
Method	`track`	record problem instance evaluations and success (target hit)
Method	`trials`	return `list` of number of recorded trials for each instance
Instance Variable	`max_sweeps`	Undocumented
Instance Variable	`params`	Undocumented
Property	`n_problems`	current number of tracked problems (different `(id_function, dimension)`)
Method	`_assert_consistencies`	Undocumented
Method	`_check`	check problem to avoid kernel crashes, raise ValueError
Method	`_empty_data`	should always return an empty list, called in the `data` method
Method	`_n_instance_data`	Undocumented
Method	`_to_key`	return `data`-key of problem, namely `(id_function, dimension)`
Instance Variable	`_calls`	Undocumented
Instance Variable	`_data`	a `dict[(fun, dim)]` of `dict[iinst]` of `list` of `(evaluations, success)` `tuples`
Instance Variable	`_dimension_offset`	budget = (dimension + offset) * budget_multiplier
Instance Variable	`_sweeps`	Undocumented

def __init__(self, budget_multiplier, min_successes=11): ¶

min_successes=11 for instances 1-5 provokes at least three

sweeps, hence 15 trials, given the budget_multiplier is large enough and the algorithm terminates early enough before the budget is exhausted. 3 x 1-5 is the instance-setup from BBOB 2009, however the choice of the initial solution still slightly differs.

def all(self, method): ¶

return dict with method(key) as values and

with all (id_function, dimension) as keys.

Works with the instance methods evaluations, successes, budget_exhausted, trials, or instances as argument or the respective strings.

def budget_exhausted(self, problem): ¶

compare average evaluations per instance with budget_from_dimension

def budget_from_dimension(self, problem_or_dimension): ¶

return budget_multiplier x ((problem.dimension or dimension) + self.offset).

By default, offset == 0.

def data(self, problem): ¶

return reference to dict with instance as keys,

where nonexisting entries are returned as empty but not permanently created.

def data_of_instance(self, problem): ¶

return list of (evaluations, success) of problem instance

def done(self, problem=None, message=True): ¶

return False iff this/any problem instance remains to be (re-)run.

When problem is not None, return True if this problem instance does not require another trial, however False as long as done has not been called twice without argument (thereby assuming that the loop starts with while repeater.done()).

When problem is None, return True if no single recorded problem requires another trial.

Return invariably False before the second call of done() without argument, which is considered to happen right before the second sweep when the first sweep is finished.

Details

done() without argument gives only consistent results before or after a full first sweep. In particular, during the first sweep it cannot account for problems that have not yet been run once.

Calling done() increments the sweep counter iff it returns False, the default for max_sweeps is 1e4. The attribute can be directly reassigned at any time.