cocopp.pproc.DataSet

Compare indexEntry instances.

Instantiate a DataSet.

The first three input arguments correspond to three consecutive lines of an index file (.info extension).

Undocumented

Undocumented

return minimum size not smaller than sample_size such that modulo self.nbRuns() == 0

Sets the attributes ert and target from the attribute evals.

checks consistency of data set according to - number of instances - instances used

Returns a dictionary of the instances.

The key is the instance Id, the value is a list of index.

Returns a dictionary of the instances and their count.

The keys are instance id and the values are the number of repetitions of such instance.

Remove all runs with id instance_number.

Fix (hopefully) all related attributes, that is, drop the respective values in lists or arrays.

Determine the average number of f-evals for each target in targets list.

The average is weighted correcting for imbalanced trial instances.

If a target is not reached within trial itrail, self.maxevals[itrial] contributes to the average.

Equals to sum(evals(target)) / nbruns. If ERT is finite this equals to ERT * psucc == (sum(evals) / ntrials / psucc) * psucc, where ERT, psucc, and evals are a function of target.

Details: this should be the same as the precomputed ert property.

Determine the expected running time (ERT) to reach target values. The value is numpy.inf, if the target was never reached.

Details: uses attribute self.ert.

return len(targets) data rows self.evals[i, 1:].

If bootstrap, the "data rows" are len(self.evals[i, 1:]) values drawn with replacement from self.evals[i, 1:]. This may be useful to estimate variances (at some point).

If complement_unsuccessful (given append_instances is False), replace nonfinite values (namely nan) either with the respective maxevals * complement_unsuccessful or with complement_unsucccessful(maxevals) if complement_unsucccessful is callable. This allows, for example, to return unsuccessful "runtimes" as their negative values by passing complement_unsucccessful=-1 which can then be used with toolsstats.simulated_evals.

Returned rows are those with the closest target that is not larger than the given target so that self.evals[i, 0] <= target and, if i > 0, self.evals[i - 1, 0] > target. When the target was never reached the row is np.array(self.nbRuns() * [np.nan]).

Makes by default a copy of the data, however this might change in future.

Details:

maxevals are usually read from the .info files, however with constraints they are read from the .tdat files as the .info files do not contain g-evaluations.

If instances_are_appended, the maxevals attribute is not consistent with evals_appended. The latter is abandoned anyways and superseded by evals_balanced (which is more or less automatic)!?

return result of detEvals for each instance individually

in an OrderedDict whos keys are all elements of instancenumbers.

raw_data=True means no instance balancing/repetitions.

See detEvals for further keyword arguments.

return the number of successful runs for each target.

Unless bool(raw_values) is True, the number of runs are for each instance expanded to their least common multiplier if genericsettings.balance_instances, hence the success events are not necessarily independent in this case.

Details: if raw_values is an int, only the first raw_values columns of the data set are used. If raw_values is True, all data without any balancing repetitions are used.

See also detSuccessRates.

return a np.array with the success rate for each target in targets, easiest target first.

If genericsetting.balance_instances, the rate is weighted such that each instance has the same weight independently of how often it was repeated.

Return a list of arrays of "simulated runtimes"

with an interface analogous to detEvals. Elements of the returned list correspond to the elements of targets.

TODO: return unsuccessful optionally as negative numbers instead of nan?

samplesize is by default the smallest multiple of nbRuns() that is not smaller than 45 and it remains the same even when instances are not uniform and hence simulated separately. If samplesize == 'individual' and instances are uniform, an adhoc sample size is computed for each target separately based on the data.

bootstrap is passed to detEvals such that the simulated runs use a bootstrapped subset. This will increase the variance from repeated evals_with_simulated_restarts calls. This may become useful to measure dispersion of runtime profiles.

instance, when given, returns only the simulated runtimes from this instance. The default samplesize remains the same based on the overall .nbRuns().

np.sort(np.concatenate(return_value)) provides the combined sorted ECDF data over all targets which can be plotted with pyplot.step like:

x = np.sort(np.concatenate(return_value))
pyplot.step(np.concatenate([x, [x[-1]]]), np.linspace(0, 1, len(x) + 1))

Don't use 'individual' sample sizes when the data are meant to be combined.

Unsuccessful data are represented as np.nan.

Simulated restarts are used for unsuccessful runs. The usage of detEvals or evals_with_simulated_restarts should be largely interchangeable, while the latter has a "success" rate of either 0 or 1 when instances_are_uniform.

>>> import numpy as np
>>> import cocopp  # candidate data sets: 13/cma (fast) or 13/ipop (more diverse)
>>> print('load data set'); dsl = cocopp.load2('b/2009/bay')  # doctest:+ELLIPSIS
load data set...
>>> for t in [1, 0.1, 1e-8]:
...     evals = ds.detEvals([t], instance=None)[0]
...     evals.sort()  # sort nan to the end
...     simevals = ds.evals_with_simulated_restarts([t],
...                             samplesize=ds.nbRuns())[0]
...     simevals.sort()
...     nans = sum(np.isnan(evals))
...     if nans:
...         evals, simevals = evals[:-nans], simevals[:-nans]
...     # works only when the unsuccessful are the longest
...     assert all(evals == simevals), (evals, simevals)
>>> instance = np.random.randint(5) + 1
>>> target = 10**(10 * np.random.rand() - 8)
>>> d = ds.detEvals([target], instance=instance)[0]
>>> simevals = ds.evals_with_simulated_restarts([target],
...                 samplesize=5, instance=instance)[0]
>>> if all(np.isfinite(d)):  # check that exactly one value is not repeated
...     assert sum(simevals) in [2 * sum(d) - di for di in d], (simevals, d)

Details:

• For targets where all runs were successful, samplesize=nbRuns() is sufficient (and preferable) if randint is derandomized.
• A single successful running length is computed by adding uniformly randomly chosen running lengths until the first time a successful one is chosen.
• if samplesize >> nbRuns and nsuccesses is large, the data representation becomes somewhat inefficient. Similarly, for nsuccesses==0 it is samplesize * [np.nan].
• When not instances_are_uniform, simulated restarts are taken from the same instance only and the samplesize is (almost) the same for all instances.

TODO: revisit the adhoc best_samplesize and

_byinstance_samplesize computations

Determine the running lengths for reaching the target values.

Undocumented

print text info to stdout

return print info as string

return OrderedDict of index lists for each instance.

raw_values means no instance balancing, otherwise the indices refer to instancenumbers_balanced whos first indices are the same as in instancenumbers.

The default for raw_values seems more appropriate for internal usage, in particular as .instancenumbers_balanced equals .instancenumbers when evals were not balanced?

The index starts with 0 conforming with instancenumbers, maxevals, detEvals and others. However in the evals array, column 0 contains f-values and the instance indices start with 1.

Undocumented

keep only instances with id in instance_ids,

return list of deleted instance ids.

DEPRECATED: use rather .percentiles(targets, 50) or

np.median(ds.detEvals(targets, complement_unsuccessful=np.inf), 1) instead of ds.median_evals(targets, append_instances=False). .percentiles applies simulated restarts, within-instance only when instances are not uniform, instead of appending runs with the same instance number.

return median for each row in self.evals, unsuccessful runs count.

If target_values is not None compute the median evaluations to reach the given target values.

Return np.nan if the median run was unsuccessful.

If append_instances and self.evals_are_appended, append all instances from the same instance numbers as if the algorithm was restarted. self.evals_are_appended is True if the resulting number of (unique) instances is at least genericsettings.appended_evals_minimal_trials and if testbedsettings.current_testbed.instances_are_uniform.

CAVEAT: instance appending is not fully consistent with the ert attribute and with simulated restarts, both of which are computed over all instance.

Details: copies the evals attribute and sets nan to inf in order to get the median with nan values in the sorting.

Returns the maximum number of function evaluations over all runs (trials), obsolete and replaced by attribute max_eval

Returns the number of runs depending on genericsettings.balance_instances.

return the evaluations percentiles for each target in target_values.

Unsuccessful runs are indicated with np.nan and always sorted to the end (largest values and/or large percentiles).

If and only if prctiles is a scalar, the resulting type for each target is a scalar, otherwise it is a list.

The evaluations (runtimes) are measured with simulated restarts. The specific behavior depends on instances_are_uniform. If instances_are_uniform is None its value is inferred from cocopp.testbedsettings.current_testbed.instances_are_uniform. Then, if instances_are_uniform, runtimes (evaluations) are determined with simulated restarts from all instances. Otherwise, simulated restarts come only from the same instance, akin to within-experiment restarts. In the former case, _all_ percentiles are either finite or np.nan. In the latter case, any number of the larger percentiles may be np.nan.

>>> import numpy as np
>>> import cocopp  # candidate data sets: 13/cma (fast) or 13/ipop (more diverse)
>>> print('load data set'); dsl = cocopp.load2('b/2009/bay')  # doctest:+ELLIPSIS
load data set...
>>> pp0 = dsl[0].percentiles
>>> pt = pp0([1, 0.1, 0.01], 50)  # median evals of three targets
>>> [float(p) for p in pt]
[16.0, 105.0, 200.0]
>>> pt = pp0([1e0])  # default percentiles (5, 25, 50,...) of target 1e0
>>> [float(p) for p in pt[0]]
[3.0, 10.25, 16.0, 38.5, 53.75]
>>> pt = pp0([1], instances_are_uniform=False)
>>> [float(p) for p in pt[0]]
[3.0, 10.0, 16.0, 39.0, 55.0]
>>> assert np.isfinite(pp0([1e-2], 100)[0])  # with simulated restarts
>>> # within instance restarts: each instance has a succesful trial, hence
>>> bool(np.isfinite(pp0([1e-2], 100, False)[0]))
True
>>> assert np.isnan(pp0([1e-9], 50)[0])

Details: data to determine the percentiles are generated by the .evals_with_simulated_restarts method.

Save this instance to a pickle file.

Saves this instance to a (by default gzipped) pickle file. If not specified by argument outputdir, the location of the pickle is given by the location of the first index file associated to this instance.

This method will overwrite existing files.

This method is not in use. Instead, get_DataSetList pickles a DataSetList.

plot all data from evals attribute and the median.

Plotted are Delta f-value vs evaluations. The sort for the color heatmap is based on the final performance.

color_map is a list or generator with self.nbRuns() colors and used as iter(color_map). The maps can be generated with the matplotlib.colors.LinearSegmentedColormap attributes of module matplotlib.cm. Default is brg between 0 and 0.5, like plt.cm.brg(np.linspace(0, 0.5, self.nbRuns())).

**plot_kwargs is updated with plot_formats and passed to plot_function (for convenience).

Small dots indicate the last evaluation for runs that did no reach the final/smallest target.

plot data of funvals attribute, versatile

TODO: seems outdated on 19/8/2016 and 05/2019 (would fail as it was

using "isfinite" instead of "np.isfinite" and is not called from anywhere)

Splits the post-processed data arrays by trials.

return OrderedDict with number of successes for each instance