cocopp.pproc.DataSet : API documentation

Known subclasses: cocopp.bestalg.BestAlgSet

Unit element for the COCO post-processing.

An instance of this class is created from one unit element of
experimental data. One unit element would correspond to data for a
given algorithm (a given :py:attr:`algId` and a :py:attr:`comment`
line) and a given problem (:py:attr:`funcId` and
:py:attr:`dimension`).

Class attributes:

  - *funcId* -- function Id (integer)
  - *dim* -- dimension (integer)
  - *indexFiles* -- associated index files (list of strings)
  - *dataFiles* -- associated data files (list of strings)
  - *comment* -- comment for the setting (string)
  - *targetFuncValue* -- final target function value (float), might be missing
  - *precision* -- final ftarget - fopt (float), data with 
                   target[idat] < precision are optional and not relevant.  
  - *algId* -- algorithm name (string)
  - *evals* -- data aligned by function values (2xarray, list of data rows [f_val, eval_run1, eval_run2,...]), cave: in a portfolio data rows can have different lengths
  - *funvals* -- data aligned by function evaluations (2xarray)
  - *maxevals* -- maximum number of function evaluations (array)
  - *finalfunvals* -- final function values (array)
  - *readmaxevals* -- maximum number of function evaluations read
                      from index file (array)
  - *readfinalFminusFtarget* -- final function values - ftarget read
                                from index file (array)
  - *pickleFile* -- associated pickle file name (string)
  - *target* -- target function values attained (array)
  - *ert* -- ert for reaching the target values in target (array)
  - *instancenumbers* -- list of numbers corresponding to the instances of
                 the test function considered (list of int)
  - *isFinalized* -- list of bool for if runs were properly finalized

:py:attr:`evals` and :py:attr:`funvals` are arrays of data collected
from :py:data:`N` data sets.

Both have the same format: zero-th column is the value on which the
data of a row is aligned, the :py:data:`N` subsequent columns are
either the numbers of function evaluations for :py:attr:`evals` or
function values for :py:attr:`funvals`.

A short example::

    >>> from __future__ import print_function    
    >>> import sys
    >>> import os
    >>> import urllib
    >>> import tarfile
    >>> import cocopp
    >>> cocopp.genericsettings.verbose = False # ensure to make doctests work
    >>> def setup(infoFile):
    ...     if not os.path.exists(infoFile):
    ...         filename = cocopp.archives.bbob.get_one('2009/BIPOP-CMA-ES_hansen')
    ...         tarfile.open(filename).extractall(cocopp.archives.bbob.local_data_path)
    >>> infoFile = os.path.join(cocopp.archives.bbob.local_data_path, 'BIPOP-CMA-ES', 'bbobexp_f2.info')
    >>> print('get'); setup(infoFile) # doctest:+ELLIPSIS
    get...
    >>> dslist = cocopp.load(infoFile)
      Data consistent according to consistency_check() in pproc.DataSet
    >>> print(dslist)  # doctest:+ELLIPSIS
    [DataSet(BIPOP-CMA-ES on f2 2-D), ..., DataSet(BIPOP-CMA-ES on f2 40-D)]
    >>> type(dslist)
    <class 'cocopp.pproc.DataSetList'>
    >>> len(dslist)
    6
    >>> ds = dslist[3]  # a single data set of type DataSet
    >>> ds
    DataSet(BIPOP-CMA-ES on f2 10-D)
    >>> for d in dir(ds):  # doctest:+ELLIPSIS
    ...     if '_old_' not in d:
    ...         print(d)
    _DataSet__parseHeader
    __class__
    __delattr__
    __dict__
    ...
    __getattribute__
    ...
    __module__
    __ne__
    __new__
    __reduce__
    __reduce_ex__
    __repr__
    __setattr__
    __sizeof__
    __str__
    __subclasshook__
    __weakref__
    _argsort
    _attributes
    _complement_data
    _cut_data
    _detEvals2
    _detMaxEvals
    _evals
    _extra_attr
    algId
    algs
    comment
    computeERTfromEvals
    consistency_check
    createDictInstance
    createDictInstanceCount
    dataFiles
    detAverageEvals
    detERT
    detEvals
    detSuccessRates
    detSuccesses
    dim
    ert
    evals
    evals_
    evals_with_simulated_restarts
    finalfunvals
    funcId
    funvals
    generateRLData
    get_data_format
    get_suite
    get_testbed_name
    indexFiles
    info
    instancenumbers
    isBiobjective
    isFinalized
    mMaxEvals
    max_eval
    maxevals
    median_evals
    nbRuns
    pickle
    plot
    plot_funvals
    precision
    readfinalFminusFtarget
    readmaxevals
    reference_values
    splitByTrials
    success_ratio
    target
    testbed_name
    >>> all(ds.evals[:, 0] == ds.target)  # first column of ds.evals is the "target" f-value
    True
    >>> # investigate row 0,10,20,... and of the result columns 0,5,6, index 0 is ftarget
    >>> ev = ds.evals[0::10, (0,5,6)]  # doctest:+ELLIPSIS  
    >>> assert 3.98107170e+07 <= ev[0][0] <= 3.98107171e+07 
    >>> assert ev[0][1] == 1
    >>> assert ev[0][2] == 1
    >>> assert 6.07000000e+03 <= ev[-1][-1] <= 6.07000001e+03
    >>> # show last row, same columns
    >>> ev = ds.evals[-1,(0,5,6)]  # doctest:+ELLIPSIS
    >>> assert ev[0] == 1e-8
    >>> assert 5.67600000e+03 <= ev[1] <= 5.67600001e+03
    >>> ds.info()  # prints similar data more nicely formated 
    Algorithm: BIPOP-CMA-ES
    Function ID: 2
    Dimension DIM = 10
    Number of trials: 15
    Final target Df: 1e-08
    min / max number of evals per trial: 5676 / 6346
       evals/DIM:  best     15%     50%     85%     max |  aRT/DIM  nsucc
      ---Df---|-----------------------------------------|----------------
      1.0e+03 |     102     126     170     205     235 |    164.2  15
      1.0e+01 |     278     306     364     457     480 |    374.5  15
      1.0e-01 |     402     445     497     522     536 |    490.8  15
      1.0e-03 |     480     516     529     554     567 |    532.8  15
      1.0e-05 |     513     546     563     584     593 |    562.5  15
      1.0e-08 |     568     594     611     628     635 |    609.6  15

    >>> import numpy as np
    >>> idx = list(range(0, 50, 10)) + [-1]
    >>> # get aRT average runtime for some targets
    >>> t = np.array([idx, ds.target[idx], ds.ert[idx]]).T  # doctest:+ELLIPSIS  
    >>> assert t[0][0] == 0
    >>> assert t[0][2] == 1
    >>> assert t[-1][-2] == 1e-8
    >>> assert 6.09626666e+03 <= t[-1][-1] <= 6.09626667e+03

    Note that the load of a data set depends on the set of instances
    specified in testbedsettings' TestBed class (or its children)
    (None means all instances are read in):
    >>> import sys
    >>> import os
    >>> import urllib
    >>> import tarfile
    >>> import cocopp
    >>> cocopp.genericsettings.verbose = False # ensure to make doctests work
    >>> infoFile = os.path.join(cocopp.archives.bbob.local_data_path, 'BIPOP-CMA-ES', 'bbobexp_f2.info')
    >>> if not os.path.exists(infoFile):
    ...   filename = cocopp.archives.bbob.get_one('bbob/2009/BIPOP-CMA-ES_hansen')
    ...   tarfile.open(filename).extractall(cocopp.archives.bbob.local_data_path)
    >>> dslist = cocopp.load(infoFile)
      Data consistent according to consistency_check() in pproc.DataSet
    >>> dslist[2].instancenumbers
    [1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5]
    >>> dslist[2].evals[-1]  # doctest:+ELLIPSIS
    array([...
    >>> assert (dslist[2].evals[-1])[0] == 1.0e-8
    >>> assert 2.01200000e+03 <= (dslist[2].evals[-1])[-1] <= 2.01200001e+03
    >>> # because testbedsettings.GECCOBBOBTestbed.settings['instancesOfInterest'] was None
    >>> cocopp.testbedsettings.GECCOBBOBTestbed.settings['instancesOfInterest'] = [1, 3]
    >>> cocopp.config.config('GECCOBBOBTestbed') # make sure that settings are used
    >>> dslist2 = cocopp.load(infoFile)
      Data consistent according to consistency_check() in pproc.DataSet
    >>> dslist2[2].instancenumbers
    [1, 1, 1, 3, 3, 3]
    >>> dslist2[2].evals[-1]  # doctest:+ELLIPSIS
    array([...
    >>> assert (dslist2[2].evals[-1])[0] == 1.0e-8
    >>> assert 2.20700000e+03 <= (dslist2[2].evals[-1])[-1] <= 2.20700001e+03
    >>> # set things back to cause no troubles elsewhere:
    >>> cocopp.testbedsettings.GECCOBBOBTestbed.settings['instancesOfInterest'] = None
    >>> cocopp.config.config('GECCOBBOBTestbed') # make sure that settings are used

Method	isBiobjective	Undocumented
Method	get_testbed_name	Undocumented
Method	get_data_format	Undocumented
Method	get_suite	Undocumented
Method	__init__	Instantiate a DataSet.
Method	evals_ 0	Shall become `evals` attribute in future.
Method	evals_ 1	Undocumented
Method	evals_	Undocumented
Method	consistency_check	checks consistency of data set according to - number of instances - instances used
Method	computeERTfromEvals	Sets the attributes ert and target from the attribute evals.
Method	evals_with_simulated_restarts	Return a len(targets) list of `samplesize` "simulated" run lengths (#evaluations, sorted).
Method	__eq__	Compare indexEntry instances.
Method	__ne__	Undocumented
Method	__repr__	Undocumented
Method	info	print text info to stdout
Method	mMaxEvals	Returns the maximum number of function evaluations over all runs (trials), obsolete and replaced by attribute `max_eval`
Method	max_eval	maximum number of function evaluations over all runs (trials),
Method	nbRuns	Returns the number of runs.
Method	pickle	Save this instance to a pickle file.
Method	createDictInstance	Returns a dictionary of the instances.
Method	createDictInstanceCount	Returns a dictionary of the instances and their count.
Method	splitByTrials	Splits the post-processed data arrays by trials.
Method	generateRLData	Determine the running lengths for reaching the target values.
Method	detAverageEvals	Determine the average number of f-evals for each target in `targets` list. If a target is not reached within trial itrail, self.maxevals[itrial] contributes to the average.
Method	detSuccesses	Determine for each target in targets the number of successful runs, keeping in return list the order in targets.
Method	detSuccessRates	return a np.array with the success rate for each target in targets, easiest target first
Method	detERT	Determine the average running time to reach target values. The value is numpy.inf, if the target was never reached.
Method	detEvals	No summary
Method	plot_funvals	plot data of `funvals` attribute, versatile
Method	median_evals	return median for each target, unsuccessful runs count.
Method	plot	plot all data from `evals` attribute and the median.
Method	_cut_data	No summary
Method	_complement_data	insert a line for each target value
Method	_detMaxEvals	computes for each data column the (maximal) evaluation until final_target was reached, or `self.maxevals` otherwise.
Method	__parseHeader	Extract data from a header line in an index entry.
Method	_detEvals2	Determine the number of evaluations to reach target values.
Method	_argsort	return index array for a sorted order of trials.
Method	_old_plot	plot data from `evals` attribute.

def isBiobjective(self):

Undocumented

def get_testbed_name(self):

Undocumented

def get_data_format(self):

Undocumented

def get_suite(self):

Undocumented

def __init__(self, header, comment, data, indexfile):

overridden in cocopp.bestalg.BestAlgSet

Instantiate a DataSet.

The first three input arguments correspond to three consecutive lines of an index file (.info extension).

Parameters	string header	information of the experiment
	string comment	more information on the experiment
	string data	information on the runs of the experiment
	string indexfile	string for the file name from where the information come

@property
def evals_ 0(self):

Shall become evals attribute in future.

evals are the central data. Each line evals[i] has a (target) function value in evals[i][0] and the function evaluation for which this target was reached the first time in trials 1,... in evals[i][1:].

@evals_.setter
def evals_ 1(self, value):

Undocumented

@evals_.deleter
def evals_(self):

Undocumented

def _cut_data(self):

attributes target, evals, and ert are truncated to target values not much smaller than defined in attribute precision (typically 1e-8). Attribute maxevals is recomputed for columns that reach the final target precision. Note that in the bi-objective case the attribute precision does not exist.

def _complement_data(self, step=10**0.2, final_target=1e-08):

insert a line for each target value

def consistency_check(self):

checks consistency of data set according to - number of instances - instances used

def computeERTfromEvals(self):

Sets the attributes ert and target from the attribute evals.

def evals_with_simulated_restarts(self, targets, samplesize=genericsettings.simulated_runlength_bootstrap_sample_size, randintfirst=toolsstats.randint_derandomized, randintrest=toolsstats.randint_derandomized, bootstrap=False):

Return a len(targets) list of samplesize "simulated" run lengths (#evaluations, sorted).

np.sort(np.concatenate(return_value)) provides the combined sorted ECDF data over all targets which may be plotted with pyplot.step (missing the last step).

Unsuccessful data are represented as np.nan.

Simulated restarts are used for unsuccessful runs. The usage of detEvals or evals_with_simulated_restarts should be largely interchangeable, while the latter has a "success" rate of either 0 or 1.

TODO: change this: To get a bootstrap sample for estimating dispersion use min_samplesize=0, randint=np.random.randint.

Details:

For targets where all runs were successful, samplesize=nbRuns() is sufficient (and preferable) if randint is derandomized.
A single successful running length is computed by adding uniformly randomly chosen running lengths until the first time a successful one is chosen. In case of no successful run the result is None.

TODO: if samplesize >> nbRuns and nsuccesses is large, the data representation becomes somewhat inefficient.

TODO: it may be useful to make the samplesize dependent on the number of successes and supply the multipliers max(samplesizes) / samplesizes.

def __eq__(self, other):

overridden in cocopp.bestalg.BestAlgSet

Compare indexEntry instances.

def __ne__(self, other):

overridden in cocopp.bestalg.BestAlgSet

Undocumented

def __repr__(self):

overridden in cocopp.bestalg.BestAlgSet

Undocumented

def info(self, targets=None):

print text info to stdout

def mMaxEvals(self):

Returns the maximum number of function evaluations over all runs (trials), obsolete and replaced by attribute max_eval

def _detMaxEvals(self, final_target=None):

computes for each data column the (maximal) evaluation until final_target was reached, or self.maxevals otherwise.

@property
def max_eval(self):

maximum number of function evaluations over all runs (trials),

return max(self.maxevals)

def nbRuns(self):

Returns the number of runs.

def __parseHeader(self, header):

Extract data from a header line in an index entry.

def pickle(self, outputdir=None, gzipped=True):

overridden in cocopp.bestalg.BestAlgSet

Save this instance to a pickle file.

Saves this instance to a (by default gzipped) pickle file. If not specified by argument outputdir, the location of the pickle is given by the location of the first index file associated to this instance.

This method will overwrite existing files.

def createDictInstance(self):

overridden in cocopp.bestalg.BestAlgSet

Returns a dictionary of the instances.

The key is the instance Id, the value is a list of index.

def createDictInstanceCount(self):

Returns a dictionary of the instances and their count.

The keys are instance id and the values are the number of repetitions of such instance.

def splitByTrials(self, whichdata=None):

Splits the post-processed data arrays by trials.

Parameters	string whichdata	either 'evals' or 'funvals' determines the output
Returns	this method returns dictionaries of arrays, the key of the dictionaries being the instance id, the value being a smaller post-processed data array corresponding to the instance Id. If whichdata is 'evals' then the array contains function evaluations (1st column is alignment targets). Else if whichdata is 'funvals' then the output data contains function values (1st column is alignment budgets). Otherwise this method returns a tuple of these two arrays in this order.

def generateRLData(self, targets):

Determine the running lengths for reaching the target values.

:keyword list targets: target function values of interest

:returns: dict of arrays, one array for each target. Each array
          are copied from attribute :py:attr:`evals` of
          :py:class:`DataSetList`: first element is a target
          function value smaller or equal to the element of
          targets considered and has for other consecutive
          elements the corresponding number of function
          evaluations.

def detAverageEvals(self, targets):

Determine the average number of f-evals for each target in targets list. If a target is not reached within trial itrail, self.maxevals[itrial] contributes to the average.

Equals to sum(evals(target)) / nbruns. If aRT is finite this equals to aRT * psucc == (sum(evals) / ntrials / psucc) * psucc, where aRT, psucc, and evals are a function of target.

def detSuccesses(self, targets):

Determine for each target in targets the number of successful runs, keeping in return list the order in targets.

dset.SuccessRates(targets) == np.array(dset.detNbSuccesses(targets)) / dset.nbRuns()

are the respective success rates.

def detSuccessRates(self, targets):

return a np.array with the success rate for each target in targets, easiest target first

def detERT(self, targets):

overridden in cocopp.bestalg.BestAlgSet

Determine the average running time to reach target values. The value is numpy.inf, if the target was never reached.

Parameters	list targets	target function values of interest
Returns	list of average running times (# f-evals) for the respective targets.

def detEvals(self, targets, copy=True, bootstrap=False):

overridden in cocopp.bestalg.BestAlgSet

returns len(targets) data rows self.evals[idata, 1:] each row with the closest but not larger target such that self.evals[idata, 0] <= target, and self.evals[idata-1, 0] > target or in the "limit" cases the idata==0 line or the line np.array(self.nbRuns() * [np.nan]).

Makes by default a copy of the data, however this might change in future.

def _detEvals2(self, targets):

Determine the number of evaluations to reach target values.

Parameters	seq or float targets	target precisions
Returns	list of len(targets) values, each being an array of nbRuns FEs values

def _argsort(self, smallest_target_value=-np.inf):

return index array for a sorted order of trials.

Sorted from best to worst, for unsuccessful runs successively larger target values are queried to determine which is better.

Returned indices range from 1 to self.nbRuns() referring to columns in self.evals.

Target values smaller than smallest_target_value are not considered.

Details: if two runs have the exact same evaluation profile, they are sorted identically, however we could account for final f-values which seems only to make sense for smallest_target_value<=final_target_value.

def plot_funvals(self, **kwargs):

plot data of funvals attribute, versatile

TODO: seems outdated on 19/8/2016 and 05/2019 (would fail as it was: using "isfinite" instead of "np.isfinite" and is not called from anywhere)

def _old_plot(self, **kwargs):

plot data from evals attribute.

**kwargs is passed to matplolib.loglog.

TODO: seems outdated on 19/8/2016 ("np.isfinite" was "isfinite" hence raising an error)

def median_evals(self, target_values=None):

return median for each target, unsuccessful runs count.

target_values is not in effect for now, the median is computed for all target values (rows in evals attribute).

Return np.nan if the median run was unsuccessful.

Details: copies the evals attribute and sets nan to inf in order to get the median with nan values in the sorting.

def plot(self, plot_function=plt.semilogy, smallest_target=8e-09, median_format='k--', color_map=None, **kwargs):

plot all data from evals attribute and the median.

Plotted are Delta f-value vs evaluations. The sort for the color heatmap is based on the final performance.

color_map is a list or generator with self.nbRuns() colors and used as iter(color_map). The maps can be generated with the matplotlib.colors.LinearSegmentedColormap attributes of module matplotlib.cm. Default is brg between 0 and 0.5, like plt.cm.brg(np.linspace(0, 0.5, self.nbRuns())).

**kwargs is passed to plot_function.

API Documentation for cocopp, generated by pydoctor at 2020-01-21 16:27:37.

cocopp.pproc.DataSet(object) class documentation

`cocopp.pproc.DataSet(object)` class documentation