cocopp.pproc.DataSet(object)
class documentationcocopp.pproc
(View In Hierarchy)
Known subclasses: cocopp.bestalg.BestAlgSet
Unit element for the COCO post-processing. An instance of this class is created from one unit element of experimental data. One unit element would correspond to data for a given algorithm (a given :py:attr:`algId` and a :py:attr:`comment` line) and a given problem (:py:attr:`funcId` and :py:attr:`dimension`). Class attributes: - *funcId* -- function Id (integer) - *dim* -- dimension (integer) - *indexFiles* -- associated index files (list of strings) - *dataFiles* -- associated data files (list of strings) - *comment* -- comment for the setting (string) - *targetFuncValue* -- final target function value (float), might be missing - *precision* -- final ftarget - fopt (float), data with target[idat] < precision are optional and not relevant. - *algId* -- algorithm name (string) - *evals* -- data aligned by function values (2xarray, list of data rows [f_val, eval_run1, eval_run2,...]), cave: in a portfolio data rows can have different lengths - *funvals* -- data aligned by function evaluations (2xarray) - *maxevals* -- maximum number of function evaluations (array) - *finalfunvals* -- final function values (array) - *readmaxevals* -- maximum number of function evaluations read from index file (array) - *readfinalFminusFtarget* -- final function values - ftarget read from index file (array) - *pickleFile* -- associated pickle file name (string) - *target* -- target function values attained (array) - *ert* -- ert for reaching the target values in target (array) - *instancenumbers* -- list of numbers corresponding to the instances of the test function considered (list of int) - *isFinalized* -- list of bool for if runs were properly finalized :py:attr:`evals` and :py:attr:`funvals` are arrays of data collected from :py:data:`N` data sets. Both have the same format: zero-th column is the value on which the data of a row is aligned, the :py:data:`N` subsequent columns are either the numbers of function evaluations for :py:attr:`evals` or function values for :py:attr:`funvals`. A short example:: >>> from __future__ import print_function >>> import sys >>> import os >>> import urllib >>> import tarfile >>> import cocopp >>> cocopp.genericsettings.verbose = False # ensure to make doctests work >>> def setup(infoFile): ... if not os.path.exists(infoFile): ... filename = cocopp.archives.bbob.get_one('2009/BIPOP-CMA-ES_hansen') ... tarfile.open(filename).extractall(cocopp.archives.bbob.local_data_path) >>> infoFile = os.path.join(cocopp.archives.bbob.local_data_path, 'BIPOP-CMA-ES', 'bbobexp_f2.info') >>> print('get'); setup(infoFile) # doctest:+ELLIPSIS get... >>> dslist = cocopp.load(infoFile) Data consistent according to consistency_check() in pproc.DataSet >>> print(dslist) # doctest:+ELLIPSIS [DataSet(BIPOP-CMA-ES on f2 2-D), ..., DataSet(BIPOP-CMA-ES on f2 40-D)] >>> type(dslist) <class 'cocopp.pproc.DataSetList'> >>> len(dslist) 6 >>> ds = dslist[3] # a single data set of type DataSet >>> ds DataSet(BIPOP-CMA-ES on f2 10-D) >>> for d in dir(ds): # doctest:+ELLIPSIS ... if '_old_' not in d: ... print(d) _DataSet__parseHeader __class__ __delattr__ __dict__ ... __getattribute__ ... __module__ __ne__ __new__ __reduce__ __reduce_ex__ __repr__ __setattr__ __sizeof__ __str__ __subclasshook__ __weakref__ _argsort _attributes _complement_data _cut_data _detEvals2 _detMaxEvals _evals _extra_attr algId algs comment computeERTfromEvals consistency_check createDictInstance createDictInstanceCount dataFiles detAverageEvals detERT detEvals detSuccessRates detSuccesses dim ert evals evals_ evals_with_simulated_restarts finalfunvals funcId funvals generateRLData get_data_format get_suite get_testbed_name indexFiles info instancenumbers isBiobjective isFinalized mMaxEvals max_eval maxevals median_evals nbRuns pickle plot plot_funvals precision readfinalFminusFtarget readmaxevals reference_values splitByTrials success_ratio target testbed_name >>> all(ds.evals[:, 0] == ds.target) # first column of ds.evals is the "target" f-value True >>> # investigate row 0,10,20,... and of the result columns 0,5,6, index 0 is ftarget >>> ev = ds.evals[0::10, (0,5,6)] # doctest:+ELLIPSIS >>> assert 3.98107170e+07 <= ev[0][0] <= 3.98107171e+07 >>> assert ev[0][1] == 1 >>> assert ev[0][2] == 1 >>> assert 6.07000000e+03 <= ev[-1][-1] <= 6.07000001e+03 >>> # show last row, same columns >>> ev = ds.evals[-1,(0,5,6)] # doctest:+ELLIPSIS >>> assert ev[0] == 1e-8 >>> assert 5.67600000e+03 <= ev[1] <= 5.67600001e+03 >>> ds.info() # prints similar data more nicely formated Algorithm: BIPOP-CMA-ES Function ID: 2 Dimension DIM = 10 Number of trials: 15 Final target Df: 1e-08 min / max number of evals per trial: 5676 / 6346 evals/DIM: best 15% 50% 85% max | aRT/DIM nsucc ---Df---|-----------------------------------------|---------------- 1.0e+03 | 102 126 170 205 235 | 164.2 15 1.0e+01 | 278 306 364 457 480 | 374.5 15 1.0e-01 | 402 445 497 522 536 | 490.8 15 1.0e-03 | 480 516 529 554 567 | 532.8 15 1.0e-05 | 513 546 563 584 593 | 562.5 15 1.0e-08 | 568 594 611 628 635 | 609.6 15 >>> import numpy as np >>> idx = list(range(0, 50, 10)) + [-1] >>> # get aRT average runtime for some targets >>> t = np.array([idx, ds.target[idx], ds.ert[idx]]).T # doctest:+ELLIPSIS >>> assert t[0][0] == 0 >>> assert t[0][2] == 1 >>> assert t[-1][-2] == 1e-8 >>> assert 6.09626666e+03 <= t[-1][-1] <= 6.09626667e+03 Note that the load of a data set depends on the set of instances specified in testbedsettings' TestBed class (or its children) (None means all instances are read in): >>> import sys >>> import os >>> import urllib >>> import tarfile >>> import cocopp >>> cocopp.genericsettings.verbose = False # ensure to make doctests work >>> infoFile = os.path.join(cocopp.archives.bbob.local_data_path, 'BIPOP-CMA-ES', 'bbobexp_f2.info') >>> if not os.path.exists(infoFile): ... filename = cocopp.archives.bbob.get_one('bbob/2009/BIPOP-CMA-ES_hansen') ... tarfile.open(filename).extractall(cocopp.archives.bbob.local_data_path) >>> dslist = cocopp.load(infoFile) Data consistent according to consistency_check() in pproc.DataSet >>> dslist[2].instancenumbers [1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5] >>> dslist[2].evals[-1] # doctest:+ELLIPSIS array([... >>> assert (dslist[2].evals[-1])[0] == 1.0e-8 >>> assert 2.01200000e+03 <= (dslist[2].evals[-1])[-1] <= 2.01200001e+03 >>> # because testbedsettings.GECCOBBOBTestbed.settings['instancesOfInterest'] was None >>> cocopp.testbedsettings.GECCOBBOBTestbed.settings['instancesOfInterest'] = [1, 3] >>> cocopp.config.config('GECCOBBOBTestbed') # make sure that settings are used >>> dslist2 = cocopp.load(infoFile) Data consistent according to consistency_check() in pproc.DataSet >>> dslist2[2].instancenumbers [1, 1, 1, 3, 3, 3] >>> dslist2[2].evals[-1] # doctest:+ELLIPSIS array([... >>> assert (dslist2[2].evals[-1])[0] == 1.0e-8 >>> assert 2.20700000e+03 <= (dslist2[2].evals[-1])[-1] <= 2.20700001e+03 >>> # set things back to cause no troubles elsewhere: >>> cocopp.testbedsettings.GECCOBBOBTestbed.settings['instancesOfInterest'] = None >>> cocopp.config.config('GECCOBBOBTestbed') # make sure that settings are used
Method | isBiobjective | Undocumented |
Method | get_testbed_name | Undocumented |
Method | get_data_format | Undocumented |
Method | get_suite | Undocumented |
Method | __init__ | Instantiate a DataSet. |
Method | evals_ 0 | Shall become evals attribute in future. |
Method | evals_ 1 | Undocumented |
Method | evals_ | Undocumented |
Method | consistency_check | checks consistency of data set according to - number of instances - instances used |
Method | computeERTfromEvals | Sets the attributes ert and target from the attribute evals. |
Method | evals_with_simulated_restarts | Return a len(targets) list of samplesize "simulated" run lengths (#evaluations, sorted). |
Method | __eq__ | Compare indexEntry instances. |
Method | __ne__ | Undocumented |
Method | __repr__ | Undocumented |
Method | info | print text info to stdout |
Method | mMaxEvals | Returns the maximum number of function evaluations over all runs (trials), obsolete and replaced by attribute max_eval |
Method | max_eval | maximum number of function evaluations over all runs (trials), |
Method | nbRuns | Returns the number of runs. |
Method | pickle | Save this instance to a pickle file. |
Method | createDictInstance | Returns a dictionary of the instances. |
Method | createDictInstanceCount | Returns a dictionary of the instances and their count. |
Method | splitByTrials | Splits the post-processed data arrays by trials. |
Method | generateRLData | Determine the running lengths for reaching the target values. |
Method | detAverageEvals | Determine the average number of f-evals for each target in targets list. If a target is not reached within trial itrail, self.maxevals[itrial] contributes to the average. |
Method | detSuccesses | Determine for each target in targets the number of successful runs, keeping in return list the order in targets. |
Method | detSuccessRates | return a np.array with the success rate for each target in targets, easiest target first |
Method | detERT | Determine the average running time to reach target values. The value is numpy.inf, if the target was never reached. |
Method | detEvals | No summary |
Method | plot_funvals | plot data of funvals attribute, versatile |
Method | median_evals | return median for each target, unsuccessful runs count. |
Method | plot | plot all data from evals attribute and the median. |
Method | _cut_data | No summary |
Method | _complement_data | insert a line for each target value |
Method | _detMaxEvals | computes for each data column the (maximal) evaluation until final_target was reached, or self.maxevals otherwise. |
Method | __parseHeader | Extract data from a header line in an index entry. |
Method | _detEvals2 | Determine the number of evaluations to reach target values. |
Method | _argsort | return index array for a sorted order of trials. |
Method | _old_plot | plot data from evals attribute. |
Instantiate a DataSet.
The first three input arguments correspond to three consecutive lines of an index file (.info extension).
Parameters | string header | information of the experiment |
string comment | more information on the experiment | |
string data | information on the runs of the experiment | |
string indexfile | string for the file name from where the information come |
Shall become evals attribute in future.
evals are the central data. Each line evals[i] has a (target) function value in evals[i][0] and the function evaluation for which this target was reached the first time in trials 1,... in evals[i][1:].
target
, evals
, and ert
are truncated to target values not
much smaller than defined in attribute precision
(typically 1e-8).
Attribute maxevals
is recomputed for columns that reach the final target
precision. Note that in the bi-objective case the attribute precision
does not exist.Return a len(targets) list of samplesize "simulated" run lengths (#evaluations, sorted).
np.sort(np.concatenate(return_value)) provides the combined
sorted ECDF data over all targets which may be plotted with
pyplot.step
(missing the last step).
Unsuccessful data are represented as np.nan
.
Simulated restarts are used for unsuccessful runs. The usage of
detEvals
or evals_with_simulated_restarts
should be largely
interchangeable, while the latter has a "success" rate of either
0 or 1.
TODO: change this: To get a bootstrap sample for estimating dispersion use min_samplesize=0, randint=np.random.randint.
Details:
randint
is derandomized.None
.TODO: if samplesize
>> nbRuns
and nsuccesses is large,
the data representation becomes somewhat inefficient.
TODO: it may be useful to make the samplesize dependent on the number of successes and supply the multipliers max(samplesizes) / samplesizes.
max_eval
maximum number of function evaluations over all runs (trials),
return max(self.maxevals)
Save this instance to a pickle file.
Saves this instance to a (by default gzipped) pickle file. If not specified by argument outputdir, the location of the pickle is given by the location of the first index file associated to this instance.
This method will overwrite existing files.
Returns a dictionary of the instances.
The key is the instance Id, the value is a list of index.
Returns a dictionary of the instances and their count.
The keys are instance id and the values are the number of repetitions of such instance.
Parameters | string whichdata | either 'evals' or 'funvals' determines the output |
Returns | this method returns dictionaries of arrays, the key of the dictionaries being the instance id, the value being a smaller post-processed data array corresponding to the instance Id. If whichdata is 'evals' then the array contains function evaluations (1st column is alignment targets). Else if whichdata is 'funvals' then the output data contains function values (1st column is alignment budgets). Otherwise this method returns a tuple of these two arrays in this order. |
Determine the running lengths for reaching the target values. :keyword list targets: target function values of interest :returns: dict of arrays, one array for each target. Each array are copied from attribute :py:attr:`evals` of :py:class:`DataSetList`: first element is a target function value smaller or equal to the element of targets considered and has for other consecutive elements the corresponding number of function evaluations.
Determine the average number of f-evals for each target in targets list. If a target is not reached within trial itrail, self.maxevals[itrial] contributes to the average.
Equals to sum(evals(target)) / nbruns. If aRT is finite this equals to aRT * psucc == (sum(evals) / ntrials / psucc) * psucc, where aRT, psucc, and evals are a function of target.
Determine for each target in targets the number of successful runs, keeping in return list the order in targets.
dset.SuccessRates(targets) == np.array(dset.detNbSuccesses(targets)) / dset.nbRuns()
are the respective success rates.
Parameters | list targets | target function values of interest |
Returns | list of average running times (# f-evals) for the respective targets. |
returns len(targets) data rows self.evals[idata, 1:] each row with the closest but not larger target such that self.evals[idata, 0] <= target, and self.evals[idata-1, 0] > target or in the "limit" cases the idata==0 line or the line np.array(self.nbRuns() * [np.nan]).
Makes by default a copy of the data, however this might change in future.
Parameters | seq or float targets | target precisions |
Returns | list of len(targets) values, each being an array of nbRuns FEs values |
return index array for a sorted order of trials.
Sorted from best to worst, for unsuccessful runs successively larger target values are queried to determine which is better.
Returned indices range from 1 to self.nbRuns() referring to columns in self.evals.
Target values smaller than smallest_target_value are not considered.
Details: if two runs have the exact same evaluation profile, they are sorted identically, however we could account for final f-values which seems only to make sense for smallest_target_value<=final_target_value.
plot data of funvals
attribute, versatile
plot data from evals
attribute.
**kwargs
is passed to matplolib.loglog
.
TODO: seems outdated on 19/8/2016 ("np.isfinite" was "isfinite" hence raising an error)
return median for each target, unsuccessful runs count.
target_values
is not in effect for now, the median is computed
for all target values (rows in evals
attribute).
Return np.nan
if the median run was unsuccessful.
Details: copies the evals attribute and sets nan
to inf
in
order to get the median with nan
values in the sorting.
plot all data from evals
attribute and the median.
Plotted are Delta f-value vs evaluations. The sort for the color heatmap is based on the final performance.
color_map
is a list
or generator
with self.nbRuns()
colors
and used as iter(color_map). The maps can be generated with the
matplotlib.colors.LinearSegmentedColormap
attributes of module
matplotlib.cm
. Default is brg
between 0 and 0.5, like
plt.cm.brg(np.linspace(0, 0.5, self.nbRuns())).
**kwargs
is passed to plot_function
.