mlos_bench.storage.base_experiment_data.ExperimentData

class mlos_bench.storage.base_experiment_data.ExperimentData(*, experiment_id: str, description: str, root_env_config: str, git_repo: str, git_commit: str)

Base interface for accessing the stored experiment benchmark data.

An experiment groups together a set of trials that are run with a given set of scripts and mlos_bench configuration files.

Attributes:

default_tunable_config_id

Retrieves the (tunable) config id for the default tunable values for this experiment.

description

Description of the experiment.

experiment_id

ID of the experiment.

objectives

Retrieve the experiment’s objectives data from the storage.

results_df

Retrieve all experimental results as a single DataFrame.

root_env_config

Root environment configuration.

trials

Retrieve the experiment’s trials’ data from the storage.

tunable_config_trial_groups

Retrieve the Experiment’s (Tunable) Config Trial Group data from the storage.

tunable_configs

Retrieve the experiment’s (tunable) configs’ data from the storage.

__init__(*, experiment_id: str, description: str, root_env_config: str, git_repo: str, git_commit: str)

CONFIG_COLUMN_PREFIX = 'config.'

RESULT_COLUMN_PREFIX = 'result.'

property default_tunable_config_id: int | None

Retrieves the (tunable) config id for the default tunable values for this experiment.

Note: this is by default the first trial executed for this experiment. However, it is currently possible that the user changed the tunables config in between resumptions of an experiment.

Returns:

int

property description: str

Description of the experiment.

property experiment_id: str

ID of the experiment.

abstract property objectives: Dict[str, Literal['min', 'max']]

Retrieve the experiment’s objectives data from the storage.

Returns:

objectivesDict[str, objective]

A dictionary of the experiment’s objective names (optimization_targets) and their directions (e.g., min or max).

abstract property results_df: DataFrame

Retrieve all experimental results as a single DataFrame.

Returns:

resultspandas.DataFrame

A DataFrame with configurations and results from all trials of the experiment. Has columns [trial_id, tunable_config_id, tunable_config_trial_group_id, ts_start, ts_end, status] followed by tunable config parameters (prefixed with “config.”) and trial results (prefixed with “result.”). The latter can be NULLs if the trial was not successful.

property root_env_config: Tuple[str, str, str]

Root environment configuration.

Returns:

root_env_configTuple[str, str, str]

A tuple of (root_env_config, git_repo, git_commit) for the root environment.

abstract property trials: Dict[int, TrialData]

Retrieve the experiment’s trials’ data from the storage.

Returns:

trialsDict[int, TrialData]

A dictionary of the trials’ data, keyed by trial id.

abstract property tunable_config_trial_groups: Dict[int, TunableConfigTrialGroupData]

Retrieve the Experiment’s (Tunable) Config Trial Group data from the storage.

Returns:

trialsDict[int, TunableConfigTrialGroupData]

A dictionary of the trials’ data, keyed by (tunable) by config id.

abstract property tunable_configs: Dict[int, TunableConfigData]

Retrieve the experiment’s (tunable) configs’ data from the storage.

Returns:

trialsDict[int, TunableConfigData]

A dictionary of the configs’ data, keyed by (tunable) config id.