mlos_bench.storage.sql.experiment_data module

An interface to access the experiment benchmark data stored in SQL DB.

class mlos_bench.storage.sql.experiment_data.ExperimentSqlData(*, engine: Engine, schema: DbSchema, experiment_id: str, description: str, root_env_config: str, git_repo: str, git_commit: str)

Bases: ExperimentData

SQL interface for accessing the stored experiment benchmark data.

An experiment groups together a set of trials that are run with a given set of scripts and mlos_bench configuration files.

Attributes:

default_tunable_config_id

Retrieves the (tunable) config id for the default tunable values for this experiment.

description

Description of the experiment.

experiment_id

ID of the experiment.

objectives

Retrieve the experiment’s objectives data from the storage.

results_df

Retrieve all experimental results as a single DataFrame.

root_env_config

Root environment configuration.

trials

Retrieve the experiment’s trials’ data from the storage.

tunable_config_trial_groups

Retrieve the Experiment’s (Tunable) Config Trial Group data from the storage.

tunable_configs

Retrieve the experiment’s (tunable) configs’ data from the storage.

property default_tunable_config_id: int | None

Retrieves the (tunable) config id for the default tunable values for this experiment.

Note: this is by default the first trial executed for this experiment. However, it is currently possible that the user changed the tunables config in between resumptions of an experiment.

Returns:

int

property objectives: Dict[str, Literal['min', 'max']]

Retrieve the experiment’s objectives data from the storage.

Returns:

objectivesDict[str, objective]

A dictionary of the experiment’s objective names (optimization_targets) and their directions (e.g., min or max).

property results_df: DataFrame

Retrieve all experimental results as a single DataFrame.

Returns:

resultspandas.DataFrame

A DataFrame with configurations and results from all trials of the experiment. Has columns [trial_id, tunable_config_id, tunable_config_trial_group_id, ts_start, ts_end, status] followed by tunable config parameters (prefixed with “config.”) and trial results (prefixed with “result.”). The latter can be NULLs if the trial was not successful.

property trials: Dict[int, TrialData]

Retrieve the experiment’s trials’ data from the storage.

Returns:

trialsDict[int, TrialData]

A dictionary of the trials’ data, keyed by trial id.

property tunable_config_trial_groups: Dict[int, TunableConfigTrialGroupData]

Retrieve the Experiment’s (Tunable) Config Trial Group data from the storage.

Returns:

trialsDict[int, TunableConfigTrialGroupData]

A dictionary of the trials’ data, keyed by (tunable) by config id.

property tunable_configs: Dict[int, TunableConfigData]

Retrieve the experiment’s (tunable) configs’ data from the storage.

Returns:

trialsDict[int, TunableConfigData]

A dictionary of the configs’ data, keyed by (tunable) config id.