mlos_bench.storage.sql.storage module

Saving and restoring the benchmark data in SQL database.

class mlos_bench.storage.sql.storage.SqlStorage(config: dict, global_config: dict | None = None, service: Service | None = None)

Bases: Storage

An implementation of the Storage interface using SQLAlchemy backend.

Attributes:

experiments

Retrieve the experiments’ data from the storage.

Methods

Experiment(*, tunables, experiment_id, ...)	Base interface for storing the results of the experiment.
Trial(*, tunables, experiment_id, trial_id, ...)	Base interface for storing the results of a single run of the experiment.
experiment(*, experiment_id, trial_id, ...)	Create a new experiment in the storage.

experiment(*, experiment_id: str, trial_id: int, root_env_config: str, description: str, tunables: TunableGroups, opt_targets: Dict[str, Literal['min', 'max']]) → Experiment

Create a new experiment in the storage.

We need the opt_target parameter here to know what metric to retrieve when we load the data from previous trials. Later we will replace it with full metadata about the optimization direction, multiple objectives, etc.

Parameters:

experiment_idstr

Unique identifier of the experiment.

trial_idint

Starting number of the trial.

root_env_configstr

A path to the root JSON configuration file of the benchmarking environment.

descriptionstr

Human-readable description of the experiment.

tunablesTunableGroups

opt_targetsDict[str, Literal[“min”, “max”]]

Names of metrics we’re optimizing for and the optimization direction {min, max}.

Returns:

experimentStorage.Experiment

An object that allows to update the storage with the results of the experiment and related data.

property experiments: Dict[str, ExperimentData]

Retrieve the experiments’ data from the storage.

Returns:

experimentsDict[str, ExperimentData]

A dictionary of the experiments’ data, keyed by experiment id.