mlos_bench.storage.sql.experiment module

Saving and restoring the benchmark data using SQLAlchemy.

class mlos_bench.storage.sql.experiment.Experiment(*, engine: Engine, schema: DbSchema, tunables: TunableGroups, experiment_id: str, trial_id: int, root_env_config: str, description: str, opt_targets: Dict[str, Literal['min', 'max']])

Bases: Experiment

Logic for retrieving and storing the results of a single experiment.

Attributes:

description

Get the Experiment’s description.

experiment_id

Get the Experiment’s ID.

opt_targets

Get the Experiment’s optimization targets and directions.

trial_id

Get the current Trial ID.

tunables

Get the Experiment’s tunables.

Methods

load([last_trial_id])	Load (tunable values, benchmark scores, status) to warm-up the optimizer.
load_telemetry(trial_id)	Retrieve the telemetry data for a given trial.
load_tunable_config(config_id)	Load tunable values for a given config ID.
merge(experiment_ids)	Merge in the results of other (compatible) experiments trials.
new_trial(tunables[, ts_start, config])	Create a new experiment run in the storage.
pending_trials(timestamp, *, running)	Return an iterator over the pending trials that are scheduled to run on or before the specified timestamp.

load(last_trial_id: int = -1) → Tuple[List[int], List[dict], List[Dict[str, Any] | None], List[Status]]

Load (tunable values, benchmark scores, status) to warm-up the optimizer.

If last_trial_id is present, load only the data from the (completed) trials that were scheduled after the given trial ID. Otherwise, return data from ALL merged-in experiments and attempt to impute the missing tunable values.

Parameters:

last_trial_idint

(Optional) Trial ID to start from.

Returns:

(trial_ids, configs, scores, status)([int], [dict], [Optional[dict]], [Status])

Trial ids, Tunable values, benchmark scores, and status of the trials.

load_telemetry(trial_id: int) → List[Tuple[datetime, str, Any]]

Retrieve the telemetry data for a given trial.

Parameters:

trial_idint

Trial ID.

Returns:

metricsList[Tuple[datetime, str, Any]]

Telemetry data.

load_tunable_config(config_id: int) → Dict[str, Any]

Load tunable values for a given config ID.

merge(experiment_ids: List[str]) → None

Merge in the results of other (compatible) experiments trials. Used to help warm up the optimizer for this experiment.

Parameters:

experiment_idsList[str]

List of IDs of the experiments to merge in.

pending_trials(timestamp: datetime, *, running: bool) → Iterator[Trial]

Return an iterator over the pending trials that are scheduled to run on or before the specified timestamp.

Parameters:

timestampdatetime

The time in UTC to check for scheduled trials.

runningbool

If True, include the trials that are already running. Otherwise, return only the scheduled trials.

Returns:

trialsIterator[Storage.Trial]

An iterator over the scheduled (and maybe running) trials.