mlos_bench.environments.Environment¶
- class mlos_bench.environments.Environment(*, name: str, config: dict, global_config: dict | None = None, tunables: TunableGroups | None = None, service: Service | None = None)¶
An abstract base of all benchmark environments.
- Attributes:
parametersKey/value pairs of all environment parameters (i.e., const_args and tunable_params).
tunable_paramsGet the configuration space of the given environment.
Methods
new(*, env_name, class_name, config[, ...])Factory method for a new environment with a given config.
pprint([indent, level])Pretty-print the environment configuration.
run()Execute the run script for this environment.
setup(tunables[, global_config])Set up a new benchmark environment, if necessary.
status()Check the status of the benchmark environment.
teardown()Tear down the benchmark environment.
- __init__(*, name: str, config: dict, global_config: dict | None = None, tunables: TunableGroups | None = None, service: Service | None = None)¶
Create a new environment with a given config.
- Parameters:
- name: str
Human-readable name of the environment.
- configdict
Free-format dictionary that contains the benchmark environment configuration. Each config must have at least the “tunable_params” and the “const_args” sections.
- global_configdict
Free-format dictionary of global parameters (e.g., security credentials) to be mixed in into the “const_args” section of the local config.
- tunablesTunableGroups
A collection of groups of tunable parameters for all environments.
- service: Service
An optional service object (e.g., providing methods to deploy or reboot a VM/Host, etc.).
- classmethod new(*, env_name: str, class_name: str, config: dict, global_config: dict | None = None, tunables: TunableGroups | None = None, service: Service | None = None) Environment¶
Factory method for a new environment with a given config.
- Parameters:
- env_name: str
Human-readable name of the environment.
- class_name: str
FQN of a Python class to instantiate, e.g., “mlos_bench.environments.remote.HostEnv”. Must be derived from the Environment class.
- configdict
Free-format dictionary that contains the benchmark environment configuration. It will be passed as a constructor parameter of the class specified by name.
- global_configdict
Free-format dictionary of global parameters (e.g., security credentials) to be mixed in into the “const_args” section of the local config.
- tunablesTunableGroups
A collection of groups of tunable parameters for all environments.
- service: Service
An optional service object (e.g., providing methods to deploy or reboot a VM/Host, etc.).
- Returns:
- envEnvironment
An instance of the Environment class initialized with config.
- property parameters: Dict[str, int | float | str | None]¶
Key/value pairs of all environment parameters (i.e., const_args and tunable_params). Note that before .setup() is called, all tunables will be set to None.
- Returns:
- parametersDict[str, TunableValue]
Key/value pairs of all environment parameters (i.e., const_args and tunable_params).
- pprint(indent: int = 4, level: int = 0) str¶
Pretty-print the environment configuration. For composite environments, print all children environments as well.
- Parameters:
- indentint
Number of spaces to indent the output. Default is 4.
- levelint
Current level of indentation. Default is 0.
- Returns:
- prettystr
Pretty-printed environment configuration. Default output is the same as __repr__.
- run() Tuple[Status, datetime, Dict[str, int | float | str | None] | None]¶
Execute the run script for this environment.
For instance, this may start a new experiment, download results, reconfigure the environment, etc. Details are configurable via the environment config.
- Returns:
- (status, timestamp, output)(Status, datetime, dict)
3-tuple of (Status, timestamp, output) values, where output is a dict with the results or None if the status is not COMPLETED. If run script is a benchmark, then the score is usually expected to be in the score field.
- setup(tunables: TunableGroups, global_config: dict | None = None) bool¶
Set up a new benchmark environment, if necessary. This method must be idempotent, i.e., calling it several times in a row should be equivalent to a single call.
- Parameters:
- tunablesTunableGroups
A collection of tunable parameters along with their values.
- global_configdict
Free-format dictionary of global parameters of the environment that are not used in the optimization process.
- Returns:
- is_successbool
True if operation is successful, false otherwise.
- status() Tuple[Status, datetime, List[Tuple[datetime, str, Any]]]¶
Check the status of the benchmark environment.
- Returns:
- (benchmark_status, timestamp, telemetry)(Status, datetime, list)
3-tuple of (benchmark status, timestamp, telemetry) values. timestamp is UTC time stamp of the status; it’s current time by default. telemetry is a list (maybe empty) of (timestamp, metric, value) triplets.
- teardown() None¶
Tear down the benchmark environment.
This method must be idempotent, i.e., calling it several times in a row should be equivalent to a single call.
- property tunable_params: TunableGroups¶
Get the configuration space of the given environment.
- Returns:
- tunablesTunableGroups
A collection of covariant groups of tunable parameters.