mlos_bench.environments

Tunable Environments for mlos_bench.

Overview

Environments are classes that represent an execution setting (i.e., environment) for running a benchmark or tuning process.

For instance, a LocalEnv represents a local execution environment, a RemoteEnv represents a remote execution environment, a VMEnv represents a virtual machine, etc.

An Environment goes through a series of phases (e.g., setup(), run(), teardown(), etc.) that can be used to prepare a VM, workload, etc.; run a benchmark, script, etc.; and clean up afterwards. Often, what these phases do (e.g., what commands to execute) will depend on the specific Environment and the configs that Environment was loaded with. This lets Environments be very flexible in what they can accomplish.

Environments can be stacked together with the CompositeEnv class to represent complex setups (e.g., an appication running on a remote VM with a benchmark running from a local machine).

See below for the set of Environments currently available in this package.

Note that additional ones can also be created by extending the base Environment class and referencing them in the json configs using the class key.

Environment Tunables

Each environment can use TunableGroups to specify the set of configuration parameters that can be optimized or searched. At each iteration of the optimization process, the optimizer will generate a set of values for the Tunables that the environment can use to configure itself.

At a python level, this happens by passing a TunableGroups() object to the tunable_groups parameter of the Environment constructor, but that is typically handled by the load_environment() method of the ConfigPersistenceService() invoked by the mlos_bench command line tool’s mlos_bench.launcher.Launcher class.

In the typical json user level configs, this is specified in the include_tunables section of the Environment config to include the TunableGroups definitions from other json files when the Launcher processes the initial set of config files.

The tunable_params setting in the config section of the Environment config can also be used to limit which of the TunableGroups should be used for the Environment.

Since json configs also support $variable substitution in the values using the globals mechanism, this setting can used to dynamically change the set of active TunableGroups for a given Experiment using only globals, allowing for configs to be more modular and composable.

Environment Services

Environments can also reference services that provide the necessary support to perform the actions that environment needs for each of its phases depending upon where its being deployed (e.g., local machine, remote machine, cloud provider VM, etc.)

Although this can be done in the Environment config directly with the include_services key, it is often more useful to do it in the global or cli config to allow for the same Environment to be used in different settings (e.g., local machine, SSH accessible machine, Azure VM, etc.) without having to change the Environment config.

Variable Propagation

TODO: Document how variable propagation works in the script environments using required_args, const_args, etc.

Examples

While this documentation is generated from the source code and is intended to be a useful reference on the internal details, most users will be more interested in generating json configs to be used with the mlos_bench command line tool.

For a simple working user oriented example please see the test_local_env_bench.jsonc file or other examples in the source tree linked below.

For more developer oriented examples please see the mlos_bench/tests/environments directory in the source tree.

Notes

See also

mlos_bench.config

Overview of the configuration system.

mlos_bench.services

Overview of the Services available to the Environments and their configurations.

mlos_bench.tunables

Overview of the Tunables available to the Environments and their configurations.

Submodules