mlos_bench.environments package

Tunable Environments for mlos_bench.

class mlos_bench.environments.CompositeEnv(*, name: str, config: dict, global_config: dict | None = None, tunables: TunableGroups | None = None, service: Service | None = None)

Bases: Environment

Composite benchmark environment.

Attributes:

children

Return the list of child environments.

parameters

Key/value pairs of all environment parameters (i.e., const_args and tunable_params).

tunable_params

Get the configuration space of the given environment.

Methods

new(*, env_name, class_name, config[, ...])	Factory method for a new environment with a given config.
pprint([indent, level])	Pretty-print the environment and its children.
run()	Submit a new experiment to the environment.
setup(tunables[, global_config])	Set up the children environments.
status()	Check the status of the benchmark environment.
teardown()	Tear down the children environments.

property children: List[Environment]

Return the list of child environments.

pprint(indent: int = 4, level: int = 0) → str

Pretty-print the environment and its children.

Parameters:

indentint

Number of spaces to indent the output at each level. Default is 4.

levelint

Current level of indentation. Default is 0.

Returns:

prettystr

Pretty-printed environment configuration.

run() → Tuple[Status, datetime, Dict[str, int | float | str | None] | None]

Submit a new experiment to the environment. Return the result of the last child environment if successful, or the status of the last failed environment otherwise.

Returns:

(status, timestamp, output)(Status, datetime, dict)

3-tuple of (Status, timestamp, output) values, where output is a dict with the results or None if the status is not COMPLETED. If run script is a benchmark, then the score is usually expected to be in the score field.

setup(tunables: TunableGroups, global_config: dict | None = None) → bool

Set up the children environments.

Parameters:

tunablesTunableGroups

A collection of tunable parameters along with their values.

global_configdict

Free-format dictionary of global parameters of the environment that are not used in the optimization process.

Returns:

is_successbool

True if all children setup() operations are successful, false otherwise.

status() → Tuple[Status, datetime, List[Tuple[datetime, str, Any]]]

Check the status of the benchmark environment.

Returns:

(benchmark_status, timestamp, telemetry)(Status, datetime, list)

3-tuple of (benchmark status, timestamp, telemetry) values. timestamp is UTC time stamp of the status; it’s current time by default. telemetry is a list (maybe empty) of (timestamp, metric, value) triplets.

teardown() → None

Tear down the children environments.

This method is idempotent, i.e., calling it several times is equivalent to a single call. The environments are being torn down in the reverse order.

class mlos_bench.environments.Environment(*, name: str, config: dict, global_config: dict | None = None, tunables: TunableGroups | None = None, service: Service | None = None)

Bases: object

An abstract base of all benchmark environments.

Attributes:

parameters

Key/value pairs of all environment parameters (i.e., const_args and tunable_params).

tunable_params

Get the configuration space of the given environment.

Methods

new(*, env_name, class_name, config[, ...])	Factory method for a new environment with a given config.
pprint([indent, level])	Pretty-print the environment configuration.
run()	Execute the run script for this environment.
setup(tunables[, global_config])	Set up a new benchmark environment, if necessary.
status()	Check the status of the benchmark environment.
teardown()	Tear down the benchmark environment.

classmethod new(*, env_name: str, class_name: str, config: dict, global_config: dict | None = None, tunables: TunableGroups | None = None, service: Service | None = None) → Environment

Factory method for a new environment with a given config.

Parameters:

env_name: str

Human-readable name of the environment.

class_name: str

FQN of a Python class to instantiate, e.g., “mlos_bench.environments.remote.HostEnv”. Must be derived from the Environment class.

configdict

Free-format dictionary that contains the benchmark environment configuration. It will be passed as a constructor parameter of the class specified by name.

global_configdict

Free-format dictionary of global parameters (e.g., security credentials) to be mixed in into the “const_args” section of the local config.

tunablesTunableGroups

A collection of groups of tunable parameters for all environments.

service: Service

An optional service object (e.g., providing methods to deploy or reboot a VM/Host, etc.).

Returns:

envEnvironment

An instance of the Environment class initialized with config.

property parameters: Dict[str, int | float | str | None]

Key/value pairs of all environment parameters (i.e., const_args and tunable_params). Note that before .setup() is called, all tunables will be set to None.

Returns:

parametersDict[str, TunableValue]

Key/value pairs of all environment parameters (i.e., const_args and tunable_params).

pprint(indent: int = 4, level: int = 0) → str

Pretty-print the environment configuration. For composite environments, print all children environments as well.

Parameters:

indentint

Number of spaces to indent the output. Default is 4.

levelint

Current level of indentation. Default is 0.

Returns:

prettystr

Pretty-printed environment configuration. Default output is the same as __repr__.

run() → Tuple[Status, datetime, Dict[str, int | float | str | None] | None]

Execute the run script for this environment.

For instance, this may start a new experiment, download results, reconfigure the environment, etc. Details are configurable via the environment config.

Returns:

(status, timestamp, output)(Status, datetime, dict)

3-tuple of (Status, timestamp, output) values, where output is a dict with the results or None if the status is not COMPLETED. If run script is a benchmark, then the score is usually expected to be in the score field.

setup(tunables: TunableGroups, global_config: dict | None = None) → bool

Set up a new benchmark environment, if necessary. This method must be idempotent, i.e., calling it several times in a row should be equivalent to a single call.

Parameters:

tunablesTunableGroups

A collection of tunable parameters along with their values.

global_configdict

Free-format dictionary of global parameters of the environment that are not used in the optimization process.

Returns:

is_successbool

True if operation is successful, false otherwise.

status() → Tuple[Status, datetime, List[Tuple[datetime, str, Any]]]

Check the status of the benchmark environment.

Returns:

(benchmark_status, timestamp, telemetry)(Status, datetime, list)

3-tuple of (benchmark status, timestamp, telemetry) values. timestamp is UTC time stamp of the status; it’s current time by default. telemetry is a list (maybe empty) of (timestamp, metric, value) triplets.

teardown() → None

Tear down the benchmark environment.

This method must be idempotent, i.e., calling it several times in a row should be equivalent to a single call.

property tunable_params: TunableGroups

Get the configuration space of the given environment.

Returns:

tunablesTunableGroups

A collection of covariant groups of tunable parameters.

class mlos_bench.environments.LocalEnv(*, name: str, config: dict, global_config: dict | None = None, tunables: TunableGroups | None = None, service: Service | None = None)

Bases: ScriptEnv

Scheduler-side Environment that runs scripts locally.

Attributes:

parameters

Key/value pairs of all environment parameters (i.e., const_args and tunable_params).

tunable_params

Get the configuration space of the given environment.

Methods

new(*, env_name, class_name, config[, ...])	Factory method for a new environment with a given config.
pprint([indent, level])	Pretty-print the environment configuration.
run()	Run a script in the local scheduler environment.
setup(tunables[, global_config])	Check if the environment is ready and set up the application and benchmarks, if necessary.
status()	Check the status of the benchmark environment.
teardown()	Clean up the local environment.

run() → Tuple[Status, datetime, Dict[str, int | float | str | None] | None]

Run a script in the local scheduler environment.

Returns:

(status, timestamp, output)(Status, datetime, dict)

3-tuple of (Status, timestamp, output) values, where output is a dict with the results or None if the status is not COMPLETED. If run script is a benchmark, then the score is usually expected to be in the score field.

setup(tunables: TunableGroups, global_config: dict | None = None) → bool

Check if the environment is ready and set up the application and benchmarks, if necessary.

Parameters:

tunablesTunableGroups

A collection of tunable OS and application parameters along with their values. In a local environment these could be used to prepare a config file on the scheduler prior to transferring it to the remote environment, for instance.

global_configdict

Free-format dictionary of global parameters of the environment that are not used in the optimization process.

Returns:

is_successbool

True if operation is successful, false otherwise.

status() → Tuple[Status, datetime, List[Tuple[datetime, str, Any]]]

Check the status of the benchmark environment.

Returns:

(benchmark_status, timestamp, telemetry)(Status, datetime, list)

3-tuple of (benchmark status, timestamp, telemetry) values. timestamp is UTC time stamp of the status; it’s current time by default. telemetry is a list (maybe empty) of (timestamp, metric, value) triplets.

teardown() → None

Clean up the local environment.

class mlos_bench.environments.LocalFileShareEnv(*, name: str, config: dict, global_config: dict | None = None, tunables: TunableGroups | None = None, service: Service | None = None)

Bases: LocalEnv

Scheduler-side Environment that runs scripts locally and uploads/downloads data to the shared file storage.

Attributes:

parameters

Key/value pairs of all environment parameters (i.e., const_args and tunable_params).

tunable_params

Get the configuration space of the given environment.

Methods

new(*, env_name, class_name, config[, ...])	Factory method for a new environment with a given config.
pprint([indent, level])	Pretty-print the environment configuration.
run()	Download benchmark results from the shared storage and run post-processing scripts locally.
setup(tunables[, global_config])	Run setup scripts locally and upload the scripts and data to the shared storage.
status()	Check the status of the benchmark environment.
teardown()	Clean up the local environment.

run() → Tuple[Status, datetime, Dict[str, int | float | str | None] | None]

Download benchmark results from the shared storage and run post-processing scripts locally.

Returns:

(status, timestamp, output)(Status, datetime, dict)

3-tuple of (Status, timestamp, output) values, where output is a dict with the results or None if the status is not COMPLETED. If run script is a benchmark, then the score is usually expected to be in the score field.

setup(tunables: TunableGroups, global_config: dict | None = None) → bool

Run setup scripts locally and upload the scripts and data to the shared storage.

Parameters:

tunablesTunableGroups

A collection of tunable OS and application parameters along with their values. In a local environment these could be used to prepare a config file on the scheduler prior to transferring it to the remote environment, for instance.

global_configdict

Free-format dictionary of global parameters of the environment that are not used in the optimization process.

Returns:

is_successbool

True if operation is successful, false otherwise.

status() → Tuple[Status, datetime, List[Tuple[datetime, str, Any]]]

Check the status of the benchmark environment.

Returns:

(benchmark_status, timestamp, telemetry)(Status, datetime, list)

3-tuple of (benchmark status, timestamp, telemetry) values. timestamp is UTC time stamp of the status; it’s current time by default. telemetry is a list (maybe empty) of (timestamp, metric, value) triplets.

class mlos_bench.environments.MockEnv(*, name: str, config: dict, global_config: dict | None = None, tunables: TunableGroups | None = None, service: Service | None = None)

Bases: Environment

Scheduler-side environment to mock the benchmark results.

Attributes:

parameters

Key/value pairs of all environment parameters (i.e., const_args and tunable_params).

tunable_params

Get the configuration space of the given environment.

Methods

new(*, env_name, class_name, config[, ...])	Factory method for a new environment with a given config.
pprint([indent, level])	Pretty-print the environment configuration.
run()	Produce mock benchmark data for one experiment.
setup(tunables[, global_config])	Set up a new benchmark environment, if necessary.
status()	Check the status of the benchmark environment.
teardown()	Tear down the benchmark environment.

run() → Tuple[Status, datetime, Dict[str, int | float | str | None] | None]

Produce mock benchmark data for one experiment.

Returns:

(status, timestamp, output)(Status, datetime, dict)

3-tuple of (Status, timestamp, output) values, where output is a dict with the results or None if the status is not COMPLETED. The keys of the output dict are the names of the metrics specified in the config; by default it’s just one metric named “score”. All output metrics have the same value.

class mlos_bench.environments.RemoteEnv(*, name: str, config: dict, global_config: dict | None = None, tunables: TunableGroups | None = None, service: Service | None = None)

Bases: ScriptEnv

Environment to run benchmarks and scripts on a remote host OS.

e.g. Application Environment

Attributes:

parameters

Key/value pairs of all environment parameters (i.e., const_args and tunable_params).

tunable_params

Get the configuration space of the given environment.

Methods

new(*, env_name, class_name, config[, ...])	Factory method for a new environment with a given config.
pprint([indent, level])	Pretty-print the environment configuration.
run()	Runs the run script on the remote environment.
setup(tunables[, global_config])	Check if the environment is ready and set up the application and benchmarks on a remote host.
status()	Check the status of the benchmark environment.
teardown()	Clean up and shut down the remote environment.

run() → Tuple[Status, datetime, Dict[str, int | float | str | None] | None]

Runs the run script on the remote environment.

This can be used to, for instance, submit a new experiment to the remote application environment by (re)configuring an application and launching the benchmark, or run a script that collects the results.

Returns:

(status, timestamp, output)(Status, datetime, dict)

3-tuple of (Status, timestamp, output) values, where output is a dict with the results or None if the status is not COMPLETED. If run script is a benchmark, then the score is usually expected to be in the score field.

setup(tunables: TunableGroups, global_config: dict | None = None) → bool

Check if the environment is ready and set up the application and benchmarks on a remote host.

Parameters:

tunablesTunableGroups

A collection of tunable OS and application parameters along with their values. Setting these parameters should not require an OS reboot.

global_configdict

Free-format dictionary of global parameters of the environment that are not used in the optimization process.

Returns:

is_successbool

True if operation is successful, false otherwise.

teardown() → None

Clean up and shut down the remote environment.

class mlos_bench.environments.Status(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: Enum

Enum for the status of the benchmark/environment.

CANCELED = 5

FAILED = 6

PENDING = 1

READY = 2

RUNNING = 3

SUCCEEDED = 4

TIMED_OUT = 7

UNKNOWN = 0

is_canceled() → bool

Check if the status of the benchmark/environment is CANCELED.

is_completed() → bool

Check if the status of the benchmark/environment is one of {SUCCEEDED, CANCELED, FAILED, TIMED_OUT}.

is_failed() → bool

Check if the status of the benchmark/environment is FAILED.

is_good() → bool

Check if the status of the benchmark/environment is good.

is_pending() → bool

Check if the status of the benchmark/environment is PENDING.

is_ready() → bool

Check if the status of the benchmark/environment is READY.

is_succeeded() → bool

Check if the status of the benchmark/environment is SUCCEEDED.

is_timed_out() → bool

Check if the status of the benchmark/environment is TIMED_OUT.

Subpackages

• mlos_bench.environments.local package
  • LocalEnv
    • LocalEnv.run()
    • LocalEnv.setup()
    • LocalEnv.status()
    • LocalEnv.teardown()
  • LocalFileShareEnv
    • LocalFileShareEnv.run()
    • LocalFileShareEnv.setup()
    • LocalFileShareEnv.status()
  • Submodules
    • mlos_bench.environments.local.local_env module
      • LocalEnv
    • mlos_bench.environments.local.local_fileshare_env module
      • LocalFileShareEnv
• mlos_bench.environments.remote package
  • HostEnv
    • HostEnv.setup()
    • HostEnv.teardown()
  • NetworkEnv
    • NetworkEnv.setup()
    • NetworkEnv.teardown()
  • OSEnv
    • OSEnv.setup()
    • OSEnv.teardown()
  • RemoteEnv
    • RemoteEnv.run()
    • RemoteEnv.setup()
    • RemoteEnv.teardown()
  • SaaSEnv
    • SaaSEnv.setup()
  • VMEnv
  • Submodules
    • mlos_bench.environments.remote.host_env module
      • HostEnv
    • mlos_bench.environments.remote.network_env module
      • NetworkEnv
    • mlos_bench.environments.remote.os_env module
      • OSEnv
    • mlos_bench.environments.remote.remote_env module
      • RemoteEnv
    • mlos_bench.environments.remote.saas_env module
      • SaaSEnv
    • mlos_bench.environments.remote.vm_env module
      • VMEnv

Submodules

• mlos_bench.environments.base_environment module
  • Environment
    • Environment.new()
    • Environment.parameters
    • Environment.pprint()
    • Environment.run()
    • Environment.setup()
    • Environment.status()
    • Environment.teardown()
    • Environment.tunable_params
• mlos_bench.environments.composite_env module
  • CompositeEnv
    • CompositeEnv.children
    • CompositeEnv.pprint()
    • CompositeEnv.run()
    • CompositeEnv.setup()
    • CompositeEnv.status()
    • CompositeEnv.teardown()
• mlos_bench.environments.mock_env module
  • MockEnv
    • MockEnv.run()
• mlos_bench.environments.script_env module
  • ScriptEnv
• mlos_bench.environments.status module
  • Status
    • Status.CANCELED
    • Status.FAILED
    • Status.PENDING
    • Status.READY
    • Status.RUNNING
    • Status.SUCCEEDED
    • Status.TIMED_OUT
    • Status.UNKNOWN
    • Status.is_canceled()
    • Status.is_completed()
    • Status.is_failed()
    • Status.is_good()
    • Status.is_pending()
    • Status.is_ready()
    • Status.is_succeeded()
    • Status.is_timed_out()