The “experiments.yml” reference

Simexpal needs an experiments.yml file to to automatically execute your experiments on the desired instances. In this page we describe the structure of the experiments.yml file. The experiments.yml is a YAML file that contains a dictionary with several keys:

instances: list of all the instances that will be used for the experiments
instdir: path to the directory that stores all the instances
experiments: list of all the experiments that will be executed on the instances

There are also further keys that allow for customization of the experiments and to automatically build the binaries the experiments run from:

builds: list of Git builds, which also include build instructions
revisions: list of Git revisions of builds
variants: list of additional input parameters for experiments
matrix: specifies which combinations of experiments, instances, variants and revisions are run

Instances

This entry is a list of instances that will be used for experiments. The following keys are used for specifying instances:

extensions: list of extensions that the instance has
files: list of files the instance consists of
items: list of instances
name: name of the instance (used when dealing with instances that consist of unrelated files)
repo: source of instances
set: list of sets the instance belongs to

For detailed usage examples, see the Instances page.

Builds

This entry is a list of builds that will be used for revisions. The following keys are used for specifying builds:

configure: list of dictionaries containing configuration parameters
compile: list of dictionaries containing compilation parameters
environ: dictionary of (environment variable, value)-pairs
git: link to the Git repository
install: list of dictionaries containing installation parameters
name: (arbitrary) name of the build
recursive-clone: boolean (true/false) - whether to pull git submodules recursively or not
regenerate: list of dictionaries containing regeneration parameters
requires: list of required builds
workdir: path of the working directory

For detailed usage examples, see the Builds page.

Revisions

This entry is a list of revisions that will be used for experiments. The following keys are used for specifying revisions:

build_version: dictionary of (build, SHA-1 hash/branch)-pairs
develop: boolean (true/false) - whether this revision is a develop revision or not
name: (arbitrary) name of the revision

For detailed usage examples, see the Revisions page.

Experiments

This entry is a list of experiments that will be executed on all the instances. Each experiment includes three keys:

args: list of experiment arguments
environ: dictionary of (environment variable, value)-pairs
name: name of the experiment
num_nodes: number of nodes on which to run
num_threads: number of cpus required per task
output: dictionary containing all output file extensions
procs_per_node: number of tasks to invoke on each node
repeat: integer - number of times an experiment is repeated
slurm_args: list of additional sbatch arguments
stdout: extension of the output file
timeout: integer - timeout in seconds
use_builds: list of used build names

For detailed usage examples, see the Experiments page.

Variants

This entry is a list of variants that will be used for experiments. The following keys are used for specifying variants:

axis: name of the variant axis
enum: list of variant values
environ: dictionary of (environment variable, value)-pairs
extra_args: list of variant arguments
items: list of dictionaries, which specify variants belonging to the same axis.
name: name of the variant
num_nodes: number of nodes on which to run
num_threads: number of cpus required per task
procs_per_node: number of tasks to invoke on each node
range: list with start and stop element
steps: step size

For detailed usage examples, see the Variants page.

Run Matrix

This entry is a list of desired experiment combinations. The following keys are used for specifying desired experiment combinations:

axes: list of included axis names
experiments: list of included experiment names
include: list of dictionaries, which specify included experiment combinations
instsets: list of included instance set names
repetitions: integer - number of times all combinations of an include entry are repeated
revisions: list of included revision names
variants: list of included variant names

For detailed usage examples, see the Run Matrix page.