Run Matrix
You might want to take a look at the following pages before exploring the run matrix:
Simexpal will build every possible combination of experiment,
instance, variant and revision.
There are cases where this is not desired. For example, you might only want to run certain
instance/variant combinations. On this page we describe how to use the matrix
key in the
experiments.yml
file and will mainly use the
C++ example from
the Quick Start guide in order to do so.
Include Experiments
Currently the matrix
key works by specifying all the experiment combinations that you want
to include. Therefore, we will specify the
include
: list of dictionaries, which specify included experiment combinations
key.
Each entry of include
can contain the following keys:
experiments
: list of included experiment namesinstsets
: list of included instance set namesaxes
: list of included axis namesvariants
: list of included variant namesrevisions
: list of included revision names
Simexpal will then build the cross product of each experiment, instance set, variant and revision
for each entry in include
. Omitting a key in include
will select all possible elements of
the respective key, e.g., omitting revisions
is equivalent to specifying the revisions
key
with all possible revisions.
The matrix
key of
1matrix:
2 include:
3 - experiments: [quick-sort]
4 variants: [ba-insert, bs32]
5 revisions: [main]
6 - experiments: [quick-sort]
7 variants: [ba-bubble]
8 revisions: [main]
(The full experiments.yml
can be found here .)
results in the following experiments (using simex experiments list
to display them):
Experiment Instance Status
---------- -------- ------
quick-sort ~ ba-bubble, bs32 @ main uniform-n1000-s1 [0] not submitted
quick-sort ~ ba-bubble, bs32 @ main uniform-n1000-s2 [0] not submitted
quick-sort ~ ba-bubble, bs64 @ main uniform-n1000-s1 [0] not submitted
quick-sort ~ ba-bubble, bs64 @ main uniform-n1000-s2 [0] not submitted
quick-sort ~ ba-insert, bs32 @ main uniform-n1000-s1 [0] not submitted
quick-sort ~ ba-insert, bs32 @ main uniform-n1000-s2 [0] not submitted
Note
If the axes
key is omitted and the variants
key does not contain any variant of an axis, then
every variant of this axis will be selected. If this is not intended you need to specify the axes
key with the names of the desired axes.
If we wanted the second entry of the include
key in the experiments.yml
above, to only include
the variant ba-bubble
(without any other variants of other axes), we would need to specify the
axes
key like this:
1matrix:
2 include:
3 - experiments: [quick-sort]
4 variants: [ba-insert, bs32]
5 revisions: [main]
6 - experiments: [quick-sort]
7 axes: [block-algo]
8 variants: [ba-bubble]
9 revisions: [main]
In this way we obtain the experiments:
Experiment Instance Status
---------- -------- ------
quick-sort ~ ba-bubble @ main uniform-n1000-s1 [0] not submitted
quick-sort ~ ba-bubble @ main uniform-n1000-s2 [0] not submitted
quick-sort ~ ba-insert, bs32 @ main uniform-n1000-s1 [0] not submitted
quick-sort ~ ba-insert, bs32 @ main uniform-n1000-s2 [0] not submitted
Repetitions
Note
Repetitions specified in the run matrix will override the values of repeat
specified in the
experiments
stanza.
Similarly to repeating experiments we can repeat all experiment combinations
of an include
entry by specifying
repetitions
: integer - number of times all combinations of aninclude
entry are repeated.
To repeat experiments twice, we can define the key as follows:
1matrix:
2 include:
3 - experiments: [...]
4 variants: [...]
5 ...
6 repetitions: 2
The default value of repetitions
is 1
.
More Examples
Experiments with Different Instance Types
When you are dealing with experiments that have different instance types, e.g. Multiple Extensions
Instances and Local Instances, you need to use Instance Sets and the instsets
key
appropriately. This means:
First, you have to assign your instances with different types to separate instance sets. For example, you
can assign your multiple extension instances to the instance set multiple_extension_set
and your local
instances to local_instance_set
. Assuming you have defined two experiments multiple_ext_experiment
and local_experiment
that take multiple extension and local instances as input respectively, you can
specify your run matrix as follows:
1matrix:
2 include:
3 - experiments: [multiple_ext_experiment]
4 instsets: [multiple_extension_set]
5 ...
6 - experiments: [local_experiment]
7 instsets: [local_instance_set]
8 ...
In this way, we can assure that the right instance paths are passed to each experiment.