Run Matrix

You might want to take a look at the following pages before exploring the run matrix:

Simexpal will build every possible combination of experiment, instance, variant and revision. There are cases where this is not desired. For example, you might only want to run certain instance/variant combinations. On this page we describe how to use the matrix key in the experiments.yml file and will mainly use the C++ example from the Quick Start guide in order to do so.

Include Experiments

Currently the matrix key works by specifying all the experiment combinations that you want to include. Therefore, we will specify the

  • include: list of dictionaries, which specify included experiment combinations

key.

Each entry of include can contain the following keys:

  • experiments: list of included experiment names

  • instsets: list of included instance set names

  • axes: list of included axis names

  • variants: list of included variant names

  • revisions: list of included revision names

Simexpal will then build the cross product of each experiment, instance set, variant and revision for each entry in include. Omitting a key in include will select all possible elements of the respective key, e.g., omitting revisions is equivalent to specifying the revisions key with all possible revisions.

The matrix key of

1matrix:
2  include:
3    - experiments: [quick-sort]
4      variants: [ba-insert, bs32]
5      revisions: [main]
6    - experiments: [quick-sort]
7      variants: [ba-bubble]
8      revisions: [main]

(The full experiments.yml can be found here .)

results in the following experiments (using simex experiments list to display them):

Experiment                                    Instance                     Status
----------                                    --------                     ------
quick-sort ~ ba-bubble, bs32 @ main           uniform-n1000-s1             [0] not submitted
quick-sort ~ ba-bubble, bs32 @ main           uniform-n1000-s2             [0] not submitted
quick-sort ~ ba-bubble, bs64 @ main           uniform-n1000-s1             [0] not submitted
quick-sort ~ ba-bubble, bs64 @ main           uniform-n1000-s2             [0] not submitted
quick-sort ~ ba-insert, bs32 @ main           uniform-n1000-s1             [0] not submitted
quick-sort ~ ba-insert, bs32 @ main           uniform-n1000-s2             [0] not submitted

Note

If the axes key is omitted and the variants key does not contain any variant of an axis, then every variant of this axis will be selected. If this is not intended you need to specify the axes key with the names of the desired axes.

If we wanted the second entry of the include key in the experiments.yml above, to only include the variant ba-bubble (without any other variants of other axes), we would need to specify the axes key like this:

1matrix:
2  include:
3    - experiments: [quick-sort]
4      variants: [ba-insert, bs32]
5      revisions: [main]
6    - experiments: [quick-sort]
7      axes: [block-algo]
8      variants: [ba-bubble]
9      revisions: [main]

In this way we obtain the experiments:

Experiment                                    Instance                     Status
----------                                    --------                     ------
quick-sort ~ ba-bubble @ main                 uniform-n1000-s1             [0] not submitted
quick-sort ~ ba-bubble @ main                 uniform-n1000-s2             [0] not submitted
quick-sort ~ ba-insert, bs32 @ main           uniform-n1000-s1             [0] not submitted
quick-sort ~ ba-insert, bs32 @ main           uniform-n1000-s2             [0] not submitted

Repetitions

Note

Repetitions specified in the run matrix will override the values of repeat specified in the experiments stanza.

Similarly to repeating experiments we can repeat all experiment combinations of an include entry by specifying

  • repetitions: integer - number of times all combinations of an include entry are repeated.

To repeat experiments twice, we can define the key as follows:

How to specify repetitions in the matrix key of an experiments.yml file.
1matrix:
2  include:
3    - experiments: [...]
4      variants: [...]
5      ...
6      repetitions: 2

The default value of repetitions is 1.

More Examples

Experiments with Different Instance Types

When you are dealing with experiments that have different instance types, e.g. Multiple Extensions Instances and Local Instances, you need to use Instance Sets and the instsets key appropriately. This means:

First, you have to assign your instances with different types to separate instance sets. For example, you can assign your multiple extension instances to the instance set multiple_extension_set and your local instances to local_instance_set. Assuming you have defined two experiments multiple_ext_experiment and local_experiment that take multiple extension and local instances as input respectively, you can specify your run matrix as follows:

How to specify experiments with different instance types in the run matrix of the experiments.yml file.
1matrix:
2  include:
3    - experiments: [multiple_ext_experiment]
4      instsets: [multiple_extension_set]
5      ...
6    - experiments: [local_experiment]
7      instsets: [local_instance_set]
8      ...

In this way, we can assure that the right instance paths are passed to each experiment.