Variants

You might want to take a look at the following pages before exploring variants:

When benchmarking algorithms, it is often useful to compare different variants or parameter configurations of the same algorithm. Simexpal can manage those variants without requiring you to duplicate the experiments stanza multiple times.

Specifying Variants

There are two types of variants:

  1. static variants

  2. dynamic variants

Static variants have to be specified individually and thus allow for more customization e.g. setting different environment variables for each individual variant. In cases where such customization is not necessary, dynamic variations can be used. By using @-Variables and syntactic sugar, dynamic variations also simplify writing the experiments.yml.

Static Variants

To specify static variants we will use the following keys:

  • axis: name of the variant axis

  • items: list of dictionaries, which specify variants belonging to the same axis.

For the variants in the items list we have to specify the

  • name: name of the variant

  • extra_args: list of variant arguments

keys.

How to specify static variants in the experiments.yml file.
 1 variants:
 2   - axis: 'block-algo'
 3     items:
 4       - name: 'ba-insert'
 5         extra_args: ['insertion_sort']
 6       - name: 'ba-bubble'
 7         extra_args: ['bubble_sort']
 8   - axis: 'block-size'
 9     items:
10       - name: 'bs32'
11         extra_args: ['32']
12       - name: 'bs64'
13         extra_args: ['64']

In this way we created the two variant axes block-algo and block-size. The former axis has the variants ba-insert and ba-bubble and the latter has the variants bs32 and bs64. If the extra_args contain more than one argument, e.g., a keyword argument --block-algo ba-insert, then the arguments are stated separately in a list, i.e. ['--block-algo', 'ba-insert'].

Experiments will then be created from the cross product of the variants of each axis, i.e, (ba-insert, bs32), (ba-insert, bs64), (ba-bubble, bs32) and (ba-bubble, bs32) for the experiments.yml above.

Note

The elements in each cross product will be sorted lexicographically. This plays a role when the extra_args contain positional arguments as they are resolved in the order of the elements in a cross product.

Later we will see how to choose only certain combinations of variants (Run Matrix).

Dynamic Variants

Dynamic variants work similar to static variants. Some differences are the syntax, automatic assignment of variant names of the form <axis_name>:<variant_value> to the variants and the introduction of the @-variable @VARIANT_VALUE:<axis_name>@.

range Variants

In cases where the values of variants are integers with regular step sizes, we can use range variants. In order to do so, we will use the

  • axis: name of the variant axis

  • range: list with start and stop element

  • steps: step size

  • extra_args: list of variant arguments

keys.

The range and steps keys will generate values i equivalent to a for(int i = start; i <= stop; i += steps) loop in C.

The experiments.yml file in the static variants example can then be written in the following way:

How to specify range variants in the experiments.yml file.
 1 variants:
 2   - axis: 'block-algo'
 3     items:
 4       - name: 'ba-insert'
 5         extra_args: ['insertion_sort']
 6       - name: 'ba-bubble'
 7         extra_args: ['bubble_sort']
 8   - axis: 'block-size'
 9     range: [32,64]
10     steps: 32
11     extra_args: ['@VARIANT_VALUE:block-size@']

The only differences to the static variants example is that we created the variant axis block-size with variants named block-size:32 and block-size:64. Also, we need to specify the @VARIANT_VALUE:<axis_name>@ @-variable in the extra_args key to access the values of the variants.

enum Variants

enum variants can be seen as a generalization of range variants. With enum variants, we can specify arbitrary values explicitly. In order to so, we will use the

  • axis: name of the variant axis

  • enum: list of variant values

  • extra_args: list of variant arguments

keys.

The experiments.yml file in the range variants example can then be written in the following way:

How to specify enum variants in the experiments.yml file.
1 variants:
2   - axis: 'block-algo'
3     enum: ['insertion_sort', 'bubble_sort']
4     extra_args: ['@VARIANT_VALUE:block-algo@']
5   - axis: 'block-size'
6     range: [32,64]
7     steps: 32
8     extra_args: ['@VARIANT_VALUE:block-size@']

The only differences to the range variants example is that we created the variant axis block-algo with variants named block-algo:insertion_sort and block-algo:bubble_sort.

Setting Environment Variables

Note

Environment variables specified in variants will override the values of environment variables specified in the experiments stanza (if the same variable occurs).

Setting environment variables for variants works similar to setting environment variables for experiments. For static variants we set the

  • environ: dictionary of (environment variable, value)-pairs

key for each item in the items key. For dynamic variants we set the environ key on the same level as the axis key. For example you can specify environment variables as follows:

How to specify environment variables for variants in the experiments.yml file.
 1variants:
 2  - axis: num_threads
 3    items:
 4      - name: ONT2
 5        ...
 6        environ:
 7          OMP_NUM_THREADS: 2
 8
 9      - name: ONT4
10        ...
11        environ:
12          OMP_NUM_THREADS: 4
13  - axis: block-algo
14    enum: ...
15    ...
16    environ:
17      foo1: 'bar'
18  - axis: block-size
19    range: ...
20    steps: ...
21    ...
22    environ:
23      foo2: 'baz'

With dynamic variants it is also possible to use the @-variable @VARIANT_VALUE:<axis_name>@ as value for the environment variables.

Note

Dynamic variants of the same axis will share the environment variables set by environ.

Slurm: --ntasks-per-node, -c, -N

Note

Supported Slurm arguments specified in variants will override the values of supported Slurm arguments specified in the experiments stanza (if the same slurm argument occurs).

The supported Slurm arguments in experiments are also supported for variants. For static variants we set the

  • procs_per_node: number of tasks to invoke on each node (slurm: --ntasks-per-node=n)

  • num_threads: number of cpus required per task (slurm: -c, --cpus-per-task=ncpus)

  • num_nodes: number of nodes on which to run (N = min[-max]) (slurm: -N, --nodes=N)

keys for each item in the items key.

How to specify supported Slurm parameters for static variants in the experiments.yml file.
 1variants:
 2  - axis: num_cores
 3    items:
 4      - name: c24
 5        num_nodes: 1
 6        procs_per_node: 24
 7        num_threads: 2
 8        extra_args: []           # empty extra_args as we only want to benchmark with
 9                                 # different node/cpu settings
10      - name: c48
11        num_nodes: 2
12        procs_per_node: 24
13        num_threads: 2
14        extra_args: []

When launching your experiments with slurm, the variant c24 will append -N 1 --ntasks-per-node 24 -c 2 to the sbatch command. Analogously for the experiment with the variant c48.

For dynamic variants we set procs_per_node, num_threads and num_nodes on the same level as the axis key:

How to specify supported Slurm parameters for dynamic variants in the experiments.yml file.
1variants:
2  - axis: 'block-size'
3    range: [32,64]
4    steps: 32
5    extra_args: ['@VARIANT_VALUE:block-size@']
6    num_nodes: 1
7    procs_per_node: 24
8    num_threads: 2

Note

Dynamic variants of the same axis will share the values set by procs_per_node, num_threads and num_nodes.

Also, the @-variable @VARIANT_VALUE:<axis_name>@ can be specified as value of these keys. E.g. the following is equivalent to the static case above:

How to specify supported Slurm parameters as dynamic variants in the experiments.yml file.
1variants:
2- axis: 'num-cores'
3  range: [1,2]
4  steps: 1
5  num_nodes: '@VARIANT_VALUE:num-cores@'
6  procs_per_node: 24
7  num_threads: 2

Next

Now that you have entirely set up your experiments, you can modify the experiment combinations that you want to run. Visit the Run Matrix page for a detailed explanation.