Variants
You might want to take a look at the following pages before exploring variants:
When benchmarking algorithms, it is often useful to compare different variants or parameter
configurations of the same algorithm. Simexpal can manage those variants without requiring
you to duplicate the experiments
stanza multiple times.
Specifying Variants
There are two types of variants:
static variants
dynamic variants
Static variants have to be specified individually and thus allow for more customization e.g.
setting different environment variables for each individual
variant. In cases where such customization is not necessary, dynamic variations can be used. By using
@-Variables and syntactic sugar, dynamic variations also simplify writing the experiments.yml
.
Static Variants
To specify static variants we will use the following keys:
axis
: name of the variant axisitems
: list of dictionaries, which specify variants belonging to the same axis.
For the variants in the items
list we have to specify the
name
: name of the variantextra_args
: list of variant arguments
keys.
1 variants:
2 - axis: 'block-algo'
3 items:
4 - name: 'ba-insert'
5 extra_args: ['insertion_sort']
6 - name: 'ba-bubble'
7 extra_args: ['bubble_sort']
8 - axis: 'block-size'
9 items:
10 - name: 'bs32'
11 extra_args: ['32']
12 - name: 'bs64'
13 extra_args: ['64']
In this way we created the two variant axes block-algo
and block-size
. The former
axis has the variants ba-insert
and ba-bubble
and the latter has the variants
bs32
and bs64
. If the extra_args
contain more than one argument, e.g., a keyword
argument --block-algo ba-insert
, then the arguments are stated separately in a list, i.e.
['--block-algo', 'ba-insert']
.
Experiments will then be created from the cross product of the variants of each axis, i.e,
(ba-insert, bs32)
, (ba-insert, bs64)
, (ba-bubble, bs32)
and (ba-bubble, bs32)
for the experiments.yml
above.
Note
The elements in each cross product will be sorted lexicographically. This plays a role when
the extra_args
contain positional arguments as they are resolved in the order of the
elements in a cross product.
Later we will see how to choose only certain combinations of variants (Run Matrix).
Dynamic Variants
Dynamic variants work similar to static variants. Some differences are the syntax, automatic
assignment of variant names of the form <axis_name>:<variant_value>
to the variants and the introduction
of the @-variable @VARIANT_VALUE:<axis_name>@
.
range
Variants
In cases where the values of variants are integers with regular step sizes, we can use range
variants. In order to do so, we will use the
axis
: name of the variant axisrange
: list with start and stop elementsteps
: step sizeextra_args
: list of variant arguments
keys.
The range
and steps
keys will generate values i
equivalent to a
for(int i = start; i <= stop; i += steps)
loop in C.
The experiments.yml
file in the static variants example can then be written in the
following way:
1 variants:
2 - axis: 'block-algo'
3 items:
4 - name: 'ba-insert'
5 extra_args: ['insertion_sort']
6 - name: 'ba-bubble'
7 extra_args: ['bubble_sort']
8 - axis: 'block-size'
9 range: [32,64]
10 steps: 32
11 extra_args: ['@VARIANT_VALUE:block-size@']
The only differences to the static variants example is that we created the variant axis
block-size
with variants named block-size:32
and block-size:64
. Also, we need to specify the
@VARIANT_VALUE:<axis_name>@
@-variable in the extra_args
key to access the values of
the variants.
enum
Variants
enum
variants can be seen as a generalization of range variants. With enum
variants, we can specify arbitrary values explicitly. In order to so, we will use the
axis
: name of the variant axisenum
: list of variant valuesextra_args
: list of variant arguments
keys.
The experiments.yml
file in the range variants example can then be written in the
following way:
1 variants:
2 - axis: 'block-algo'
3 enum: ['insertion_sort', 'bubble_sort']
4 extra_args: ['@VARIANT_VALUE:block-algo@']
5 - axis: 'block-size'
6 range: [32,64]
7 steps: 32
8 extra_args: ['@VARIANT_VALUE:block-size@']
The only differences to the range variants example is that we created the variant axis
block-algo
with variants named block-algo:insertion_sort
and block-algo:bubble_sort
.
Setting Environment Variables
Note
Environment variables specified in variants will override the values of environment variables
specified in the experiments
stanza (if the same variable occurs).
Setting environment variables for variants works similar to setting environment variables for experiments. For static variants we set the
environ
: dictionary of (environment variable, value)-pairs
key for each item in the items
key. For dynamic variants we set the environ
key on the same
level as the axis
key. For example you can specify environment variables as follows:
1variants:
2 - axis: num_threads
3 items:
4 - name: ONT2
5 ...
6 environ:
7 OMP_NUM_THREADS: 2
8
9 - name: ONT4
10 ...
11 environ:
12 OMP_NUM_THREADS: 4
13 - axis: block-algo
14 enum: ...
15 ...
16 environ:
17 foo1: 'bar'
18 - axis: block-size
19 range: ...
20 steps: ...
21 ...
22 environ:
23 foo2: 'baz'
With dynamic variants it is also possible to use the @-variable @VARIANT_VALUE:<axis_name>@
as value for the environment variables.
Note
Dynamic variants of the same axis
will share the environment variables set by environ
.
Slurm: --ntasks-per-node
, -c
, -N
Note
Supported Slurm arguments specified in variants will override the values of supported Slurm
arguments specified in the experiments
stanza (if the same slurm argument occurs).
The supported Slurm arguments in experiments are also supported for variants. For static variants we set the
procs_per_node
: number of tasks to invoke on each node (slurm:--ntasks-per-node=n
)num_threads
: number of cpus required per task (slurm:-c
,--cpus-per-task=ncpus
)num_nodes
: number of nodes on which to run (N = min[-max]) (slurm:-N
,--nodes=N
)
keys for each item in the items
key.
1variants:
2 - axis: num_cores
3 items:
4 - name: c24
5 num_nodes: 1
6 procs_per_node: 24
7 num_threads: 2
8 extra_args: [] # empty extra_args as we only want to benchmark with
9 # different node/cpu settings
10 - name: c48
11 num_nodes: 2
12 procs_per_node: 24
13 num_threads: 2
14 extra_args: []
When launching your experiments with slurm, the variant c24
will append
-N 1 --ntasks-per-node 24 -c 2
to the sbatch command. Analogously for the experiment with
the variant c48
.
For dynamic variants we set procs_per_node
, num_threads
and num_nodes
on the same
level as the axis
key:
1variants:
2 - axis: 'block-size'
3 range: [32,64]
4 steps: 32
5 extra_args: ['@VARIANT_VALUE:block-size@']
6 num_nodes: 1
7 procs_per_node: 24
8 num_threads: 2
Note
Dynamic variants of the same axis
will share the values set by procs_per_node
, num_threads
and num_nodes
.
Also, the @-variable @VARIANT_VALUE:<axis_name>@
can be specified as value of
these keys. E.g. the following is equivalent to the static case above:
1variants:
2- axis: 'num-cores'
3 range: [1,2]
4 steps: 1
5 num_nodes: '@VARIANT_VALUE:num-cores@'
6 procs_per_node: 24
7 num_threads: 2
Next
Now that you have entirely set up your experiments, you can modify the experiment combinations that you want to run. Visit the Run Matrix page for a detailed explanation.