Benchmarks¶
The benchmarks measure the loss and the time of scimba’s solvers with
benchopt. A benchmark family is a directory
benchmarks/benchmarks_<backend>/<family>/ (<backend> is jax or
torch) holding a benchopt objective.py, its datasets and solvers, and a
Makefile with two targets: run for the full benchmark (hours) and
run_test for a short one.
The commands below are run from the repository root, in an environment with scimba and benchopt:
uv sync --group scimba_jax --group benchmarks
Running the benchmarks without SLURM¶
make -C benchmarks/benchmarks_jax run_test # every jax family
make -C benchmarks/benchmarks_jax run_test FAMILIES="pinns fem" # some families
Use run instead of run_test for the full benchmarks. The results go to
each family’s outputs/, with the output of the run in outputs/run.log,
where the failures are read from when archiving. Before a run meant to be archived, remove the
previous results with make -C benchmarks/benchmarks_jax clean: the merged
result file of a family takes in every result present in its outputs/.
nonregression/Makefile
runs both backends at once (make bench-clean, make bench-test,
make bench, with BACKENDS and FAMILIES to restrict them).
Running the benchmarks with SLURM¶
benchmarks/slurm/submit.sh -w hpc-n932 -n run_test # print the sbatch command only
benchmarks/slurm/submit.sh -w hpc-n932 run_test # every jax family
benchmarks/slurm/submit.sh -w 'hpc-n[932-934]' run jax pinns fem # first free node of a list
The job takes a whole node and runs one family per GPU. Its logs, and the
state of each family, are in benchmarks/slurm/logs/<run>/. The results go to
each family’s outputs/, as without SLURM. Setting up the environment on the
cluster, the options of submit.sh and what happens when a job is preempted
are described in
slurm/README.md.
Archiving the results¶
From benchmarks/nonregression, right after the run, from the same checkout
and the same environment:
make archive-test # after run_test
make archive # after run
make show # build the comparison page and open it
The results are copied to results/dev/<commit>/<machine>/, where
<commit> is the commit checked out. The page compares any two archived runs,
loss and time side by side.
The archive is kept in the git repository: add it and commit it, then push, so that it is part of the history the others compare against:
git add results
git commit -m "benchmarks: archive the results of <commit> on <machine>"
git push
The archive keeps the label of the commit that ran, whatever commit adds it to the repository.
Attaching an archive to a release¶
A full run is attached to a release in two steps that follow the release
procedure, from benchmarks/nonregression:
# once the version is bumped to 1.3.5 on develop:
make attach VERSION=1.3.5 FROM=<commit of the run>
git add results/releases.json
git commit -m "benchmarks: attach <commit of the run> to 1.3.5"
git push
# once develop is merged into main and main is tagged v1.3.5:
make confirm VERSION=1.3.5
git add results/releases.json
git commit -m "benchmarks: confirm the runs of 1.3.5"
git push
The attachments are recorded in results/releases.json, which is in the git
repository as the archive is: commit it after each step, on develop (the
next merge brings it to main). These commits do not prevent the
confirmation: confirm only compares the code the runs depend on. The run is then
shown as release 1.3.5 in the comparison page, marked as not tagged yet
until it is confirmed. attach refuses a run whose code differs from
develop (in the sources or the benchmark the run depends on, the version
number aside), a run archived from a modified checkout and a run_test run;
confirm refuses when that code changed between develop and the tag.
FORCE=1 goes on anyway and records why. So the simplest is to run the
benchmarks on develop once it holds the code to release.
How the archive is stored, what the comparison page shows and the details of
attach are in
nonregression/README.md.