Running on a cluster

Running on a cluster#

The pulse CLI is designed so one config.toml can drive an array-job parameter sweep, and a long run can survive a wall-time kill and be continued (--restart), possibly on a different node allocation. This page assumes SLURM but the same ideas (override flags, exit codes, restart) apply to any scheduler.

Mesh once, run many#

Generating a mesh (especially a realistic lv_ellipsoid/biv_ellipsoid/ukb geometry) can take much longer than a short SLURM job’s queue-wait budget wants, and every array-job task sharing one config.toml would otherwise regenerate (or race to regenerate) it independently. Build it once, on a login node or a short single-task job, before submitting the sweep:

pulse validate-config config.toml   # parse-time checks only: no mesh, no MPI-collective work
pulse geometry config.toml          # build/cache the mesh once

pulse geometry is exactly the same step pulse run would do lazily on first use, cached in geometry.folder/<hash>/, a subfolder keyed by a hash of the [geometry] section — so running it up front is purely an optimization, not a separate code path; every array-job task’s own pulse run reuses the cache rather than rebuilding it. A sweep over a geometry parameter (e.g. --set geometry.char_length=... per task) is safe too: each distinct geometry gets its own subfolder, so tasks never overwrite or delete each other’s mesh. Tasks that need the same, not yet cached mesh may each generate it, but every job generates into a private temporary folder that is atomically renamed into place; whichever finishes first wins, and the others reuse its mesh and discard their own copy. Warming the cache with pulse geometry first (once per distinct geometry) avoids that duplicated work. It’s also the cheapest way to surface a marker-name typo ([[load]], [bcs]) ahead of the timed sweep: that check needs the actual mesh, so it only runs once pulse run has built or loaded the geometry (see Exit codes) — with the mesh already cached, that happens within seconds rather than after a from-scratch mesh generation inside the job.

A SLURM array job for a parameter sweep#

--output-folder and --set let one base config.toml be varied per array-job task without copying files:

#!/bin/bash
#SBATCH --job-name=pulse-sweep
#SBATCH --array=0-9
#SBATCH --ntasks=64
#SBATCH --time=04:00:00

DT=(0.5 1 2 5 10 0.5 1 2 5 10)
srun pulse run config.toml \
    --output-folder "runs/${SLURM_ARRAY_TASK_ID}" \
    --set "time.dt=\"${DT[$SLURM_ARRAY_TASK_ID]} ms\"" \
    --overwrite

--output-folder resolves against the current working directory the job runs in (unlike every path inside the config file, which resolves against the config file’s own directory — see Overrides), so runs/${SLURM_ARRAY_TASK_ID} above lands next to wherever the job script itself runs from. --overwrite makes the task safe to resubmit: if that array index’s output folder already has results in it (e.g. a resubmit after a scheduler-level failure, or resubmitting the whole array because task 3 failed), pulse would otherwise refuse to touch it rather than silently deleting a previous task’s output from under a differently-indexed rerun.

Prestress cache#

With [prestress] the unloaded reference configuration is computed once and cached in prestress.cache_folder/<hash>/, keyed by the physics that determines it (geometry, material, compressibility, boundary conditions, targets, ramp steps, function spaces), not by [solver] or [circulation]. Runs of the same physics therefore share one entry, so for array jobs put prestress.cache_folder on shared storage, as for the geometry cache. Concurrent tasks may each compute a missing entry; the first to finish installs it by an atomic rename, and a valid entry is never replaced: the others run with their own result (the same up to round-off), discard it and leave the installed entry for later runs and restarts. Only a stale or unreadable entry is replaced. Run one task (or pulse run once with a short end_time) first to warm it. --overwrite never deletes the cache.

Surviving a wall-time kill: --restart#

For a run whose simulated time exceeds what a single job’s wall-time allows, set a checkpoint interval and let the job resubmit itself onto the same output folder:

#!/bin/bash
#SBATCH --job-name=pulse-longrun
#SBATCH --time=24:00:00
#SBATCH --ntasks=64

# output.checkpoint_every = "0.1 s" in config.toml
if [ -f output/restart.json ]; then FLAG=--restart; else FLAG=--overwrite; fi
srun pulse run config.toml $FLAG

Resubmit the same script (e.g. from a scheduler dependency chain, sbatch --dependency=afterany, or cron) until run.json: status == "finished". Each resubmission picks up --restart automatically once output/restart.json exists (written after the first successful checkpoint). Before that — the first submission, or a job killed before its first checkpoint, which leaves a results.bp but no restart.json — there is nothing to continue from, so the script starts over with --overwrite (without it, pulse refuses to touch the existing results.bp, saying that no restart checkpoint exists yet). --overwrite is safe here: it only deletes pulse’s own artifacts, and only after the config has been validated.

What may change across a restart, and what may not: pulse refuses --restart if the run’s physics has changed since the last checkpoint, comparing a hash of the whole resolved config except the run length (time.end_time/num_steps) and everything under [output]/[postprocess]/[solver]. So between restarts you may freely:

  • Extend time.end_time, or change num_steps, to run longer than originally configured — as long as the effective dt ((end_time - start_time) / num_steps when num_steps is given) stays the same; changing num_steps without a matching end_time change is refused rather than silently mixing time steps.

  • Change anything under [output] (save_every, checkpoint_every, performance, log_every) or [postprocess] (vtx, fields, points, vertex_tags, plots).

  • Change anything under [solver] (max_halvings, petsc_options, or --petsc-options): they change how a step is solved, not the physics. This is the way to continue after a solver failure (exit code 2) — e.g. --restart --set solver.max_halvings=8.

  • Run on a different number of MPI ranks than the original job used (the checkpoint is read and redistributed across however many ranks the new job has).

But not, without pulse refusing with an error naming the mismatch:

  • time.start_time, time.dt (or the effective dt derived from num_steps), [geometry] (except geometry.folder for a generated geometry type, which is just a cache location there, not the physics itself — it does count for geometry.type = "folder", where it’s the actual mesh being simulated), [material], [active], [compressibility], [viscoelasticity], [bcs], [[load]] (including a [load.profile] file table’s contents, not just its path — editing that CSV also counts as a physics change), [problem]. So a solver failure that needs a smaller time.dt or a gentler load cannot be continued: rerun it with --overwrite (or into a new output folder).

Exit codes for job-script branching#

0 success, 1 a configuration error (ConfigError, a command-line usage error, or a missing cli extra), 2 a runtime failure (Newton failing to converge within solver.max_halvings, or any other unexpected error, e.g. from mesh generation or I/O). A parse-time mistake (bad TOML, wrong units, an unknown key) is always caught before any mesh is built or loaded; a marker-name mistake (which needs the actual mesh to check) is caught right after that — still well before the collective solve loop, but only cheap in wall-time if the geometry was already built/cached ahead of time (see Mesh once, run many). Either way, a job script can branch on the exit code directly:

srun pulse run config.toml $FLAG
case $? in
  0) echo "done" ;;
  1) echo "config or usage error, not retrying" >&2; exit 1 ;;
  2) echo "runtime failure (solver, I/O, mesh generation): check output/run.json and the log" >&2; exit 1 ;;
esac

output/run.json (status: "running"|"finished"|"failed", plus n_ranks, wall-clock start/end, and — on a failure — the error) is the same information in a form a later step or a monitoring script can read back out of the output folder itself. It is only written once the run has started: a failure while setting up the simulation (config, geometry, markers, boundary conditions) leaves the output folder untouched, so check the exit code and the job’s own log for those.

Env var overrides with scheduler-provided variables#

PULSE_<SECTION>__<KEY> env vars (matched case-insensitively against the schema, e.g. PULSE_TIME__DT) sit below --set and above the TOML file in precedence — convenient for piping a scheduler’s own environment straight through without constructing a --set string in the job script:

export PULSE_OUTPUT__FOLDER="runs/${SLURM_ARRAY_TASK_ID}"
export PULSE_TIME__DT="${DT_MS} ms"
srun pulse run config.toml

(Note PULSE_OUTPUT__FOLDER here still resolves against the config file’s directory like any other in-file path, since it’s not the dedicated --output-folder flag — use that flag instead if you need current-working-directory-relative resolution, as in the array-job example above.)

Solver advice for large meshes#

solver.petsc_options (and the matching --petsc-options flag) are merged over pulse’s problem defaults, so a large mesh (a fine realistic ventricular geometry, or a sweep run at production resolution) that needs a different linear solver can set it without editing Python:

[solver]
petsc_options = { ksp_type = "cg", pc_type = "hypre", pc_hypre_type = "boomeramg" }

Algebraic multigrid scales much better across MPI ranks and mesh sizes than the direct (MUMPS) factorization pulse otherwise defaults to for the small-to-moderate meshes in the templates. Fine-tune further with --petsc-options/solver.petsc_options if the default iterative tolerances need adjusting for a particular mesh.