Performance tuning#

A solprop run spends its time in a handful of operations repeated at every stage of every step: fast Fourier transforms between time and frequency, dense matrix products between modal amplitudes and the transverse field (for multimode and general modal guides), the pointwise nonlinear response (Kerr, plasma, Raman) at every quadrature node, and the Raman convolution. Each of these runs on a thread pool, and the pools belong to different libraries: the backend’s own kernels, FFTW, and a BLAS. This page explains which pools exist, how each is sized by default, which environment variables change them, and how to choose settings for one large run, for a scan of many runs, and on a GPU.

Two rules apply throughout:

Set threads before Python starts. Every pool below is sized when its library is loaded or when the backend is built. Some libraries allow a smaller active thread count later, but this is not a portable way to reconfigure all pools. Put the variables on the command line or in the job script, not in os.environ halfway through a notebook.
Measure your own problem. The defaults come from measurements on representative problems, but the best settings depend on the grid, the number of modes or nodes, the responses switched on and the machine. Time about ten steps of your real problem (a short fibre, n_save=2) at two or three settings before a long campaign.

None of the settings on this page changes the physics. Thread counts can change the order of floating-point sums inside the transforms, so results from different thread counts can differ by rounding (a few units in the last place); a run repeated with the same settings on the same build is reproducible.

Which backend#

The backend decides which pools exist at all (see Choosing a backend):

backend	parallel pools	notes
`numpy_ref`	NumPy’s BLAS only	the reference; single-threaded apart from matrix products
`numba_cpu`	Numba kernels, FFTW, BLAS (and FFTW for the transverse transforms of free-space runs)	sizes each pool from the problem
native CPU (`native`)	OpenMP kernels, FFTW (modal and spatial transforms), the Raman convolution transforms (Kokkos-FFT over FFTW, with selected macOS shapes on Accelerate’s vDSP), a vendor BLAS	uses every CPU in the process’s affinity mask unless told otherwise
native GPU (`native`, CUDA or HIP build)	the device	CPU pools are irrelevant; one device per process

backend="auto" chooses the native backend when solprop-native is installed and numba_cpu otherwise. The largest single gain is usually installing a locally built native backend (The native backend), which links a vendor BLAS and compiles for the machine’s own instruction set; which backend to use compares the backends by problem.

Which stepper#

The default integrator, Multistep(), evaluates the nonlinear response once per step; RungeKutta() usually six times, in fewer, longer steps. Keep the default wherever the response dominates a step, and compare the two on a short stretch of your own problem when it matters: the call counts and the time per call are in the run’s step history and log. Choosing a stepper sets out when each is faster, with the measurements.

Many save planes. Landing on every save plane caps the step at the plane spacing. With hundreds of planes, interpolate_planes=True steps across them and interpolates each inside its step instead, and step_planes=k saves the march’s own steps at no extra cost (the solver, “Choosing the saved planes”).

The `numba_cpu` pools#

numba_cpu sizes three pools from the problem when the backend is built, and records what it chose:

Kernel threads (the Numba parallel pool). The response pass over quadrature nodes runs in parallel only for large problems: above \(10^6\) elements (\(n_{to} \times n_\text{modes} \times n_\text{pol} \times n_\text{nodes}\), with \(n_{to}\) the number of samples on the oversampled time grid), or above 262 144 for a guided run whose response is evaluated at more than one transverse node. Below that threshold the kernels run serially (the small regime), because waking a pool costs more than the work saves. Above it (the large regime) the pool takes every CPU available to the process.

FFTW threads. A batched transform of at least 65 536 elements (\(n_\text{columns} \times n_{to}\)) gets up to four threads; smaller transforms get one. More than four threads rarely repays the synchronisation on the problem sizes solprop runs; transforms well above half a million elements can gain from eight. The Raman convolution transforms and, for free-space runs, the transverse transforms are sized by the same rule.

BLAS threads. The modal synthesis and projection products get two BLAS threads at or above 262 144 elements (\(n_\text{rows} \times n_{to}\)), one below. The BLAS pool is set through threadpoolctl; Apple’s Accelerate exposes no thread control at all, and is recorded as uncontrolled.

“Every CPU available” means the process’s thread budget, the smallest of:

its CPU affinity mask, the CPUs it may run on (narrowed by taskset, numactl or a batch scheduler’s allocation);
the physical cores that hold those CPUs, unless SOLPROP_THREADS is set: a core’s second hardware thread shares its arithmetic units, and on 36 cores of two Intel Xeon Gold 6240 one thread per core ran every benchmark case 6 to 31 % faster than two (read on Linux and Windows; Apple silicon has one thread per core);
the tightest Linux control-group CPU quota on its path (containers), rounded down: a quota limits CPU time, and a team of threads that exceeds it is throttled whole, so a quota of 1.5 CPUs runs one thread;
SOLPROP_THREADS, if set.

Both CPU backends size every pool within this budget, and clamp an explicit count (threads= or a variable below) to it rather than refusing it; the pools take turns, so each may use all of it. Set SOLPROP_THREADS to each process’s share when several processes share CPUs without being pinned apart. That includes several solprop processes in one CPU-limited container (a Kubernetes CPU limit, Docker --cpus): the quota is the container’s, and each process would otherwise take all of it. A cloud virtual machine or a Slurm job has no quota by default (cat /sys/fs/cgroup/cpu.max prints max); there the CPUs it may run on are the limit. SOLPROP_OVERSUBSCRIBE=1 lets an explicit count exceed the CPUs and the quota (never SOLPROP_THREADS), for a test that compares thread counts on a small machine; a run has no reason to set it. The budget and the limit that set it are recorded in the run’s metadata (thread_budget, thread_budget_source).

Each pool can be fixed from the environment:

variable	pool	default
`SOLPROP_THREADS`	the process’s thread budget, which every pool below (and the native backend’s) stays within; it may exceed the physical cores, up to the CPUs	the affinity mask, its physical cores and the quota
`SOLPROP_OVERSUBSCRIBE`	`1` lets an explicit count exceed the CPUs and the quota	`0`
`SOLPROP_NUMBA_THREADS`	Numba kernel threads	all available CPUs above the size threshold, else 1
`SOLPROP_NUMBA_REGIME`	`small` (serial node loop) or `large` (parallel node loop)	chosen from the size threshold
`SOLPROP_FFTW_THREADS`	FFTW threads for the temporal and Raman transforms	every available CPU from 1 048 576 elements, up to 4 from 65 536, else 1
`SOLPROP_TRANSVERSE_FFTW_THREADS`	FFTW threads for the transverse transforms of free-space runs	the same rule
`SOLPROP_BLAS_THREADS`	BLAS threads	2 above 262 144 elements, else 1
`SOLPROP_FFTW_PLAN`	FFTW planner effort: `estimate`, `measure` or `patient`	`measure`
`SOLPROP_FFTW_WISDOM`	a file of FFTW wisdom to import before planning and export after	none

Free-space node groups. A free-space, line or planar evaluation works through the plane’s columns in node chunks, each paying a fixed cost for its launches, its temporal transforms and its gather and scatter. Their default budget is therefore 1 GiB, against 64 MiB on modal routes; SimulationSettings.node_budget sets it explicitly. On a laptop short of memory a smaller budget saves memory at some speed; on a large device a larger one can still help.

Where a free-space step goes. SOLPROP_PASS_TIMERS=1, read when a numba_cpu backend is built, gives it per-pass wall-clock timers for the Cartesian, line and planar evaluations: the transverse and temporal transforms, the packs, copies, clears, the response and the crops of each node chunk, and the steppers’ traversals around them (NumbaCpuBackend.pass_timers, from solprop.backends.numba_cpu.pass_timers). It is a measurement aid for SEP 28 and its benchmarks; unset, nothing is timed and nothing is added to a pass. On the native backend the same stages are profiling regions (solprop_native.free.*) that any Kokkos Tools timer reports.

Every explicit count is clamped to the thread budget. NUMBA_NUM_THREADS, Numba’s own variable, sets the size of the pool Numba creates at import, and the kernel pool, chosen or explicit, is clamped to it as well; the FFTW and BLAS pools are not, since they are not Numba’s. NUMBA_THREADING_LAYER chooses Numba’s threading layer (omp, tbb or workqueue) and is normally best left alone.

FFTW’s idle threads on Linux. pyFFTW’s Linux wheels thread FFTW through their own copy of GNU OpenMP. If that runtime’s idle threads spin, they take the cores the Numba kernels need between transforms. On 8 cores of an x86-64 server that made a four-mode capillary step five times slower whenever FFTW had more than one thread. numba_cpu therefore imports pyFFTW with OMP_WAIT_POLICY=passive set for that runtime alone, and the environment is restored straight away (the native backend’s own choice, above, is the one that stays). This is skipped if you have set OMP_WAIT_POLICY or GOMP_SPINCOUNT, because then your choice applies, and active brings the slowdown back. It is also skipped when your script imported pyfftw before solprop. The run’s metadata records what applied as fftw_wait_policy.

Numba’s own threads on Linux. Numba’s OpenMP threading layer loads a second GNU OpenMP runtime when its threads first start, and with every core in the pool its idle workers spin on the cores FFTW’s threads need: on a 64-core AMD server the comb-to-continuum benchmark window took 17 s, against 5.4 s with the workers waiting passively. numba_cpu therefore starts Numba’s threads with OMP_WAIT_POLICY=passive for that runtime. When the runtime is already loaded under settings solprop cannot change (the native backend’s build may load the same library first), it chooses Numba’s workqueue layer instead, which uses no OpenMP and was as fast (5.5 s). Your own OMP_WAIT_POLICY, GOMP_SPINCOUNT or NUMBA_THREADING_LAYER is respected. The run’s metadata records what applied as numba_openmp.

OpenMP binding and numba_cpu. OMP_PROC_BIND (or OMP_PLACES, which binds under GNU OpenMP even alone) made the same window take 45-47 s, whatever Numba’s layer: the bound runtimes place their threads on the same few cores. On Linux numba_cpu therefore loads pyFFTW’s runtime and Numba’s with binding off, and your variables stay set for any other runtime, such as the native backend’s. When the native backend, imported first under binding, has bound the main thread to one core, numba_cpu gives the main thread the process’s CPUs back before it plans its transforms, which recovers modal runs; free-space runs in that combination still had FFTW transforms several times slower on Linux, so numba_cpu warns. If pyfftw was imported before solprop under binding, solprop cannot undo it and warns too. In short: do not set OMP_PROC_BIND or OMP_PLACES for numba_cpu runs. Binding brought the native backend no measurable gain on one socket either; on several sockets, place whole processes with numactl or taskset instead.

FFTW planning. measure times candidate algorithms once per transform shape and keeps the fastest; that pays off on long runs, but the timing is only meaningful on an otherwise idle machine, and planning a large multimode problem can take seconds. Use SOLPROP_FFTW_PLAN=estimate for short runs, tests and busy shared machines, and SOLPROP_FFTW_WISDOM=~/.cache/solprop/wisdom to plan once and reuse the plans across runs of the same shapes (the wisdom’s digest is recorded in each run, since plans chosen from different wisdom are different executions).

The native backend’s pools#

The native backend reads its thread settings once, when solprop_native is first imported:

variable	pool	default
`SOLPROP_NATIVE_THREADS`	the size of Kokkos’ OpenMP pool, an explicit count within the thread budget (above the CPUs only with `SOLPROP_OVERSUBSCRIBE=1`, above `SOLPROP_THREADS` never)	the thread budget
`SOLPROP_NATIVE_KERNEL_THREADS`	response, stage and reduction kernels	sized per operation, up to the cap
`SOLPROP_NATIVE_FFT_THREADS`	FFTW threads for the modal and spatial transforms	sized per plan, up to the cap
`SOLPROP_NATIVE_RAMAN_FFT_THREADS`	threads for the Raman convolution transforms of complete node chunks	sized per plan, up to the cap
`SOLPROP_NATIVE_RAMAN_TAIL_FFT_THREADS`	threads for the Raman transforms of the last, partial node chunk	sized from active rows, up to the cap
`SOLPROP_NATIVE_THREADED_FFT_MIN_SAMPLES`	the real-side samples per execution from which a modal or spatial FFT plan is threaded	16,384 (32,768 on macOS)
`SOLPROP_NATIVE_BAND_TILES`	the field plasma column’s band form: unset for the build’s own (tiled on a GPU, a team per node on a CPU), `0` for the team column, a positive `N` for the tiled column with host teams of `N` threads	unset

The four pool variables are optional positive integers, clamped to the cap; an invalid value raises ValueError at import. An explicit pool setting forces its clamped width and bypasses automatic sizing, including for short FFTs. Setting only the global cap retains automatic sizing. GPU builds ignore these CPU pool settings.

On OpenMP CPU builds, operation dimensions select serial, intermediate or full-width execution. Stage and reduction work, packing and temporal copies, nonlinear responses, native dense transforms, coupled matrix functions and FFT plans have separate size rules. Response sizing accounts for both the total work, active columns and fine samples per column. The choices and plans are cached when the backend is constructed; no runtime tuning occurs during propagation. Partial chunks and fused batch updates use their own work sizes. Vendor BLAS pools retain their process-start settings. Non-OpenMP host builds preserve their execution-space policy; build_info().cpu_size_policy_enabled reports whether operation widths are supported.

The thresholds below are shared across CPU hosts, except that macOS keeps higher stage and FFT starts (8,192 and 32,768), where a short serial loop is still the faster one; C is the clamped capacity of the relevant pool, and sizes count work in the operation itself, not bytes. Between the two thresholds an operation uses min(4, C) threads; equal thresholds give only serial or full-width execution.

operation	work used for sizing	threading starts	full pool starts
stage, update and reduction	elements in the field or fused batch	4,096 (8,192 on macOS)	as threading
packing and temporal copies	entries actually copied	65,536	65,536
nonlinear response	active columns × components × fine samples	32,768	196,608, or 65,536 fine samples per column, or 40 component columns
native/custom GEMM	output rows × inner dimension × samples	1,048,576	1,048,576
general modal synthesis/projection	active nodes × components × spectral samples × modes	65,536	65,536
coupled matrix functions	active spectral bins × modes³	32,768	32,768
modal/spatial FFT	real-side samples in the actual batch	16,384, 32,768 on macOS (`SOLPROP_NATIVE_THREADED_FFT_MIN_SAMPLES`)	131,072
Raman FFT	real-side samples in the actual batch	65,536	262,144

A response column is one active node and one component; its column-length and column-count rules apply only past its threading start. General modal map updates use the two maps’ combined element count times the polynomial degree. Native GEMM sizing never changes a vendor BLAS’s width or selects a different provider.

The CPU threading measurements explain the thresholds. For example, an 8,192-sample copy remains serial even when a similarly sized arithmetic stage benefits from threading. Short response columns can use four threads while long columns or wider batches use the full pool. The backend applies these choices automatically; start by setting only the process cap when sharing a machine.

python -m solprop_native records the policy constants and explicit override mask. Its existing thread counts remain process capacities. The backend’s metadata records execution_sizing, with the chosen threads and sizing elements for each operation; batch_execution_sizing covers fused batch work. Response entries also report fine_samples and columns, because equal total work can have different column lengths. build_info() reports the resolved process values as requested_threads, runtime_threads, thread_count (kernels), fftw_threads, raman_fft_threads, raman_tail_fft_threads, threaded_fft_min_samples, blas_threads and blas_threading; cpu_size_policy_enabled and explicit_thread_pools (a bit mask: kernel 1, FFT 2, Raman 4, Raman tail 8) distinguish automatic sizing from explicit requests.

Free-space transforms are planned at SOLPROP_FFTW_PLAN’s effort (measure by default, as on numba_cpu); estimate skips the measured planning for short runs and tests. Answers at different thread counts may differ by rounding; solprop holds them to its rounding bounds, not to identical bits (SEP 25). Contiguous modal and Raman FFTW plans default to measured planning at one thread and estimate planning at wider widths; the explicit effort overrides both, including long Raman rows and partial chunks. Measured planning can add substantial construction time even when the warmed propagation is faster. Use SOLPROP_FFTW_PLAN=estimate when construction dominates a short run. On macOS with NEON FFTW, serial and long Raman rows use Kokkos-FFT, while parallel power-of-two rows of at most 16,384 samples retain vDSP. The build guide describes the coherent dependency recipe and the fallback for a library without NEON codelets.

OMP_NUM_THREADS and KOKKOS_NUM_THREADS do not size the Kokkos pool — the backend initialises Kokkos with an explicit thread count, which takes precedence; set SOLPROP_NATIVE_THREADS instead. (Kokkos prints a warning at start-up if OMP_NUM_THREADS asks for fewer threads than it was given.)

The default cap is the thread budget (above): the CPUs in the process’s affinity mask, a container’s quota and SOLPROP_THREADS. Automatic sizing avoids assigning all of those threads to small operations. On a shared machine or when several solprop processes run side by side, set SOLPROP_THREADS (which holds numba_cpu too) or SOLPROP_NATIVE_THREADS to each process’s CPU budget. The operation policy stays within that budget.

OpenMP placement and waiting#

The native CPU backend runs on OpenMP, whose standard variables apply:

OMP_PROC_BIND and OMP_PLACES pin threads to cores, which avoids threads migrating between cores and sockets: OMP_PROC_BIND=spread OMP_PLACES=cores is a good default for one run on a multi-socket machine. Kokkos prints a notice when OMP_PROC_BIND is unset; OMP_PROC_BIND=false silences it without pinning. With a build whose FFTW uses pthreads rather than OpenMP, prefer OMP_PROC_BIND=false.
OMP_WAIT_POLICY decides what idle threads do between parallel regions: active keeps them spinning, passive lets them sleep at once. The native backend chooses for you when none of OMP_WAIT_POLICY, GOMP_SPINCOUNT and KMP_BLOCKTIME is set. It runs several thread teams in turn (Kokkos’, the threaded FFTs’, the BLAS’s), and when its threads fill the CPUs one team’s spinning threads take the cores the next needs: on 16 cores of an AMD EPYC 7763 the runtime’s own waiting made the four-mode capillary 25 times slower (334 s against 13.5 s). So the import sets OMP_WAIT_POLICY=passive. The setting stays in the environment, so worker processes wait alike, and the run’s metadata records what applied as openmp_wait_policy. Where hardware threads are spare and nothing else will use them, spinning is faster: on 36 cores of two Intel Xeon Gold 6240 (72 hardware threads), with the whole machine to itself, OMP_WAIT_POLICY=active ran the supercontinuum in 49 s against 94 s and the capillary in 9.7 s against 14.7 s. It is not the default because the runtime reads the policy once, at start-up, and work that arrives on the machine later takes those spare threads; set it for a run on a machine it has to itself. LLVM’s OpenMP runtime (used on macOS) also honours KMP_BLOCKTIME, the time a thread spins before sleeping; do not set it for GNU OpenMP.
On a machine your run has to itself, also pinning the threads helps: with idle threads sleeping, OMP_PROC_BIND=close OMP_PLACES=cores made the capillary and filament cases 6 to 30 % faster on the EPYC allocation above. Leave it unset where several processes share a machine, since each would pin its threads to the same first cores.

BLAS and vendor libraries#

The BLAS matters for the dense modal transforms of multimode capillaries, coherent walls and the general modal route. A BLAS with its own thread pool beside the native backend’s oversubscribes the cores: with a pthreads OpenBLAS at the full core count, the modal and wall cases ran 10 to 75 times slower than necessary. The Linux distribution build of the native backend therefore carries an OpenBLAS built with OpenMP, which shares the backend’s runtime. At start-up the backend fits a host OpenBLAS to its own pool unless OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS or OMP_NUM_THREADS is set: a pthreads build is held to one thread, an OpenMP build capped at the Kokkos thread count. numba_cpu sizes its BLAS pool through threadpoolctl. Otherwise a BLAS’s own variables apply:

library	thread variable	notes
OpenBLAS (pthreads build)	`OPENBLAS_NUM_THREADS`	the usual Linux default
OpenBLAS (OpenMP build)	`OMP_NUM_THREADS`	ignores `OPENBLAS_NUM_THREADS`; the best choice beside the native backend, which shares its runtime
Intel MKL	`MKL_NUM_THREADS`, or per domain `MKL_DOMAIN_NUM_THREADS`	e.g. `MKL_DOMAIN_NUM_THREADS="MKL_DOMAIN_BLAS=1,MKL_DOMAIN_FFT=8"`
Apple Accelerate	`VECLIB_MAXIMUM_THREADS`	threads through Grand Central Dispatch
AMD AOCL-BLIS	`BLIS_NUM_THREADS`

NumPy’s own BLAS (used by numpy_ref and by solprop’s translation layer) answers to the same variables. A locally built native backend links a vendor BLAS (Accelerate on macOS, MKL when oneAPI is available on Linux, otherwise OpenBLAS); its build record says which (python -m solprop_native).

One OpenMP runtime per process. The native backend, FFTW and some BLAS builds each use OpenMP, and two different OpenMP runtimes loaded into one process can crash or oversubscribe the machine. On macOS, solprop_native must be imported before anything imports pyfftw, and raises an error otherwise. solprop imports pyfftw only when a numba_cpu backend is built, so this matters only for a script that imports pyfftw itself or runs numba_cpu before the native backend in the same process: put import solprop_native at the top of such a script. Building the native backend explains how to build pyFFTW against the same FFTW when a binary wheel brings its own runtime.

Recipes#

One large run on a workstation. Give the run every performance core, pin it, and let idle threads sleep if you are using the machine for anything else:

SOLPROP_NATIVE_THREADS=8 OMP_PROC_BIND=spread OMP_PLACES=cores OMP_WAIT_POLICY=passive \
    python my_run.py

Without the native backend, numba_cpu sizes itself; to force a larger problem onto more threads, for example SOLPROP_NUMBA_THREADS=8 SOLPROP_FFTW_THREADS=4 python my_run.py.

How far a single run scales. A single-mode fibre on a few thousand time samples gains little beyond two to four threads: its transforms are short and each step is mostly synchronisation. Multimode capillaries and the general modal route, whose products and node loops are large, keep gaining to eight to eighteen threads. Free-space runs with large transverse grids scale furthest.

A scan of many runs. For a scan over pressure, energy or noise seeds, many single-threaded workers almost always beat one many-threaded run: each worker keeps its own caches and nothing waits for synchronisation. solprop’s Pool executor does this for you: every worker is a newly spawned Python process started with all of NUMBA_NUM_THREADS, SOLPROP_THREADS, OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS and VECLIB_MAXIMUM_THREADS set to its threads (one by default) and OMP_WAIT_POLICY=passive, so every pool of either backend is sized within that budget, and on Linux each worker is pinned to its own CPUs. The number of workers is the number of CPUs available divided by threads, reduced if a memory probe of the first point shows that fewer fit (four fifths of the memory, less the controller’s own, divided by one worker’s footprint). run_scan runs its points one after another in the calling process unless it is given an executor, so ask for the pool, in the script or on the command line:

python my_scan.py                            # the script calls run_scan(..., executor=Pool())
python -m solprop.scan my_scan.py --workers 8    # or let the launcher choose Pool

A problem too large to fit many copies in memory can use fewer, wider workers, Pool(threads=4) (or --threads 4). To run two scans side by side on one machine, give each its own CPUs with SOLPROP_SCAN_CPU_OFFSET and SOLPROP_SCAN_CPU_COUNT. See scans and batches. At one thread per worker the choice of backend is less clear-cut than for a single large run: single-threaded numba_cpu can match or beat the single-threaded native backend on some problems and machines, so time one point of the scan on both (backend="numba_cpu" and backend="native") before a large campaign.

If you manage the parallelism yourself — a job array, GNU parallel, several scripts started by hand — each process needs every pool held to one thread explicitly, and the serial kernel regime selected, before Python starts:

NUMBA_NUM_THREADS=1 NUMBA_THREADING_LAYER=workqueue SOLPROP_NUMBA_REGIME=small \
SOLPROP_THREADS=1 OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
VECLIB_MAXIMUM_THREADS=1 OMP_WAIT_POLICY=passive OMP_PROC_BIND=false \
    python point.py

On a multi-socket server, keep a multi-threaded run’s threads within one socket (OMP_PLACES=cores with OMP_PROC_BIND=close, or taskset/numactl): crossing sockets usually costs more in memory traffic than the extra cores return for the problem sizes solprop runs, and two runs, one per socket, are usually faster than one run across both.

A cluster. Slurm writes an array job whose tasks each run a Pool inside their allocation; it respects SLURM_CPUS_PER_TASK, and its threads sets the same variables in every task.

A GPU. On a CUDA or HIP build of the native backend the CPU pools do not matter; the backend uses one device per process, chosen with CUDA_VISIBLE_DEVICES or HIP_VISIBLE_DEVICES. Small problems are limited by kernel-launch latency and often run faster on the CPU: pass backend="numba_cpu" for them. Run independent problems in separate processes to use several devices. See The native backend.

Memory. Large multimode and general modal problems can be limited by memory before threads; SOLPROP_MEMORY_BUDGET and the admission check are described in Execution resources.

Checking what was used#

Every backend logs the pools it chose at INFO level, and records them in the run’s backend record, which is saved with the output:

import solprop
from solprop.driver import prop_capillary

solprop.configure_logging(
    "INFO"
)  # prints "numba_cpu: small regime, 1 kernel threads, ..."
run = prop_capillary(
    125e-6,
    1e-3,
    "He",
    1.0,
    lambda0=800e-9,
    wavelength_limits=(200e-9, 2e-6),
    trange=100e-15,
    fwhm=10e-15,
    energy=1e-9,
    n_save=2,
    progress="none",
    backend="numba_cpu",
)
settings = run.metadata.backend.settings
print({key: settings[key] for key in ("regime", "numba_threads", "fftw_threads")})

For the native backend the same record carries runtime_threads, thread_count, fftw_threads, raman_fft_threads, blas_library and the build profile; python -m solprop_native prints the whole build record without running anything.

Profiling the native backend#

Kokkos profiling hooks are compiled into every native build, and every kernel, region and allocation carries a solprop_native.* label (for example solprop_native.transform.*, solprop_native.raman.*, solprop_native.plasma.*, solprop_native.statistics.*). Load any Kokkos Tools plugin at run time with KOKKOS_TOOLS_LIBS; nothing is linked into the package. For the simple kernel timer:

git clone --depth 1 --branch 5.2.0 https://github.com/kokkos/kokkos-tools.git
cmake -S kokkos-tools -B kokkos-tools/build -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$PWD/kokkos-tools/install"
cmake --build kokkos-tools/build --target install --parallel

KOKKOS_TOOLS_LIBS="$PWD/kokkos-tools/install/lib/libkp_kernel_timer.so" \
  python my_simulation.py

On macOS the plugin is libkp_kernel_timer.dylib. At shutdown the timer reports call counts and accumulated times for the solprop_native.* labels and for Kokkos-FFT’s KokkosFFT::exec_plan regions; the NVTX and ROCTX connectors pass the same labels to Nsight Systems and the ROCm profilers. The free-space source evaluation’s stages are regions of their own (solprop_native.free.core, .transverse_inverse, .pack, .to_time, .clear_projected, .response, .to_frequency, .transverse_forward); with no tool loaded a region costs one flag test. For the Runge–Kutta march, the backend’s metadata()["source_evaluations"] counts the native source evaluations since the initial state was set, rejected trials included.