solprop.driver.memory#

Admit a run against a declared memory budget before its backend allocates.

The estimate of solprop.backends.memory.estimate_memory() is compared with a budget before solprop.backends.registry.select() constructs the backend. Over budget, the run is refused by name – the estimate, the budget, the largest groups and what to change – or warned on, when the declaration asks for a warning. Nothing is approximated to fit: the remedy for a run that does not fit is more memory, never an approximated stored field.

The declaration. In order of precedence:

the memory_budget keyword of solprop.driver.simulation.Simulation.run(), for one run in a script or a notebook;
the environment variable MEMORY_BUDGET_VARIABLE (SOLPROP_MEMORY_BUDGET), which reaches every entry point, the keyword interfaces and the scan workers, without a change of code: the natural place for a job script’s allocation;
otherwise the machine’s memory with 20 % headroom: four fifths of the smallest of the physical memory, a cgroup limit and a Slurm allocation, the same limit and headroom the scan executors admit their workers against – warned on, not refused, so that a user who wants to try a run that may not fit can.

A declaration is a byte count, as an integer or a string with an optional unit ("16 GiB", "12GB", "8e9"), optionally prefixed "warn:" to warn instead of refusing; "none" declares no check. A declared budget refuses; the undeclared default only warns. The default checks against the machine because a persistent footprint beyond it pages or is killed after the run has started, possibly hours in; physical memory rather than free memory keeps the decision the same on a machine whatever else is running. The headroom leaves room for what the estimate does not count (the planning heap, per-stage transients) and for the rest of the process.

The output. A run kept in memory holds every plane it saves, so admit_memory() adds the store at its planned capacity – the field, the noise phase and the statistics of the entrance and every save plane – as the group output, when it is given the run’s handler (solprop.output.schema.PlaneStore.held_bytes()). A run that keeps its field on some planes only (SEP 22) is charged the statistics of every plane and the field of those alone, so a statistics-only run is charged its statistics. A file output writes each plane out and holds none of them, so it adds nothing. The step history and the between-planes rows grow with the steps taken and are not counted.

Module Attributes

`MEMORY_BUDGET_VARIABLE`	The environment variable that declares a memory budget for every run.
`DEFAULT_HEADROOM`	The fraction of the machine's memory the default budget leaves unclaimed.

Functions

`admit_guide_memory`(guide, grid, *, z_max, ...)	Admit a general modal guide's route against its budget before it is built.
`admit_memory`(problem, backend[, ...])	Estimate a run's persistent memory and admit it against its budget.
`field_bytes`(problem)	Return the bytes of one plane's field, and its noise phase, in a memory store.
`format_bytes`(size)	Return a byte count in binary units, to three significant figures.
`parse_memory_budget`(value, *[, source])	Read a memory budget declaration.
`plane_bytes`(problem)	Return the bytes one saved plane occupies in an in-memory store.
`resolve_memory_budget`([memory_budget])	Return the budget a run is admitted against.

Classes

`MemoryAdmission`(estimate, budget)	A run's estimate and the budget it was admitted against.
`MemoryLimit`(limit[, action, source])	A declared memory budget and what exceeding it does.

solprop.driver.memory.DEFAULT_HEADROOM = 0.2#: The fraction of the machine’s memory the default budget leaves unclaimed.

solprop.driver.memory.MEMORY_BUDGET_VARIABLE = 'SOLPROP_MEMORY_BUDGET'#: The environment variable that declares a memory budget for every run.

class solprop.driver.memory.MemoryAdmission(estimate, budget)[source]#

Bases: object

A run’s estimate and the budget it was admitted against.

Attributes:

estimatesolprop.backends.memory.MemoryEstimate: The persistent bytes the backend will allocate, by group.
budgetMemoryLimit: The budget the estimate was checked against.

budget: MemoryLimit#

estimate: MemoryEstimate#

record()[source]#

Return the admission as the run’s backend record carries it.

Returns:

dict: estimate (bytes), the non-empty groups (bytes), budget (bytes, or None) and budget_source.

class solprop.driver.memory.MemoryLimit(limit, action='refuse', source='keyword')[source]#

Bases: object

A declared memory budget and what exceeding it does.

Attributes:

limitint or None: The budget, bytes, positive; None declares no check.
action{‘refuse’, ‘warn’}: Whether a run over the budget is refused or warned on.
sourcestr: Where the declaration came from: "keyword", MEMORY_BUDGET_VARIABLE or "machine".

Examples

>>> from solprop.driver.memory import MemoryLimit
>>> MemoryLimit(2**30, "warn", "keyword").limit
1073741824

action: Literal['refuse', 'warn'] = 'refuse'#

limit: int | None#

source: str = 'keyword'#

solprop.driver.memory.admit_guide_memory(guide, grid, *, z_max, backend, memory_budget=None)[source]#

Admit a general modal guide’s route against its budget before it is built.

The route’s dense E/H reference and its maps are allocated when a general modal simulation is constructed (solprop.driver.simulation.Simulation.from_waveguide()), before admit_memory() can see a translated problem, and on a large guide they are nearly all of the run’s memory. This admits the three groups solprop.backends.memory.estimate_guide_memory() takes from the guide’s declared shape, under the same budget and the same refuse or warn policy, before any of it exists. The whole run is admitted again by admit_memory() once it is translated.

Parameters:

guidesolprop.modes.guide.ModalGuide: A guide built by solprop.modes.guide.ModalGuide.from_provider().
gridsolprop.grid.TimeGrid: The run’s field grid.
z_maxfloat: Where the run ends, m, positive.
backendstr: The registry name the run will select, or "auto".
memory_budgetint or str or None, optional: The declaration (resolve_memory_budget()).

Returns:

MemoryAdmission: The guide’s estimate and the budget.

Raises:

MemoryBudgetError: If the estimate exceeds a budget declared to refuse; nothing of the route has been built.

Warns:

MemoryBudgetWarning: If the estimate exceeds a budget declared warn:, or the machine’s default.

Examples

>>> from solprop.driver.memory import admit_guide_memory
>>> from solprop.grid import FieldGrid
>>> from solprop.processing._examples import example_general_run
>>> guide = example_general_run()[2]
>>> grid = FieldGrid.from_limits(1e-5, 1.03e-6, (1.0e-6, 1.06e-6), 100e-15)
>>> admit_guide_memory(
...     guide, grid, z_max=1e-5, backend="numpy_ref", memory_budget="1 kB"
... )
Traceback (most recent call last):
...
solprop.exceptions.MemoryBudgetError: the general modal guide's reference...

solprop.driver.memory.admit_memory(problem, backend, memory_budget=None, *, output=None, retention=None)[source]#

Estimate a run’s persistent memory and admit it against its budget.

Called before the backend is constructed, so nothing the backend would allocate exists yet. The general route’s reference is already resident, in the translated problem, and is counted with the rest. Given the run’s output, what the output keeps in memory is added as the group output: an in-memory store at its planned capacity, nothing for a file.

Parameters:

problemsolprop.translate.problem.KernelProblem: The translated problem.
backendstr: The registry name the run will select, or "auto".
memory_budgetint or str or None, optional: This run’s declaration (resolve_memory_budget()).
outputsolprop.output.schema.Output or None, optional: The run’s handler. A PlaneStore is charged what held_bytes() says it keeps for the entrance and every save plane; None, or another handler, is charged nothing.
retentionsolprop.output.schema.FieldRetention or None, optional: Which planes keep their field, resolved against the run’s planes; None, every plane. The output is charged the field on the kept planes only.

Returns:

MemoryAdmission: The estimate and the budget, for the run’s record.

Raises:

MemoryBudgetError: If the estimate exceeds a budget declared to refuse; the message names the estimate, the budget, the largest groups and what to change. Also, whatever the declaration but "none" or "warn:", if the Adams method’s history is what does not fit – the run without it would – in the machine’s memory or, on an accelerator, in four fifths of its free memory; the message names RungeKutta(), which keeps no history.

Warns:

MemoryBudgetWarning: If the estimate exceeds a budget declared warn:, or the undeclared machine budget for a reason other than the Adams history.

Examples

>>> from solprop.driver import Simulation
>>> from solprop.driver.memory import admit_memory
>>> simulation = Simulation.from_gnlse(
...     0.11, 0.01, [0.0, 0.0, -1.183e-26], lambda0=835e-9,
...     wavelength_limits=(500e-9, 2e-6), trange=1e-12,
...     width=100e-15, peak_power=9.8, pulseshape="sech",
... )
>>> admission = admit_memory(simulation.translate(), "numpy_ref", "1 GiB")
>>> admission.estimate.total < admission.budget.limit
True
>>> admit_memory(simulation.translate(), "numpy_ref", "1 kB")
Traceback (most recent call last):
...
solprop.exceptions.MemoryBudgetError: ...

solprop.driver.memory.field_bytes(problem)[source]#

Return the bytes of one plane’s field, and its noise phase, in a memory store.

The part of plane_bytes() a plane that keeps no field does not hold (SEP 22).

Parameters:

problemsolprop.translate.problem.KernelProblem: The translated problem.

Returns:

int: Sixteen bytes per state entry, and eight more with a noise model.

Examples

>>> from solprop.driver import Simulation
>>> from solprop.driver.memory import field_bytes
>>> simulation = Simulation.from_gnlse(
...     0.11, 0.01, [0.0, 0.0, -1.183e-26], lambda0=835e-9,
...     wavelength_limits=(500e-9, 2e-6), trange=1e-12,
...     width=100e-15, peak_power=9.8, pulseshape="sech", noise=None,
... )
>>> field_bytes(simulation.translate()) == 16 * simulation.translate().shape[-1]
True

solprop.driver.memory.format_bytes(size)[source]#

Return a byte count in binary units, to three significant figures.

Parameters:

sizeint: Bytes, non-negative.

Returns:

str: The count in B, KiB, MiB, GiB or TiB.

Examples

>>> from solprop.driver.memory import format_bytes
>>> format_bytes(3 * 2**30 // 2), format_bytes(512)
('1.50 GiB', '512 B')

solprop.driver.memory.parse_memory_budget(value, *, source='keyword')[source]#

Read a memory budget declaration.

Parameters:

valueint or str: A positive byte count; or a string: a number with an optional unit (B, kB, MB, GB, TB in powers of 1000, KiB, MiB, GiB, TiB in powers of 1024; case-insensitive), optionally prefixed warn:; or none, which declares no check.
sourcestr, optional: Where the declaration came from, for the record and the messages.

Returns:

MemoryLimit: The declared budget.

Raises:

DriverError: If the declaration cannot be read or is not positive.

Examples

>>> from solprop.driver.memory import parse_memory_budget
>>> parse_memory_budget("warn: 16 GiB")
MemoryLimit(limit=17179869184, action='warn', source='keyword')
>>> parse_memory_budget("none").limit is None
True

solprop.driver.memory.plane_bytes(problem)[source]#

Return the bytes one saved plane occupies in an in-memory store.

The field (complex128), the noise phase (float64) when the run carries a noise model, every statistic at its catalogued shape (float64; a statistic catalogued with a smaller shape for spatial beams is counted at the modal one, an upper bound of a few entries), and the position and the two per-plane counts.

Parameters:

problemsolprop.translate.problem.KernelProblem: The translated problem.

Returns:

int: Bytes per plane.

Examples

>>> from solprop.driver import Simulation
>>> from solprop.driver.memory import plane_bytes
>>> simulation = Simulation.from_gnlse(
...     0.11, 0.01, [0.0, 0.0, -1.183e-26], lambda0=835e-9,
...     wavelength_limits=(500e-9, 2e-6), trange=1e-12,
...     width=100e-15, peak_power=9.8, pulseshape="sech",
... )
>>> problem = simulation.translate()
>>> plane_bytes(problem) >= 16 * problem.shape[-1]
True

solprop.driver.memory.resolve_memory_budget(memory_budget=None)[source]#

Return the budget a run is admitted against.

Parameters:

memory_budgetint or str or None, optional: A declaration for this run (parse_memory_budget()); None takes the environment variable MEMORY_BUDGET_VARIABLE, and without it the machine’s memory less DEFAULT_HEADROOM.

Returns:

MemoryLimit: The budget and where it came from.

Examples

>>> from solprop.driver.memory import resolve_memory_budget
>>> resolve_memory_budget(2**30)
MemoryLimit(limit=1073741824, action='refuse', source='keyword')