Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,24 @@ method, non-gravitational parameters, parallel workers, …):
layup orbitfit --help
```

### Control how many CPUs layup uses

`--num-workers` (CLI) and `num_workers=` (API) default to `-1`, meaning decide
automatically: `$LAYUP_NUM_WORKERS` if set, otherwise 1 when layup is already
running inside another worker process, otherwise the CPUs available to this
process.

Set `LAYUP_NUM_WORKERS` when layup does not own the whole machine — running it
from your own process pool, or as one of several jobs on a shared node:

```
export LAYUP_NUM_WORKERS=4
```

Otherwise each copy would size its pool to the whole machine and oversubscribe
it. This is separate from `OMP_NUM_THREADS` and friends, which control threads
within a worker rather than the number of workers.

### Use the Python API

The same load → fit → convert → predict workflow is available directly from
Expand Down
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,6 @@ Notes:
:hidden:

Home page <self>
Controlling parallelism <parallelism>
API Reference <autoapi/index>
Notebooks <notebooks>
24 changes: 24 additions & 0 deletions docs/parallelism.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
Controlling parallelism
========================================================================================

``--num-workers`` (command line) and ``num_workers=`` (Python API) default to ``-1``,
meaning decide automatically:

1. ``$LAYUP_NUM_WORKERS``, if set.
2. Otherwise ``1``, when layup is already running inside another worker process.
3. Otherwise the CPUs available to this process (the affinity mask on Linux, so
``taskset`` and cgroup limits are respected).

Set ``LAYUP_NUM_WORKERS`` when layup does not own the whole machine — for example when
running it from your own process pool, or as one of several jobs on a shared node:

.. code-block:: console

>> export LAYUP_NUM_WORKERS=4

Without it, each copy sizes its pool to the whole machine and oversubscribes it.

.. note::

This is separate from ``OMP_NUM_THREADS`` and the other threadpool variables, which
control the number of threads *within* a worker rather than the number of workers.
4 changes: 3 additions & 1 deletion src/layup_cmdline/comet.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,9 @@ def main():
optional.add_argument(
"-n",
"--num-workers",
help="Number of CPU workers to use for parallel processing each chunk. -1 uses all available CPUs.",
help="Number of CPU workers for parallel processing. -1 (default) decides "
"automatically: $LAYUP_NUM_WORKERS if set, else 1 when layup is already "
"running inside another worker process, else the CPUs available to this process.",
dest="n",
type=int,
default=-1,
Expand Down
4 changes: 3 additions & 1 deletion src/layup_cmdline/convert.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ def main():
optional.add_argument(
"-n",
"--num-workers",
help="Number of CPU workers to use for parallel processing each chunk. -1 uses all available CPUs.",
help="Number of CPU workers for parallel processing. -1 (default) decides "
"automatically: $LAYUP_NUM_WORKERS if set, else 1 when layup is already "
"running inside another worker process, else the CPUs available to this process.",
dest="n",
type=int,
default=-1,
Expand Down
4 changes: 3 additions & 1 deletion src/layup_cmdline/orbitfit.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,9 @@ def main():
optional.add_argument(
"-n",
"--num-workers",
help="Number of CPU workers to use for parallel processing each chunk. -1 uses all available CPUs.",
help="Number of CPU workers for parallel processing. -1 (default) decides "
"automatically: $LAYUP_NUM_WORKERS if set, else 1 when layup is already "
"running inside another worker process, else the CPUs available to this process.",
dest="n",
type=int,
default=-1,
Expand Down
4 changes: 3 additions & 1 deletion src/layup_cmdline/predict.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,9 @@ def main():
optional.add_argument(
"-n",
"--num-workers",
help="Number of CPU workers to use for parallel processing each chunk. -1 uses all available CPUs.",
help="Number of CPU workers for parallel processing. -1 (default) decides "
"automatically: $LAYUP_NUM_WORKERS if set, else 1 when layup is already "
"running inside another worker process, else the CPUs available to this process.",
dest="n",
type=int,
default=-1,
Expand Down
Loading