aitch (pronounced like the letter H) implements a light-weight cluster-style
scheduler for use on a single workstation. Unlike parallel, xargs, and
other similar utilities, which presume that all jobs use the same amount
of resources, aitch permits one to request a different amount from an
arbitrary number of unique kinds of resources. In effect it implements a
counting semaphore for each resource and provides a means to adjust their
values by non-negative integers.
Install rust as described here. Build with
cargo build.
Say for example your machine has six cores and you want to batch a heterogeneous collection of jobs which require different numbers of cores:
hstart 6
hsubmit 2 my-two-slot-job
hsubmit 4 my-four-slot-job [and-its-args...]
hsubmit 1 my-single-slot-job
In this case, the third job is queued and remains pending until one of the first two finish.
Other kinds of resources you might want to meter out include accelerator cards, memory, and file or network I/O. So for example, if in addition your machine also has two GPUs and 32 GB of RAM, the scheduler could be configured as follows:
hstart 6,2,32
hsubmit 1,0,32 my-high-memory-app
hsubmit 2,1,8 my-visualization-app
hsubmit 6,0,1 my-compute-intensive-app
hsubmit 0,0,0 'touch helloworld'
Should your jobs need to know which specific slot of a resource they consume, an environment variable prefixed with "QUEUE" is defined:
echo "export CUDA_VISIBLE_DEVICES=$QUEUE1; \
python -m 'import tensorflow...'" > my-deeplearning-app
hsubmit 2,1,4 my-deeplearning-app
In this case, QUEUE1 would be set to either 0 or 1, assuming the hstart
from above was still in effect. QUEUE0 would similarly be two integers
between 0 and 5 separated by a comma.
One can also specify job dependencies:
dep1=`hsubmit 6,0,12 do-me-first`
hsubmit 1,0,1 --dep $dep1 wait-for-do-me-first-to-finish
and pass in environment variables:
hsubmit 1,0,0 --env FOO=bar winnie-the-pooh
and redirect the standard streams:
hsubmit 1,0,3 --out stdout.txt -err stderr.txt log-the-results
Besides the hstart and hsubmit commands, there are also hjobs, hkill,
hnslots, hstatus, and hstop. Usage information for each is displayed
with the --help flag.
Run the tests with cargo test.
Build and test on a local disk with the --target-dir <DIR> flag. Useful if
the source is on a networked fileshare which doesn't support locking.
Update rust and cargo with rustup update.
The conda-forge feedstock is at https://github.com/conda-forge/aitch-feedstock.
Check the formatting of the feedstock recipe with conda smithy recipe-lint.
Generate a new key for the source code archive with openssl sha256 <github-tar-gz-file>.