GNU Parallel, a shell tool for executing jobs in parallel using one or more computers, has been installed on the cluster. A typical workload might be handling the pre-
or post-processing of a large set of data files.
Parallel handles the splitting of the workload across the resources available
to it, and also includes logging/resume capability so you can restart partially
completed processing.
To start using it, load the most recent module:
module load parallel
A README and some example code is available in the gnu-parallel subdirectory of PIK IT's HPC cluster-examples GitLab repository:
https://gitlab.pik-potsdam.de/its/hpc/cluster-examples/tree/master/gnu-parallel
Further reading:
1) https://www.gnu.org/software/parallel/
2) man parallel (after module load)
3) man parallel_tutorial (after module load)
