Hybrid job in wholenode queue
A hybrid program combines both MPI and shared-memory to take advantage of compute clusters with multi-core compute nodes. Libraries for OpenMPI, Intel MPI (IMPI), and MVAPICH2 and compilers which include OpenMP for C, C++, and Fortran are available.
This example shows how to submit a hybrid program, this job asked for 4 MPI tasks (with 2 MPI tasks per node), each with 64 OpenMP threads for a total of 256 CPU-cores:
#!/bin/bash
# FILENAME: myjobsubmissionfile
#SBATCH -A myallocation # Allocation name
#SBATCH --nodes=2 # Total # of nodes
#SBATCH --ntasks-per-node=2 # Total # of MPI tasks per node
#SBATCH --cpus-per-task=64 # cpu-cores per task (default value is 1, >1 for multi-threaded tasks)
#SBATCH --time=1:30:00 # Total run time limit (hh:mm:ss)
#SBATCH -J myjobname # Job name
#SBATCH -o myjob.o%j # Name of stdout output file
#SBATCH -e myjob.e%j # Name of stderr error file
#SBATCH -p wholenode # Queue (partition) name
#SBATCH --mail-user=useremailaddress
#SBATCH --mail-type=all # Send email at begin and end of job
# Manage processing environment, load compilers and applications.
module purge
module load compilername
module load mpilibrary
module load applicationname
module list
# Set thread count (default value is 1).
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# Launch MPI code
mpirun -np $SLURM_NTASKS ./myexecutablefiles
The ntasks x cpus-per-task should equal to or less than the total number of CPU cores on a node.