OpenMP job in wholenode queue
A shared-memory job is a single process that takes advantage of a multi-core processor and its shared memory to achieve parallelization.
When running OpenMP programs, all threads must be on the same compute node to take advantage of shared memory. The threads cannot communicate between nodes.
To run an OpenMP program, set the environment variable OMP_NUM_THREADS to the desired number of threads. This should almost always be equal to the number of cores on a compute node. You may want to set to another appropriate value if you are running several processes in parallel in a single job or node.
This example shows how to submit an OpenMP program, this job asked for 2 MPI tasks, each with 64 OpenMP threads for a total of 128 CPU-cores:
#!/bin/bash
# FILENAME: myjobsubmissionfile
#SBATCH -A myallocation # Allocation name
#SBATCH --nodes=1 # Total # of nodes (must be 1 for OpenMP job)
#SBATCH --ntasks-per-node=2 # Total # of MPI tasks per node
#SBATCH --cpus-per-task=64 # cpu-cores per task (default value is 1, >1 for multi-threaded tasks)
#SBATCH --time=1:30:00 # Total run time limit (hh:mm:ss)
#SBATCH -J myjobname # Job name
#SBATCH -o myjob.o%j # Name of stdout output file
#SBATCH -e myjob.e%j # Name of stderr error file
#SBATCH -p wholenode # Queue (partition) name
#SBATCH --mail-user=useremailaddress
#SBATCH --mail-type=all # Send email to above address at begin and end of job
# Manage processing environment, load compilers and applications.
module purge
module load compilername
module load applicationname
module list
# Set thread count (default value is 1).
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# Launch OpenMP code
./myexecutablefiles
The ntasks x cpus-per-task should equal to or less than the total number of CPU cores on a node.
If an OpenMP program uses a lot of memory and 128 threads use all of the memory of the compute node, use fewer processor cores (OpenMP threads) on that compute node.