MPI job in wholenode queue
An MPI job is a set of processes that take advantage of multiple compute nodes by communicating with each other. OpenMPI, Intel MPI (IMPI), and MVAPICH2 are implementations of the MPI standard.
This shows an example of a job submission file of the MPI programs:
#!/bin/bash
# FILENAME: myjobsubmissionfile
#SBATCH -A myallocation # Allocation name
#SBATCH --nodes=2 # Total # of nodes
#SBATCH --ntasks=256 # Total # of MPI tasks
#SBATCH --time=1:30:00 # Total run time limit (hh:mm:ss)
#SBATCH -J myjobname # Job name
#SBATCH -o myjob.o%j # Name of stdout output file
#SBATCH -e myjob.e%j # Name of stderr error file
#SBATCH -p wholenode # Queue (partition) name
#SBATCH --mail-user=useremailaddress
#SBATCH--mail-type=all # Send email to above address at begin and end of job
# Manage processing environment, load compilers and applications.
module purge
module load compilername
module load mpilibrary
module load applicationname
module list
# Launch MPI code
mpirun -np $SLURM_NTASKS ./myexecutablefiles
SLURM can run an MPI program with the srun command. The number of processes is requested with the -n option. If you do not specify the -n option, it will default to the total number of processor cores you request from SLURM.
If the code is built with OpenMPI, it can be run with a simple srun -n command. If it is built with Intel IMPI, then you also need to add the --mpi=pmi2 option: srun --mpi=pmi2 -n 256 ./mycode.exe in this example.
Invoking an MPI program on Anvil with ./myexecutablefiles is typically wrong, since this will use only one MPI process and defeat the purpose of using MPI. Unless that is what you want (rarely the case), you should use srun which is the Slurm analog of mpirun or mpiexec, or use mpirun or mpiexec to invoke an MPI program.