Slurm
The Simple Linux Utility for Resource Management (SLURM) is a system providing job scheduling and job management on compute clusters. With SLURM, a user requests resources and submits a job to a queue. The system will then take jobs from queues, allocate the necessary nodes, and execute them.
Running Jobs
SLURM performs job scheduling. Jobs may be any type of program. You may use either the batch or interactive mode to run your jobs. Use the batch mode for finished programs; use the interactive mode only for debugging.
In this section, you'll find a few pages describing the basics of creating and submitting SLURM jobs. As well, a number of example SLURM jobs that you may be able to adapt to your own needs.
PBS to Slurm
This is a reference for the most common command, environment variables, and job specification options used by the workload management systems and their equivalents.
Quick Guide
This table lists the most common command, environment variables, and job specification options used by the workload management systems and their equivalents (adapted from http://www.schedmd.com/slurmdocs/rosetta.html).
| User Commands | PBS/Torque | Slurm |
|---|---|---|
| Job submission | qsub [script_file] |
sbatch [script_file] |
| Interactive Job | qsub -I |
sinteractive |
| Job deletion | qdel [job_id] |
scancel [job_id] |
| Job status (by job) | qstat [job_id] |
squeue [-j job_id] |
| Job status (by user) | qstat -u [user_name] |
squeue [-u user_name] |
| Job hold | qhold [job_id] |
scontrol hold [job_id] |
| Job release | qrls [job_id] |
scontrol release [job_id] |
| Queue info | qstat -Q |
squeue |
| Queue access | qlist |
slist |
| Node list | pbsnodes -l |
sinfo -Nscontrol show nodes |
| Cluster status | qstat -a |
sinfo |
| GUI | xpbsmon |
sview |
| Environment | PBS/Torque | Slurm |
| Job ID | $PBS_JOBID |
$SLURM_JOB_ID |
| Job Name | $PBS_JOBNAME |
$SLURM_JOB_NAME |
| Job Queue/Account | $PBS_QUEUE |
$SLURM_JOB_ACCOUNT |
| Submit Directory | $PBS_O_WORKDIR |
$SLURM_SUBMIT_DIR |
| Submit Host | $PBS_O_HOST |
$SLURM_SUBMIT_HOST |
| Number of nodes | $PBS_NUM_NODES |
$SLURM_JOB_NUM_NODES |
| Number of Tasks | $PBS_NP |
$SLURM_NTASKS |
| Number of Tasks Per Node | $PBS_NUM_PPN |
$SLURM_NTASKS_PER_NODE |
| Node List (Compact) | n/a | $SLURM_JOB_NODELIST |
| Node List (One Core Per Line) | LIST=$(cat $PBS_NODEFILE) |
LIST=$(srun hostname) |
| Job Array Index | $PBS_ARRAYID |
$SLURM_ARRAY_TASK_ID |
| Job Specification | PBS/Torque | Slurm |
| Script directive | #PBS |
#SBATCH |
| Queue | -q [queue] |
-A [queue] |
| Node Count | -l nodes=[count] |
-N [min[-max]] |
| CPU Count | -l ppn=[count] |
-n [count]Note: total, not per node |
| Wall Clock Limit | -l walltime=[hh:mm:ss] |
-t [min] OR-t [hh:mm:ss] OR-t [days-hh:mm:ss] |
| Standard Output FIle | -o [file_name] |
-o [file_name] |
| Standard Error File | -e [file_name] |
-e [file_name] |
| Combine stdout/err | -j oe (both to stdout) OR-j eo (both to stderr) |
(use -o without -e) |
| Copy Environment | -V |
--export=[ALL | NONE | variables]Note: default behavior is ALL |
| Copy Specific Environment Variable | -v myvar=somevalue |
--export=NONE,myvar=somevalue OR--export=ALL,myvar=somevalue |
| Event Notification | -m abe |
--mail-type=[events] |
| Email Address | -M [address] |
--mail-user=[address] |
| Job Name | -N [name] |
--job-name=[name] |
| Job Restart | -r [y|n] |
--requeue OR--no-requeue |
| Working Directory | --workdir=[dir_name] |
|
| Resource Sharing | -l naccesspolicy=singlejob |
--exclusive OR--shared |
| Memory Size | -l mem=[MB] |
--mem=[mem][M|G|T] OR--mem-per-cpu=[mem][M|G|T] |
| Account to charge | -A [account] |
-A [account] |
| Tasks Per Node | -l ppn=[count] |
--tasks-per-node=[count] |
| CPUs Per Task | --cpus-per-task=[count] |
|
| Job Dependency | -W depend=[state:job_id] |
--depend=[state:job_id] |
| Job Arrays | -t [array_spec] |
--array=[array_spec] |
| Generic Resources | -l other=[resource_spec] |
--gres=[resource_spec] |
| Licenses | --licenses=[license_spec] |
|
| Begin Time | -A "y-m-d h:m:s" |
--begin=y-m-d[Th:m[:s]] |
See the official Slurm Documentation for further details.
Notable Differences
-
Separate commands for Batch and Interactive jobs
Unlike PBS, in Slurm interactive jobs and batch jobs are launched with completely distinct commands.
Usesbatch [allocation request options] scriptto submit a job to the batch scheduler, andsinteractive [allocation request options]to launch an interactive job. sinteractive accepts most of the same allocation request options as sbatch does. -
No need for
cd $PBS_O_WORKDIRIn Slurm your batch job starts to run in the directory from which you submitted the script whereas in PBS/Torque you need to explicitly move back to that directory with
cd $PBS_O_WORKDIR. -
No need to manually export environment
The environment variables that are defined in your shell session at the time that you submit the script are exported into your batch job, whereas in PBS/Torque you need to use the
-Vflag to export your environment. -
Location of output files
The output and error files are created in their final location immediately that the job begins or an error is generated, whereas in PBS/Torque temporary files are created that are only moved to the final location at the end of the job. Therefore in Slurm you can examine the output and error files from your job during its execution.
See the official Slurm Documentation for further details.