Slurm Partitions (Queues)

Anvil provides different queues with varying job sizes and walltimes. There are also limits on the number of jobs queued and running on a per-user and queue basis. Queues and limits are subject to change based on the evaluation from the Early User Program.

Anvil Production Queues
Queue Name	Node Type	Max Nodes per Job	Max Cores per Job	Max Duration	Max running Jobs in Queue	Max running + submitted Jobs in Queue	Charging factor
debug	regular	2 nodes	256 cores	2 hrs	1	2	1
gpu-debug	gpu	1 node	2 gpus	0.5 hrs	1	2	1
wholenode	regular	16 nodes	2,048 cores	96 hrs	64	2500	1 (node-exclusive)
wide	regular	56 nodes	7,168 cores	12 hrs	5	10	1 (node-exclusive)
shared	regular	1 node	128 cores	96 hrs	6400 cores	-	1
highmem	large-memory	1 node	128 cores	48 hrs	2	4	4
gpu	gpu	-	-	48 hrs	-	-	1

For gpu queue: max of 12 GPU in use per user and max of 32 GPU in use per allocation.

Make sure to specify the desired partition when submitting your jobs (e.g. -p wholenode). If you do not specify one, the job will be directed into the default partition (shared).

If the partition is node-exclusive (e.g. the wholenode and wide queues), even if you ask for 1 core in your job submission script, your job will get allocated an entire node and would not share this node with any other jobs. Hence, it will be charged for 128 cores' worth and squeue command would show it as 128 cores, too. See SU accounting for more details.

Link to section 'Useful tools' of 'Slurm Partitions (Queues)' Useful tools

To display all Slurm partitions and their current usage, type showpartitions at the command line.

x-anvilusername@login03.anvil:[~] $ showpartitions
Partition statistics for cluster anvil at CURRENTTIME
      Partition     #Nodes     #CPU_cores  Cores_pending   Job_Nodes MaxJobTime Cores Mem/Node
      Name State Total  Idle  Total   Idle Resorc  Other   Min   Max  Day-hr:mn /node     (GB)
 wholenode    up   750   684  96000  92160      0   1408     1 infin   infinite   128     257 
    shared:*  up   250   224  32000  30208      0      0     1 infin   infinite   128     257 
      wide    up   750   684  96000  92160      0      0     1 infin   infinite   128     257 
   highmem    up    32    32   4096   4096      0      0     1 infin   infinite   128    1031 
     debug    up    17     5   2176   2176      0      0     1 infin   infinite   128     257 
       gpu    up    16    10   2048   1308      0    263     1 infin   infinite   128     515 
 gpu-debug    up    16    10   2048   1308      0      0     1 infin   infinite   128     515

To show the list of available constraint feature names for different node types, type sfeatures at the command line.

x-anvilusername@login03.anvil:[~] $ sfeatures
NODELIST     CPUS   MEMORY    AVAIL_FEATURES   GRES
a[000-999]   128    257526    A,a              (null)
b[000-031]   128    1031669   B,b              (null)
g[000-015]   128    515545    G,g,A100         gpu:4