Alternatives to array jobs

October 31, 2013

In the past, array jobs were an efficient way of submitting large number of similar jobs, sequentially iterating through a list of inputs or parameters. Array jobs were recommended in these situations largely due to limitations of the old job scheduling software, PBSPro, used by ITaP several years ago. PBSPro was unable to effectively handle large quantities of jobs submitted at once.

The current job scheduling software used by ITaP, Torque/Moab, does not have this limitation and array jobs are no longer the most efficient way to submit a large number of sequential jobs. Using array jobs today may severely limit how quickly you can run large numbers of jobs.

Using array jobs is no longer recommended and will not be supported going forward. There are several simpler and more robust ways of accomplishing what array jobs do without using array jobs.

This example submits a batch of similar jobs providing each individual job with an iterator counter. There are two sample scripts below, one for bash and one for tcsh. The code can be copied to a file, for example, called “submit”. This file should be given execute permissions with the command “chmod u+x submit” and then can be executed with the command “./submit”.

bash version:

#!/bin/bash
iterations=`seq 1 100`
for i in $iterations; do
   export JOBINDEX=$i
   qsub -lnodes=1:ppn=1 -N myjob_$i -v JOBINDEX myjob
   sleep 0.2
done 

tcsh version:

#!/bin/tcsh
set iterations = ( `seq 1 100` )
foreach i ( $iterations )
   setenv $JOBINDEX $i
   qsub -lnodes=1:ppn=1 -N myjob_$i -v JOBINDEX myjob
   sleep 0.2
end 

In this example, a list of iterators is defined in the variable $iterations. Then, for each iteration, a JOBINDEX variable is set. The job script “myjob” is submitted while giving the job a unique name “myjob_$i” and exporting the $JOBINDEX variable to the job with “-v JOBINDEX”. A small sleep between each submission should be used to limit submission rate as to not overwhelm the server. The variable $JOBINDEX can be referenced as needed inside the myjob submission script.

This particular example submits 100 jobs, iterating between 1 and 100. The first line of this script can be easily changed to submit jobs over different iterations. A few examples follow. The code for bash is on the left and the equivalent tcsh statement is on the right.

To submit 100 jobs, from 1000 to 1200, incrementing by 2:
bash tcsh
iterations=seq 1000 2 1200 set iterations = ( seq 1000 2 1200 )

The “seq” command can be used to generate more complex iterations such as floating point numbers and 0-padded numbers (ie, 001, 002, 003). See “man seq” for more details on this command.

To submit several jobs over arbitrary iterators:
bash tcsh
iterations="H He Li Be B C N O" set iterations = ( H He Li Be B C N O )
Iteration lists can also be dynamically generated using file lists:
bash tcsh
iterations=$RCAC_SCRATCH/myinputs/jobinputs* set iterations = ( $RCAC_SCRATCH/myinputs/jobinputs* )

With this method you will submit individual PBS jobs that can be processed more efficiently by the scheduling software, improving throughput and turnaround time for your workflow.

If you are using array jobs to do something more complex not covered by the above example, please contact us at rcac-help@purdue.edu. We’ll be happy to work with you to find a workflow that avoids array jobs and maximizes efficiency.

Originally posted: October 31, 2013