DokuWiki

Just have to document this. Very handy.

You can find detailed informaton at this Link

A job array makes it easy to manage a sequence of jobs with “shorthand” syntax. It could be managing 20 jobs or 2,000 jobs, managed by a single command. Once submitted the array job itself can be managed or the individual jobs that make up the job array.

MAX_JOB_ARRAY_SIZE=65534

can be set in lsb.params. The default is 1,000.

For example:

bsub -q idle -J “test[1-20]” -i ~/arrays/in.%I -o ~/arrays/out.%J.%I ~/array/my_array_job.sh

This single command will submit 20 jobs at once to queue idle. The %I iterates form 1 to 20. The %J assumes the JOBPID of the job array. If submitted, you would see something like:

bjobs 1766
JOBID  USER   STAT  QUEUE  FROM_HOST    EXEC_HOST     JOB_NAME   SUBMIT_TIME
1766   hmeij  RUN   idle   swallowtail  compute-1-1   test[1]    Sep 09 12:34
1766   hmeij  RUN   idle   swallowtail  compute-1-17  test[2]    Sep 09 12:34
1766   hmeij  RUN   idle   swallowtail  compute-2-31  test[3]    Sep 09 12:34
1766   hmeij  RUN   idle   swallowtail  compute-1-10  test[4]    Sep 09 12:34
...
1766   hmeij  PEND  idle   swallowtail       -        test[18]   Sep 09 12:34
1766   hmeij  PEND  idle   swallowtail       -        test[19]   Sep 09 12:34
1766   hmeij  PEND  idle   swallowtail       -        test[20]   Sep 09 12:34

Nifty.

Your input files are dynamically matched … so “test[1]” uses ~/arrays/in.1 and the output file is saved as ~/arrays/out.1766.1 etc.

You now can manage your job arrays: for example to abort the entire job array bkill 1766 or address individual jobs within the array bkill 1766[18].

If it was working you could use these sample files.

• create input files: ~/arrays/in.N with content (where N = 1 .. 20)

'N'

• create ~/arrays/my_array_job.sh

#!/bin/bash
read i
echo $(($i*$i))

Submit.

Back