DokuWiki

Just have to document this. This functionality is not available in Lava but is in LSF. Very handy.

LSF supports Job Arrays. You can find detialed informaton at this Link

A job array makes it easy to manage a sequence of jobs with “shorthand” syntax. It could be managing 20 jobs or 2,000 jobs, managed by a single command. Once submitted the array job itself can be managed or the individual jobs that make up the job array.

MAX_JOB_ARRAY_SIZE=65534

can be set in lsb.params. The default is 1,000.

For example:

bsub -q idle -J “test[1-20]” -i ~/arrays/in.%I -o ~/arrays/out.%J.%I ~/array/my_array_job.sh

This single command will submit 20 jobs at once to queue idle. The %I iterates form 1 to 20. The %J assumes the JOBPID of the job array. If submitted, you would see something like:

bjobs 1766
JOBID  USER   STAT  QUEUE  FROM_HOST    EXEC_HOST     JOB_NAME   SUBMIT_TIME
1766   hmeij  RUN   idle   swallowtail  compute-1-1   test[1]    Sep 09 12:34
1766   hmeij  RUN   idle   swallowtail  compute-1-17  test[2]    Sep 09 12:34
1766   hmeij  RUN   idle   swallowtail  compute-2-31  test[3]    Sep 09 12:34
1766   hmeij  RUN   idle   swallowtail  compute-1-10  test[4]    Sep 09 12:34
...
1766   hmeij  PEND  idle   swallowtail       -        test[18]   Sep 09 12:34
1766   hmeij  PEND  idle   swallowtail       -        test[19]   Sep 09 12:34
1766   hmeij  PEND  idle   swallowtail       -        test[20]   Sep 09 12:34

Nifty.

Your input files are dynamically matched … so “test[1]” uses ~/arrays/in.1 and the output file is saved as ~/arrays/out.1766.1 etc.

You now can manage your job arrays: for example to abort the entire job array bkill 1766 or address individual jobs within the array bkill 1766[18].

If it was working you could use these sample files.

• create input files: ~/arrays/in.N with content (where N = 1 .. 20)

'N'

• create ~/arrays/my_array_job.sh

#!/bin/bash
read i
echo $(($i*$i))

Submit.

Back