Platform LSF HTML Documentation

⇒ format of entries below are …

time stamp

name of config file changed

parameters changed

description of what is expected

— Henk Meij 2007/09/13 10:32

lsb.hosts

# overload -hmeij
compute-1-17.local     12    ()         ()      ()   ()   ()
compute-1-18.local     12    ()         ()      ()   ()   ()
...
compute-2-32.local     12    ()         ()      ()   ()   ()
...

Noticed that a ton of job run with extremely little memory footprints. However, each takes up one job slot, so 8 of them fill a host. But after the host is status is “Full” we still have tons of memory idle. So i'm first experimenting with raising the number of job slots on the hosts in queue 16-lwnodes (12 instead of the default 8).

— Henk Meij 2007/09/06 10:01

lsf.shared

Begin Resource
...
# below are custom resources -hmeij
   sanscratch   Numeric 30       N           (Available Disk Space in M)
   localscratch Numeric 30       N           (Available Disk Space in M)
End Resource

lsf.cluster.lava

RESOURCENAME  LOCATION
...
# one shared instance for all hosts -hmeij
sanscratch          [all]
# one instance local for each host -hmeij
localscratch        [default]
End ResourceMap

The custom perl program /opt/lava/6.1/linux2.6-glibc2.3-ia32e/etc/elim was copied to each node via cluster-fork. The above configuration files were edited followed by lsadmin reconfig, badmin mbdrestart, and badmin reconfig. This should load the custom resource into LSF. The elim program will report available disk space in megabytes to the local lim which then forwards the information to the master lim. Then it is available to end users via the resource string parameter, for example to request more than 300G of free space in /sanscratch … bsub -R “sanscratch>300000” …

— Henk Meij 2007/08/30 11:17

lsf.conf

LSB_SHORT_HOSTLIST=1

This parameter takes no blank spaces around the equal sign! Deh. What it does is present a more condensed lists of hosts when submitting parallel jobs. For example (before and after change) :

[hmeij@swallowtail gaussian]$ bjobs
JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
13914   hmeij   RUN   gaussian   swallowtail nfs-2-3:nfs-2-3:nfs-2-3:nfs-2-3:nfs-2-3:nfs-2-3:nfs-2-3:nfs-2-3 test       Aug 30 11:13

[hmeij@swallowtail gaussian]$ bjobs
JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
13914   hmeij   RUN   gaussian   swallowtail 8*nfs-2-3   test       Aug 30 11:13

— Henk Meij 2007/08/27 16:16

lsf.conf

ACCT_ARCHIVE_AGE = 30

Rotate the accounting file every 30 days. A cronjob removes monthly any files where the value is >1 … this because bacct will read all of them. For sample output view the cluster home page where you can find statistics on job processing calculated by bacct for the last 7 days (update early Am each day).

NA

none

[hmeij@swallowtail ~]# bsub -n 8 -R "span[ptile=2]" ...

This option requests 8 cores (job slots) for processing, and instructs the scheduler to assign no more than 2 cores per host. This should yield a better performance. Does not work in Lava.

NA

lsb.params

PARALLEL_SCHED_BY_SLOT = Y 

From the manual … “To submit jobs based on the number of available job slots instead of the number of processors.” If this is defined you can use the resource requirements (arguments to the -R option of bsub, see item above) … string span[hosts=1] indicates that all the processors allocated to this job must be on the same host; string span[ptile=value] indicates the number of processors (value) on each host that should be allocated to the job. Does not work in Lava.

NA

lsb.hosts

HOST_NAME              MXJ   r1m     pg    ls   tmp  DISPATCH_WINDOW
...
# overload infiniband enabled nodes 2:1 job slots/node 
compute-1-1.local      16    ()      ()    ()   ()   ()
...[all 16 nodes]...
default                 !    ()      ()    ()   ()   ()

The exclamation mark implies that the scheduler assigns as many job slots to a node as there are cores in the processors. Thus for all our nodes there will be 8 job slots. With the MXJ setting we can “overload” that setting. Since i noticed that the chemistry parallel jobs using Amber barely max the CPU out in terms of memory, we can experiment with this setting. It may be that we have to toggle back to a ratio of 1.5:1 versus 2:1 but we'll see. The output of bhosts will display the current settings per host. Works in Lava, but we can not take advantage of it with above parameter.

— Henk Meij 2007/08/28 16:07

lsb.params

MBD_SLEEP_TIME = 30       #mbatchd scheduling interval (60 secs is default)
SBD_SLEEP_TIME = 15       #sbatchd scheduling interval (30 secs is default)
JOB_ACCEPT_INTERVAL = 1   #interval for any host to accept a job 
                          # (default is 1 (one-fold of MBD_SLEEP_TIME))

The interval delay tells LSF to waits a short time between dispatching jobs to the same host of length as defined for the MAster Batch Daemon (MBD). The Slave Batch Daemon (SBD) interval defines the lenght of time in between the recalculations of resource parameters whcih are forwarded to MBD. Since we're not busy, we lower them a bit.

NA

lsf.conf

LSB_LOCALDIR=/share/apps/logs/lsb.events

Enables duplicate logging of lsb.events. Does not work in Lava.

Back