DokuWiki

When you have one or more jobs running that rely on Gaussian internal checkpoint mechanism, heavy read/write operations may result. That traffic should definitely not hit the /home file system but the /sanscratch file system. That scratch space is also NFS mounted over the Infiniband interconnects (via IPoIB). The result is that this file systems IO operations will also slow our file server down tremendously (even though /sanscratch is a 5 disk Raid 0 setup).

So we've been trying to figure out how to control, or throttle, the IO traffic. Usually the application itself will provide option for this, like rsync's –bwlimit option, but we've not found anything so far. However from the operating systems' point of view we do have a tool available: ionice - get/set program io scheduling class and priority.

So for those that rely on the generation of large Gaussian checkpoint file, please add the following lines to the very top of your submission script:

ionice -c 2 -n 7 -p $$
ionice -p $$

This instructs the compute node to schedule the IO traffic with “best effort” scheduling class and lowest priority for the scripts process ID and any processes launched from the script. This seems (surprisingly, perhaps because of IpoIB?) to have a positive effect on the client's IO traffic that hits the NFS mounted filesystem. It has a tremendous positive impact when issued on the file server itself so perhaps a monitor script is needed in the future.

Example using the /sanscratch area; you must stage your data in the temporary work directory and save results back to your home directory before the job finishes. The scheduler will automatically remove and delete that work directory.

#!/bin/bash
# submit like so: bsub < run.forked

# if writing large checkpoint files uncommnet next lines
#ionice -c 2 -n 7 -p $$
#ionice -p $$

rm -rf err* out* output.*

#BSUB -q mw256fd
#BSUB -o out
#BSUB -e err
#BSUB -J test

# job slots: match inside gaussian.com
#BSUB -n 4
# force all onto one host (shared code and data stack)
#BSUB -R "span[hosts=1]"

# unique job scratch dirs
MYSANSCRATCH=/sanscratch/$LSB_JOBID
MYLOCALSCRATCH=/localscratch/$LSB_JOBID
export MYSANSCRATCH MYLOCALSCRATCH

# cd to remote working dir
cd $MYSANSCRATCH
pwd

# environment
export GAUSS_SCRDIR="$MYSANSCRATCH"

export g09root="/share/apps/gaussian/g09root"
. $g09root/g09/bsd/g09.profile

#export gdvroot="/share/apps/gaussian/gdvh11"
#. $gdvroot/gdv/bsd/gdv.profile

# copy input data to fast disk
cp ~/jobs/forked/gaussian.com .
touch gaussian.log

# run plain vanilla
g09 < gaussian.com > gaussian.log

# run dev
#gdv < gaussian.com > gaussian.log

# save results back to homedir
cp gaussian.log ~/jobs/forked/output.$LSB_JOBID

Back