Differences

This shows you the differences between two versions of the page.

--- cluster:148 [2016/03/29 20:00]
hmeij07
+++ cluster:148 [2020/01/24 18:36] (current)
hmeij07
@@ Line 4: / Line 4: @@
 ==== BLCR Checkpoint in OL3 ====
-  * This page concerns PARALLEL mpirun jobs only
+**Deprecated since we did [[cluster:185|OS Update]] \\
+We will replace it with [[cluster:190|DMTCP]] ** \\
+ --- //[[hmeij@wesleyan.edu|Henk]] 2020/01/14 14:31//
+  * This page concerns PARALLEL mpirun jobs only; there are some restrictions
+    * all MPI threads need to be confined to one node
+    * restarted jobs must use the same node (not sure why)
   * For SERIAL jobs go here [[cluster:147|BLCR Checkpoint in OL3]]
@@ Line 11: / Line 17: @@
   * Users Guide [[https://upc-bugs.lbl.gov/blcr/doc/html/BLCR_Users_Guide.html]]
+Checkpointing parallel jobs is a bit more complex than a serial job. MPI workers (the -n) are fired off by worker 0 of ''mpirun'' and all workers may open files and perform socket to socket communications. Therefore a restart will need to restore all file IDs, process IDs, etc. A job may thus fail if a certain process ID is already running. Restarted files also behave as if the old JOBPID is running and will write results to the old STDERR and STDOUT files.
+The ''blcr_wrapper_parallel' below will manage all this for you. Like the serial wrapper only edit the top of the file and provide the information necessary. But first, your software needs to be compiled with a special "older" version of OpenMPI. MPI checkpointing support has been removed in later versions of OpenMPI.
+Here is the admin stuff.
 <code>
-[hmeij@cottontail lammps]$ bsub < blcr_wrapper
+# from eric at lbl, configure openmpi, I choose 1.6.5 (version needs to be < 1.7)
+./configure \
+            --enable-ft-thread \
+            --with-ft=cr \
+            --enable-opal-multi-threads \
+            --with-blcr=/share/apps/blcr/0.8.5/test \
+            --without-tm \
+            --prefix=/share/apps/CENTOS6/openmpi/1.6.5.cr
+make
+make install
+# next download cr_mpirun from LBL
+https://upc-bugs.lbl.gov/blcr-dist/cr_mpirun/cr_mpirun-295.tar.gz
+# configure and test cr_mpirun
+export PATH=/share/apps/CENTOS6/openmpi/1.6.5.cr/bin:$PATH
+export LD_LIBRARY_PATH=/share/apps/CENTOS6/openmpi/1.6.5.cr/lib:$LD_LIBRARY_PATH
+./configure --with-blcr=/share/apps/blcr/0.8.5/test
+make
+make check
+============================================================================
+Testsuite summary for cr_mpirun 295
+============================================================================
+# TOTAL: 3
+# PASS:  3
+# SKIP:  0
+# XFAIL: 0
+# FAIL:  0
+# XPASS: 0
+# ERROR: 0
+============================================================================
+make[1]: Leaving directory `/home/apps/src/petaltail6/cr_mpirun-295'
+# I copied cr_mpirun into /share/apps/CENTOS6/openmpi/1.6.5.cr/bin/
+# cr_mpirun needs access to all these in $PATH
+# mpirun cr_mpirun ompi-checkpoint ompi-restart cr_checkpoint cr_restart
+# next compile you parallel software using mpicc/mpicxx from the 1.6.5 distro
+</code>
+Here is what a sample run using the Openlava scheduler looks like
+<code>
+# submit as usual after editing the top of the file, see comments in that wrapper file
+[hmeij@cottontail lammps]$ bsub < blcr_wrapper_parallel
 Job <681> is submitted to queue <test>.
-[hmeij@cottontail lammps]$ bjobs
-JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
+# cr_mpirun job
-     hmeij   PEND  test       cottontail              test       Mar 29 13:23
 [hmeij@cottontail lammps]$ bjobs
 JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
@@ Line 26: / Line 86: @@
                                              petaltail
-[hmeij@cottontail lammps]$ tail ~/.lsbatch/1459272204.681.out
+# wrapper stores BLCR checkpoint file (chk.PID) in this location
-    4062132.4   -4439564.1 1.4618689e+08 2.8818933e+08    17552.743
+# and it calls the openmpi snapshot tools and stores that in
-    4395340.9   -4440084.6 1.3417499e+08 2.8818925e+08    19150.394
+# ompi_global_snapshot_SOME-PID.ckpt, also in same location
-    4711438.8   -4440426.5 1.2277977e+08 2.8818918e+08    20665.317
-    5007151.9   -4440573.2 1.1211925e+08 2.8818913e+08    22081.756
-    5279740.3     -4440516 1.0229219e+08 2.8818909e+08    23386.523
-    5527023.5   -4440257.5     93377214 2.8818906e+08    24569.109
-    5747387.7   -4439813.3     85432510 2.8818904e+08    25621.734
-    5939773.3   -4439214.4     78496309 2.8818904e+08    26539.282
-    6103647.2   -4438507.6     72587871 2.8818905e+08    27319.145
-    6238961.8   -4437755.5     67708974 2.8818907e+08    27961.064
 [hmeij@cottontail lammps]$ ll /sanscratch/checkpoints/681
 total 30572
@@ Line 49: / Line 102: @@
 drwx------ 3 hmeij its       46 Mar 29 13:28 ompi_global_snapshot_9134.ckpt
 -rw-r--r-- 1 hmeij its       16 Mar 29 13:23 pwd.9127
+# the processes running
 [hmeij@cottontail lammps]$ ssh petaltail ps -u hmeij
   PID TTY          TIME CMD
@@ Line 57: / Line 112: @@
 ?        00:00:00 1459272204.681.
 ?        00:00:00 cr_mpirun
-?        00:00:00 blcr_watcher
+?        00:00:00 blcr_watcher <--- the watcher, will terminate 9113 if 9127 is gone (ie done or crashed)
 ?        00:00:00 cr_mpirun
 ?        00:00:00 mpirun
@@ Line 69: / Line 124: @@
 ?        00:00:00 ps
 pts/2    00:00:00 bash
+# how far did the job progress?
 [hmeij@cottontail lammps]$ tail ~/.lsbatch/1459272204.681.out
     5007151.9   -4440573.2 1.1211925e+08 2.8818913e+08    22081.756
@@ Line 80: / Line 137: @@
     6425840.1   -4436426.7     60970646 2.8818913e+08    28840.129
     6479251.9   -4436021.1     59044759 2.8818917e+08    29086.075
+# simulate crash
 [hmeij@cottontail lammps]$ ssh petaltail kill 9133
+# edit the file and prep for a restart, submit again
 [hmeij@cottontail lammps]$ bsub < blcr_wrapper
 Job <684> is submitted to queue <test>.
-[hmeij@cottontail lammps]$ rm -f ../.lsbatch/*^C
-[hmeij@cottontail lammps]$ bjobs
+# so job 684 is resatrinig job 681, wrapper preps files
-JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
-     hmeij   RUN   test       cottontail  petaltail   test       Mar 29 13:48
-                                             petaltail
-                                             petaltail
-                                             petaltail
 [hmeij@cottontail lammps]$ ll ../.lsbatch/
 total 172
@@ Line 99: / Line 154: @@
 -rw------- 1 hmeij its   53 Mar 29 13:48 1459273700.684.out
 -rwxr--r-- 1 hmeij its 4270 Mar 29 13:48 1459273700.684.shell
-lrwxrwxrwx 1 hmeij its   33 Mar 29 13:48 hostfile.681 -> /home/hmeij/.lsbatch/hostfile.684
+-rwxr--r-- 1 hmeij its   33 Mar 29 13:48 hostfile.681
 -rw-r--r-- 1 hmeij its   40 Mar 29 13:48 hostfile.684
 -rw-r--r-- 1 hmeij its   40 Mar 29 13:48 hostfile.tmp.684
-[hmeij@cottontail lammps]$ less ../.lsbatch/*684.err
 [hmeij@cottontail lammps]$ ssh petaltail ps -u hmeij
   PID TTY          TIME CMD
@@ Line 115: / Line 170: @@
 ?        00:00:00 1459273700.684
 ?        00:00:00 1459273700.684.
-?        00:00:00 cr_restart
+?        00:00:00 cr_restart    <------ started everything back up
 ?        00:00:00 cr_mpirun
 ?        00:00:00 mpirun
@@ Line 129: / Line 184: @@
 pts/2    00:00:00 bash
+# and now you can watch the output picking from last checkpoint file
 [hmeij@cottontail lammps]$ tail -20 ../.lsbatch/1459272204.681.out
     5527023.5   -4440257.5     93377214 2.8818906e+08    24569.109
@@ Line 150: / Line 206: @@
     5796854.1   -4451671.6     83661722 2.8818969e+08    25640.235
     5665179.3     -4453332     88410367 2.8818972e+08    24990.519
+# let job finish
 </code>
+==== Parallel Wrapper v2 ====
+A bit more verbose and error handling. Also the blcr_wrapper or the cr_checkpoint loop code can now terminate the job.
+<code>
+#!/bin/bash -x
+rm -f err out
+# work dir and cwd
+export MYSANSCRATCH=/sanscratch/$LSB_JOBID
+cd $MYSANSCRATCH
+# at job finish, all content in /sanscratch/JOBPID
+# will be copied to /sanscratch/checkpoints/JOBPID
+# content older than 3 months will be removed
+# SCHEDULER set queue name in next TWO lines
+queue=hp12
+#BSUB -q hp12
+#BSUB -n 6
+#BSUB -J test
+#BSUB -o out
+#BSUB -e err
+# next required for mpirun checkpoint to work
+# restarts must use same node (not sure why)
+#BSUB -R "span[hosts=1]"
+#BSUB -m n5
+# CHECK POINT TIME INTERVAL: 10m (debug) 6h 12h 18h 1d
+cpti=15m
+# COPY APPLICATION TO WORK DIR $MYSANSCRATCH (cwd)
+# always stage the application (and data if needed)
+# if mpirun save_exec="n" (default)
+save_exec="n"
+pre_cmd=" scp -r
+$HOME/python/kflaherty/data/HD163296.CO32.regridded.cen15.vis
+$HOME/python/kflaherty/data/HD163296.CO32.regridded.cen15.vis.fits
+$HOME/python/kflaherty/data/lowres_ALMA_weights_calc.sav
+$HOME/python/kflaherty/co.dat
+$HOME/python/kflaherty/disk_other.py
+$HOME/python/kflaherty/disk.py
+$HOME/python/kflaherty/mol_dat.py
+$HOME/python/kflaherty/mpi_run_models.py
+$HOME/python/kflaherty/sample_co32.sh
+$HOME/python/kflaherty/single_model.py . "
+post_cmd=" scp $MYSANSCRATCH/chain*.dat $HOME/tmp/"
+# IF START OF JOB, UNCOMMENT
+# its either start or restart block
+#mode=start
+#cmd=" python mpi_run_models.py /sanscratch/$LSB_JOBID > /sanscratch/$LSB_JOBID/test2.out "
+# IF RESTART OF JOB, UNCOMMENT, MUST BE RUN ON SAME NODE
+# you must have pwd.JOBPID and chk.JOBPID in $orgjobpid/
+mode=restart
+orgjobpid=636341
+# user environment
+export PYTHONHOME=/share/apps/CENTOS6/blcr_soft/python/2.7.10
+export PYTHONPATH=/home/apps/CENTOS6/blcr_soft/python/2.7.10/lib/python2.7/site-packages
+export PATH=$PYTHONHOME/bin:$PATH
+. /home/apps/miriad/MIRRC.sh
+export PATH=$MIRBIN:$PATH
+which python
+############### NOTHING TO EDIT BELOW THIS LINE ##################
+# checkpoints
+checkpoints=/sanscratch/checkpoints
+# kernel modules
+mods=`/sbin/lsmod | grep ^blcr | wc -l`
+if [ $mods -ne 2 ]; then
+        echo "Error: BLCR modules not loaded on `hostname`"
+        kill $$
+fi
+# blcr setup
+restore_options=""
+#restore_options="--no-restore-pid --no-restore-pgid --no-restore-sid"
+if [ $save_exec == "n" ]; then
+        #save_options="--save-private --save-shared"
+        save_options="--save-none"
+else
+        save_options="--save-all"
+fi
+# environment
+export PATH=/share/apps/CENTOS6/openmpi/1.6.5.cr/bin:$PATH
+export LD_LIBRARY_PATH=/share/apps/CENTOS6/openmpi/1.6.5.cr/lib:$LD_LIBRARY_PATH
+export PATH=/share/apps/blcr/0.8.5/${queue}/bin:$PATH
+export LD_LIBRARY_PATH=/share/apps/blcr/0.8.5/${queue}/lib:$LD_LIBRARY_PATH
+which mpirun cr_mpirun ompi-checkpoint ompi-restart cr_checkpoint cr_restart
+# setup checkpoints dir
+if [ ! -d $checkpoints/$LSB_JOBID ]; then
+        mkdir -p $checkpoints/$LSB_JOBID
+else
+        echo "Error: $checkpoints/$LSB_JOBID already exists, exciting"
+        kill $$
+fi
+# save process id and path and start application
+if [ "$mode" == "start" ];  then
+        # hostfile
+        echo "${LSB_HOSTS}" > $HOME/.lsbatch/hostfile.tmp.$LSB_JOBID
+        tr '\/ ' '\r\n' < $HOME/.lsbatch/hostfile.tmp.$LSB_JOBID > $HOME/.lsbatch/hostfile.$LSB_JOBID
+        c=`wc -l $HOME/.lsbatch/hostfile.$LSB_JOBID | awk '{print $1}'`
+        for i in `seq 1 $c`; do echo '127.0.0.1' >> $HOME/.lsbatch/localhost.$LSB_JOBID; done
+        $pre_cmd
+        # why
+        rm -f /tmp/tmp??????
+        cr_mpirun -v -am ft-enable-cr --gmca snapc_base_global_snapshot_dir $checkpoints/$LSB_JOBID \
+        -x LD_LIBRARY_PATH --hostfile $HOME/.lsbatch/localhost.$LSB_JOBID $cmd 2>>$checkpoints/$LSB_JOBID/cr_mpirun.err &
+        pid=$!
+        pwd > $checkpoints/$LSB_JOBID/pwd.$pid
+        orgjobpid=0
+# otherwise restart the job
+elif [ "$mode" == "restart" ]; then
+        orgpid=`ls $checkpoints/$orgjobpid/pwd.* | awk -F\. '{print $2}'`
+        orgpwd=`cat $checkpoints/$orgjobpid/pwd.$orgpid`
+        if [ "X$orgpwd" == "X" ]; then
+                echo "Error: orgpwd problem, check error log"
+                exit
+        fi
+        # cleanup old if present
+        rm -rf /sanscratch/$orgjobpid /localscratch/$orgjobpid
+        rm -f $HOME/.lsbatch/*.$orgjobpid
+        # why
+        rm -f /tmp/tmp??????
+        # stage old
+        scp $checkpoints/$orgjobpid/*.$orgjobpid.err $checkpoints/$orgjobpid/*.$orgjobpid.out $HOME/.lsbatch/
+        scp -r $checkpoints/$orgjobpid/* $MYSANSCRATCH
+        ln -s $MYSANSCRATCH /sanscratch/$orgjobpid
+        scp $checkpoints/$orgjobpid/hostfile.$orgjobpid $HOME/.lsbatch/
+        scp -r $checkpoints/$orgjobpid/$orgjobpid/* /localscratch/$LSB_JOBID
+        # why
+        scp $checkpoints/$orgjobpid/$orgjobpid/tmp?????? /tmp/
+        ln -s /localscratch/$LSB_JOBID /localscratch/$orgjobpid
+        c=`wc -l $HOME/.lsbatch/hostfile.$orgjobpid | awk '{print $1}'`
+        for i in `seq 1 $c`; do echo '127.0.0.1' >> $HOME/.lsbatch/localhost.$orgjobpid; done
+        cr_restart --kmsg-warning $restore_options --relocate $orgpwd=$MYSANSCRATCH --cont \
+        $MYSANSCRATCH/chk.$orgpid 2>>$checkpoints/$LSB_JOBID/cr_restart.err &
+        pid=$!
+        started=`ps -u $USER | awk '{print $1}' | grep $pid | wc -l`
+        if [ $started -ne 1 ]; then
+                echo "Error: cr_restart failed, check error log"
+                kill $$
+        fi
+        pwd > $checkpoints/$LSB_JOBID/pwd.$pid
+# obviously
+else
+        echo "Error: startup mode not defined correctly"
+        kill $$
+fi
+# if $cmd disappears during $pcit, terminate wrapper
+export POST_CMD="$post_cmd"
+blcr_watcher $pid $$ $LSB_JOBID $orgjobpid &
+bw_pid=$!
+# always run this block
+while [ true ]; do
+        # checkpoint time interval
+        sleep $cpti
+        # finished?
+        no_pid=`ps -u $USER | grep $pid | awk '{print $1}'`
+        if [ "${no_pid}x" == 'x' ]; then
+                # save output
+                scp -rp $MYSANSCRATCH/* $checkpoints/$LSB_JOBID/
+                $POST_CMD
+                kill $bw_pid
+                rm -f $HOME/.lsbatch/*${orgjobpid}*
+                exit
+        fi
+        # checkpoint file outside of sanscratch
+        scp -r $MYSANSCRATCH/* $checkpoints/$LSB_JOBID/
+        scp -r /localscratch/$LSB_JOBID $checkpoints/$LSB_JOBID/
+        chmod u+w $checkpoints/$LSB_JOBID/chk.* /sanscratch/$LSB_JOBID/chk.*
+        # why
+        scp /tmp/tmp?????? $checkpoints/$LSB_JOBID/$LSB_JOBID/
+        cr_checkpoint -v --tree --cont $save_options -f $checkpoints/$LSB_JOBID/chk.$pid $pid \
+>>$checkpoints/$LSB_JOBID/cr_checkpoint.err
+        scp $HOME/.lsbatch/*.$LSB_JOBID.err $HOME/.lsbatch/*.$LSB_JOBID.out $checkpoints/$LSB_JOBID/
+        scp $HOME/.lsbatch/hostfile.$LSB_JOBID $checkpoints/$LSB_JOBID/
+        scp -r /localscratch/$LSB_JOBID $checkpoints/$LSB_JOBID/
+        # why
+        scp /tmp/tmp?????? $checkpoints/$LSB_JOBID/$LSB_JOBID/
+        date >> $checkpoints/$LSB_JOBID/cr_checkpoint.err
+done
+</code>
+==== Parallel Wrapper v1 ====
+<code>
+#!/bin/bash
+rm -f err out
+# work dir and cwd
+export MYSANSCRATCH=/sanscratch/$LSB_JOBID
+cd $MYSANSCRATCH
+# at job finish, all content in /sanscratch/JOBPID
+# will be copied to /sanscratch/checkpoints/JOBPID
+# content older than 3 months will be removed
+# SCHEDULER
+#BSUB -q test
+#BSUB -J test
+#BSUB -n 4
+#BSUB -o out
+#BSUB -e err
+# next required for mpirun checkpoint to work
+# restarts must use same node in test queue (not sure why, others ca restart anywhere)
+#BSUB -R "span[hosts=1]"
+# CHECK POINT TIME INTERVAL: 10m (debug) 6h 12h 18h 1d
+cpti=1d
+# COPY APPLICATION TO WORK DIR $MYSANSCRATCH (cwd)
+# always stage the application (and data if needed)
+# if mpirun save_exec="n" (default)
+save_exec="n"
+pre_cmd=" scp -r  $HOME/lammps/au.inp $HOME/lammps/auu3
+ $HOME/lammps/henz.dump  $HOME/lammps/data.Big11AuSAMInitial  . "
+post_cmd=" scp auout $HOME/lammps/auout.$LSB_JOBID "
+# IF START OF JOB, UNCOMMENT
+# its either start or restart block
+mode=start
+queue=test
+cmd=" lmp_mpi -c off -var GPUIDX 0 -in au.inp -l auout "
+# IF RESTART OF JOB, UNCOMMENT
+# you must have pwd.JOBPID and chk.JOBPID in $orgjobpid/
+#mode=restart
+#queue=test
+#orgjobpid=691
+# user environment
+export PATH=/share/apps/CENTOS6/blcr_soft/lammps/16Feb16/:$PATH
+export LD_LIBRARY_PATH=/share/apps/CENTOS6/blcr_soft/lammps/16Feb16/lib:$LD_LIBRARY_PATH
+#which lmp_mpi
+############### NOTHING TO EDIT BELOW THIS LINE ##################
+# checkpoints
+checkpoints=/sanscratch/checkpoints
+# kernel modules
+mods=`/sbin/lsmod | grep ^blcr | wc -l`
+if [ $mods -ne 2 ]; then
+        echo "Error: BLCR modules not loaded on `hostname`"
+        kill $$
+fi
+# blcr setup
+restore_options=""
+#restore_options="--no-restore-pid --no-restore-pgid --no-restore-sid"
+if [ $save_exec == "n" ]; then
+        #save_options="--save-private --save-shared"
+        save_options="--save-none"
+else
+        save_options="--save-all"
+fi
+# environment
+export PATH=/share/apps/CENTOS6/openmpi/1.6.5.cr/bin:$PATH
+export LD_LIBRARY_PATH=/share/apps/CENTOS6/openmpi/1.6.5.cr/lib:$LD_LIBRARY_PATH
+export PATH=/share/apps/blcr/0.8.5/${queue}/bin:$PATH
+export LD_LIBRARY_PATH=/share/apps/blcr/0.8.5/${queue}/lib:$LD_LIBRARY_PATH
+#which mpirun cr_mpirun ompi-checkpoint ompi-restart cr_checkpoint cr_restart
+# setup checkpoints dir
+if [ ! -d $checkpoints/$LSB_JOBID ]; then
+        mkdir -p $checkpoints/$LSB_JOBID
+else
+        echo "Error: $checkpoints/$LSB_JOBID already exists, exciting"
+        kill $$
+fi
+# save process id and path and start application
+if [ "$mode" == "start" ];  then
+        # hostfile
+        echo "${LSB_HOSTS}" > $HOME/.lsbatch/hostfile.tmp.$LSB_JOBID
+        tr '\/ ' '\r\n' < $HOME/.lsbatch/hostfile.tmp.$LSB_JOBID > $HOME/.lsbatch/hostfile.$LSB_JOBID
+        $pre_cmd
+        cr_mpirun -am ft-enable-cr --gmca snapc_base_global_snapshot_dir $checkpoints/$LSB_JOBID \
+                  --hostfile $HOME/.lsbatch/hostfile.$LSB_JOBID $cmd &
+        pid=$!
+        pwd > $checkpoints/$LSB_JOBID/pwd.$pid
+        orgjobpid=0
+# otherwise restart the job
+elif [ "$mode" == "restart" ]; then
+        orgpid=`ls $checkpoints/$orgjobpid/pwd.* | awk -F\. '{print $2}'`
+        orgpwd=`cat $checkpoints/$orgjobpid/pwd.$orgpid`
+        # cleanup old
+        rm -rf /sanscratch/$orgjobpid $HOME/.lsbatch/*.$orgjobpid
+        # stage old
+        scp $checkpoints/$orgjobpid/*.$orgjobpid.err $checkpoints/$orgjobpid/*.$orgjobpid.out $HOME/.lsbatch/
+        scp -r $checkpoints/$orgjobpid/* $MYSANSCRATCH
+        ln -s $MYSANSCRATCH /sanscratch/$orgjobpid
+        scp $checkpoints/$orgjobpid/hostfile.$orgjobpid $HOME/.lsbatch/
+        cr_restart --kmsg-warning $restore_options --relocate $orgpwd=$MYSANSCRATCH $MYSANSCRATCH/chk.$orgpid &
+        pid=$!
+        started=` ps -u hmeij | awk '{print $1}' | grep $pid | wc -l`
+        if [ $started -ne 1 ]; then
+                echo "Error: cr_restart failed, check error log"
+                kill $$
+        fi
+        pwd > $checkpoints/$LSB_JOBID/pwd.$pid
+# obviously
+else
+        echo "Error: startup mode not defined correctly"
+        kill $$
+fi
+# if $cmd disappears during $pcit, terminate wrapper
+export POST_CMD="$post_cmd"
+blcr_watcher $pid $$ $LSB_JOBID $orgjobpid &
+# always run this block
+while [ true ]; do
+        # checkpoint time interval
+        sleep $cpti
+        # checkpoint file outside of sanscratch
+        scp -r $MYSANSCRATCH/* $checkpoints/$LSB_JOBID/
+        cr_checkpoint --tree $save_options -f $checkpoints/$LSB_JOBID/chk.$pid $pid
+        scp $HOME/.lsbatch/*.$LSB_JOBID.err $HOME/.lsbatch/*.$LSB_JOBID.out $checkpoints/$LSB_JOBID/
+        scp $HOME/.lsbatch/hostfile.$LSB_JOBID $checkpoints/$LSB_JOBID/
+done
+</code>
 \\
 **[[cluster:0|Back]]**

DokuWiki

User Tools

Site Tools

Differences

Page Tools