DokuWiki

User Tools

Site Tools

Trace:
cluster:147

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
cluster:147 [2016/03/18 18:13]
hmeij07 		cluster:147 [2020/02/27 18:06] (current)
hmeij07
Line 3: Line 3:
  
 ==== BLCR Checkpoint in OL3 ==== ==== BLCR Checkpoint in OL3 ====
 +
 +**Deprecated since we did OS upgrades [[cluster:185|OS Update]]\\
 +We will install DMTCP as a replacement...[[cluster:190|DMTCP]]** \\
 + --- //[[hmeij@wesleyan.edu|Henk]] 2020/01/14 14:28//
 +
 +  * This page concerns SERIAL jobs only; SERIAL jobs can restart on any node
  
   * Installation and what it does [[cluster:124|BLCR]]   * Installation and what it does [[cluster:124|BLCR]]
Line 8: Line 14:
   * Users Guide [[https://upc-bugs.lbl.gov/blcr/doc/html/BLCR_Users_Guide.html]]   * Users Guide [[https://upc-bugs.lbl.gov/blcr/doc/html/BLCR_Users_Guide.html]]
  
-When we move to Openlava 3.x all queues will support checkpointing, which means you can run your job in a "wrapper" and if the job or cluster crashes you can restart your job from last checkpoint file.+All queues will support checkpointing, which means you can run your job in a "wrapper" and if the job or cluster crashes you can restart your job from last checkpoint file.
  
-Checkpointing is an expensive operation so do not checkpoint under 6 hours. For example, if your job runs for a month checkpoint once a day, if your job runs for a week checkpoint every 12 hours. From this point on I expect all users to checkpoint. Some software does this internally (Amber, Gaussian). For applications or home grown code you can use BLCR. (Too bad it does not work out of box within Openlava).+Checkpointing is an expensive operation so do not checkpoint under 6 hours. For example, if your job runs for a month checkpoint once a day, if your job runs for a week checkpoint every 12 hours. From this point on I expect all users to checkpoint. Some software does this internally (Amber, Gaussian). For applications or home grown code you can use BLCR.
  
-You need to test out checkpointing before you rely on it. I've notice that some local code, when opening files for output, BLCR does not notice it. The code below has such an example (file fid.txt). Hopefully future versions of BLCR will fix this. Or maybe we shuold open files differently, this needs investigating further.+You need to test out checkpointing before you rely on it. I've notice that some local code, when opening files for output, BLCR does not notice it. The code below has such an example (file fid.txt). Hopefully future versions of BLCR will fix this. Or maybe we should open files differently, this needs investigating further.
  
-BLCR, Berkely Lab Checkpoint and Restart, remembers file paths and process ids. The code stages the necessary STDOUT and STDERR files Openlava generates and invokes the relocation feature while ignore old process ids. If an application is large (for example 10G), and static, it is advisable to not save the application inside the checkpoint file.+BLCR, Berkely Lab Checkpoint and Restart, remembers file paths and process ids. The code stages the necessary STDOUT and STDERR files scheduler generates and invokes the relocation feature while ignore old process ids. If an application is large, and static, it is advisable to not save the application inside the checkpoint file.
  
-At the bottom of this page is the v0.1 version of ''blcr_wrapper'' program which will hide the complexity for you. ''blcr_watcher'' is a program that is in your PATH already and will terminate the wrapper if the application done inside of a check point time interval. I will work with any group interested to customize your ''blcr_wrapper'' for your lab/group.+At the bottom of this page is the current version of ''blcr_wrapper'' program which will hide the complexity for you. ''blcr_watcher'' is a program that is in your PATH already and will terminate the wrapper if the application finishes inside of a check point time interval. I will work with any group interested to customize your ''blcr_wrapper'' for your lab/group.
  
    * Here is an interactive simple sample run.    * Here is an interactive simple sample run.
-   * + 
 <code> <code>
  
Line 43: Line 49:
 Connection to petaltail lost. Connection to petaltail lost.
  
-# that was not too clever, log back in, restart application in another directory+# ooops, that was not too clever, log back in, restart application in another directory
 [hmeij@petaltail 187]$ cd .. [hmeij@petaltail 187]$ cd ..
 [hmeij@petaltail sanscratch]$ mkdir 188 [hmeij@petaltail sanscratch]$ mkdir 188
Line 88: Line 94:
 ==== Putting it all Together ==== ==== Putting it all Together ====
  
-The ''blcr_wrapper'' will perform a "change directory" to $MYSANSCRATCH which is /sanscratch/JOBPID. SO think in those terms. Copy the application (and input data) to '.' using $pre_cmd in the script. If $save_exec="n", then upon a restart the script will copy the application back.+The ''blcr_wrapper'' will perform a "change directory" to $MYSANSCRATCH which is /sanscratch/JOBPID. So think in those terms. Copy the application (and input data) to '.' using $pre_cmd in the script. If $save_exec="n", then upon a restart the script will copy the application back.
  
 You edit the top part of ''blcr_wrapper'' to match your job's needs. Then either the block of START or RESTART is uncommented. When you restart a job (some new JOBPID assigned by cluster) the script needs the old JOBPID of crashed job that has latest checkpoint file in /sanscratch/checkpoints/ You edit the top part of ''blcr_wrapper'' to match your job's needs. Then either the block of START or RESTART is uncommented. When you restart a job (some new JOBPID assigned by cluster) the script needs the old JOBPID of crashed job that has latest checkpoint file in /sanscratch/checkpoints/
Line 96: Line 102:
 <code> <code>
  
-[hmeij@cottontail ~/ynam]$ bsub < blcr_wrapper+[hmeij@cottontail ~/ynam]$ bsub < blcr_wrapper_serial
  
 </code> </code>
  
  
-==== Files v0.1 ====+==== Files v0.2 ====
  
-  * ''blcr_wrapper'' v0.1 at /home/hmeij/ynam/blcr_wrapper (copy this file, I've added comments to the script below, may have a typo in it or whatever)+  * ''blcr_wrapper_serial'' at /home/hmeij/jobs/blcr/ for non MPI jobs
  
 <code> <code>
  
-  +#!/bin/bash  
-#!/bin/bash+# work dir and cwd 
 +export MYSANSCRATCH=/sanscratch/$LSB_JOBID 
 +cd $MYSANSCRATCH
  
 # at job finish, all content in /sanscratch/JOBPID # at job finish, all content in /sanscratch/JOBPID
 # will be copied to /sanscratch/checkpoints/JOBPID # will be copied to /sanscratch/checkpoints/JOBPID
-# clean this area up once in a while +# content older than 3 months will be removed
-# content older than 6 months will be removed+
  
 # SCHEDULER  # SCHEDULER 
Line 122: Line 129:
 #BSUB -e err #BSUB -e err
  
-# CHECK POINT TIME INTERVAL: 6h 12h 18h 1d  +# CHECK POINT TIME INTERVAL: 10m (debug) 6h 12h 18h 1d  
-cpti=6h+cpti=10m
  
 # COPY APPLICATION TO WORK DIR $MYSANSCRATCH (cwd) # COPY APPLICATION TO WORK DIR $MYSANSCRATCH (cwd)
 # always stage the application (and data if needed) # always stage the application (and data if needed)
 # if application is large and static save_exec="n" # if application is large and static save_exec="n"
 +save_exec="n"
 pre_cmd=" scp $HOME/ynam/a.out . " pre_cmd=" scp $HOME/ynam/a.out . "
 +post_cmd=" scp $MYSANSCRATCH/fid.txt $HOME/ynam "
  
 # IF START OF JOB, UNCOMMENT # IF START OF JOB, UNCOMMENT
-# its either start or restart+# its either start or restart block
 mode=start mode=start
 queue=test queue=test
 cmd="./a.out" cmd="./a.out"
-save_exec="y" 
  
 # IF RESTART OF JOB, UNCOMMENT # IF RESTART OF JOB, UNCOMMENT
Line 141: Line 149:
 #mode=restart #mode=restart
 #queue=test #queue=test
-#orgjobpid=188+#orgjobpid=250 
 + 
 +# buglines: if your group/lab is mentioned set value to "y", else "n" 
 +# "y" for rblumel/ynam 
 +do_bug1a_cmd="y" 
 +do_bug1b_cmd="y" 
  
  
Line 152: Line 165:
 checkpoints=/sanscratch/checkpoints checkpoints=/sanscratch/checkpoints
  
-# work directory, avoid /home,  +# bug commands 
-MYSANSCRATCH=/sanscratch/$LSB_JOBID +bug1a_cmd="scp $MYSANSCRATCH/fid.txt $checkpoints/$LSB_JOBID/" 
-MYLOCALSCRATCH=/localscratch/$LSB_JOBID +bug1b_cmd="scp $checkpoints/$orgjobpid/fid.txt $MYSANSCRATCH"
-export MYSANSCRATCH MYLOCALSCRATCH +
-cd $MYSANSCRATCH+
  
 # kernel modules # kernel modules
Line 162: Line 173:
 if [ $mods -ne 2 ]; then if [ $mods -ne 2 ]; then
         echo "Error: BLCR modules not loaded on `hostname`"         echo "Error: BLCR modules not loaded on `hostname`"
-        exit+        kill $$
 fi fi
  
Line 180: Line 191:
 else else
         echo "Error: $checkpoints/$LSB_JOBID already exists, exciting"         echo "Error: $checkpoints/$LSB_JOBID already exists, exciting"
-        exit+        kill $$
 fi fi
  
Line 190: Line 201:
         pwd > $checkpoints/$LSB_JOBID/pwd.$pid         pwd > $checkpoints/$LSB_JOBID/pwd.$pid
         orgjobpid=0         orgjobpid=0
 +        if [ "X$do_bug1a_cmd" == "Xy" ]; then
 +                $bug1a_cmd
 +        fi
  
 # otherwise restart the job # otherwise restart the job
Line 195: Line 209:
         orgpid=`ls $checkpoints/$orgjobpid/pwd.* | awk -F\. '{print $2}'`         orgpid=`ls $checkpoints/$orgjobpid/pwd.* | awk -F\. '{print $2}'`
         orgpwd=`cat $checkpoints/$orgjobpid/pwd.$orgpid`         orgpwd=`cat $checkpoints/$orgjobpid/pwd.$orgpid`
 +        #if [ "X$orgpid" == "X" -o "X$orgpwd" == "X" ]; then
 +        #       echo "Error: problem with missing orgpid or orgpwd values"
 +        #       kill $$
 +        #fi
         scp $checkpoints/$orgjobpid/*.$orgjobpid.err $checkpoints/$orgjobpid/*.$orgjobpid.out $HOME/.lsbatch/         scp $checkpoints/$orgjobpid/*.$orgjobpid.err $checkpoints/$orgjobpid/*.$orgjobpid.out $HOME/.lsbatch/
         if [ $save_exec == "n" ]; then         if [ $save_exec == "n" ]; then
                 $pre_cmd                 $pre_cmd
         fi         fi
-        # why? --save-all? +        if [ "X$do_bug1b_cmd" == "Xy" ]; then 
-        scp $checkpoints/$orgjobpid/fid.txt $MYSANSCRATCH+                $bug1b_cmd 
 +        fi
         cr_restart $restore_options --relocate $orgpwd=$MYSANSCRATCH $checkpoints/$orgjobpid/chk.$orgpid &         cr_restart $restore_options --relocate $orgpwd=$MYSANSCRATCH $checkpoints/$orgjobpid/chk.$orgpid &
         pid=$!         pid=$!
Line 208: Line 227:
 else else
         echo "Error: startup mode not defined correctly"         echo "Error: startup mode not defined correctly"
-        exit+        kill $$
 fi fi
  
 # if $cmd disappears during $pcit, terminate wrapper # if $cmd disappears during $pcit, terminate wrapper
 +export POST_CMD="$post_cmd"
 blcr_watcher $pid $$ $LSB_JOBID $orgjobpid & blcr_watcher $pid $$ $LSB_JOBID $orgjobpid &
  
Line 221: Line 241:
         cr_checkpoint $save_options -f $checkpoints/$LSB_JOBID/chk.$pid $pid         cr_checkpoint $save_options -f $checkpoints/$LSB_JOBID/chk.$pid $pid
         scp $HOME/.lsbatch/*.$LSB_JOBID.err $HOME/.lsbatch/*.$LSB_JOBID.out $checkpoints/$LSB_JOBID/         scp $HOME/.lsbatch/*.$LSB_JOBID.err $HOME/.lsbatch/*.$LSB_JOBID.out $checkpoints/$LSB_JOBID/
-        # why? --save-all? +        if [ "X$do_bug1a_cmd" == "Xy" ]; then 
-        scp $MYSANSCRATCH/fid.txt $checkpoints/$LSB_JOBID/ +                $bug1a_cmd 
 +        fi
 done done
  
Line 260: Line 281:
  
 </code> </code>
 +
 +
 +==== Matlab ====
 +
 +  * https://www.bu.edu/tech/support/research/software-and-programming/common-languages/matlab/matlab-batch/checkpointing/
 +
  
  
cluster/147.1458324809.txt.gz · Last modified: 2016/03/18 18:13 by hmeij07

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki