This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:147 [2016/03/17 19:19] hmeij07 [BLCR Checkpoint in OL3] |
cluster:147 [2017/05/24 18:32] hmeij07 |
||
---|---|---|---|
Line 3: | Line 3: | ||
==== BLCR Checkpoint in OL3 ==== | ==== BLCR Checkpoint in OL3 ==== | ||
+ | |||
+ | * This page concerns SERIAL jobs only; SERIAL jobs can restart on any node | ||
* Installation and what it does [[cluster: | * Installation and what it does [[cluster: | ||
Line 12: | Line 14: | ||
Checkpointing is an expensive operation so do not checkpoint under 6 hours. For example, if your job runs for a month checkpoint once a day, if your job runs for a week checkpoint every 12 hours. From this point on I expect all users to checkpoint. Some software does this internally (Amber, Gaussian). For applications or home grown code you can use BLCR. (Too bad it does not work out of box within Openlava). | Checkpointing is an expensive operation so do not checkpoint under 6 hours. For example, if your job runs for a month checkpoint once a day, if your job runs for a week checkpoint every 12 hours. From this point on I expect all users to checkpoint. Some software does this internally (Amber, Gaussian). For applications or home grown code you can use BLCR. (Too bad it does not work out of box within Openlava). | ||
- | You need to test out checkpointing before you rely on it. I've notice that some local code, when opening files for output, BLCR does not notice it. The code below has such an example (file fid.txt). Hopefully future versions of BLCR will fix this. Or maybe we shuold | + | You need to test out checkpointing before you rely on it. I've notice that some local code, when opening files for output, BLCR does not notice it. The code below has such an example (file fid.txt). Hopefully future versions of BLCR will fix this. Or maybe we should |
- | BLCR, Berkely Lab Checkpoint and Restart, remembers file paths and process ids. The code stages the necessary STDOUT and STDERR files Openlava generates and invokes the relocation feature while ignore old process ids. If an application is large (for example 10G), and static, it is advisable to not save the application inside the checkpoint file. | + | BLCR, Berkely Lab Checkpoint and Restart, remembers file paths and process ids. The code stages the necessary STDOUT and STDERR files Openlava generates and invokes the relocation feature while ignore old process ids. If an application is large, and static, it is advisable to not save the application inside the checkpoint file. |
- | At the bottom of this page is the v0.1 version of '' | + | At the bottom of this page is the current |
* Here is an interactive simple sample run. | * Here is an interactive simple sample run. | ||
- | | + | |
< | < | ||
Line 43: | Line 45: | ||
Connection to petaltail lost. | Connection to petaltail lost. | ||
- | # that was not too clever, log back in, restart application in another directory | + | # ooops, |
[hmeij@petaltail 187]$ cd .. | [hmeij@petaltail 187]$ cd .. | ||
[hmeij@petaltail sanscratch]$ mkdir 188 | [hmeij@petaltail sanscratch]$ mkdir 188 | ||
Line 84: | Line 86: | ||
</ | </ | ||
- | + | ||
+ | |||
+ | ==== Putting it all Together ==== | ||
+ | |||
+ | The '' | ||
+ | |||
+ | You edit the top part of '' | ||
+ | |||
+ | Then submit to scheduler as usual | ||
+ | |||
+ | < | ||
+ | |||
+ | [hmeij@cottontail ~/ynam]$ bsub < blcr_wrapper_serial | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | ==== Files v0.2 ==== | ||
+ | |||
+ | * '' | ||
+ | |||
+ | < | ||
+ | |||
+ | #!/bin/bash | ||
+ | # work dir and cwd | ||
+ | export MYSANSCRATCH=/ | ||
+ | cd $MYSANSCRATCH | ||
+ | |||
+ | # at job finish, all content in / | ||
+ | # will be copied to / | ||
+ | # content older than 3 months will be removed | ||
+ | |||
+ | # SCHEDULER | ||
+ | #BSUB -q test | ||
+ | #BSUB -n 1 | ||
+ | #BSUB -J test | ||
+ | #BSUB -o out | ||
+ | #BSUB -e err | ||
+ | |||
+ | # CHECK POINT TIME INTERVAL: 10m (debug) 6h 12h 18h 1d | ||
+ | cpti=10m | ||
+ | |||
+ | # COPY APPLICATION TO WORK DIR $MYSANSCRATCH (cwd) | ||
+ | # always stage the application (and data if needed) | ||
+ | # if application is large and static save_exec=" | ||
+ | save_exec=" | ||
+ | pre_cmd=" | ||
+ | post_cmd=" | ||
+ | |||
+ | # IF START OF JOB, UNCOMMENT | ||
+ | # its either start or restart block | ||
+ | mode=start | ||
+ | queue=test | ||
+ | cmd=" | ||
+ | |||
+ | # IF RESTART OF JOB, UNCOMMENT | ||
+ | # you must have pwd.JOBPID and chk.JOBPID in $orgjobpid/ | ||
+ | # | ||
+ | # | ||
+ | # | ||
+ | |||
+ | # buglines: if your group/lab is mentioned set value to " | ||
+ | # " | ||
+ | do_bug1a_cmd=" | ||
+ | do_bug1b_cmd=" | ||
+ | |||
+ | |||
+ | |||
+ | ############### | ||
+ | |||
+ | |||
+ | |||
+ | # checkpoints | ||
+ | checkpoints=/ | ||
+ | |||
+ | # bug commands | ||
+ | bug1a_cmd=" | ||
+ | bug1b_cmd=" | ||
+ | |||
+ | # kernel modules | ||
+ | mods=`lsmod | grep ^blcr | wc -l` | ||
+ | if [ $mods -ne 2 ]; then | ||
+ | echo " | ||
+ | kill $$ | ||
+ | fi | ||
+ | |||
+ | # blcr setup | ||
+ | restore_options=" | ||
+ | if [ $save_exec == " | ||
+ | save_options=" | ||
+ | else | ||
+ | save_options=" | ||
+ | fi | ||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | |||
+ | # setup checkpoints dir | ||
+ | if [ ! -d $checkpoints/ | ||
+ | mkdir -p $checkpoints/ | ||
+ | else | ||
+ | echo " | ||
+ | kill $$ | ||
+ | fi | ||
+ | |||
+ | # save process id and path and start application | ||
+ | if [ " | ||
+ | $pre_cmd | ||
+ | cr_run $cmd & | ||
+ | pid=$! | ||
+ | pwd > $checkpoints/ | ||
+ | orgjobpid=0 | ||
+ | if [ " | ||
+ | $bug1a_cmd | ||
+ | fi | ||
+ | |||
+ | # otherwise restart the job | ||
+ | elif [ " | ||
+ | orgpid=`ls $checkpoints/ | ||
+ | orgpwd=`cat $checkpoints/ | ||
+ | #if [ " | ||
+ | # echo " | ||
+ | # kill $$ | ||
+ | #fi | ||
+ | scp $checkpoints/ | ||
+ | if [ $save_exec == " | ||
+ | $pre_cmd | ||
+ | fi | ||
+ | if [ " | ||
+ | $bug1b_cmd | ||
+ | fi | ||
+ | cr_restart $restore_options --relocate $orgpwd=$MYSANSCRATCH $checkpoints/ | ||
+ | pid=$! | ||
+ | pwd > $checkpoints/ | ||
+ | |||
+ | # obviously | ||
+ | else | ||
+ | echo " | ||
+ | kill $$ | ||
+ | fi | ||
+ | |||
+ | # if $cmd disappears during $pcit, terminate wrapper | ||
+ | export POST_CMD=" | ||
+ | blcr_watcher $pid $$ $LSB_JOBID $orgjobpid & | ||
+ | |||
+ | # always run this block | ||
+ | while [ true ]; do | ||
+ | # checkpoint time interval | ||
+ | sleep $cpti | ||
+ | # checkpoint file outside of sanscratch | ||
+ | cr_checkpoint $save_options -f $checkpoints/ | ||
+ | scp $HOME/ | ||
+ | if [ " | ||
+ | $bug1a_cmd | ||
+ | fi | ||
+ | done | ||
+ | |||
+ | </ | ||
+ | |||
+ | * '' | ||
+ | |||
+ | < | ||
+ | |||
+ | # | ||
+ | |||
+ | # watch a process during check point time interval | ||
+ | # if it disappears, terminate the blcr_wrapper | ||
+ | |||
+ | checkpoints=/ | ||
+ | |||
+ | watch_pid=$1 | ||
+ | watch_wrapper=$2 | ||
+ | jobpid=$3 | ||
+ | orgjobpid=$4 | ||
+ | |||
+ | while [ $watch_pid -gt 0 ]; do | ||
+ | sleep 600 | ||
+ | nopid=`ps -u $USER | grep $watch_pid | awk ' | ||
+ | if [ " | ||
+ | # save output | ||
+ | scp -rp $MYSANSCRATCH/ | ||
+ | if [ $orgjobpid -gt 0 ]; then | ||
+ | rm -f $HOME/ | ||
+ | fi | ||
+ | $POST_CMD | ||
+ | kill $watch_wrapper | ||
+ | exit; | ||
+ | fi | ||
+ | done | ||
+ | |||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | ==== Matlab ==== | ||
+ | |||
+ | * https:// | ||