User Tools

Site Tools


cluster:147

This is an old revision of the document!



Back

BLCR Checkpoint in OL3

  • Installation and what it does BLCR

When we move to Openlava 3.x all queues will support checkpointing, which means you can run your job in a “wrapper” and if the job or cluster crashes you can restart your job from last checkpoint file.

Checkpointing is an expensive operation so do not checkpoint under 6 hours. For example, if your job runs for a month checkpoint once a day, if your job runs for a week checkpoint every 12 hours. From this point on I expect all users to checkpoint. Some software does this internally (Amber, Gaussian). For applications or home grown code you can use BLCR. (Too bad it does not work out of box within Openlava).

You need to test out checkpointing before you rely on it. I've notice that some local code, when opening files for output, BLCR does not notice it. The code below has such an example (file fid.txt). Hopefully future versions of BLCR will fix this. Or maybe we shuold open files differently, this needs investigating further.

BLCR, Berkely Lab Checkpoint and Restart, remembers file paths and process ids. The code stages the necessary STDOUT and STDERR files Openlava generates and invokes the relocation feature while ignore old process ids. If an application is large (for example 10G), and static, it is advisable to not save the application inside the checkpoint file.

At the bottom of this page is the v0.1 version of blcr_wrapper program which will hide the complexity for you. blcr_watcher is a program that is in your PATH already and will terminate the wrapper if the application done inside of a check point time interval. I will work with any group interested to customize your blcr_wrapper for your lab/group.

  • Here is an interactive simple sample run.
# location
[hmeij@petaltail 187]$ pwd
/sanscratch/187

# start the application with BLCR
[hmeij@petaltail 187]$ cr_run ./a.out &
[1] 15559

# the application opens a file
[hmeij@petaltail 187]$ ll
total 1104
-rwxr--r-- 1 hmeij its 1126428 Mar 15 15:38 a.out
-rw-r--r-- 1 hmeij its       0 Mar 15 15:41 fid.txt

# wait 5 mins, make a checkpoint file, pid 155559 is a.out running
[hmeij@petaltail 187]$ cr_checkpoint --save-all -f /home/hmeij/checkpoints/chk.15559 15559

#  the application runs for an hour, after a few mins pull the rug from underneath it
[hmeij@petaltail 187]$ pkill -u hmeij
Connection to petaltail lost.

# that was not too clever, log back in, restart application in another directory
[hmeij@petaltail 187]$ cd ..
[hmeij@petaltail sanscratch]$ mkdir 188
[hmeij@petaltail sanscratch]$ cd 188

# restart the application from checkpoint file in background
# note the relocate directive
[1]+ cr_restart --relocate /sanscratch/187=/sanscratch/188 \
--no-restore-pid /home/hmeij/checkpoints/chk.15559 &

# note that a.out is missing from directory
[hmeij@petaltail 188]$ ll
total 0
-rw-r--r-- 1 hmeij its 0 Mar 15 15:45 fid.txt

# but a.out is running upon restart 
# On of these sleep processes is determining when next checkpoint gets created (set in blcr_wrapper)
# the other sleep process determines when blcr_watcher next checks if application has finished (every 10 mins)

[hmeij@petaltail 188]$ ps
  PID TTY          TIME CMD                                                      
24936 ?        00:00:00 res                                                      
24942 ?        00:00:00 1458238185.218                                           
24945 ?        00:00:00 1458238185.218.                                          
24960 ?        00:00:00 cr_restart                                               
24961 ?        00:00:00 blcr_watcher                                             
24962 ?        00:00:00 sleep                                                    
24964 ?        00:00:09 a.out                                                    
24965 ?        00:00:00 sleep                                                    
24983 ?        00:00:00 sshd                                                     
24984 ?        00:00:00 ps 

# after an hour
[1]+  Done                    cr_restart --relocate /sanscratch/187=/sanscratch/188 \
 --no-restore-pid /home/hmeij/checkpoints/chk.15559

# the result
[hmeij@petaltail 188]$ cat fid.txt
  0.999577738717693       8.112048998602431E-005


Back

cluster/147.1458242361.txt.gz · Last modified: 2016/03/17 19:19 by hmeij07