Differences

This shows you the differences between two versions of the page.

--- cluster:124 [2013/10/31 17:31]
hmeij created
+++ cluster:124 [2013/10/31 18:21]
hmeij
@@ Line 1: / Line 1: @@
-BLCR
+\\
+**[[cluster:0|Back]]**
-are modules loaded (done via /etc/rc.local)
+==== BLCR ====
+So we need a day of down time to switch file server functionality from greentail to sharptail. It would be nice if everybody did not loose any computational progress.  To do that, we need to learn to checkpoint at the application level.  If a node crashes or power is lost, those applications can then restart the job from the last checkpoint.
+I've decided to support one checkpoint/restart utility, The Berkeley Laboratory Checkpoint/Restart tool. Hence this page.
+BLCR consists of two kernel modules, some user-level libraries, and several command-line executables. No kernel patching is required. Modules are loading upon boot via /etc/rc.local. The modules are dependent on the kernel source where the compilation took place.  So for our first supported BLRC modules I've chosen the mw256 queue and nodes. Here is some documentation on BLCR
+  * [[https://upc-bugs.lbl.gov/blcr/doc/html/BLCR_Admin_Guide.html|BLCR_Admin_Guide.html]]
+  * [[https://upc-bugs.lbl.gov/blcr/doc/html/BLCR_Users_Guide.html|BLCR_Users_Guide.html]]
+First lets test on a node to grasp the concept.
+<code>
+# are modules loaded
 [hmeij@n33 blcr]$ lsmod | grep blcr
 blcr                  115529  0
 blcr_imports           10715  1 blcr
-set env
-  export PATH=/share/apps/blcr/0.8.5/mw256/bin:$PATH
-  export LD_LIBRARY_PATH=/share/apps/blcr/0.8.5/mw256/lib:$LD_LIBRARY_PATH
+# set env
+export PATH=/share/apps/blcr/0.8.5/mw256/bin:$PATH
+export LD_LIBRARY_PATH=/share/apps/blcr/0.8.5/mw256/lib:$LD_LIBRARY_PATH
+# is it all working
 [hmeij@n33 blcr]$ cr_checkpoint --help
 Usage: cr_checkpoint [options] ID
@@ Line 23: / Line 39: @@
 ...
+# and here is our application and output (one extra character per second)
+[hmeij@n33 blcr]$ ./t-20001030-01
+*
+**
+***
+****
+*****
+******
+...
+</code>
+So now lets run this under BLCR and observe what happens.  After we define the proper environment, we use ''cr_run'' to launch our application. Standard output and error are written into the output file ''context''. We then observe the PID of our process and use ''cr_checkpoint'' to write a checkpoint file and immediately terminate the process.
+<code>
+# start application
 [hmeij@n33 blcr]$ cr_run ./t-20001030-01 > context 2>&1 &
 [1] 12789
+# observe PID
 [hmeij@n33 blcr]$ ps
   PID TTY          TIME CMD
@@ Line 32: / Line 65: @@
 pts/29   00:00:00 bash
+# wait, then checkpoint and terminate process
 [hmeij@n33 blcr]$ sleep 30
 [hmeij@n33 blcr]$ cr_checkpoint --term 12789
 [1]+  Terminated              cr_run ./t-20001030-01 > context 2>&1
+# save the output
 [hmeij@n33 blcr]$ mv context context.save
+</code>
+Ok.  Next we use ''cr_restart'' to restart our application by pointing it to the checkpoint file generated.  Then we'll wait a bit and terminate the restart.
+<code>
+# restart in background
 [hmeij@n33 blcr]$ cr_restart ./context.12789 > context 2>&1 &
 [1] 13579
+# wait and terminate the restart
 [hmeij@n33 blcr]$ sleep 30
 [hmeij@n33 blcr]$ kill %1
 [1]+  Terminated              cr_restart ./context.12789 > context 2>&1
+</code>
+So what we're interested in is the boundary between first termination and subsequent restart.  It alooks like this:
+<code>
 [hmeij@n33 blcr]$ tail context.save
@@ Line 68: / Line 117: @@
 ************************************************************
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
+# pretty nifty!
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
+# but be forewarned that there are binary characters lurking at this boundary
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
+# you can strip them out with ''sed'' or ''tr''
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
+# it looks like this
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
+^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@***************************************************
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
+</code>
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
-^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@***************************************************
@@ Line 133: / Line 169: @@
 lrwxrwxrwx 1 hmeij its 40 Oct 31 11:06 1383231850.62322.shell ->
 /sanscratch/62322/1383231850.62322.shell
+\\
+**[[cluster:0|Back]]**

DokuWiki

User Tools

Site Tools

Differences

Page Tools