DMTCP Checkpoint/Restart allows one to transparently checkpoint to disk a distributed computation. It works under Linux, with no modifications to the Linux kernel nor to the application binaries. It can be used by unprivileged users (no root privilege needed). One can later restart from a checkpoint, or even migrate the processes by moving the checkpoint files to another host prior to restarting.
This is a replacement for THE BLCR methods we used BLCR Checkpoint in OL3 -serial or BLCR Checkpoint in OL3 - parallel … BLCR is not being developed anymore. Today's brief power outage removed the BLCR kernel module HPCC wide. So learn DMTCP. I have not provided wrappers but you can follow the same logic as we used with BLCR.
Write your checkpoint files in
/sanscratch/checkpoints/JOBPID so it does not add into your quota. The scheduler will not create this directory for you, you must do this in your submit job. Directories will automatically be delete if 120 days old.
# sample steps to put in your scripts # make a directory (first one done by scheduler, second one done by your script) mkdir -p /sanscratch/111 /sanscratch/checkpoints/111 cd /sanscratch/111 # invoke sample rpogram which generates one line of output time ./a.out + Done time ./a.out real 64m0.513s user 64m0.813s sys 0m0.022s total 1108 -rwxr--r-- 1 hmeij its 1126428 Jan 14 14:36 a.out -rw-r--r-- 1 hmeij its 49 Jan 14 15:41 fid.txt [hmeij@cottontail2 111]$ cat fid.txt 0.999578495067887 7.978578277139264E-005 # for example, with 24 hours checkpoint interval # launch new coordinator on random port, log port, 24 hour checkpoints # make sure you create destination dir for checkpoints dmtcp_launch --new-coordinator \ --coord-port 0 --port-file port.txt --interval 86400 \ --ckptdir /sanscratch/checkpoints/111 \ time ./a.out # run command above with interval 300 (every 5 mins) [hmeij@cottontail2 111]$ ps PID TTY TIME CMD 20004 pts/1 00:00:00 time 20008 pts/1 00:00:00 dmtcp_coordinat 20010 pts/1 00:03:02 a.out 20060 pts/1 00:00:00 ps 20891 pts/1 00:00:00 bash # the coordinator to which application process was attached 1 S hmeij 20008 1 0 80 0 - 4665 ep_pol 07:46 pts/1 00:00:00 \\ /usr/bin/dmtcp_coordinator --quiet --exit-on-last --daemon # the random port (in case somebody else is also checkpointing on this host [hmeij@cottontail2 111]$ cat port.txt 38913 # 5 mins checkpoints, little impact [hmeij@cottontail2 111]$ sleep 6m; ll /sanscratch/checkpoints/111/ total 2960 -rw------- 1 hmeij its 3011726 Jan 16 13:53 ckpt_a.out_24945f6ae3823bbf-40000-fb2d136b9b544.dmtcp\\ -rwxr--r-- 1 hmeij its 12440 Jan 16 13:53 dmtcp_restart_script_24945f6ae3823bbf-40000-fb2d11ea62fa0.sh\\ lrwxrwxrwx 1 hmeij its 60 Jan 16 13:53 dmtcp_restart_script.sh -> dmtcp_restart_script_24945f6ae3823bbf-40000-fb2d11ea62fa0.sh\\ [hmeij@cottontail2 111]$ real 63m57.287s user 63m54.703s sys 0m0.444s # lets try a termination of processes and restart [hmeij@cottontail2 111]$ time dmtcp_launch \\ --new-coordinator --coord-port 0 --port-file port.txt \\ -i 300 --ckptdir /sanscratch/checkpoints/111 ./a.out &  29201 [hmeij@cottontail2 111]$ ps PID TTY TIME CMD 20891 pts/1 00:00:00 bash 29201 pts/1 00:00:00 bash 29202 pts/1 00:00:01 a.out 29210 pts/1 00:00:00 dmtcp_coordinat 29212 pts/1 00:00:00 ps # terminate half way through [hmeij@cottontail2 111]$ sleep 32m; kill -9 29202 29210 # restart from checkpoints directory (or copy files into desired location) cd /sanscratch/checkpoints/111 ./dmtcp_restart_script.sh # ps 0 S hmeij 20891 20890 -bash 0 R hmeij 31136 20891 [DMTCP:a.out] 1 S hmeij 31201 1 /usr/bin/dmtcp_coordinator --quiet --exit-on-last --daemon # You must make sure the old directory and file exists, otherwise  ERROR at fileconnection.cpp:737 in refill; REASON='JASSERT(jalib::Filesystem::FileExists(_path)) failed' _path = /sanscratch/111/fid.txt Message: File not found. a.out (40000): Terminating... # that is because at checkpoint time that file was opened by a.out # The process will pick up from last checkpoint # and write output to original work directory
From tarball. Handy reference.
# Overview of DMTCP To install DMTCP, see [INSTALL.md](INSTALL.md). ## Concepts: DMTCP Checkpoint/Restart allows one to transparently checkpoint to disk a distributed computation. It works under Linux, with no modifications to the Linux kernel nor to the application binaries. It can be used by unprivileged users (no root privilege needed). One can later restart from a checkpoint, or even migrate the processes by moving the checkpoint files to another host prior to restarting. There is one DMTCP coordinator for each computation that you wish to checkpoint. By specifying `--coord-host` and `--coord-port` (or the environment variables `DMTCP_COORD_HOST` and `DMTCP_COORD_PORT`), you can add a process to a coordinator different from the default coordinator. If you don't specify, the default coordinator is always at (`localhost:7779`). A DMTCP coordinator process is started on one host. Application binaries are started under the `dmtcp_launch` command, causing them to connect to the coordinator upon startup. As threads are spawned, child processes are forked, remote processes are spawned via ssh, libraries are dynamically loaded, DMTCP transparently and automatically tracks them. By default, DMTCP uses gzip to compress the checkpoint images. This can be turned off (`dmtcp_launch --no-gzip` ; or setting an environment variable to 0: `DMTCP_GZIP=0`). This will be faster, and if your memory is dominated by incompressible data, this can be helpful. Gzip can add seconds for large checkpoint images. Typically, checkpoint and restart is less than one second without gzip. A DMTCP checkpoint image includes any libraries (`.so` files) that it may have been using. This strategy is used for greater portability of the checkpoint images --- and in some cases, it even allows migration of the checkpoint images (and hence, processes) to hosts with different Linux distributions, different Linux kernels, etc. To run a program with checkpointing: 1. Run `dmtcp_coordinator` in a separate terminal/window ``` bin/dmtcp_coordinator ``` 2. In separate terminal(s), replace each command(s) with `dmtcp_launch [command]` ``` bin/dmtcp_launch ./a.out ``` 3. To checkpoint, type `c<return>` into `dmtcp_coordinator` In `dmtcp_coordinator` window: ``` h<return> for help c<return> for checkpoint l<return> for list of processes to be checkpointed k<return> to kill processes to be checkpointed q<return> to kill processes to be checkpointed and quit the coordinator ``` 4. Restart: Creating a checkpoint causes the `dmtcp_coordinator` to write a script, `dmtcp_restart_script.sh`, along with a checkpoint file (file type: `.dmtcp`) for each client process. The simplest way to restart a previously checkpointed computation is: ``` ./bin/dmtcp_restart_script.sh ``` * `./dmtcp_restart_script.sh` usually works "as is", but it can be edited. * Alternatively, if all processes were on the same processor, and there were no .dmtcp files prior to this checkpoint: ``` ./bin/dmtcp_restart ckpt_*.dmtcp ``` ## Convenience commands and debugging restarted processes: 1. Help exists: ```bash # bin/dmtcp_coordinator --help ; bin/dmtcp_launch --help ; # bin/dmtcp_command --help, etc. # Automatically start a coordinator in background bin/dmtcp_launch ./a.out & # Checkpoint all processes of the default coordinator bin/dmtcp_command --checkpoint # Kill a.out, and optionally kill coordinator process bin/dmtcp_command --kill # Kill a.out, and optionally kill coordinator process bin/dmtcp_command --quit # Restart directly from local checkpoint images (.dmtcp files) ./dmtcp_restart_script.sh # Or else, directly restart from the ckpt images in the current directory. # (Be sure there are no old ckpt_a.out_*.dmtcp files. # Ensure that the restarted process is running, and not suspended.) bin/dmtcp_restart ckpt_a.out_*.dmtcp & # Have gdb attach to a restarted process, and debug # NOTE: You must specify 'mtcp_restart', not 'dmtcp_restart' gdb ./a.out `pgrep -n MTCP` # force a.out to exit any low level libraries and return to a known location # set a breakpoint on a common function and continue: (gdb) break write (gdb) continue ``` 2. To enable debug statements for DMTCP, configure with: `./configure --enable-debug` (or `./configure --help`, in general). The flag `--enable-debug` both prints to stderr and writes files. ``` $DMTCP_TMPDIR/dmtcp-$USER@$HOST/jassertlog.* ``` where `$DMTCP_TMPDIR` is `/tmp` by default on most distributions. In reading this, it's useful to know that DMTCP sets up barriers so that all processes proceed to the following states together during checkpoint: `RUNNING`, `SUSPENDED`, `FD_LEADER_ELECTION`, `DRAINED`, `CHECKPOINTED`, `REFILLED`, `RUNNING`. 3. `util/gdb-add-symbol-file` may be a useful debugging tool. It computes the arguments for the `add-symbol-file` command of gdb, to import symbol information about a dynamic library. It is most useful in combination with *-dbg Linux packages and prefix to `dmtcp_launch`: ``` env LD_LIBRARY_PATH=/usr/lib/debug dmtcp_launch ... ``` followed by `attach` in gdb. ## Command-line options: `dmtcp_launch`, `dmtcp_command`, and `dmtcp_restart` print their options when run with no command-line arguments. `dmtcp_coordinator` offers help when run (Type `h<return>` for help.). Options through environment variables: 1. `dmtcp_coordinator`: * `DMTCP_CHECKPOINT_INTERVAL=<time in seconds>` (default: `0`, disabled) * `DMTCP_COORD_PORT=<coordinator listener port>` (default: `7779`) * `DMTCP_CHECKPOINT_DIR=<where restart script is written>` (default: `./`) * `DMTCP_TMPDIR=<where temporary files are written>` (default: environment variable `TMPDIR` or `/tmp`) 2. `dmtcp_launch` / `dmtcp_restart`: * `DMTCP_COORD_HOST=<hostname where coordinator will run>` (default: `localhost`) * `DMTCP_COORD_PORT=<coordinator listener port>` (default: `7779`) * `DMTCP_GZIP=<0: disable compression of checkpoint image>` (default: `1`, compression enabled) * `DMTCP_CHECKPOINT_DIR=<location to store checkpoints>` (default: `./`) * `DMTCP_SIGCKPT=<internal signal number>` (default: `12(SIGUSR2)`) * `DMTCP_TMPDIR=<where temporary files are written>` (default: environment variable `TMPDIR` or `/tmp`) 3. `dmtcp_command: * `DMTCP_COORD_HOST=<hostname where coordinator will run>` (default: `localhost`) * `DMTCP_COORD_PORT=<coordinator listener port>` (default: `7779`) ## Adapting DMTCP to application requirements and to external environments: 1. Application control over checkpoints: * `dmtcp_is_enabled()` [Returns 0 (false) or 1 (true)] * `dmtcp_checkpoint()` [with return values to distinguish resume and restart] * `dmtcp_disable_ckpt()` [Temporarily delay checkpoint during critical section] * `dmtcp_enable_ckpt()` [Resume previously delayed checkpoint] [ See `test/plugin/applic-delayed-ckpt/` and `test/plugin/applic-initiated-ckpt/` for small, easily executed examples. ] 2. User-defined plugin modules may be added to DMTCP. For examples, see the `test/plugin` directory of the source distribution. * Plugins provide user hooks for many common events, including: checkpoint, resume, restart, thread-start, thread-exit, exec program, etc. * Plugins provide for user-defined wrappers around library calls and system calls, to allow users to virtualize the external environment. * Plugins provide a plublish/subscribe service by which multiple processes may exchange information at the time of restart. [ See `doc/plugin-tutorial.pdf` for further informaiton, and see `test/plugin` for several small, easily executed examples. ] ## Short notes: 1. A restarted process sees the shared libraries and environment variables that existed prior to checkpoint. These are contained in the .dmtcp checkpoint file. 2. At restart time, one can choose either to use the original `dmtcp_coordinator` or else to start a new coordinator. Each process restarted by the `dmtcp_restart` command needs to know the host and port used by `dmtcp_coordinator`. These default to `localhost` and port `7779`. The coordinator can be specified to use port 0, in which case the coordinator chooses arbitrary port, and prints it to stdout. Setting `DMTCP_COORD_PORT` in the environment seen by the four main commands (`dmtcp_coordinator`, `dmtcp_launch`, `dmtcp_restart` and `dmtcp_command`) will override the default port. Similarly, setting `DMTCP_COORD_HOST` for `dmtcp_launch` and `dmtcp_restart` is needed if they start on a different host than that of the coordinator. 3. It often works to migrate processes by moving the checkpoint files to another host and editing `dmtcp_restart_script.sh` prior to restarting. Whether it works is affected by how different are the corresponding versions for the kernel and glibc. 4. Checkpoint is implemented by sending a signal to each user thread. As with all well-written code, your system calls should be prepared for an error return of `EINTR` (interrupted, due to a simultaneous checkpoint invocation or other kernel activity), in which case you can call the system call again. 5. Matlab should be invoked without graphics, and to be extra safe, without the JVM. The -nodisplay and -nojvm flags for Matlab suffice: ``` bin/dmtcp_launch matlab -nodisplay -nojvm ``` 6. For Ubuntu/Debian Linux, ensure that `build-essential` is installed. For OpenSuse Linux, ensure that `linux-kernel-headers` is installed. 7. By default, successive checkpoints of the same process write to the same checkpoint image filename. If you prefer that successive checkpoint be written to distinct filenames, then use: ``` ./configure --enable-unique-checkpoint-filenames ``` 8. Using Ctrl+C to kill a computation could leave some stale processes that remain connected to the coordinator. This can affect the restart. To ensure that this isn't the case, verify that there are no stale processes from your computation using `dmtcp_command` or `dmtcp_coordinator`. An alternative to Ctrl+C is to use the kill command in `dmtcp_command` or `dmtcp_coordinator`. ## Checkpointing Open MPI Verify that `mpirun` works. Verify `dmtcp_launch`, `dmtcp_restart`, etc. commands are in your path: ``` ssh <REMOTE-HOST> which dmtcp_launch ``` If they are not in your path, adjust your shell initialization file to extend your path. Verify `ssh <REMOTE-HOST>` works without password otherwise do the following: ``` ssh-keygen -t dsa [accept default values] ssh-keygen -t rsa [accept default values] cat ~/.ssh/id*.pub >> ~/.ssh/authorized_keys ``` ``` make clean make make check dmtcp_launch mpirun ./hello_mpi dmtcp_command --checkpoint ./dmtcp_restart_script.sh ``` DMTCP uses SIGUSR2 as default and so do older versions of Open MPI. If you have an older version (e.g < 1.3), try choosing a different value of SIGNUM for DMTCP as follows: ``` dmtcp_launch --ckpt-signal <SIGNUM> mpirun ./hello_mpi ``` ## Using DMTCP with X-Window: Note that this method does not work with X extensions like OpenGL. If someone wishes to extend this method to OpenGL, we have some ideas for an approach that we can share. Also, this method does not currently successfully checkpoint an xterm, for reasons that we do not fully understand. We will look further into this later when time and resources permit. ### TightVNC Install TightVNC (either as a package from your Linux distro, or at: http://www.tightvnc.com with installation instructions at: http://www.tightvnc.com/doc/unix/README.txt If the server fails to start, you may need to specify the location of fonts on your system. Do this by editing the `vncserver` Perl script (which you put in your path above). Modify the `$fontPath` variable to point to your font directories. For example, you can list all of the subdirectories of `/usr/share/fonts/` in the `fontPath`. The processes started up automatically by the VNC server are listed in the `~/.vnc/xstartup` file. Use the following as your `~/.vnc/xstartup`, where we use the blackbox window manager and an `x_app` application as an example: ``` #!/bin/csh blackbox & x_app ``` You should test that you can use the `vncserver` and `vncviewer` now. This example uses desktop number 1: ``` vncserver :1 vncviewer localhost:1 # Kill vncviewer window manually, and then: vncserver -kill :1 ``` ### Launch TightVNC with DMTCP Make sure the executables `dmtcp_launch`, `dmtcp_coordinator`, `dmtcp_restart`, and `dmtcp_command` are in your path. Note that if the VNC server is killed without using the `vncserver -kill`, there will be some temporary files left over that prevent the server from restarting. If this occurs, remove them: ``` rm -rf /tmp/.X1-lock /tmp/.X11-unix/X1 ``` where X1 corresponds to starting the server on port 1. Now, start the VNC server under checkpointing control: ``` dmtcp_launch vncserver :1 ``` Use the VNC viewer to view your `x_app` application in the blackbox window manager: ``` vncviewer localhost:1 ``` Before checkpointing, close any xterm windows. Also, close the vncviewer itself. They can be reopened again after the checkpoint has completed. [Optional] To verify that vncserver is running under checkpoint control: ``` dmtcp_command h dmtcp_command s ``` To checkpoint the VNC server, `x_app`, and any other processes running under the VNC server, remove any old checkpoint files, and type: ``` rm -f ckpt_*.dmtcp dmtcp_command --checkpoint ``` This creates a file of the form `ckpt_*.dmtcp` for each process being checkpointed. To kill the vncviewer and restart, use the restart script: ``` vncserver -kill :1 # This script assumes dmtcp_restart is in your path. If not, # modify the script to replace dmtcp_restart by a full path to it. ./dmtcp_restart_script.sh ``` Alternatively, you may prefer to directly use the `dmtcp_restart` command: ``` vncserver -kill :1 dmtcp_restart ckpt_*.dmtcp ``` Note: if checkpointing doesn't fully complete, make sure you're not out of disk space, and that there are no other file system problems. ### Other applications with X Window support Certain applications, such as some shells, vim, etc., try to recognize mouse events from the X11 windows system. While DMTCP successfully checkpoints and restarts these applications, it does so by disabling the connection to X11. Mouse events are not recognized. ## Support for incremental and differential checkpoint using HBICT [HBICT (Hash Based Incremental Checkpointing Tool)](http://hbict.sourceforge.net/projects/hbict) provides DMTCP support for delta-compression (relative to the previous checkpoint) which is then additionally compressed using gzip. To enable it: 1. Download and install HBICT somewhere in your `PATH`. For example, to test it in a single user's account: ``` $ tar zxvf hbict-1.0.tar.gz; cd hbict-1.0; ./configure; make $ export PATH=$PWD/src:$PATH ``` To see the options of 'hbict', execute: ``` $ hbict --help ``` Note, in particular, the `-r` option to create a new full checkpoint from the existing delta-compressed checkpoint files. 2. In the DMTCP directory, configure and re-make DMTCP for HBICT: ``` $ ./configure --enable-delta-compression; make clean; make ``` 3. The `--no-hbict` and `--no-gzip` options of `dmtcp_launch` will control what type of compression is used. a. `dmtcp_launch <my_prog>` causes successive checkpoints to be delta-compressed relative to previous checkpoints (HBICT compression) and then also compressed with gzip. b. `dmtcp_launch --no-gzip <my_prog>` causes successive checkpoints to be delta-compressed relative to previous checkpoints (using HBICT compression). c. `dmtcp_launch --no-hbict <my_prog>` causes gzipped checkpoints to be created with no delta-compression. d. `dmtcp_launch --no-hbict --no-gzip <my_prog>` disables all compression. If delta-compression is used you'll see the several checkpoint files in your `DMTCP_CHECKPOINT_DIR` as in the following example using HBICT+gzip compression: ``` $ dmtcp_launch test/dmtcp1 $ ls -1 -sh 4,0K ckpt_dmtcp1_6cbb52b0-12935-4ddcf7a0.dmtcp 1,8M ckpt_dmtcp1_6cbb52b0-12935-4ddcf7a0.hbict.0 56K ckpt_dmtcp1_6cbb52b0-12935-4ddcf7a0.hbict.1 52K ckpt_dmtcp1_6cbb52b0-12935-4ddcf7a0.hbict.2 ... ``` In this example, the size of the delta-compressed checkpoints are 32 times smaller, since test/dmtcp1 is mostly unchanging text (code), with little data to change. The actualy delta-compression ratio depends strongly on the particular application. 4. To restart the application you'll need the file `ckpt_dmtcp1_6cbb52b0-12935-4ddcf7a0.dmtcp`, along with all the `*.hbict.N` files to be present in the same directory. Then execute: ``` $ dmtcp_restart ckpt_dmtcp1_6cbb52b0-12935-4ddcf7a0.dmtcp ``` and the `*.hbict.N` files will automaticaly be discovered and used by the HBICT tool. ## Directory layout: - `src`: DMTCP source code. - `src/plugin`: source for DMTCP internal plugins. - `jalib`: small pkg used by DMTCP for assertions, warnings, tracing code, etc. - `bin`: DMTCP binaries (`dmtcp_launch`, `dmtcp_restart`, `dmtcp_coordinator`, etc.) - `lib`: DMTCP internal libraries, including internal plugins that are not exposed to the end user. - `test`: Used by 'make check' - `test/plugin`: Simple examples for learning DMTCP plugins - `plugin`: (top-level directory of optional plugins; they must be invoked by a command line flag of `dmtcp_launch`) - `doc`: Random documentation on aspects of the DMTCP design Note especially: doc/plugin-tutorial.pdf - `include`: Contains dmtcp.h ; useful for third-party plugins - `contrib`: Contributed plugins and other add-ons. The more popular ones will eventually be migrated to the top-level plugin directory as they become mature; Contrib plugins are not built by default. - `util`: random utilities, useful mostly for experts