User Tools

Site Tools


cluster:234

This is an old revision of the document!



Back

Cuda Checkpoint

  • located in n91:/projects/cuda-checkpoint-main/580.65.06
  • utility linked in n91:/usr/local/bin/cuda-checkpoint
  • do not find a way to download 550 features version
  • for our mwgpu256 gpu compute nodes

Put your checkpoints in /sanscratch/checkpoints/$SLURM_JOB_ID directory. You need to make that directory in your submit script but do not have to clean it up. Checkpoints older than 120 days are automatically removed.

[root@n91 ~]# rpm -qa criu
criu-3.18-5.module+el8.10.0+2001+6a33db9f.x86_64

[root@n91 ~]# rpm -ql criu
/run/criu
/usr/lib/.build-id
/usr/lib/.build-id/9b
/usr/lib/.build-id/9b/603c4fff3830838f46feaf4bd1204adc49421e
/usr/lib/tmpfiles.d/criu.conf
/usr/sbin/criu
/usr/share/doc/criu
/usr/share/doc/criu/COPYING
/usr/share/doc/criu/README.md
/usr/share/man/man8/criu.8.gz

An example of performing a checkpoint/restore is
/projects/cuda-checkpoint-main/580.65.06/src/example.sh
the counter does not spit out output so hard to test….

Actually works a little different.

To get these output files, the process follows these standard steps:

1. Compile and Run

nvcc counter.cu -o counter

./counter &
PID=$!


2. Suspend GPU Work: 
You use cuda-checkpoint to freeze and extract 
the GPU memory layout into host memory space

cuda-checkpoint --toggle --pid $PID

3. Dump to Files: 
You use CRIU to write the combined state out 
into your output directory

criu dump --shell-job --images-dir demo --tree $PID

Once completed, the target counter process is terminated, 
and your demo/ directory 
will be packed with the binary image (.img) files representing 
the exact frozen state of your application.



[hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID
CRIU needs to have the CAP_SYS_ADMIN or the CAP_CHECKPOINT_RESTORE capability:

# as root set
# Note: CRIU, Check/Restore In Userspace; so why is this not the default behavior?

[root@n91 ~]# setcap cap_checkpoint_restore+eip /usr/bin/criu

# as user, needed?
[hmeij@n91 src]$ SKIP_SETCAP=true
[hmeij@n91 src]$ SKIP_CHOWN=true

Running as non-root requires '--unprivileged'
Please consult the documentation for limitations when running as non-root

[hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID --unprivileged

[hmeij@n91 src]$ ll demo
total 608388
-rw-r--r-- 1 hmeij its      2713 Oct  6 10:57 core-1583155.img
-rw-r--r-- 1 hmeij its      1561 Oct  6 10:57 core-1583158.img
-rw-r--r-- 1 hmeij its      1763 Oct  6 10:57 core-1583163.img
-rw-r--r-- 1 hmeij its       296 Oct  6 10:57 fdinfo-2.img
-rw-r--r-- 1 hmeij its      1791 Oct  6 10:57 files.img
-rw-r--r-- 1 hmeij its        18 Oct  6 10:57 fs-1583155.img
-rw-r--r-- 1 hmeij its        36 Oct  6 10:57 ids-1583155.img
-rw-r--r-- 1 hmeij its        44 Oct  6 10:57 inventory.img
-rw-r--r-- 1 hmeij its       148 Oct  6 10:57 memfd.img
-rw-r--r-- 1 hmeij its      2942 Oct  6 10:57 mm-1583155.img
-rw-r--r-- 1 hmeij its     11832 Oct  6 10:57 pagemap-1583155.img
-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959861.img
-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959862.img
-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959863.img
-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959864.img
-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-1.img
-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-2.img
-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-3.img
-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-4.img
-rw-r--r-- 1 hmeij its 614502400 Oct  6 10:57 pages-5.img
-rw-r--r-- 1 hmeij its        53 Oct  6 10:57 pipes-data.img
-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pstree.img
-rw-r--r-- 1 hmeij its        12 Oct  6 10:57 seccomp.img
-rw-r--r-- 1 hmeij its        56 Oct  6 10:57 stats-dump
-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 timens-0.img
-rw-r--r-- 1 hmeij its       213 Oct  6 10:57 tty-info.img


[hmeij@n91 src]$ criu restore --shell-job --restore-detached --images-dir demo --unprivileged

Warn  (criu/kerndat.c:1103): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file
Warn  (criu/kerndat.c:1421): Can't get pidfd
Warn  (criu/kerndat.c:1103): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file
1583155: Error (criu/tty.c:872): tty: Can't setup uid 0 gid 5 on 0xf: Operation not permitted
1583155: Error (criu/files.c:1213): Unable to open fd=0 id=0xf
Error (criu/cr-restore.c:2547): Restoring FAILED.
Error (criu/cr-restore.c:1514): 1583155 killed by signal 9: Killed

BUMMER!


Back

cluster/234.1791309157.txt.gz · Last modified: by hmeij07