\\
**[[cluster:0|Back]]**
===== Cuda Checkpoint =====
* source and user manual at
* https://github.com/NVIDIA/cuda-checkpoint
* https://developer.nvidia.com/blog/checkpointing-cuda-applications-with-criu/
* located in n91:/projects/cuda-checkpoint-main/580.65.06
* utility linked in n91:/usr/local/bin/cuda-checkpoint
* do not find a way to download 550 features version
* for our mwgpu256 gpu compute nodes
Put your checkpoints in /sanscratch/checkpoints/$SLURM_JOB_ID directory. You need to make that directory in your submit script but do not have to clean it up. Checkpoints older than 120 days are automatically removed.
[root@n91 ~]# rpm -qa criu
criu-3.18-5.module+el8.10.0+2001+6a33db9f.x86_64
[root@n91 ~]# rpm -ql criu
/run/criu
/usr/lib/.build-id
/usr/lib/.build-id/9b
/usr/lib/.build-id/9b/603c4fff3830838f46feaf4bd1204adc49421e
/usr/lib/tmpfiles.d/criu.conf
/usr/sbin/criu
/usr/share/doc/criu
/usr/share/doc/criu/COPYING
/usr/share/doc/criu/README.md
/usr/share/man/man8/criu.8.gz
An example of performing a checkpoint/restore is\\
/projects/cuda-checkpoint-main/580.65.06/src/example.sh\\
the counter does not spit out output so hard to test....\\
Actually works a little different.
To get these output files, the process follows these standard steps:
1. Compile and Run
nvcc counter.cu -o counter
./counter & # nvidia-smi confirms counter is on GPU
PID=$!; echo $PID
2262128
echo hello | nc -u localhost 10000 -w 1
2. Suspend GPU Work:
You use cuda-checkpoint to freeze and extract
the GPU memory layout into host memory space
...this terminates the counter on GPU
cuda-checkpoint --toggle --pid $PID
3. Dump to Files:
You use CRIU to write the combined state out
into your output directory
...this terminates the counter on CPU
criu dump --shell-job --images-dir demo --tree $PID --unprivileged
Once completed, the target counter process is terminated,
and your demo/ directory
will be packed with the binary image (.img) files representing
the exact frozen state of your application.
# as root set
# Note: CRIU, Check/Restore In Userspace; so why is this not the default behavior?
[root@n91 ~]# setcap cap_checkpoint_restore+eip /usr/bin/criu
# as user, needed?
[hmeij@n91 src]$ SKIP_SETCAP=true
[hmeij@n91 src]$ SKIP_CHOWN=true
Running as non-root requires '--unprivileged'
Please consult the documentation for limitations when running as non-root
[hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID --unprivileged
[hmeij@n91 src]$ ll demo
total 608388
-rw-r--r-- 1 hmeij its 2713 Oct 6 10:57 core-1583155.img
-rw-r--r-- 1 hmeij its 1561 Oct 6 10:57 core-1583158.img
-rw-r--r-- 1 hmeij its 1763 Oct 6 10:57 core-1583163.img
-rw-r--r-- 1 hmeij its 296 Oct 6 10:57 fdinfo-2.img
-rw-r--r-- 1 hmeij its 1791 Oct 6 10:57 files.img
-rw-r--r-- 1 hmeij its 18 Oct 6 10:57 fs-1583155.img
-rw-r--r-- 1 hmeij its 36 Oct 6 10:57 ids-1583155.img
-rw-r--r-- 1 hmeij its 44 Oct 6 10:57 inventory.img
-rw-r--r-- 1 hmeij its 148 Oct 6 10:57 memfd.img
-rw-r--r-- 1 hmeij its 2942 Oct 6 10:57 mm-1583155.img
-rw-r--r-- 1 hmeij its 11832 Oct 6 10:57 pagemap-1583155.img
-rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959861.img
-rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959862.img
-rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959863.img
-rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959864.img
-rw-r--r-- 1 hmeij its 2097152 Oct 6 10:57 pages-1.img
-rw-r--r-- 1 hmeij its 2097152 Oct 6 10:57 pages-2.img
-rw-r--r-- 1 hmeij its 2097152 Oct 6 10:57 pages-3.img
-rw-r--r-- 1 hmeij its 2097152 Oct 6 10:57 pages-4.img
-rw-r--r-- 1 hmeij its 614502400 Oct 6 10:57 pages-5.img
-rw-r--r-- 1 hmeij its 53 Oct 6 10:57 pipes-data.img
-rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pstree.img
-rw-r--r-- 1 hmeij its 12 Oct 6 10:57 seccomp.img
-rw-r--r-- 1 hmeij its 56 Oct 6 10:57 stats-dump
-rw-r--r-- 1 hmeij its 38 Oct 6 10:57 timens-0.img
-rw-r--r-- 1 hmeij its 213 Oct 6 10:57 tty-info.img
[hmeij@n91 src]$ criu restore --shell-job --restore-detached --images-dir demo --unprivileged
2262128: Error (criu/tty.c:831): tty: Can't set tty params on 0xf: Operation not permitted
2262128: Error (criu/files.c:1213): Unable to open fd=0 id=0xf
Error (criu/cr-restore.c:2547): Restoring FAILED.
BUMMER!
Note: running the criu dump/restore commands via sudo
does not fix the issue. "In Userspace" !
* Good write up and exercise. **This does work with sudo on CPU side**
* https://www.marcusfolkesson.se/blog/checkpoint-restore/
# counter2 is counting in console 1 until the dump happens
42
43
44
45
46
47
48
49
50
51
Killed
# the dump populates demo2
[hmeij@n91 demo2]$ sudo criu dump -t 1595356 --shell-job
[sudo] password for hmeij:
Can't check shutdown state of inet socket
Warn (compel/arch/x86/src/lib/infect.c:356): Will restore 1595356 with interrupted system call
[hmeij@n91 demo2]$ ll
total 164
-rw-r--r-- 1 root root 3526 Oct 6 14:59 cgroup.img
-rw-r--r-- 1 root root 2477 Oct 6 14:59 core-1595356.img
-rw-r--r-- 1 root root 44 Oct 6 14:59 fdinfo-2.img
-rw-r--r-- 1 root root 476 Oct 6 14:59 files.img
-rw-r--r-- 1 root root 18 Oct 6 14:59 fs-1595356.img
-rw-r--r-- 1 root root 36 Oct 6 14:59 ids-1595356.img
-rw-r--r-- 1 root root 46 Oct 6 14:59 inventory.img
-rw-r--r-- 1 root root 785 Oct 6 14:59 mm-1595356.img
-rw-r--r-- 1 root root 197 Oct 6 14:59 pagemap-1595356.img
-rw-r--r-- 1 root root 110592 Oct 6 14:59 pages-1.img
-rw-r--r-- 1 root root 30 Oct 6 14:59 pstree.img
-rw-r--r-- 1 root root 12 Oct 6 14:59 seccomp.img
-rw-r--r-- 1 root root 46 Oct 6 14:59 stats-dump
-rw-r--r-- 1 root root 38 Oct 6 14:59 timens-0.img
-rw-r--r-- 1 root root 203 Oct 6 14:59 tty-info.img
# the restore does not miss a beat in console2
[hmeij@n91 demo2]$ sudo criu restore --shell-job
52
53
54
55
56
57
58
59
60
...
\\
**[[cluster:0|Back]]**