cluster:234
This is an old revision of the document!
Cuda Checkpoint
- source and user manual at
- located in n91:/projects/cuda-checkpoint-main/580.65.06
- utility linked in n91:/usr/local/bin/cuda-checkpoint
- do not find a way to download 550 features version
- for our mwgpu256 gpu compute nodes
Put your checkpoints in /sanscratch/checkpoints/$SLURM_JOB_ID directory. You need to make that directory in your submit script but do not have to clean it up. Checkpoints older than 120 days are automatically removed.
[root@n91 ~]# rpm -qa criu criu-3.18-5.module+el8.10.0+2001+6a33db9f.x86_64 [root@n91 ~]# rpm -ql criu /run/criu /usr/lib/.build-id /usr/lib/.build-id/9b /usr/lib/.build-id/9b/603c4fff3830838f46feaf4bd1204adc49421e /usr/lib/tmpfiles.d/criu.conf /usr/sbin/criu /usr/share/doc/criu /usr/share/doc/criu/COPYING /usr/share/doc/criu/README.md /usr/share/man/man8/criu.8.gz
An example of performing a checkpoint/restore is
/projects/cuda-checkpoint-main/580.65.06/src/example.sh
the counter does not spit out output so hard to test….
Actually works a little different.
To get these output files, the process follows these standard steps: 1. Compile and Run nvcc counter.cu -o counter ./counter & PID=$! 2. Suspend GPU Work: You use cuda-checkpoint to freeze and extract the GPU memory layout into host memory space cuda-checkpoint --toggle --pid $PID 3. Dump to Files: You use CRIU to write the combined state out into your output directory criu dump --shell-job --images-dir demo --tree $PID Once completed, the target counter process is terminated, and your demo/ directory will be packed with the binary image (.img) files representing the exact frozen state of your application. [hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID CRIU needs to have the CAP_SYS_ADMIN or the CAP_CHECKPOINT_RESTORE capability: # as root set # Note: CRIU, Check/Restore In Userspace; so why is this not the default behavior? [root@n91 ~]# setcap cap_checkpoint_restore+eip /usr/bin/criu # as user, needed? [hmeij@n91 src]$ SKIP_SETCAP=true [hmeij@n91 src]$ SKIP_CHOWN=true Running as non-root requires '--unprivileged' Please consult the documentation for limitations when running as non-root [hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID --unprivileged [hmeij@n91 src]$ ll demo total 608388 -rw-r--r-- 1 hmeij its 2713 Oct 6 10:57 core-1583155.img -rw-r--r-- 1 hmeij its 1561 Oct 6 10:57 core-1583158.img -rw-r--r-- 1 hmeij its 1763 Oct 6 10:57 core-1583163.img -rw-r--r-- 1 hmeij its 296 Oct 6 10:57 fdinfo-2.img -rw-r--r-- 1 hmeij its 1791 Oct 6 10:57 files.img -rw-r--r-- 1 hmeij its 18 Oct 6 10:57 fs-1583155.img -rw-r--r-- 1 hmeij its 36 Oct 6 10:57 ids-1583155.img -rw-r--r-- 1 hmeij its 44 Oct 6 10:57 inventory.img -rw-r--r-- 1 hmeij its 148 Oct 6 10:57 memfd.img -rw-r--r-- 1 hmeij its 2942 Oct 6 10:57 mm-1583155.img -rw-r--r-- 1 hmeij its 11832 Oct 6 10:57 pagemap-1583155.img -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959861.img -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959862.img -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959863.img -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959864.img -rw-r--r-- 1 hmeij its 2097152 Oct 6 10:57 pages-1.img -rw-r--r-- 1 hmeij its 2097152 Oct 6 10:57 pages-2.img -rw-r--r-- 1 hmeij its 2097152 Oct 6 10:57 pages-3.img -rw-r--r-- 1 hmeij its 2097152 Oct 6 10:57 pages-4.img -rw-r--r-- 1 hmeij its 614502400 Oct 6 10:57 pages-5.img -rw-r--r-- 1 hmeij its 53 Oct 6 10:57 pipes-data.img -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pstree.img -rw-r--r-- 1 hmeij its 12 Oct 6 10:57 seccomp.img -rw-r--r-- 1 hmeij its 56 Oct 6 10:57 stats-dump -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 timens-0.img -rw-r--r-- 1 hmeij its 213 Oct 6 10:57 tty-info.img [hmeij@n91 src]$ criu restore --shell-job --restore-detached --images-dir demo --unprivileged Warn (criu/kerndat.c:1103): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file Warn (criu/kerndat.c:1421): Can't get pidfd Warn (criu/kerndat.c:1103): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file 1583155: Error (criu/tty.c:872): tty: Can't setup uid 0 gid 5 on 0xf: Operation not permitted 1583155: Error (criu/files.c:1213): Unable to open fd=0 id=0xf Error (criu/cr-restore.c:2547): Restoring FAILED. Error (criu/cr-restore.c:1514): 1583155 killed by signal 9: Killed BUMMER!
- Good write up and exercise
cluster/234.1791307893.txt.gz · Last modified: by hmeij07
