User Tools

Site Tools


cluster:234

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
cluster:234 [2026/10/05 19:18] – [Cuda Checkpoint] hmeij07cluster:234 [2026/10/09 13:09] (current) – hmeij07
Line 6: Line 6:
   * source and user manual at   * source and user manual at
   * https://github.com/NVIDIA/cuda-checkpoint   * https://github.com/NVIDIA/cuda-checkpoint
 +  * https://developer.nvidia.com/blog/checkpointing-cuda-applications-with-criu/
  
   * located in n91:/projects/cuda-checkpoint-main/580.65.06   * located in n91:/projects/cuda-checkpoint-main/580.65.06
   * utility linked in n91:/usr/local/bin/cuda-checkpoint   * utility linked in n91:/usr/local/bin/cuda-checkpoint
 +
 +  * do not find a way to download 550 features version 
 +  * for our mwgpu256 gpu compute nodes
 +
 +Put your checkpoints in /sanscratch/checkpoints/$SLURM_JOB_ID directory. You need to make that directory in your submit script but do not have to clean it up. Checkpoints older than 120 days are automatically removed. 
  
 <code> <code>
Line 32: Line 38:
  /projects/cuda-checkpoint-main/580.65.06/src/example.sh\\  /projects/cuda-checkpoint-main/580.65.06/src/example.sh\\
 the counter does not spit out output so hard to test....\\ the counter does not spit out output so hard to test....\\
 +
 +Actually works a little different.
 +
 +<code>
 +
 +To get these output files, the process follows these standard steps:
 +
 +1. Compile and Run
 +
 +nvcc counter.cu -o counter
 +
 +./counter & # nvidia-smi confirms counter is on GPU
 +
 +PID=$!; echo $PID
 +2262128
 +
 +echo hello | nc -u localhost 10000 -w 1
 +
 +2. Suspend GPU Work: 
 +You use cuda-checkpoint to freeze and extract 
 +the GPU memory layout into host memory space
 +...this terminates the counter on GPU
 +
 +cuda-checkpoint --toggle --pid $PID
 +
 +3. Dump to Files: 
 +You use CRIU to write the combined state out 
 +into your output directory
 +...this terminates the counter on CPU
 +
 +criu dump --shell-job --images-dir demo --tree $PID --unprivileged
 +
 +Once completed, the target counter process is terminated, 
 +and your demo/ directory 
 +will be packed with the binary image (.img) files representing 
 +the exact frozen state of your application.
 +
 +# as root set
 +# Note: CRIU, Check/Restore In Userspace; so why is this not the default behavior?
 +
 +[root@n91 ~]# setcap cap_checkpoint_restore+eip /usr/bin/criu
 +
 +# as user, needed?
 +[hmeij@n91 src]$ SKIP_SETCAP=true
 +[hmeij@n91 src]$ SKIP_CHOWN=true
 +
 +Running as non-root requires '--unprivileged'
 +Please consult the documentation for limitations when running as non-root
 +
 +[hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID --unprivileged
 +
 +[hmeij@n91 src]$ ll demo
 +total 608388
 +-rw-r--r-- 1 hmeij its      2713 Oct  6 10:57 core-1583155.img
 +-rw-r--r-- 1 hmeij its      1561 Oct  6 10:57 core-1583158.img
 +-rw-r--r-- 1 hmeij its      1763 Oct  6 10:57 core-1583163.img
 +-rw-r--r-- 1 hmeij its       296 Oct  6 10:57 fdinfo-2.img
 +-rw-r--r-- 1 hmeij its      1791 Oct  6 10:57 files.img
 +-rw-r--r-- 1 hmeij its        18 Oct  6 10:57 fs-1583155.img
 +-rw-r--r-- 1 hmeij its        36 Oct  6 10:57 ids-1583155.img
 +-rw-r--r-- 1 hmeij its        44 Oct  6 10:57 inventory.img
 +-rw-r--r-- 1 hmeij its       148 Oct  6 10:57 memfd.img
 +-rw-r--r-- 1 hmeij its      2942 Oct  6 10:57 mm-1583155.img
 +-rw-r--r-- 1 hmeij its     11832 Oct  6 10:57 pagemap-1583155.img
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959861.img
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959862.img
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959863.img
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959864.img
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-1.img
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-2.img
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-3.img
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-4.img
 +-rw-r--r-- 1 hmeij its 614502400 Oct  6 10:57 pages-5.img
 +-rw-r--r-- 1 hmeij its        53 Oct  6 10:57 pipes-data.img
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pstree.img
 +-rw-r--r-- 1 hmeij its        12 Oct  6 10:57 seccomp.img
 +-rw-r--r-- 1 hmeij its        56 Oct  6 10:57 stats-dump
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 timens-0.img
 +-rw-r--r-- 1 hmeij its       213 Oct  6 10:57 tty-info.img
 +
 +
 +[hmeij@n91 src]$ criu restore --shell-job --restore-detached --images-dir demo --unprivileged
 +2262128: Error (criu/tty.c:831): tty: Can't set tty params on 0xf: Operation not permitted
 +2262128: Error (criu/files.c:1213): Unable to open fd=0 id=0xf
 +Error (criu/cr-restore.c:2547): Restoring FAILED.
 +
 +
 +BUMMER!
 +
 +Note: running the criu dump/restore commands via sudo
 +does not fix the issue. "In Userspace" !
 +
 +</code>
 +
 +  * Good write up and exercise. **This does work with sudo on CPU side**
 +  * https://www.marcusfolkesson.se/blog/checkpoint-restore/
 +
 +<code>
 +
 +# counter2 is counting in console 1 until the dump happens
 +
 +42
 +43
 +44
 +45
 +46
 +47
 +48
 +49
 +50
 +51
 +Killed
 +
 +# the dump populates demo2
 +
 +[hmeij@n91 demo2]$ sudo criu dump -t 1595356 --shell-job
 +[sudo] password for hmeij:
 +Can't check shutdown state of inet socket
 +Warn  (compel/arch/x86/src/lib/infect.c:356): Will restore 1595356 with interrupted system call
 +
 +[hmeij@n91 demo2]$ ll
 +total 164
 +-rw-r--r-- 1 root root   3526 Oct  6 14:59 cgroup.img
 +-rw-r--r-- 1 root root   2477 Oct  6 14:59 core-1595356.img
 +-rw-r--r-- 1 root root     44 Oct  6 14:59 fdinfo-2.img
 +-rw-r--r-- 1 root root    476 Oct  6 14:59 files.img
 +-rw-r--r-- 1 root root     18 Oct  6 14:59 fs-1595356.img
 +-rw-r--r-- 1 root root     36 Oct  6 14:59 ids-1595356.img
 +-rw-r--r-- 1 root root     46 Oct  6 14:59 inventory.img
 +-rw-r--r-- 1 root root    785 Oct  6 14:59 mm-1595356.img
 +-rw-r--r-- 1 root root    197 Oct  6 14:59 pagemap-1595356.img
 +-rw-r--r-- 1 root root 110592 Oct  6 14:59 pages-1.img
 +-rw-r--r-- 1 root root     30 Oct  6 14:59 pstree.img
 +-rw-r--r-- 1 root root     12 Oct  6 14:59 seccomp.img
 +-rw-r--r-- 1 root root     46 Oct  6 14:59 stats-dump
 +-rw-r--r-- 1 root root     38 Oct  6 14:59 timens-0.img
 +-rw-r--r-- 1 root root    203 Oct  6 14:59 tty-info.img
 +
 +# the restore does not miss a beat in console2
 +
 +[hmeij@n91 demo2]$ sudo criu restore --shell-job
 +52
 +53
 +54
 +55
 +56
 +57
 +58
 +59
 +60
 +...
 +
 +
 +</code>
  
 \\ \\
cluster/234.1791227895.txt.gz · Last modified: by hmeij07