User Tools

Site Tools


cluster:234

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
cluster:234 [2026/10/06 14:43] – hmeij07cluster:234 [2026/10/09 13:09] (current) – hmeij07
Line 6: Line 6:
   * source and user manual at   * source and user manual at
   * https://github.com/NVIDIA/cuda-checkpoint   * https://github.com/NVIDIA/cuda-checkpoint
 +  * https://developer.nvidia.com/blog/checkpointing-cuda-applications-with-criu/
  
   * located in n91:/projects/cuda-checkpoint-main/580.65.06   * located in n91:/projects/cuda-checkpoint-main/580.65.06
Line 44: Line 45:
 To get these output files, the process follows these standard steps: To get these output files, the process follows these standard steps:
  
-1. Compile and Run:bash+1. Compile and Run
  
 nvcc counter.cu -o counter nvcc counter.cu -o counter
  
-./counter & +./counter & # nvidia-smi confirms counter is on GPU
-PID=$!+
  
 +PID=$!; echo $PID
 +2262128
  
-2. Suspend GPU Work: You use cuda-checkpoint to freeze and extract the GPU memory layout into host memory space+echo hello | nc -u localhost 10000 -w 1 
 + 
 +2. Suspend GPU Work:  
 +You use cuda-checkpoint to freeze and extract  
 +the GPU memory layout into host memory space 
 +...this terminates the counter on GPU
  
 cuda-checkpoint --toggle --pid $PID cuda-checkpoint --toggle --pid $PID
  
-3. Dump to Files: You use CRIU to write the combined state out into your output directory+3. Dump to Files:  
 +You use CRIU to write the combined state out  
 +into your output directory 
 +...this terminates the counter on CPU
  
-criu dump --shell-job --images-dir demo --tree $PID+criu dump --shell-job --images-dir demo --tree $PID --unprivileged
  
-Once completed, the target counter process is terminated, and your demo/ directory will be packed with the binary image (.img) files representing the exact frozen state of your application.+Once completed, the target counter process is terminated,  
 +and your demo/ directory  
 +will be packed with the binary image (.img) files representing  
 +the exact frozen state of your application.
  
-BUMMER+# as root set 
 +# Note: CRIU, Check/Restore In Userspace; so why is this not the default behavior?
  
-[hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID +[root@n91 ~]# setcap cap_checkpoint_restore+eip /usr/bin/criu
-CRIU needs to have the CAP_SYS_ADMIN or the CAP_CHECKPOINT_RESTORE capability: +
-setcap cap_checkpoint_restore+eip criu +
- +
-[hmeij@n91 src]$ setcap cap_checkpoint_restore+eip criu +
-unable to set CAP_SETFCAP effective capability: Operation not permitted+
  
 +# as user, needed?
 [hmeij@n91 src]$ SKIP_SETCAP=true [hmeij@n91 src]$ SKIP_SETCAP=true
- 
 [hmeij@n91 src]$ SKIP_CHOWN=true [hmeij@n91 src]$ SKIP_CHOWN=true
  
-[hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID +Running as non-root requires '--unprivileged' 
-CRIU needs to have the CAP_SYS_ADMIN or the CAP_CHECKPOINT_RESTORE capability: +Please consult the documentation for limitations when running as non-root 
-setcap cap_checkpoint_restore+eip criu+ 
 +[hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID --unprivileged 
 + 
 +[hmeij@n91 src]$ ll demo 
 +total 608388 
 +-rw-r--r-- 1 hmeij its      2713 Oct  6 10:57 core-1583155.img 
 +-rw-r--r-- 1 hmeij its      1561 Oct  6 10:57 core-1583158.img 
 +-rw-r--r-- 1 hmeij its      1763 Oct  6 10:57 core-1583163.img 
 +-rw-r--r-- 1 hmeij its       296 Oct  6 10:57 fdinfo-2.img 
 +-rw-r--r-- 1 hmeij its      1791 Oct  6 10:57 files.img 
 +-rw-r--r-- 1 hmeij its        18 Oct  6 10:57 fs-1583155.img 
 +-rw-r--r-- 1 hmeij its        36 Oct  6 10:57 ids-1583155.img 
 +-rw-r--r-- 1 hmeij its        44 Oct  6 10:57 inventory.img 
 +-rw-r--r-- 1 hmeij its       148 Oct  6 10:57 memfd.img 
 +-rw-r--r-- 1 hmeij its      2942 Oct  6 10:57 mm-1583155.img 
 +-rw-r--r-- 1 hmeij its     11832 Oct  6 10:57 pagemap-1583155.img 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959861.img 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959862.img 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959863.img 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959864.img 
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-1.img 
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-2.img 
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-3.img 
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-4.img 
 +-rw-r--r-- 1 hmeij its 614502400 Oct  6 10:57 pages-5.img 
 +-rw-r--r-- 1 hmeij its        53 Oct  6 10:57 pipes-data.img 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pstree.img 
 +-rw-r--r-- 1 hmeij its        12 Oct  6 10:57 seccomp.img 
 +-rw-r--r-- 1 hmeij its        56 Oct  6 10:57 stats-dump 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 timens-0.img 
 +-rw-r--r-- 1 hmeij its       213 Oct  6 10:57 tty-info.img 
 + 
 + 
 +[hmeij@n91 src]$ criu restore --shell-job --restore-detached --images-dir demo --unprivileged 
 +2262128: Error (criu/tty.c:831): tty: Can't set tty params on 0xf: Operation not permitted 
 +2262128: Error (criu/files.c:1213): Unable to open fd=0 id=0xf 
 +Error (criu/cr-restore.c:2547): Restoring FAILED. 
 + 
 + 
 +BUMMER! 
 + 
 +Note: running the criu dump/restore commands via sudo 
 +does not fix the issue. "In Userspace" ! 
 + 
 +</code> 
 + 
 +  * Good write up and exercise. **This does work with sudo on CPU side** 
 +  * https://www.marcusfolkesson.se/blog/checkpoint-restore/ 
 + 
 +<code> 
 + 
 +# counter2 is counting in console 1 until the dump happens 
 + 
 +42 
 +43 
 +44 
 +45 
 +46 
 +47 
 +48 
 +49 
 +50 
 +51 
 +Killed 
 + 
 +# the dump populates demo2 
 + 
 +[hmeij@n91 demo2]$ sudo criu dump -t 1595356 --shell-job 
 +[sudo] password for hmeij: 
 +Can't check shutdown state of inet socket 
 +Warn  (compel/arch/x86/src/lib/infect.c:356): Will restore 1595356 with interrupted system call 
 + 
 +[hmeij@n91 demo2]$ ll 
 +total 164 
 +-rw-r--r-- 1 root root   3526 Oct  6 14:59 cgroup.img 
 +-rw-r--r-- 1 root root   2477 Oct  6 14:59 core-1595356.img 
 +-rw-r--r-- 1 root root     44 Oct  6 14:59 fdinfo-2.img 
 +-rw-r--r-- 1 root root    476 Oct  6 14:59 files.img 
 +-rw-r--r-- 1 root root     18 Oct  6 14:59 fs-1595356.img 
 +-rw-r--r-- 1 root root     36 Oct  6 14:59 ids-1595356.img 
 +-rw-r--r-- 1 root root     46 Oct  6 14:59 inventory.img 
 +-rw-r--r-- 1 root root    785 Oct  6 14:59 mm-1595356.img 
 +-rw-r--r-- 1 root root    197 Oct  6 14:59 pagemap-1595356.img 
 +-rw-r--r-- 1 root root 110592 Oct  6 14:59 pages-1.img 
 +-rw-r--r-- 1 root root     30 Oct  6 14:59 pstree.img 
 +-rw-r--r-- 1 root root     12 Oct  6 14:59 seccomp.img 
 +-rw-r--r-- 1 root root     46 Oct  6 14:59 stats-dump 
 +-rw-r--r-- 1 root root     38 Oct  6 14:59 timens-0.img 
 +-rw-r--r-- 1 root root    203 Oct  6 14:59 tty-info.img 
 + 
 +# the restore does not miss a beat in console2 
 + 
 +[hmeij@n91 demo2]$ sudo criu restore --shell-job 
 +52 
 +53 
 +54 
 +55 
 +56 
 +57 
 +58 
 +59 
 +60 
 +... 
  
 </code> </code>
cluster/234.1791297815.txt.gz · Last modified: by hmeij07