User Tools

Site Tools


cluster:234

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
cluster:234 [2026/10/06 15:02] – hmeij07cluster:234 [2026/10/09 13:09] (current) – hmeij07
Line 6: Line 6:
   * source and user manual at   * source and user manual at
   * https://github.com/NVIDIA/cuda-checkpoint   * https://github.com/NVIDIA/cuda-checkpoint
 +  * https://developer.nvidia.com/blog/checkpointing-cuda-applications-with-criu/
  
   * located in n91:/projects/cuda-checkpoint-main/580.65.06   * located in n91:/projects/cuda-checkpoint-main/580.65.06
Line 44: Line 45:
 To get these output files, the process follows these standard steps: To get these output files, the process follows these standard steps:
  
-1. Compile and Run:bash+1. Compile and Run
  
 nvcc counter.cu -o counter nvcc counter.cu -o counter
  
-./counter & +./counter & # nvidia-smi confirms counter is on GPU
-PID=$!+
  
 +PID=$!; echo $PID
 +2262128
  
-2. Suspend GPU Work: You use cuda-checkpoint to freeze and extract the GPU memory layout into host memory space+echo hello | nc -u localhost 10000 -w 1 
 + 
 +2. Suspend GPU Work:  
 +You use cuda-checkpoint to freeze and extract  
 +the GPU memory layout into host memory space 
 +...this terminates the counter on GPU
  
 cuda-checkpoint --toggle --pid $PID cuda-checkpoint --toggle --pid $PID
  
-3. Dump to Files: You use CRIU to write the combined state out into your output directory +3. Dump to Files:  
- +You use CRIU to write the combined state out  
-criu dump --shell-job --images-dir demo --tree $PID +into your output directory 
- +...this terminates the counter on CPU
-Once completed, the target counter process is terminated, and your demo/ directory will be packed with the binary image (.img) files representing the exact frozen state of your application.+
  
 +criu dump --shell-job --images-dir demo --tree $PID --unprivileged
  
- +Once completed, the target counter process is terminated,  
-[hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID +and your demo/ directory  
-CRIU needs to have the CAP_SYS_ADMIN or the CAP_CHECKPOINT_RESTORE capability: +will be packed with the binary image (.img) files representing  
-setcap cap_checkpoint_restore+eip criu+the exact frozen state of your application.
  
 # as root set # as root set
 +# Note: CRIU, Check/Restore In Userspace; so why is this not the default behavior?
 +
 [root@n91 ~]# setcap cap_checkpoint_restore+eip /usr/bin/criu [root@n91 ~]# setcap cap_checkpoint_restore+eip /usr/bin/criu
  
Line 77: Line 86:
 Running as non-root requires '--unprivileged' Running as non-root requires '--unprivileged'
 Please consult the documentation for limitations when running as non-root Please consult the documentation for limitations when running as non-root
 +
 [hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID --unprivileged [hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID --unprivileged
  
Line 109: Line 119:
  
  
 +[hmeij@n91 src]$ criu restore --shell-job --restore-detached --images-dir demo --unprivileged
 +2262128: Error (criu/tty.c:831): tty: Can't set tty params on 0xf: Operation not permitted
 +2262128: Error (criu/files.c:1213): Unable to open fd=0 id=0xf
 +Error (criu/cr-restore.c:2547): Restoring FAILED.
 +
 +
 +BUMMER!
 +
 +Note: running the criu dump/restore commands via sudo
 +does not fix the issue. "In Userspace" !
  
 </code> </code>
  
-  * Good write up and exercise+  * Good write up and exercise. **This does work with sudo on CPU side**
   * https://www.marcusfolkesson.se/blog/checkpoint-restore/   * https://www.marcusfolkesson.se/blog/checkpoint-restore/
 +
 +<code>
 +
 +# counter2 is counting in console 1 until the dump happens
 +
 +42
 +43
 +44
 +45
 +46
 +47
 +48
 +49
 +50
 +51
 +Killed
 +
 +# the dump populates demo2
 +
 +[hmeij@n91 demo2]$ sudo criu dump -t 1595356 --shell-job
 +[sudo] password for hmeij:
 +Can't check shutdown state of inet socket
 +Warn  (compel/arch/x86/src/lib/infect.c:356): Will restore 1595356 with interrupted system call
 +
 +[hmeij@n91 demo2]$ ll
 +total 164
 +-rw-r--r-- 1 root root   3526 Oct  6 14:59 cgroup.img
 +-rw-r--r-- 1 root root   2477 Oct  6 14:59 core-1595356.img
 +-rw-r--r-- 1 root root     44 Oct  6 14:59 fdinfo-2.img
 +-rw-r--r-- 1 root root    476 Oct  6 14:59 files.img
 +-rw-r--r-- 1 root root     18 Oct  6 14:59 fs-1595356.img
 +-rw-r--r-- 1 root root     36 Oct  6 14:59 ids-1595356.img
 +-rw-r--r-- 1 root root     46 Oct  6 14:59 inventory.img
 +-rw-r--r-- 1 root root    785 Oct  6 14:59 mm-1595356.img
 +-rw-r--r-- 1 root root    197 Oct  6 14:59 pagemap-1595356.img
 +-rw-r--r-- 1 root root 110592 Oct  6 14:59 pages-1.img
 +-rw-r--r-- 1 root root     30 Oct  6 14:59 pstree.img
 +-rw-r--r-- 1 root root     12 Oct  6 14:59 seccomp.img
 +-rw-r--r-- 1 root root     46 Oct  6 14:59 stats-dump
 +-rw-r--r-- 1 root root     38 Oct  6 14:59 timens-0.img
 +-rw-r--r-- 1 root root    203 Oct  6 14:59 tty-info.img
 +
 +# the restore does not miss a beat in console2
 +
 +[hmeij@n91 demo2]$ sudo criu restore --shell-job
 +52
 +53
 +54
 +55
 +56
 +57
 +58
 +59
 +60
 +...
 +
 +
 +</code>
  
 \\ \\
cluster/234.1791298960.txt.gz · Last modified: by hmeij07