User Tools

Site Tools


cluster:234

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
cluster:234 [2026/10/06 15:12] – hmeij07cluster:234 [2026/10/09 13:09] (current) – hmeij07
Line 6: Line 6:
   * source and user manual at   * source and user manual at
   * https://github.com/NVIDIA/cuda-checkpoint   * https://github.com/NVIDIA/cuda-checkpoint
 +  * https://developer.nvidia.com/blog/checkpointing-cuda-applications-with-criu/
  
   * located in n91:/projects/cuda-checkpoint-main/580.65.06   * located in n91:/projects/cuda-checkpoint-main/580.65.06
Line 48: Line 49:
 nvcc counter.cu -o counter nvcc counter.cu -o counter
  
-./counter & +./counter & # nvidia-smi confirms counter is on GPU
-PID=$!+
  
 +PID=$!; echo $PID
 +2262128
 +
 +echo hello | nc -u localhost 10000 -w 1
  
 2. Suspend GPU Work:  2. Suspend GPU Work: 
 You use cuda-checkpoint to freeze and extract  You use cuda-checkpoint to freeze and extract 
 the GPU memory layout into host memory space the GPU memory layout into host memory space
 +...this terminates the counter on GPU
  
 cuda-checkpoint --toggle --pid $PID cuda-checkpoint --toggle --pid $PID
Line 61: Line 66:
 You use CRIU to write the combined state out  You use CRIU to write the combined state out 
 into your output directory into your output directory
 +...this terminates the counter on CPU
  
-criu dump --shell-job --images-dir demo --tree $PID+criu dump --shell-job --images-dir demo --tree $PID --unprivileged
  
 Once completed, the target counter process is terminated,  Once completed, the target counter process is terminated, 
Line 68: Line 74:
 will be packed with the binary image (.img) files representing  will be packed with the binary image (.img) files representing 
 the exact frozen state of your application. the exact frozen state of your application.
- 
- 
- 
-[hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID 
-CRIU needs to have the CAP_SYS_ADMIN or the CAP_CHECKPOINT_RESTORE capability: 
  
 # as root set # as root set
Line 116: Line 117:
 -rw-r--r-- 1 hmeij its        38 Oct  6 10:57 timens-0.img -rw-r--r-- 1 hmeij its        38 Oct  6 10:57 timens-0.img
 -rw-r--r-- 1 hmeij its       213 Oct  6 10:57 tty-info.img -rw-r--r-- 1 hmeij its       213 Oct  6 10:57 tty-info.img
 +
 +
 +[hmeij@n91 src]$ criu restore --shell-job --restore-detached --images-dir demo --unprivileged
 +2262128: Error (criu/tty.c:831): tty: Can't set tty params on 0xf: Operation not permitted
 +2262128: Error (criu/files.c:1213): Unable to open fd=0 id=0xf
 +Error (criu/cr-restore.c:2547): Restoring FAILED.
 +
 +
 +BUMMER!
 +
 +Note: running the criu dump/restore commands via sudo
 +does not fix the issue. "In Userspace" !
  
 </code> </code>
  
-  * Good write up and exercise+  * Good write up and exercise. **This does work with sudo on CPU side**
   * https://www.marcusfolkesson.se/blog/checkpoint-restore/   * https://www.marcusfolkesson.se/blog/checkpoint-restore/
 +
 +<code>
 +
 +# counter2 is counting in console 1 until the dump happens
 +
 +42
 +43
 +44
 +45
 +46
 +47
 +48
 +49
 +50
 +51
 +Killed
 +
 +# the dump populates demo2
 +
 +[hmeij@n91 demo2]$ sudo criu dump -t 1595356 --shell-job
 +[sudo] password for hmeij:
 +Can't check shutdown state of inet socket
 +Warn  (compel/arch/x86/src/lib/infect.c:356): Will restore 1595356 with interrupted system call
 +
 +[hmeij@n91 demo2]$ ll
 +total 164
 +-rw-r--r-- 1 root root   3526 Oct  6 14:59 cgroup.img
 +-rw-r--r-- 1 root root   2477 Oct  6 14:59 core-1595356.img
 +-rw-r--r-- 1 root root     44 Oct  6 14:59 fdinfo-2.img
 +-rw-r--r-- 1 root root    476 Oct  6 14:59 files.img
 +-rw-r--r-- 1 root root     18 Oct  6 14:59 fs-1595356.img
 +-rw-r--r-- 1 root root     36 Oct  6 14:59 ids-1595356.img
 +-rw-r--r-- 1 root root     46 Oct  6 14:59 inventory.img
 +-rw-r--r-- 1 root root    785 Oct  6 14:59 mm-1595356.img
 +-rw-r--r-- 1 root root    197 Oct  6 14:59 pagemap-1595356.img
 +-rw-r--r-- 1 root root 110592 Oct  6 14:59 pages-1.img
 +-rw-r--r-- 1 root root     30 Oct  6 14:59 pstree.img
 +-rw-r--r-- 1 root root     12 Oct  6 14:59 seccomp.img
 +-rw-r--r-- 1 root root     46 Oct  6 14:59 stats-dump
 +-rw-r--r-- 1 root root     38 Oct  6 14:59 timens-0.img
 +-rw-r--r-- 1 root root    203 Oct  6 14:59 tty-info.img
 +
 +# the restore does not miss a beat in console2
 +
 +[hmeij@n91 demo2]$ sudo criu restore --shell-job
 +52
 +53
 +54
 +55
 +56
 +57
 +58
 +59
 +60
 +...
 +
 +
 +</code>
  
 \\ \\
cluster/234.1791299521.txt.gz · Last modified: by hmeij07