User Tools

Site Tools


cluster:234

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
cluster:234 [2026/10/05 19:16] – hmeij07cluster:234 [2026/10/09 13:09] (current) – hmeij07
Line 6: Line 6:
   * source and user manual at   * source and user manual at
   * https://github.com/NVIDIA/cuda-checkpoint   * https://github.com/NVIDIA/cuda-checkpoint
 +  * https://developer.nvidia.com/blog/checkpointing-cuda-applications-with-criu/
  
   * located in n91:/projects/cuda-checkpoint-main/580.65.06   * located in n91:/projects/cuda-checkpoint-main/580.65.06
-  * linked in n91:/usr/local/bin/cuda-checkpoint+  * utility linked in n91:/usr/local/bin/cuda-checkpoint 
 + 
 +  * do not find a way to download 550 features version  
 +  * for our mwgpu256 gpu compute nodes 
 + 
 +Put your checkpoints in /sanscratch/checkpoints/$SLURM_JOB_ID directory. You need to make that directory in your submit script but do not have to clean it up. Checkpoints older than 120 days are automatically removed. 
  
 <code> <code>
Line 29: Line 35:
 </code> </code>
  
-An example of performing a checkpoint/restore is +An example of performing a checkpoint/restore is\\ 
- /projects/cuda-checkpoint-main/580.65.06/src/example.sh + /projects/cuda-checkpoint-main/580.65.06/src/example.sh\\ 
-the counter does not spit out output so hard to test....+the counter does not spit out output so hard to test....\\ 
 + 
 +Actually works a little different. 
 + 
 +<code> 
 + 
 +To get these output files, the process follows these standard steps: 
 + 
 +1. Compile and Run 
 + 
 +nvcc counter.cu -o counter 
 + 
 +./counter & # nvidia-smi confirms counter is on GPU 
 + 
 +PID=$!; echo $PID 
 +2262128 
 + 
 +echo hello | nc -u localhost 10000 -w 1 
 + 
 +2. Suspend GPU Work:  
 +You use cuda-checkpoint to freeze and extract  
 +the GPU memory layout into host memory space 
 +...this terminates the counter on GPU 
 + 
 +cuda-checkpoint --toggle --pid $PID 
 + 
 +3. Dump to Files:  
 +You use CRIU to write the combined state out  
 +into your output directory 
 +...this terminates the counter on CPU 
 + 
 +criu dump --shell-job --images-dir demo --tree $PID --unprivileged 
 + 
 +Once completed, the target counter process is terminated,  
 +and your demo/ directory  
 +will be packed with the binary image (.img) files representing  
 +the exact frozen state of your application. 
 + 
 +# as root set 
 +# Note: CRIU, Check/Restore In Userspace; so why is this not the default behavior? 
 + 
 +[root@n91 ~]# setcap cap_checkpoint_restore+eip /usr/bin/criu 
 + 
 +# as user, needed? 
 +[hmeij@n91 src]$ SKIP_SETCAP=true 
 +[hmeij@n91 src]$ SKIP_CHOWN=true 
 + 
 +Running as non-root requires '--unprivileged' 
 +Please consult the documentation for limitations when running as non-root 
 + 
 +[hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID --unprivileged 
 + 
 +[hmeij@n91 src]$ ll demo 
 +total 608388 
 +-rw-r--r-- 1 hmeij its      2713 Oct  6 10:57 core-1583155.img 
 +-rw-r--r-- 1 hmeij its      1561 Oct  6 10:57 core-1583158.img 
 +-rw-r--r-- 1 hmeij its      1763 Oct  6 10:57 core-1583163.img 
 +-rw-r--r-- 1 hmeij its       296 Oct  6 10:57 fdinfo-2.img 
 +-rw-r--r-- 1 hmeij its      1791 Oct  6 10:57 files.img 
 +-rw-r--r-- 1 hmeij its        18 Oct  6 10:57 fs-1583155.img 
 +-rw-r--r-- 1 hmeij its        36 Oct  6 10:57 ids-1583155.img 
 +-rw-r--r-- 1 hmeij its        44 Oct  6 10:57 inventory.img 
 +-rw-r--r-- 1 hmeij its       148 Oct  6 10:57 memfd.img 
 +-rw-r--r-- 1 hmeij its      2942 Oct  6 10:57 mm-1583155.img 
 +-rw-r--r-- 1 hmeij its     11832 Oct  6 10:57 pagemap-1583155.img 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959861.img 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959862.img 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959863.img 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pagemap-shmem-86959864.img 
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-1.img 
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-2.img 
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-3.img 
 +-rw-r--r-- 1 hmeij its   2097152 Oct  6 10:57 pages-4.img 
 +-rw-r--r-- 1 hmeij its 614502400 Oct  6 10:57 pages-5.img 
 +-rw-r--r-- 1 hmeij its        53 Oct  6 10:57 pipes-data.img 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 pstree.img 
 +-rw-r--r-- 1 hmeij its        12 Oct  6 10:57 seccomp.img 
 +-rw-r--r-- 1 hmeij its        56 Oct  6 10:57 stats-dump 
 +-rw-r--r-- 1 hmeij its        38 Oct  6 10:57 timens-0.img 
 +-rw-r--r-- 1 hmeij its       213 Oct  6 10:57 tty-info.img 
 + 
 + 
 +[hmeij@n91 src]$ criu restore --shell-job --restore-detached --images-dir demo --unprivileged 
 +2262128: Error (criu/tty.c:831): tty: Can't set tty params on 0xf: Operation not permitted 
 +2262128: Error (criu/files.c:1213): Unable to open fd=0 id=0xf 
 +Error (criu/cr-restore.c:2547): Restoring FAILED. 
 + 
 + 
 +BUMMER! 
 + 
 +Note: running the criu dump/restore commands via sudo 
 +does not fix the issue. "In Userspace" ! 
 + 
 +</code> 
 + 
 +  * Good write up and exercise. **This does work with sudo on CPU side** 
 +  * https://www.marcusfolkesson.se/blog/checkpoint-restore/ 
 + 
 +<code> 
 + 
 +# counter2 is counting in console 1 until the dump happens 
 + 
 +42 
 +43 
 +44 
 +45 
 +46 
 +47 
 +48 
 +49 
 +50 
 +51 
 +Killed 
 + 
 +# the dump populates demo2 
 + 
 +[hmeij@n91 demo2]$ sudo criu dump -t 1595356 --shell-job 
 +[sudo] password for hmeij: 
 +Can't check shutdown state of inet socket 
 +Warn  (compel/arch/x86/src/lib/infect.c:356): Will restore 1595356 with interrupted system call 
 + 
 +[hmeij@n91 demo2]$ ll 
 +total 164 
 +-rw-r--r-- 1 root root   3526 Oct  6 14:59 cgroup.img 
 +-rw-r--r-- 1 root root   2477 Oct  6 14:59 core-1595356.img 
 +-rw-r--r-- 1 root root     44 Oct  6 14:59 fdinfo-2.img 
 +-rw-r--r-- 1 root root    476 Oct  6 14:59 files.img 
 +-rw-r--r-- 1 root root     18 Oct  6 14:59 fs-1595356.img 
 +-rw-r--r-- 1 root root     36 Oct  6 14:59 ids-1595356.img 
 +-rw-r--r-- 1 root root     46 Oct  6 14:59 inventory.img 
 +-rw-r--r-- 1 root root    785 Oct  6 14:59 mm-1595356.img 
 +-rw-r--r-- 1 root root    197 Oct  6 14:59 pagemap-1595356.img 
 +-rw-r--r-- 1 root root 110592 Oct  6 14:59 pages-1.img 
 +-rw-r--r-- 1 root root     30 Oct  6 14:59 pstree.img 
 +-rw-r--r-- 1 root root     12 Oct  6 14:59 seccomp.img 
 +-rw-r--r-- 1 root root     46 Oct  6 14:59 stats-dump 
 +-rw-r--r-- 1 root root     38 Oct  6 14:59 timens-0.img 
 +-rw-r--r-- 1 root root    203 Oct  6 14:59 tty-info.img 
 + 
 +# the restore does not miss a beat in console2 
 + 
 +[hmeij@n91 demo2]$ sudo criu restore --shell-job 
 +52 
 +53 
 +54 
 +55 
 +56 
 +57 
 +58 
 +59 
 +60 
 +... 
 + 
 + 
 +</code>
  
 \\ \\
cluster/234.1791227812.txt.gz · Last modified: by hmeij07