cluster:234
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
| cluster:234 [2026/10/05 19:12] – created hmeij07 | cluster:234 [2026/10/09 13:09] (current) – hmeij07 | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| + | \\ | ||
| + | **[[cluster: | ||
| + | ===== Cuda Checkpoint ===== | ||
| * source and user manual at | * source and user manual at | ||
| * https:// | * https:// | ||
| + | * https:// | ||
| * located in n91:/ | * located in n91:/ | ||
| - | * linked in n91:/ | + | * utility |
| + | |||
| + | * do not find a way to download 550 features version | ||
| + | * for our mwgpu256 gpu compute nodes | ||
| + | |||
| + | Put your checkpoints in / | ||
| < | < | ||
| Line 27: | Line 35: | ||
| </ | </ | ||
| - | An example of performing a checkpoint/ | + | An example of performing a checkpoint/ |
| - | / | + | / |
| + | the counter does not spit out output so hard to test....\\ | ||
| + | |||
| + | Actually works a little different. | ||
| + | |||
| + | < | ||
| + | |||
| + | To get these output files, the process follows these standard steps: | ||
| + | |||
| + | 1. Compile and Run | ||
| + | |||
| + | nvcc counter.cu -o counter | ||
| + | |||
| + | ./counter & # nvidia-smi confirms counter is on GPU | ||
| + | |||
| + | PID=$!; echo $PID | ||
| + | 2262128 | ||
| + | |||
| + | echo hello | nc -u localhost 10000 -w 1 | ||
| + | |||
| + | 2. Suspend GPU Work: | ||
| + | You use cuda-checkpoint to freeze and extract | ||
| + | the GPU memory layout into host memory space | ||
| + | ...this terminates the counter on GPU | ||
| + | |||
| + | cuda-checkpoint --toggle --pid $PID | ||
| + | |||
| + | 3. Dump to Files: | ||
| + | You use CRIU to write the combined state out | ||
| + | into your output directory | ||
| + | ...this terminates the counter on CPU | ||
| + | |||
| + | criu dump --shell-job --images-dir demo --tree $PID --unprivileged | ||
| + | |||
| + | Once completed, the target counter process is terminated, | ||
| + | and your demo/ directory | ||
| + | will be packed with the binary image (.img) files representing | ||
| + | the exact frozen state of your application. | ||
| + | |||
| + | # as root set | ||
| + | # Note: CRIU, Check/ | ||
| + | |||
| + | [root@n91 ~]# setcap cap_checkpoint_restore+eip / | ||
| + | |||
| + | # as user, needed? | ||
| + | [hmeij@n91 src]$ SKIP_SETCAP=true | ||
| + | [hmeij@n91 src]$ SKIP_CHOWN=true | ||
| + | |||
| + | Running as non-root requires ' | ||
| + | Please consult the documentation for limitations when running as non-root | ||
| + | |||
| + | [hmeij@n91 src]$ criu dump --shell-job --images-dir demo --tree $PID --unprivileged | ||
| + | |||
| + | [hmeij@n91 src]$ ll demo | ||
| + | total 608388 | ||
| + | -rw-r--r-- 1 hmeij its 2713 Oct 6 10:57 core-1583155.img | ||
| + | -rw-r--r-- 1 hmeij its 1561 Oct 6 10:57 core-1583158.img | ||
| + | -rw-r--r-- 1 hmeij its 1763 Oct 6 10:57 core-1583163.img | ||
| + | -rw-r--r-- 1 hmeij its 296 Oct 6 10:57 fdinfo-2.img | ||
| + | -rw-r--r-- 1 hmeij its 1791 Oct 6 10:57 files.img | ||
| + | -rw-r--r-- 1 hmeij its 18 Oct 6 10:57 fs-1583155.img | ||
| + | -rw-r--r-- 1 hmeij its 36 Oct 6 10:57 ids-1583155.img | ||
| + | -rw-r--r-- 1 hmeij its 44 Oct 6 10:57 inventory.img | ||
| + | -rw-r--r-- 1 hmeij its 148 Oct 6 10:57 memfd.img | ||
| + | -rw-r--r-- 1 hmeij its 2942 Oct 6 10:57 mm-1583155.img | ||
| + | -rw-r--r-- 1 hmeij its 11832 Oct 6 10:57 pagemap-1583155.img | ||
| + | -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959861.img | ||
| + | -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959862.img | ||
| + | -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959863.img | ||
| + | -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pagemap-shmem-86959864.img | ||
| + | -rw-r--r-- 1 hmeij its | ||
| + | -rw-r--r-- 1 hmeij its | ||
| + | -rw-r--r-- 1 hmeij its | ||
| + | -rw-r--r-- 1 hmeij its | ||
| + | -rw-r--r-- 1 hmeij its 614502400 Oct 6 10:57 pages-5.img | ||
| + | -rw-r--r-- 1 hmeij its 53 Oct 6 10:57 pipes-data.img | ||
| + | -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 pstree.img | ||
| + | -rw-r--r-- 1 hmeij its 12 Oct 6 10:57 seccomp.img | ||
| + | -rw-r--r-- 1 hmeij its 56 Oct 6 10:57 stats-dump | ||
| + | -rw-r--r-- 1 hmeij its 38 Oct 6 10:57 timens-0.img | ||
| + | -rw-r--r-- 1 hmeij its 213 Oct 6 10:57 tty-info.img | ||
| + | |||
| + | |||
| + | [hmeij@n91 src]$ criu restore --shell-job --restore-detached --images-dir demo --unprivileged | ||
| + | 2262128: Error (criu/ | ||
| + | 2262128: Error (criu/ | ||
| + | Error (criu/ | ||
| + | |||
| + | |||
| + | BUMMER! | ||
| + | |||
| + | Note: running the criu dump/ | ||
| + | does not fix the issue. "In Userspace" | ||
| + | |||
| + | </ | ||
| + | |||
| + | * Good write up and exercise. **This does work with sudo on CPU side** | ||
| + | * https:// | ||
| + | |||
| + | < | ||
| + | |||
| + | # counter2 is counting in console 1 until the dump happens | ||
| + | |||
| + | 42 | ||
| + | 43 | ||
| + | 44 | ||
| + | 45 | ||
| + | 46 | ||
| + | 47 | ||
| + | 48 | ||
| + | 49 | ||
| + | 50 | ||
| + | 51 | ||
| + | Killed | ||
| + | |||
| + | # the dump populates demo2 | ||
| + | |||
| + | [hmeij@n91 demo2]$ sudo criu dump -t 1595356 --shell-job | ||
| + | [sudo] password for hmeij: | ||
| + | Can't check shutdown state of inet socket | ||
| + | Warn (compel/ | ||
| + | |||
| + | [hmeij@n91 demo2]$ ll | ||
| + | total 164 | ||
| + | -rw-r--r-- 1 root root 3526 Oct 6 14:59 cgroup.img | ||
| + | -rw-r--r-- 1 root root 2477 Oct 6 14:59 core-1595356.img | ||
| + | -rw-r--r-- 1 root root 44 Oct 6 14:59 fdinfo-2.img | ||
| + | -rw-r--r-- 1 root root 476 Oct 6 14:59 files.img | ||
| + | -rw-r--r-- 1 root root 18 Oct 6 14:59 fs-1595356.img | ||
| + | -rw-r--r-- 1 root root 36 Oct 6 14:59 ids-1595356.img | ||
| + | -rw-r--r-- 1 root root 46 Oct 6 14:59 inventory.img | ||
| + | -rw-r--r-- 1 root root 785 Oct 6 14:59 mm-1595356.img | ||
| + | -rw-r--r-- 1 root root 197 Oct 6 14:59 pagemap-1595356.img | ||
| + | -rw-r--r-- 1 root root 110592 Oct 6 14:59 pages-1.img | ||
| + | -rw-r--r-- 1 root root 30 Oct 6 14:59 pstree.img | ||
| + | -rw-r--r-- 1 root root 12 Oct 6 14:59 seccomp.img | ||
| + | -rw-r--r-- 1 root root 46 Oct 6 14:59 stats-dump | ||
| + | -rw-r--r-- 1 root root 38 Oct 6 14:59 timens-0.img | ||
| + | -rw-r--r-- 1 root root 203 Oct 6 14:59 tty-info.img | ||
| + | |||
| + | # the restore does not miss a beat in console2 | ||
| + | |||
| + | [hmeij@n91 demo2]$ sudo criu restore --shell-job | ||
| + | 52 | ||
| + | 53 | ||
| + | 54 | ||
| + | 55 | ||
| + | 56 | ||
| + | 57 | ||
| + | 58 | ||
| + | 59 | ||
| + | 60 | ||
| + | ... | ||
| + | |||
| + | |||
| + | </ | ||
| + | \\ | ||
| + | **[[cluster: | ||
cluster/234.1791227562.txt.gz · Last modified: by hmeij07
