\\
**[[cluster:0|Back]]**

===== Cuda Checkpoint =====

  * source and user manual at
  * https://github.com/NVIDIA/cuda-checkpoint

  * located in n91:/projects/cuda-checkpoint-main/580.65.06
  * utility linked in n91:/usr/local/bin/cuda-checkpoint

  * do not find a way to download 550 features version 
  * for our mwgpu256 gpu compute nodes

Put your checkpoints in /sanscratch/checkpoints/$SLURM_JOB_ID directory. You need to make that directory in your submit script but do not have to clean it up. Checkpoints older than 120 days are automatically removed. 

<code>

[root@n91 ~]# rpm -qa criu
criu-3.18-5.module+el8.10.0+2001+6a33db9f.x86_64

[root@n91 ~]# rpm -ql criu
/run/criu
/usr/lib/.build-id
/usr/lib/.build-id/9b
/usr/lib/.build-id/9b/603c4fff3830838f46feaf4bd1204adc49421e
/usr/lib/tmpfiles.d/criu.conf
/usr/sbin/criu
/usr/share/doc/criu
/usr/share/doc/criu/COPYING
/usr/share/doc/criu/README.md
/usr/share/man/man8/criu.8.gz

</code>

An example of performing a checkpoint/restore is\\
 /projects/cuda-checkpoint-main/580.65.06/src/example.sh\\
the counter does not spit out output so hard to test....\\

\\
**[[cluster:0|Back]]**