Why I thought this was an easy problem to solve I do not know. CPU checkpoint/restart has come a long way with DMTCP for serial and parallel jobs (including multi-host). But the CPU/GPU environment adds much complexity.
A good overview of the history of GPU checkpoint/restart efforts can be found at this presentation
An excellent in depth explanation can be found in this article
Some quotes from the article … CRAC provides the ability to save and restore the state of CUDA by first using CUDA-specific save/restore operations, and then delegating to a traditional checkpoint-restart package…. In the end, the support of DMTCP for process virtualization and plugins makes it easier to add modular support for CUDA without having to excessively understand details of the internals of the host checkpointing package. This plugin will likely only work with recent gpu models.