This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:187 [2019/12/10 13:17] hmeij07 |
cluster:187 [2020/01/17 07:46] hmeij07 |
||
---|---|---|---|
Line 1: | Line 1: | ||
\\ | \\ | ||
**[[cluster: | **[[cluster: | ||
+ | |||
+ | Useful links. | ||
+ | |||
+ | * https:// | ||
+ | * https:// | ||
+ | * scheduler wrapper, inside container: NV_GPU=2,3 nvidia-docker run ... | ||
+ | * (container sees host gpu 2,3 as container gpu 0,1) | ||
+ | * https:// | ||
+ | * https:// | ||
+ | * https:// | ||
+ | * | ||
==== NGC Docker Containers ==== | ==== NGC Docker Containers ==== | ||
+ | |||
+ | Trying to understand how to leverage GPU ready applications on the Nvidia NGC web site (Nvidia GPU Cloud). Download docker containers and build your own on premise catalog. Run GPU ready software on compute nodes with docker containers. Can't wrap myself around the problem of how to integrate containers with the Openlava scheduler yet. | ||
* https:// | * https:// | ||
Line 9: | Line 22: | ||
< | < | ||
- | # Assumes CentOS | + | # get docker on centos |
- | # Assumes NVIDIA Driver is installed as per requirements ( < 340.29 ) | + | curl -fsSL https:// |
- | # Install DOCKER | + | ./ |
- | sudo curl -fsSL https:// | + | |
- | # Start DOCKER | + | # systemctl |
- | sudo systemctl start docker | + | systemctl enable docker |
- | # Add dockeruser, usermod change | + | systemctl start docker |
- | sudo adduser dockeruser | + | |
+ | # dockeruser, usermod change | ||
+ | adduser dockeruser | ||
usermod -aG docker dockeruser | usermod -aG docker dockeruser | ||
- | # Install NV-DOCKER | + | |
- | # GET NVIDIA-DOCKER | + | # get nvidia-docker |
- | wget -P /tmp https:// | + | # wget -P /tmp https:// |
- | # INSTALL | + | wget https:// |
- | sudo rpm -i / | + | |
- | # Start NV-DOCKER Service | + | # rpm -i / |
+ | # make nvidia-docker | ||
+ | |||
+ | # systemctl | ||
+ | systemctl enable nvidia-docker | ||
systemctl start nvidia-docker | systemctl start nvidia-docker | ||
Line 35: | Line 54: | ||
# or | # or | ||
+ | |||
docker pull nvidia/cuda | docker pull nvidia/cuda | ||
Line 46: | Line 66: | ||
NGC Deep Learning Ready Docker Containers: | NGC Deep Learning Ready Docker Containers: | ||
+ | |||
NVIDIA DIGITS - nvcr.io/ | NVIDIA DIGITS - nvcr.io/ | ||
TensorFlow - nvcr.io/ | TensorFlow - nvcr.io/ | ||
Line 57: | Line 78: | ||
# in the catalog you can also find | # in the catalog you can also find | ||
+ | |||
docker pull nvcr.io/ | docker pull nvcr.io/ | ||
docker pull nvcr.io/ | docker pull nvcr.io/ | ||
docker pull nvcr.io/ | docker pull nvcr.io/ | ||
docker pull nvcr.io/ | docker pull nvcr.io/ | ||
- | # not all at the latest versions | + | |
+ | # not all at the latest versions | ||
# and amber would have to be custom build on top of nvidia/cuda | # and amber would have to be custom build on top of nvidia/cuda | ||
Line 71: | Line 94: | ||
# DIGITS example | # DIGITS example | ||
- | # if you passed GPU ID 2,3 for example, | + | # if you passed |
NV_GPU=0,1 nvidia-docker run --name digits -d -p 5000:5000 nvidia/ | NV_GPU=0,1 nvidia-docker run --name digits -d -p 5000:5000 nvidia/ | ||
Line 78: | Line 102: | ||
</ | </ | ||
+ | |||
+ | There are some other issues... | ||
+ | |||
+ | * inside the container the user invoked application runs as root so copying files back and forth is a problem | ||
+ | * file systems, home directory and scratch spaces need to be mounted inside container | ||
+ | * GPUs need to be reserved via scheduler on a host then made available to container (see above) | ||
+ | |||
+ | Some notes from https:// | ||
+ | |||
+ | < | ||
+ | |||
+ | # NGC containers are hosted in a repository called nvcr.io | ||
+ | # A Docker container is the running instance of a Docker image. | ||
+ | |||
+ | # All NGC Container images are based on the CUDA platform layer (nvcr.io/ | ||
+ | |||
+ | # mount host directory to container location | ||
+ | |||
+ | -v $HOME:/ | ||
+ | |||
+ | # pull images | ||
+ | |||
+ | docker pull nvcr.io/ | ||
+ | docker images | ||
+ | |||
+ | # detailed information of container | ||
+ | |||
+ | / | ||
+ | |||
+ | # specifying a user | ||
+ | |||
+ | -u $(id -u):$(id -g) | ||
+ | |||
+ | # allocate GPUs | ||
+ | |||
+ | NV_GPU=0,1 nvidia-docker run ... | ||
+ | |||
+ | # custom build images ... | ||
+ | # looks complex based on Dockerfile config file commands | ||
+ | # see link | ||
+ | |||
+ | </ | ||
+ | |||
\\ | \\ | ||
**[[cluster: | **[[cluster: | ||