DokuWiki

User Tools

Site Tools

Trace: • 159 • 167 • 83
cluster:193

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
cluster:193 [2020/03/05 18:59] – [Pull Images] hmeij07cluster:193 [2024/09/17 16:51] (current) – [What's Running?] hmeij07
Line 2: Line 2:
 **[[cluster:0|Back]]** **[[cluster:0|Back]]**
  
-===== Docker Usage =====+===== Docker Containers Usage =====
  
 Page build up from the bottom to top.  We're not making a traditional "MPI" docker integration with our scheduler.  We'll see what usage patterns will emergence and go from there. I can help with workflow.  If more containers are desired please let me know which ones to ''pull''. Page build up from the bottom to top.  We're not making a traditional "MPI" docker integration with our scheduler.  We'll see what usage patterns will emergence and go from there. I can help with workflow.  If more containers are desired please let me know which ones to ''pull''.
  
-If users want to run web enabled applications in the container one simple workflow would be to submit a job that reserves a GPU then loops checking a lock file until removed.  Then ssh to the node and access the application via ''firefox http://localhost:PORT/''. For example DIGITS and JupyterLab below.+If users want to run web enabled applications in the container one simple workflow would be to submit a job that reserves a GPU then loops checking a lock file until removed.  Then ssh to the node and access the application via ''firefox http://localhost:PORT/''. For example DIGITS and JupyterLab described below. 
 + 
 +==== Readings ==== 
 + 
 +Interesting reads... 
 + 
 +  * https://www.stackhpc.com/k8s-mpi.html 
 +    * PMI(x), Slurm 
 + 
 +  * https://www.stackhpc.com/the-state-of-hpc-containers.html 
 +    * Docker, Kubernetes, Singularity, Shifter, CharleiCloud
    
 +  * https://en.wikipedia.org/wiki/HAProxy
 +    * HA load balancing with Docker images for CentOS
 ==== Scheduler Runs ==== ==== Scheduler Runs ====
  
-Next add to the script the scheduler syntax that is needed. Request a gpu resource ''gpu4'', memory and submit jobs. Example is located at ''home/hmeij/jobs/docker/run.docker''.  Notice that we use ''nvidia-docker'' [[https://thenewstack.io/primer-nvidia-docker-containers-meet-gpus/|External Link]].  Nvidia-Docker is basically a wrapper around the docker CLI that transparently provisions a container with the necessary dependencies to execute code on the GPU.+Next add to the script the scheduler syntax that is needed. Request a gpu resource ''gpu4'', memory and submit jobs. Example is located at ''/home/hmeij/jobs/docker/run.docker''.  Notice that we use ''nvidia-docker'' [[https://thenewstack.io/primer-nvidia-docker-containers-meet-gpus/|External Link]].  Nvidia-Docker is basically a wrapper around the docker CLI that transparently provisions a container with the necessary dependencies to execute code on the GPU.
  
 <code> <code>
Line 25: Line 37:
 #BSUB -R "rusage[gpu4=1:mem=6288],span[hosts=1]" #BSUB -R "rusage[gpu4=1:mem=6288],span[hosts=1]"
  
 +# should add a check we get an integer back in the 0-3 range
 gpuid="` gpu-free | sed "s/,/\n/g" | shuf | head -1 ` " gpuid="` gpu-free | sed "s/,/\n/g" | shuf | head -1 ` "
 echo ""; echo "running on gpu $HOSTNAME:$gpuid"; echo "" echo ""; echo "running on gpu $HOSTNAME:$gpuid"; echo ""
Line 38: Line 51:
 --model=resnet50 \ --model=resnet50 \
 --variable_update=parameter_server --variable_update=parameter_server
 +# or run_tests.py
 +
 +</code>
 +
 +To make the ''imports'' work edit that python file
 +
 +<code>
  
 +import sys
 +sys.path.insert(0, '/mnt/hmeij/jobs/docker/benchmarks-master/scripts/tf_cnn_benchmarks/')
  
 </code> </code>
Line 44: Line 66:
 ==== GPU Runs ==== ==== GPU Runs ====
  
-We put the tensorflow benchmark example in a script. It will find a free gpu, set it in a environment variable ''NV__GPU'' and then run the tensorflow application on the GPU allocated.+We put the tensorflow benchmark example in a script. It will find a free gpu, set it in a environment variable ''NV_GPU'' and then run the tensorflow application on the GPU allocated.
  
 <code> <code>
Line 63: Line 85:
 Container image Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved. Container image Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.
 Copyright 2017-2019 The TensorFlow Authors.  All rights reserved. Copyright 2017-2019 The TensorFlow Authors.  All rights reserved.
-(deleted content...)+(snip output...)
  
 # details # details
Line 84: Line 106:
 Initializing graph Initializing graph
 Running warm up Running warm up
-(deleted content...it crashes but we can see it running)+(snip output...)
  
 # query what is running on gpus ... D8 is gpu 3 (ssh n79 nvidia-smi to verify) # query what is running on gpus ... D8 is gpu 3 (ssh n79 nvidia-smi to verify)
Line 129: Line 151:
 ==== Pull Images ==== ==== Pull Images ====
  
-Pull more images from the Nvidia Gpu Cloud Catalog.  There are also models.+Pull more images from the Nvidia Gpu Cloud Catalog.  There are also models.  As you can tell, not all containers applications are up to date.  Only pulled on node ''n79'', not expecting any usage.  It is nice to pull esoteric software like the deep learning stack (digits, tensorflow, pytorch, caffe, rapidsai).
  
 <code> <code>
Line 279: Line 301:
 nvcr.io/nvidia/caffe               19.09-py2                      b52fbbef7e6b        6 months ago        5.15GB nvcr.io/nvidia/caffe               19.09-py2                      b52fbbef7e6b        6 months ago        5.15GB
 nvcr.io/nvidia/rapidsai/rapidsai   0.9-cuda10.0-runtime-centos7   22b5dc2f7e84        6 months ago        5.84GB nvcr.io/nvidia/rapidsai/rapidsai   0.9-cuda10.0-runtime-centos7   22b5dc2f7e84        6 months ago        5.84GB
 +
 +adding 09/17/2024
 +https://hub.docker.com/r/mobigroup/pygmtsar-large
 +docker pull mobigroup/pygmtsar-large
 +Status: Downloaded newer image for mobigroup/pygmtsar-large:latest
 +docker.io/mobigroup/pygmtsar-large:latest
 +
  
 # running containers (persistent across boot events) # running containers (persistent across boot events)
cluster/193.1583434756.txt.gz · Last modified: by hmeij07