This is an old revision of the document!
Table of Contents
EXX96
A page for me on how these 12 nodes were build up after they arrived. To make them “ala n37” which was the test node in redoing our K20 nodes, see K20 Redo and K20 Redo Usage
Page best followed bottom to top if interested in the whole process.
The Usage section below is HPCC users wnatig to use queue exx96.
Usage
The new queue exx96 will be comprised of nodes n79-n90. Each node holds 4x RTX2080S gpus, 2x Xeon Silver 4214 2.2 Ghz 12-core cpus, 96 GB memory and a 1TB SSD. /localscratch is around 800 GB.
A new static resource is introduced for all nodes holding gpus. n78 in queue amber128 and n33-n37 in queue mwgpu and the nodes mentioned above. The name of this resource is gpu4. Moving forward please use it instead of gpu or gputest.
The wrappers provided assume your cpu:gpu ratio is 1:1 hence in your submit code you will have #BSUB -n 1 and in your resource allocation line gpu4=1. If your ratio is something else you can set CPU_GPU_REQUEST. For example CPU_GPU_REQUEST=4:2 expects the lines #BSUB -n 4 and gpu4=2 in your submit script. Sample script at /home/hmeij/k20redo/run.rtx
The wrappers (n78.mpich3.wrapper for n78, and n37.openmpi.wrapper for all others) are located in /usr/local/bin and will set up your environment and start either of these applications: amber, lammps, gromacs, matlab and namd from /usr/local.
# command that shows gpu reservations
bhosts -l n79
gputest gpu4
Total 0 3
Reserved 0.0 1.0
# old way of doing that
lsload -l n79
HOST_NAME status r15s r1m r15m ut pg io ls it tmp swp mem gpu
n79 ok 0.0 0.0 0.0 0% 0.0 0 0 2e+08 826G 10G 90G 3.0
Peer to peer communication is possible (via PCIe rather than NVlink) with this hardware. This will get rather messy in setting up. Some quick off the cuff performance data reveals some impact. Generally in our environment the gains are not worth the effort. Using Amber and pmemd.cuda.MPI
cpu:gpu mdout.325288:| Master Total CPU time: 982.60 seconds 0.27 hours 1:1 mdout.325289:| Master Total CPU time: 611.08 seconds 0.17 hours 4:2 mdout.326208:| Master Total CPU time: 537.97 seconds 0.15 hours 36:4
Miscellaneous
Install scheduler RPM for CentOS7, reconfigure (hosts, queue, static resource), elim. Test it out with old wrapper.
Edit the n37.openmpi.wrapper for n33-n37 and n79-90 and the one on n78 for the new static resource gpu4.
Add nodes to ZenOSS hpcmon.
Propagate global known_hosts files in users ~/.ssh/ dirs.
Look at how accounting ties in with resource request gpu4= versus gpu= …
# propagate global passwd, shadow, group, hosts file # add to date_ctt2.sh script, get and set date NOW=`/bin/date +%m%d%H%M%Y.%S` for i in `seq 79 90`; do echo n$i; ssh n$i date $NOW; done # crontab # ionice gaussian 0,15,30,45 * * * * /share/apps/scripts/ionice_lexes.sh > /dev/null 2>&1 # cpu temps 40 * * * * /share/apps/scripts/lm_sensors.sh > /dev/null 2>&1 # rc.local, chmod o+x /etc/rc.d/rc.local, then add # for mapd, 'All On' enable graphicsrendering support #/usr/bin/nvidia-smi --gom=0 # for amber16 -pm=ENABLED -c=EXCLUSIVE_PROCESS #nvidia-smi --persistence-mode=1 #nvidia-smi --compute-mode=1 # for mwgpu/exx96 -pm=ENABLED -c=DEFAULT nvidia-smi --persistence-mode=1 nvidia-smi --compute-mode=0 # turn ECC off (memory scrubbing) #/usr/bin/nvidia-smi -e 0 # lm_sensor modprobe coretemp modprobe tmp401 #modprobe w83627ehf reboot
Recipe
Steps. “Ala n37” … so the RTX nodes are similar to the K20 nodes and we can put the local software in place. See K20 Redo page. First we add these packages and clean up.
# hook up VDI-D cable to GPU port (offboard video) # login as root check some things out... free -g nvidia-smi docker images docker ps # set local time zone mv /etc/localtime /etc/localtime.backup ln -s /usr/share/zoneinfo/America/New_York /etc/localtime # change passwords for root and vendor account passwd passwd exx # set hostname hostnamectl set-hostname n79 # configure private subnets and ping file server cd /etc/sysconfig/network-scripts/ vi ifcfg-enp1s0f0 vi ifcfg-enp1s0f1 systemctl restart network ping -c 3 192.168.102.42 ping -c 3 10.10.102.42 # make internet connection for yum ifdown enp1s0f0 vi ifcfg-enp1s0f0 systemctl restart network dig google.com yum install -y iptables-services vi /etc/sysconfig/iptables systemctl start iptables iptables -L systemctl stop firewalld systemctl disable firewalld # other configs vi /etc/selinux/config (disabled) mv /home /usr/local/ mkdir /home vi /etc/passwd (exx, dockeruser $HOME) mkdir /sanscratch /localscratch chmod ugo+rwx /sanscratch /localscratch chmod o+t /sanscratch /localscratch ln -s /home /share ssh-keygen -t rsa scp 10.10.102.253:/root/.ssh/authorized_keys /root/.ssh/ /etc/ssh/sshd_config (PermitRootLogin) echo "relayhost = 192.168.102.42" >> /etc/postfix/main.cf # add packages and update yum install epel-release -y yum install tcl tcl-devel dmtcp -y yum install freeglut-devel libXi-devel libXmu-devel \ make mesa-libGLU-devel -y yum install blas blas-devel lapack lapack-devel boost boost-devel -y yum install tkinter lm_sensors lm_sensors-libs -y yum install zlib-devel bzip2-devel bzip bzip-devel -y yum install openmpi openmpi-devel perl-ExtUtils-MakeMaker -y yum install cmake cmake-devel -y yum install libjpeg libjpeg-devel libjpeg-turbo-devel -y yum update -y yum clean all # remove internet, bring private back up ifdown enp1s0f0 vi ifcfg-enp1s0f0 ifup enp1s0f0 # passwd, shadow, group, hosts, fstab mkdir /homeextra1 /homeextra2 /home33 /mindstore cd /etc/ # backup files to -orig versions scp 192.168.102.89:/etc/passwd /etc/passwd (and others) scp 10.10.102.89:/etc/fstab /tmp vi /etc/fstab mount -a; df -h # pick the kernel vendor used for now grep ^menuentry /etc/grub2.cfg grub2-set-default 1 ls -d /sys/firmware/efi && echo "EFI" || echo "Legacy" grub2-mkconfig -o /boot/grub2/grub.cfg #grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg # old level 3 systemctl set-default multi-user.target reboot # switch to VGA cd /usr/local/src/ tar zxf n37.chroot-keep.ul.tar.gz cd usr/local/ mv amber16/ fsl-5.0.10/ gromacs-2018/ lammps-22Aug18/ /usr/local/ mv cuda-9.2/ /usr/local/n37-cuda-9.2/ cd /usr/local/bin/ rsync -vac 10.10.102.89:/usr/local/bin/ /usr/local/bin/ # test scripts gpu-free, gpu-info, gpu-process 0,1,2,3 id,name,temp.gpu,mem.used,mem.free,util.gpu,util.mem 0, GeForce RTX 2080 SUPER, 25, 126 MiB, 7855 MiB, 0 %, 0 % 1, GeForce RTX 2080 SUPER, 24, 11 MiB, 7971 MiB, 0 %, 0 % 2, GeForce RTX 2080 SUPER, 23, 11 MiB, 7971 MiB, 0 %, 0 % 3, GeForce RTX 2080 SUPER, 23, 11 MiB, 7971 MiB, 0 %, 0 % gpu_name, gpu_bus_id, pid, process_name GeForce RTX 2080 SUPER, 00000000:3B:00.0, 3109, python # done
What We Purchased
- 12 nodes yielding a total of
- 24 cpus
- 288 cpu cores
- 1,152 gb cpu mem
- ~20 Tflops (dpfp)
- 48 gpus
- 384 gpu mem
- ~700 Tflops (mixed mode)
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nvcr.io/nvidia/cuda 10.1-devel 9e47e9dfcb9a 2 months ago 2.83GB
portainer/portainer latest ff4ee4caaa23 2 months ago 81.6MB
nvidia/cuda 9.2-devel 1874839f75d5 2 months ago 2.35GB
nvcr.io/nvidia/cuda 9.2-devel 1874839f75d5 2 months ago 2.35GB
nvcr.io/nvidia/cuda 10.0-devel f765411c4ae6 2 months ago 2.29GB
nvcr.io/nvidia/digits 19.09-tensorflow b08982c9545c 4 months ago 8.85GB
nvcr.io/nvidia/tensorflow 19.09-py2 b82bcb185286 4 months ago 7.88GB
nvcr.io/nvidia/pytorch 19.09-py3 9d6f9ccfbe31 5 months ago 9.15GB
nvcr.io/nvidia/caffe 19.09-py2 b52fbbef7e6b 5 months ago 5.15GB
nvcr.io/nvidia/rapidsai/rapidsai 0.9-cuda10.0-runtime-centos7 22b5dc2f7e84 5 months ago 5.84GB
# free -g
total used free shared buff/cache available
Mem: 92 2 88 0 1 89
# nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.31 Driver Version: 440.31 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... Off | 00000000:3B:00.0 Off | N/A |
| 24% 24C P8 8W / 250W | 275MiB / 7981MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce RTX 208... Off | 00000000:5E:00.0 Off | N/A |
| 25% 24C P8 10W / 250W | 12MiB / 7982MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce RTX 208... Off | 00000000:AF:00.0 Off | N/A |
| 24% 23C P8 4W / 250W | 12MiB / 7982MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 GeForce RTX 208... Off | 00000000:D8:00.0 Off | N/A |
| 25% 22C P8 13W / 250W | 12MiB / 7982MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3127 C python 115MiB |
| 0 5715 G /usr/bin/X 84MiB |
| 0 6307 G /usr/bin/gnome-shell 70MiB |
+-----------------------------------------------------------------------------+
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 47G 0 47G 0% /dev
tmpfs 47G 0 47G 0% /dev/shm
tmpfs 47G 13M 47G 1% /run
tmpfs 47G 0 47G 0% /sys/fs/cgroup
/dev/nvme0n1p3 929G 42G 840G 5% /
/dev/nvme0n1p1 477M 199M 249M 45% /boot
overlay 929G 42G 840G 5% /var/lib/docker/overlay2
/6f7af00a8eb8b5ede68fd6bc9be5f7220525bdde21c14e6f1643a2a7debc454b/merged
overlay 929G 42G 840G 5% /var/lib/docker/overlay2
/9cf895d8a17106f16ba997e6025f5912abb988a512779caa2c35e2da3e7d196a/merged
tmpfs 9.3G 28K 9.3G 1% /run/user/0
# yum repolist
repo id repo name status
base/7/x86_64 CentOS-7 - Base 10,097
docker-ce-stable/x86_64 Docker CE Stable - x86_64 63
extras/7/x86_64 CentOS-7 - Extras 323
libnvidia-container libnvidia-container 65
nvidia-container-runtime nvidia-container-runtime 54
nvidia-docker nvidia-docker 50
updates/7/x86_64 CentOS-7 - Updates 1,117
Pictures
Yea, found 1T SSD
ports on gpu
GPU detail, blower model
Back, gpus stacked 2 on 2
Front, all drive bays empty
Racking
Boxes