Upgrading Cuda to latest drivers and tooltkit that supports our oldest gpu model the K20m gpus found in nodes n33-n37 (queue mwgpu). Consult the page on previous K20m upgrade K20 Redo
For legacy hardware find the latest legacy driver here
Then download the driver series selected here
Then download the latest toolkit supported (11.2) for K20m on centos7
# install drivers (uninstalls existing drivers, accept defaults) cd /usr/local/src DRIVER_VERSION=470.199.02 curl -fSsl -O $BASE_URL/$DRIVER_VERSION/NVIDIA-Linux-x86_64-$DRIVER_VERSION.run sh ./NVIDIA-Linux-x86_64-470.199.02.run # install toolkit wget https://developer.download.nvidia.com/compute/cuda/11.2.1/local_installers/cuda_11.2.1_460.32.03_linux.run sh cuda_11.2.1_460.32.03_linux.run # prompts │ CUDA Installer │ │ - [X] Driver │ │ [X] 460.32.03 │ │ + [X] CUDA Toolkit 11.2 │ │ [ ] CUDA Samples 11.2 │ │ [ ] CUDA Demo Suite 11.2 │ │ [ ] CUDA Documentation 11.2 │ │ Options │ │ Install # update link to this version: yes # no -silent -driver ... =========== = Summary = =========== Driver: Installed Toolkit: Installed in /usr/local/cuda-11.2/ Samples: Not Selected Please make sure that - PATH includes /usr/local/cuda-11.2/bin - LD_LIBRARY_PATH includes /usr/local/cuda-11.2/lib64, or, add /usr/local/cuda-11.2/lib64 to /etc/ld.so.conf and run ldconfig as root To uninstall the CUDA Toolkit, run cuda-uninstaller in /usr/local/cuda-11.2/bin To uninstall the NVIDIA Driver, run nvidia-uninstall Logfile is /var/log/cuda-installer.log # no nvidia_modprobe? ls -l /dev/nvidia? reboot # then export PATH=/usr/local/cuda/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH [hmeij@n33 ~]$ nvidia-smi Tue Sep 5 14:43:15 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla K20m On | 00000000:02:00.0 Off | 0 | | N/A 26C P8 25W / 225W | 0MiB / 4743MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 Tesla K20m On | 00000000:03:00.0 Off | 0 | | N/A 27C P8 26W / 225W | 0MiB / 4743MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 2 Tesla K20m On | 00000000:83:00.0 Off | 0 | | N/A 25C P8 24W / 225W | 0MiB / 4743MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 3 Tesla K20m On | 00000000:84:00.0 Off | 0 | | N/A 26C P8 25W / 225W | 0MiB / 4743MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ # startup slurm, finds gpus? yes # old compiled code compatible? test
Script ~hmeij/slurm/run.centos, cuda 11.2, pmemd.cuda of local install of amber20 with
For some reason this yields cpus=8 which is different behavior (expected cpu=1). Slurm is overriding the above settings with partition setting of DefCpuPerGPU=8. Slurm has not changed but cuda version has. Odd. Good news is Amber runs fine, no need to recompile.
# from slurmd.log [2023-09-05T14:51:00.691] Gres Name=gpu Type=tesla_k20m Count=4 JOBID PARTITION NAME USER ST TIME NODES CPUS MIN_MEMORY NODELIST(REASON) 1053052 mwgpu test hmeij R 0:09 1 8 0 n33 [hmeij@cottontail2 slurm]$ ssh n33 gpu-info id,name,temp.gpu,mem.used,mem.free,util.gpu,util.mem 0, Tesla K20m, 36, 95 MiB, 4648 MiB, 100 %, 25 % 1, Tesla K20m, 26, 0 MiB, 4743 MiB, 0 %, 0 % 2, Tesla K20m, 25, 0 MiB, 4743 MiB, 0 %, 0 % 3, Tesla K20m, 26, 0 MiB, 4743 MiB, 0 %, 0 % [hmeij@cottontail2 slurm]$ ssh n33 gpu-process gpu_name, gpu_id, pid, process_name Tesla K20m, 0, 28394, pmemd.cuda
Adding this does force Slurm to allocate just a single cpu. Now try 4 gpu jobs per node. No need for CUDA_VISIBLE_DEVICES setting.
JOBID PARTITION NAME USER ST TIME NODES CPUS MIN_MEMORY NODELIST(REASON) 1053992 mwgpu test hmeij R 0:04 1 1 0 n33 [hmeij@cottontail2 slurm]$ for i in `seq 1 6`; do sbatch run.centos; sleep 30; squeue | grep hmeij; done # output Submitted batch job 1054000 1054000 mwgpu test hmeij R 0:30 1 1 0 n33 Submitted batch job 1054001 1054001 mwgpu test hmeij R 0:30 1 1 0 n33 1054000 mwgpu test hmeij R 1:00 1 1 0 n33 Submitted batch job 1054002 1054002 mwgpu test hmeij R 0:30 1 1 0 n33 1054001 mwgpu test hmeij R 1:00 1 1 0 n33 1054000 mwgpu test hmeij R 1:30 1 1 0 n33 Submitted batch job 1054003 1054003 mwgpu test hmeij R 0:30 1 1 0 n33 1054002 mwgpu test hmeij R 1:00 1 1 0 n33 1054001 mwgpu test hmeij R 1:30 1 1 0 n33 1054000 mwgpu test hmeij R 2:00 1 1 0 n33 Submitted batch job 1054004 1054004 mwgpu test hmeij PD 0:00 1 1 0 (Resources) 1054003 mwgpu test hmeij R 1:00 1 1 0 n33 1054002 mwgpu test hmeij R 1:30 1 1 0 n33 1054001 mwgpu test hmeij R 2:00 1 1 0 n33 1054000 mwgpu test hmeij R 2:30 1 1 0 n33 Submitted batch job 1054005 1054005 mwgpu test hmeij PD 0:00 1 1 0(Nodes required f 1054004 mwgpu test hmeij PD 0:00 1 1 0 (Resources) 1054003 mwgpu test hmeij R 1:30 1 1 0 n33 1054002 mwgpu test hmeij R 2:00 1 1 0 n33 1054001 mwgpu test hmeij R 2:30 1 1 0 n33 1054000 mwgpu test hmeij R 3:00 1 1 0 n33 [hmeij@cottontail2 slurm]$ ssh n33 gpu-info id,name,temp.gpu,mem.used,mem.free,util.gpu,util.mem 0, Tesla K20m, 40, 95 MiB, 4648 MiB, 100 %, 25 % 1, Tesla K20m, 40, 95 MiB, 4648 MiB, 94 %, 23 % 2, Tesla K20m, 35, 95 MiB, 4648 MiB, 93 %, 21 % 3, Tesla K20m, 28, 95 MiB, 4648 MiB, 97 %, 25 %
Other software does need to be recompiled as it links to specific version of libraries rather than the generic libName.so (lammps).
Script ~hmeij/slurm/run.centos.lammps, setup env, get help page.
/share/apps/intel/parallel_studio_xe_2016_update3/compilers_and_libraries_2016.3.210/linux/bin/intel64/ifort
/usr/local/cuda/bin/nvcc
/share/apps/CENTOS7/openmpi/4.0.4/bin/mpirun
/share/apps/CENTOS7/python/3.8.3/bin/python
/share/apps/CENTOS7/lammps/25Apr2023/cuda-11.2/lmp_mpi-cuda-single-single
linux-vdso.so.1 => (0x00007ffd714ec000)
libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007fe443b9a000)
libcudart.so.11.0 => /usr/local/cuda/lib64/libcudart.so.11.0 (0x00007fe44390b000)
libcuda.so.1 => /lib64/libcuda.so.1 (0x00007fe442223000)
libcufft.so.10 => /usr/local/cuda/lib64/libcufft.so.10 (0x00007fe436a74000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fe436870000)
libmpi.so.40 => /share/apps/CENTOS7/openmpi/4.0.4/lib/libmpi.so.40 (0x00007fe43655b000)
libstdc++.so.6 => /share/apps/CENTOS7/gcc/6.5.0/lib64/libstdc++.so.6 (0x00007fe4361d9000)
libm.so.6 => /lib64/libm.so.6 (0x00007fe435ed7000)
libgcc_s.so.1 => /share/apps/CENTOS7/gcc/6.5.0/lib64/libgcc_s.so.1 (0x00007fe435cc0000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe435aa4000)
libc.so.6 => /lib64/libc.so.6 (0x00007fe4356d6000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe443def000)
librt.so.1 => /lib64/librt.so.1 (0x00007fe4354ce000)
libopen-rte.so.40 => /share/apps/CENTOS7/openmpi/4.0.4/lib/libopen-rte.so.40 (0x00007fe435218000)
libopen-pal.so.40 => /share/apps/CENTOS7/openmpi/4.0.4/lib/libopen-pal.so.40 (0x00007fe434f09000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fe434d06000)
libz.so.1 => /lib64/libz.so.1 (0x00007fe434af0000)
Large-scale Atomic/Molecular Massively Parallel Simulator - 28 Mar 2023 - Development
Usage example: lmp_mpi-cuda-single-single -var t 300 -echo screen -in in.alloy
List of command line options supported by this LAMMPS executable:
<snip>
# hmmm, using -suffix gpu it does not jump on gpus, generic non-gpu libthread error
# same version rocky8/cuda-11.6 works, centos7/cuda-10.2 works, all "make" compiles
# try "cmake" compile on n33-n36
# libspace tarball download fails on file hash and
# yields a status: [1;"Unsupported protocol"] error for ML-PACE
# without ML-SPACE hash fails for opencl-loarder third partty, bad url
# https://download.lammps.org/thirdparty/opencl-loader-opencl-loadewer-version...tgz
# then extract in _deps/ dir
# and added -D GPU_LIBRARY=../lib/gpu/libgpu.a ala QUIP_LIBRARY
# that works, cmake compile binary jumps on multiple gpus
[hmeij@n35 sharptail]$ mpirun -n 2 \
/share/apps/CENTOS7/lammps/25Apr2023/cuda-11.2/cmake/single-single/lmp \
-suffix gpu -in in.colloid
[root@greentail52 ~]# ssh n35 gpu-process
gpu_name, gpu_id, pid, process_name
Tesla K20m, 0, 9911, /share/apps/CENTOS7/lammps/25Apr2023/cuda-11.2/cmake/single-single/lmp
Tesla K20m, 1, 9912, /share/apps/CENTOS7/lammps/25Apr2023/cuda-11.2/cmake/single-single/lmp
# some stats, colloid example
1 cpu, 1 gpu
Total wall time: 0:05:49
2 cpus, 2 gpus
Total wall time: 0:03:58
4 cpus, 4 gpus
Total wall time: 0:02:23
8 cpus, 4 gpus
Total wall time: 0:02:23
# but the ML-PACE hash error is different, so no go there