User Tools

Site Tools


cluster:225

This is an old revision of the document!


Table of Contents


Back

Cuda

Upgrading Cuda to latest drivers and tooltkit that supports our GeForce RTX 2080 SUPER (and Ti) gpu models (queues exx96 and amber128). Before we embark doing all nodes, we need to test backward compatibility and assess how troublesome the upgrade might be.

First look at what is end-of-life these days … middle of page go to table table GPUs supported, scroll down keeping an eye on Turing and Lovelace columns. Cuda version 10.2 (our RTX2080S) is still supported in version 12.4 so we can install latest toolkit and drivers.

And it looks like Tesla P4, P100 and V100 are next on the end of extended life list, consult table “A. End-of-Software-Support Dates for GPUs Supported by NVIDIA vGPU Software”

Find the drivers at this page (non-legacy gpu models as opposed to legacy models cuda toolkit), look for link Linux x86_64/AMD64/EM64T Latest Production Branch Version: 550.67

Linux x86_64/AMD64/EM64T
Latest Production Branch Version: 550.67
Supported Products Tab
(make sure the models are listed – they are)

Agree&Download the NVIDIA-Linux-x86_64-550.67.run driver file.

Now obtain the latest toolkit as our gpu models are supported as verified above

and you will get instructions

# centos 7
wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda_12.4.0_550.54.14_linux.run
sudo sh cuda_12.4.0_550.54.14_linux.run

# rocky 8 (same file)
wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda_12.4.0_550.54.14_linux.run
sudo sh cuda_12.4.0_550.54.14_linux.run

The complete Linux Installer Guide is at

Installation

  • On CentOS 7 hop on vlan52, yum update followed by reboot (EOL 06/30/2024)
    • skip; libnvidia-container repo has problems and may be crucial
  • On Rocky 8
    • yum update; skip this step for now …
  • Make both files executable (same files for both)
  • Install driver first (Nvidia-Linux-*.run, reboot)
  • Then install toolkit (cuda-12.4-*.run, reboot)
# n78 first ...
# n79 second (went fine)
# make sure /usr/src/kernels/$(uname -r) exists else 
# scp into place from n100 (centos8, possibly caused by warewulf...)

sh ./NVIDIA-Linux-x86_64-550.67.run

32 bit compat? no
dkms build? yes
rebuild initramfs? yes
error nvidia module can not be loaded
reboot fixed that

[root@n78 ~]# nvidia-smi
Mon Apr  1 14:50:33 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.67                 Driver Version: 550.67         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1080 Ti     Off |   00000000:02:00.0 Off |                  N/A |
| 19%   38C    P0             59W /  250W |       0MiB /  11264MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce GTX 1080 Ti     Off |   00000000:03:00.0 Off |                  N/A |
| 16%   25C    P0             58W /  250W |       0MiB /  11264MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce GTX 1080 Ti     Off |   00000000:81:00.0 Off |                  N/A |
| 18%   28C    P0             59W /  250W |       0MiB /  11264MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce GTX 1080 Ti     Off |   00000000:82:00.0 Off |                  N/A |
| 19%   29C    P0             58W /  250W |       0MiB /  11264MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+


rm -f /usr/local/cuda # soft link to current version
/usr/bin/nvidia_uninstall  # probably not needed, made no difference > Continue

sh cuda_12.4.0_550.54.14_linux.run

===========
= Summary =
===========

Driver:   Installed
Toolkit:  Installed in /usr/local/cuda-12.4/

Please make sure that
 -   PATH includes /usr/local/cuda-12.4/bin
 -   LD_LIBRARY_PATH includes /usr/local/cuda-12.4/lib64, or, add /usr/local/cuda-12.4/lib64 to /etc/ld.so.conf and run ldconfig as root

To uninstall the CUDA Toolkit, run cuda-uninstaller in /usr/local/cuda-12.4/bin
To uninstall the NVIDIA Driver, run nvidia-uninstall
Logfile is /var/log/cuda-installer.log

# check /usr/local/cuda link, clean up others

[root@n78 ~]# export PATH=/usr/local/cuda/bin:$PATH
[root@n78 ~]# export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
[root@n78 ~]# which nvidia-smi
/usr/bin/nvidia-smi

[root@n78 ~]# which nvcc
/usr/local/cuda/bin/nvcc

[root@n78 ~]# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Feb_27_16:19:38_PST_2024
Cuda compilation tools, release 12.4, V12.4.99
Build cuda_12.4.r12.4/compiler.33961263_0

# slurm finds gpus? yes...

[2024-01-18T12:46:29.322] debug:  gpu/generic: init: init: GPU Generic plugin loaded
[2024-01-18T12:46:29.322] Gres Name=gpu Type=geforce_gtx_1080_ti Count=4

Testing

Rocky 8 on n78

Previous software compiled using OpenHPC modules pulls in cuda/11.6 module. So lets test if that works with new 550 driver and 11.2 toolkit. You can follow these steps to test any other software.

# ssh to compute node and load module
[hmeij@n78 ~]$ module load amber/22

# identify object of interest
[hmeij@n78 ~]$ which pmemd.cuda
/share/apps/CENTOS8/ohpc/software/amber/22/bin/pmemd.cuda

# feed to ldd, look for missing libraries

[hmeij@n78 ~]$ ldd `which pmemd.cuda`
        linux-vdso.so.1 (0x00007ffeb51ea000)
        libemil.so => /share/apps/CENTOS8/ohpc/software/amber/22//lib/libemil.so (0x00007f99ae5aa000)
        libnetcdff.so.6 => /share/apps/CENTOS8/ohpc/software/amber/22//lib/libnetcdff.so.6 (0x00007f99ae315000)
        libkmmd.so => /share/apps/CENTOS8/ohpc/software/amber/22//lib/libkmmd.so (0x00007f99ae10a000)
        libcufft.so.10 => /usr/local/cuda/lib64/libcufft.so.10 (0x00007f99a567d000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f99a5479000)
        libcurand.so.10 => /usr/local/cuda/lib64/libcurand.so.10 (0x00007f999f033000)
        libcublas.so.11 => /usr/local/cuda/lib64/libcublas.so.11 (0x00007f99955bf000)
        libnetcdf.so.15 => /lib64/libnetcdf.so.15 (0x00007f9995273000)
        libarpack.so => /share/apps/CENTOS8/ohpc/software/amber/22//lib/libarpack.so (0x00007f9995056000)
        libopenblas.so.0 => /lib64/libopenblas.so.0 (0x00007f999328e000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f9992f0c000)
<snip>

# does it work?

[hmeij@n78 ~]$ pmemd.cuda --version
pmemd.cuda: Version 22.0

So you'll notice libraries are pulled in from …/amber/22/… and …/cuda/lib64/…. The latter is technically wrong because that generic path points to 12.4 now not 11.6. The problem was the module did not specify the absolute path to cuda-11.6. Fixed that and provided soft links for individual libraries in cuda-12.4. Not the way to do it but it works. So toolkit 11.6 is compatible with 12.4 using the newly installed 550 driver. Amber 20/22 and Lammps 25Apr2023/7Feb2024 all work. See script ~hmeij/slurm/run.rocky.2.

pmemd.cuda: error while loading shared libraries: libcufft.so.10
...argh, Amber links in specific version of libraries

# not the way to do things...amber 20&22
  cd /usr/local/cuda-12.4/targets/x86_64-linux/lib/
  ln -s /usr/local/cuda-11.6/lib64/libcufft.so.10 
  ln -s /usr/local/cuda-11.6/lib64/libcublas.so.11
  ln -s /usr/local/cuda-11.6/lib64/libcublasLt.so.11

# not the way to do things...lammps 7Feb2024 and 25Apr2023
  ln -s /usr/local/cuda-11.6/lib64/libcudart.so.11.0 

CentOS 7 on n79

Next test is to see if older software runs compatible with newer drivers. We test that by running a gpu program against new 550 driver and cuda toolkit 9.2 and see if it works (~hmeij/slurm/run.centos7.2).

# same steps as above using n79
# set up the environment the script uses

[hmeij@n79 ~]$ which nvcc
/usr/local/n37-cuda-9.2/bin/nvcc

[hmeij@n79 ~]$ source /usr/local/amber20/amber.sh 

[hmeij@n79 ~]$ which pmemd.cuda
/usr/local/amber20/bin/pmemd.cuda

[hmeij@n79 ~]$ pmemd.cuda --version
pmemd.cuda: Version 20.0

# going further back in time

[hmeij@n79 ~]$ source /usr/local/amber16/amber.sh 

[hmeij@n79 ~]$ which pmemd.cuda
/usr/local/amber16/bin/pmemd.cuda

[hmeij@n79 ~]$ pmemd.cuda --version
pmemd.cuda: Version 14.0

So it appears that all our cuda versions can use the new 550 driver that comes with cuda-12.4 toolkit. Two other cuda versions have not been tested but should function as well (cuda-11.2 in mwgpu and cuda-10.2 in exx96). But in these queues cuda-9.2 is present on local hard disk and software was compiled against that toolkit so which queue to use did not matter. (compilations against 10.2 did not run in 9.2, as expected). I was able to test and run lammps in 11.2 consult the file ~hmeij/slurm/centos.2

These compatibility results are way, way better than expected. Yea.

Any new software compilations will use module cuda/12.4.


Back

cluster/225.1712327584.txt.gz · Last modified: 2024/04/05 14:33 by hmeij07