Differences

This shows you the differences between two versions of the page.

--- cluster:182 [2019/08/12 17:05]
hmeij07 [Lammps]
+++ cluster:182 [2019/08/12 17:47]
hmeij07 [Scripts]
@@ Line 72: / Line 72: @@
 Precision for GPU calculations
-  * DD -D_DOUBLE_DOUBLE  # Double precision for all calculations
+  * [DD] -D_DOUBLE_DOUBLE  # Double precision for all calculations
-  * SD -D_SINGLE_DOUBLE  # Accumulation of forces, etc. in double
+  * [SD] -D_SINGLE_DOUBLE  # Accumulation of forces, etc. in double
-  * SS -D_SINGLE_SINGLE  # Single precision for all calculations
+  * [SS] -D_SINGLE_SINGLE  # Single precision for all calculations
@@ Line 85: / Line 85: @@
 But the T4 shines in this application. The mixed or single precision modes compete well given the T4's price and  wattage consumption.
+==== Gromacs ====
+Gromacs was build on each of the nodes locally letting it select the optimal CPU (AVX, SSE) and GPU accelerators. The ''cmake'' flag -DGMX_BUILD_OWN_FFTW=ON yields a mixed precision compilation which is recommended. The we ran multidir options 01-04 on single GPU, and 01-08 and 01-16 on all 4 GPUs when possible.
+^  ns/day  ^  P100[1]  ^  P100[4]  ^  RTX[1]  ^  T4[1]  ^  T4[4]  ^  Notes  ^
+|  Mixed  |  |  |  254|  |  |  gpu=1, 01-04  |
+|  Mixed  |  |  551|  |  |  546|  gpu=4, 01-04  |
+|  Mixed  |  |  |  |  |  650|  gpu=4, 01-08  |
+|  Mixed  |  |  |  |  |  733|  gpu=4, 01-16  |
 ==== Scripts ====
@@ Line 147: / Line 159: @@
 </code>
+  * Gromacs
+<code>
+#!/bin/bash
+#SBATCH --nodes=1
+#SBATCH --nodelist=node9
+#SBATCH --job-name="T4 dd"
+#SBATCH --ntasks-per-node=32
+#SBATCH --gres=gpu:4
+#SBATCH --exclusive
+export PATH=$HOME/gromacs-2018/bin:$PATH
+export LD_LIBRARY_PATH=$HOME/gromacs-2018/lib:$LD_LIBRARY_PATH
+. $HOME/gromacs-2018/bin/GMXRC.bash
+rm -f gpu/??/c* gpu/??/e* gpu/??/s* gpu/??/traj* gpu/??/#* gpu/??/m*
+cd gpu
+# T4
+#export CUDA_VISIBLE_DEVICES=0123
+mpirun -np 8 gmx_mpi mdrun -maxh 1 -gpu_id 0123 \
+-nsteps 1000000 -multidir 05 06 07 08 \
+-ntmpi 0 -npme 0 -s topol.tpr -ntomp 0 -pin on -nb gpu
+</code>
+And GPU utilization was outstanding.
+[heme@login1 gromacs-2018]$ ssh node9 ./gpu-info\\
+id,name,temp.gpu,mem.used,mem.free,util.gpu,util.mem\\
+, Tesla T4, 66, 866 MiB, 14213 MiB, 98 %, 9 %\\
+, Tesla T4, 67, 866 MiB, 14213 MiB, 98 %, 9 %\\
+, Tesla T4, 66, 866 MiB, 14213 MiB, 99 %, 9 %\\
+, Tesla T4, 64, 866 MiB, 14213 MiB, 97 %, 9 %\\
 \\
 **[[cluster:0|Back]]**

DokuWiki

User Tools

Site Tools

Differences

Page Tools