This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:175 [2018/09/22 12:46] hmeij07 [Lammps] |
cluster:175 [2018/11/29 17:57] hmeij07 |
||
---|---|---|---|
Line 3: | Line 3: | ||
**[[cluster: | **[[cluster: | ||
- | ==== GTX vs P100 & K20 ==== | + | As of |
+ | --- // | ||
+ | The P100 with 12 GB is end of life, replaced by the P100 16 GB and \\ | ||
+ | The GTX 1080Ti will be replaced by the GTX2080Ti (no specs yet and not certified for Amber18, yet)\\ | ||
+ | As of | ||
+ | --- // | ||
+ | New GROMACS performance benchmarks featuring 2x and 4x NVIDIA RTX 2080 GPUs are now available. The RTX 2080 graphics card utilizes the new NVIDIA Turing GPU architecture and provides up to 6x the performance of the previous generation. (Exxact newsletter) | ||
- | Comparing these GPUs yields the following | + | |
+ | ==== P100 vs GTX & K20 ==== | ||
+ | |||
+ | ^ ^ P100 ^ GTX ^ K20 ^ ^ | ||
+ | | cores | 3,584 | 3,584 | 2,496 | count | | ||
+ | | mem | 12/16 | 11 | 5 | gb | | ||
+ | | ghz | 2.6 | 1.6 | 0.7 | speed | | ||
+ | | flops | 4.7/ | ||
+ | |||
+ | Comparing these GPUs yields the following | ||
Credits: This work was made possible, in part, through HPC time donated by Microway, Inc. We gratefully acknowledge Microway for providing access to their GPU-accelerated compute cluster. | Credits: This work was made possible, in part, through HPC time donated by Microway, Inc. We gratefully acknowledge Microway for providing access to their GPU-accelerated compute cluster. | ||
Line 30: | Line 45: | ||
</ | </ | ||
+ | |||
+ | Look at these gpu temperatures, | ||
==== Lammps ==== | ==== Lammps ==== | ||
- | We can also not complain about gpu utilization in this example. | + | We can also not complain about gpu utilization in this example. |
On our GTX server best performance was a ratio of 16:4 cpu:gpu for 932,493 tau/day (11x faster than our K20). However scaling the job to a ratio of cpu:gpu of 4:2 yields 819,207 tau/day which means a quad server can deliver about 1.6 million tau/day. | On our GTX server best performance was a ratio of 16:4 cpu:gpu for 932,493 tau/day (11x faster than our K20). However scaling the job to a ratio of cpu:gpu of 4:2 yields 819,207 tau/day which means a quad server can deliver about 1.6 million tau/day. | ||
- | A single P100 beat this easily coming in at 2.6 million tau/day. Spreading the problem over more gpus did raise overall performance to 3.3 million tau/day. However, four cpu:gpu 1:1 jobs would achieve slightly over 10 million tau/day. That is almost 10x faster than the GTX server. | + | A single P100 gpu beats this easily coming in at 2.6 million tau/day. Spreading the problem over more gpus did raise overall performance to 3.3 million tau/day. However, four cpu:gpu 1:1 jobs would achieve slightly over 10 million tau/day. That is almost 10x faster than our GTX server. |
< | < | ||
Line 61: | Line 78: | ||
</ | </ | ||
+ | |||
+ | ==== Lammps (PMMA) ==== | ||
+ | |||
+ | Using a material called PMMA (https:// | ||
+ | |||
+ | ^ gpu ^ cpus ^ ns/ | ||
+ | | 1 P100 | 4 | 89 | x4 | 356 | | ||
+ | | 1 GTX | 6 | 90 | x4 | 360 | | ||
+ | | 1 K20 | 6 | 47 | x4 | 188 | | ||
+ | |||
+ | That means the P100 works as well as the GTX. The K20 works at 50% the performance level of the others which is impressive for this old gpu. | ||
+ | |||
+ | |||
+ | |||
==== Gromacs ==== | ==== Gromacs ==== | ||
Line 66: | Line 97: | ||
Gromacs has shown vastly improved performance between versions. v5 delivered about 20 ns/day per K20 server and 350 ns/day on GTX server. v2018 delivered 75 ns/day per K20 server and 900 ns/day on GTX server. A roughly 3x improvement. | Gromacs has shown vastly improved performance between versions. v5 delivered about 20 ns/day per K20 server and 350 ns/day on GTX server. v2018 delivered 75 ns/day per K20 server and 900 ns/day on GTX server. A roughly 3x improvement. | ||
- | On the P100 I could not invoke the multidir option of gromacs (have run it on GTX, weird). The utilization of the gpu drops as more and more gpus are deployed. | + | On the P100 test node, I could not invoke the multidir option of gromacs (have run it on GTX, weird). The utilization of the gpu drops as more and more gpus are deployed. |
< | < | ||
Line 75: | Line 106: | ||
localhost, | localhost, | ||
localhost, | localhost, | ||
- | gmx_mpi mdrun -gpu_id 0123 -ntmpi 0 \ | + | gmx_mpi mdrun -gpu_id 0123 -ntmpi |
-s topol.tpr -ntomp 4 -npme 1 -nsteps 20000 -pin on -nb gpu | -s topol.tpr -ntomp 4 -npme 1 -nsteps 20000 -pin on -nb gpu | ||
Line 83: | Line 114: | ||
| | ||
- | gpu=4 mpi=25 ntomp=4 -npme 1 | + | gpu=4 mpi=25 |
Performance: | Performance: | ||
gpu=3 (same) | gpu=3 (same) | ||
Line 96: | Line 127: | ||
</ | </ | ||
+ | |||
+ | ==== Gromacs 2018.3 ==== | ||
+ | |||
+ | The multidir not running in Gromacs 2018 is a bug in the code clashing with the call MPI_Barrier (communication timing error). | ||
+ | |||
+ | < | ||
+ | |||
+ | # multidir -gpu_id 0123 with four simultaneous gromacs processes | ||
+ | |||
+ | -np 8 -ntomp | ||
+ | 01/ | ||
+ | 02/ | ||
+ | 03/ | ||
+ | 04/ | ||
+ | |||
+ | -np 16 -ntomp | ||
+ | 01/ | ||
+ | 02/ | ||
+ | 03/ | ||
+ | 04/ | ||
+ | |||
+ | multidir -gpu_id 00112233 with eight simultaneous gromacs processes | ||
+ | sharing the gpus, 2 processes per gpu | ||
+ | |||
+ | -np 8 -ntomp | ||
+ | Error in user input: | ||
+ | The string of available GPU device IDs ' | ||
+ | device IDs | ||
+ | |||
+ | </ | ||
+ | |||
+ | That last error when loading multiple processes per gpu is *not* according to their documentation. So the multidir performance is similar to previous single dir performance but still lags GTX performance by quite a bit. Albeit, there is room in utilization rate of the gpus. | ||
+ | |||
+ | |||
+ | ==== What to Buy ==== | ||
+ | |||
+ | * Amber folks: does not matter | ||
+ | * Lammps folks: P100 nodes please, although in hybrid runs does not matter | ||
+ | * Gromacs folks: GTX nodes please | ||
+ | |||
+ | Remember that GTX gpus placed in Data Center voids their warranty. | ||
+ | |||
\\ | \\ |