Back

2018 GPU Expansion

Important notes … about GeForce GTX1080Ti

☎ From Nvidia web site: Warranted Product is intended for consumer end user purposes only, and is not intended for datacenter use and/or GPU cluster commercial deployments (“Enterprise Use”). Any use of Warranted Product for Enterprise Use shall void this warranty.

☎ From Exxact web site: Exxact AMBER Certified AMBER MD Workstations and Clusters, in addition to being numerically validated using custom GPU validation suites, also come with an optimized version of AMBER 16 that has been developed in a collaboration between lead AMBER developer Ross Walker (SDSC), NVIDIA and Exxact.

So it is a grey area if GTX1080Ti data center research use is warranted or not. Most quotes come with language that all warranty issues will be the handled between card owner and card issuer. If we go with consumer grade GTX we should add some self spares…dev budget? $700 apiece. Tesla P100 is enterprise level, $6,000 apiece.

Quotes then …

Vendor A B C D Notes
Quote #1 #2 #3 #1 #2 #1 #2 #1 #2
Nodes 11 9 8 9 7 8 5 6 6 5-22 U
Cpus 22 18 16 18 14 16 10 12 12
Cores 220 180 224 216 168 192 120 168 144 physical
Gpus 22 36 32 18 28 32 10 12 12
Cores 79 129 115 66 100 115 36 43 43 k physical
Teraflops 7.7+7.8 6.3+13 18+11 18+6.4 14+10 16+11 10+47 12+56 12+56 cpu+gpu (dpfp)
$/TFlop 6,335 5,073 3,510 4,299 3,946 3,822 1,795 1,422 1,638 hpc 38+25
Per Node
Chassis 2U 2U 2U 2U 2U 1U 1U 1U 1U depth of rails?
CPU 2 2 2 2 2 2 2 2 2 “skylake”
4114 4114 5120 6126 6126 6126 6126 5120 6126 model
10+10 10+10 14+14 12+12 12+12 12+12 12+12 14+14 12+12 phys+log/ical
2.2 2.2 2.2 2.6 2.6 2.6 2.6 2.2 2.6 Ghz max 3-3.7
85 85 105 125 125 125 125 105 125 Watts
13.75 13.75 19.25 19.25 19.25 19.25 19.25 19.25 19.25 L3 cache MB
16 16 32 32 32 32 32 32 32 dflops/cycle
DDR4 192 192 192 192 192 192 192 192 192 GB memory
2666 2666 2666 2666 2666 2666 2666 2666 2666 Mhz
Drives 480 480 480 2×240 2×240 1024 1024 480 480 GB storage
2.5“s 2.5”s 2.5“s 3.5”s 3.5“s 2.5”h 2.5“h 2.5”s 2.5“s SSD/HDD
960 960 GB scratch
GPU 2 4 4 2 4 4 2 2 2
GTX GTX GTX GTX GTX GTX Tesla Tesla Telsa warranty note
1080 1080 1080 1080 1080 1080 P100 P100 P100 model
11 11 11 11 11 8! 12 12 16! GB memory
1.6 1.6 1.6 1.6 1.6 1.6 1.9 1.9 1.9 Ghz, max 1.9
250 250 250 250 250 250 250 250 250 watts
Image 1 1 1 ? ? ? ? 1 1 MPI flavors
CentOS7 y y y ? ? y y y y + all software
Nics 2 2 2 2 2 2 2 2 2 gigabit ethernet
Warranty 3 3 3 3 3 3 3 3 3 excl gtx cards
n78? n y y n y y n n n matches?
-1.8 -3.1 +1.8 +4.9 -5.8 +4.7 +2.3 -3.3 +11.4 Δ
Remember to add self spare gtx cards! No spares

GFLOPS = #chassis * #nodes/chassis * #sockets/node * #cores/socket * GHz/core * FLOPs/cycle

Note that the use of a GHz processor yields GFLOPS of theoretical performance. Divide GFLOPS by 1000 to get TeraFLOPS or TFLOPS.

http://en.community.dell.com/techcenter/high-performance-computing/w/wiki/2329

Find dpflops/cycle here https://www.aspsys.com/solutions/hpc-processors/intel-xeon-skylake/

Request

Early fall 2017 we purchased a 1U server from Exxact with four GTX1080Ti gpus and 128 gb memory with dual 8 core E5-2620 v4 cpus. These gpus have been performing well for us with speed ups (compared to our K20s); amber 5x, gromacs 2x, lammps 11x, and fsl's bedpostx a whopping 118x. Our “sweet spot cpu:gpu” ratios for the type of jobs we are running are;

Most of our jobs will only use one gpu at a time. We also have a dying cpu only jobs queue of HP blade servers. This expansion would run gpu jobs but also cpu only jobs. Hence we'd like to have more servers, which would allow a good mix of cpu only and cpu/gpu jobs.

We will supply:

Open to suggestions, modifications, substitutions. We'd prefer to go with the GTX1080Ti gpus which are still listed as certified for Amber 18.


Back