cluster:176

HPC Power

As part of our reevaluation of our data center cooling capacity and overhaul, we need to get a handle on non-emergency power consumption in data center. This will be done by a third party consultant by clamping power cables in the penthouse of Excley. So I bought myself a metered PDU and have been busy plugging entire racks into it one at a time. I then measure “idle” and “peak” amperage consumption. The end picture should reveal how much Kw the HPC consumes of all non-emergency power (that is not on enterprise UPS). And we'll figure out how much Physical Plant spends supporting the HPC. Thanks PP!

“Idle” means not very busy, most of the time something is happening anyway (ie backups)
“Peak” means very busy, most queues entirely full.
Metered Amps * 208V = Watts
Watts * (24 * 365)/1000 = KwH/Y
kwH/Y * $0.125 = $/Y (mix cogen and utility, 2013 assumption by Pete of PP)
Double the $/Y to include cooling costs (Pete of PP assumption)

HPC		Idle	Peak	Idle	Peak	Idle	Peak	Idle	Peak
Rack	L6-30P	Amps	Amps	Watts	Watts	KwH/Y	KwH/Y	$/Y	$/Y	Note
R4R04	3	22	37	4,576	7,696	40,085	67,416	5,010	8,427	“sharptail rack”
R4R05	4	17	34	3,536	7,072	30,975	61,950	3,872	7,743	“greentail rack”
R4R06	2	6		1,204		8,970		1,121		empty
R4R07	2	14	34	2,912	7,072	25,509	61,950	3,189	7,744	“mw128/tinymem”
R4R08	2	4		832		7,288		911		mostly empty
Total	13	63	105	13,060	21,840	114,405	191,318	14,300	23,915

In 2013 we found for our Dell Racks purchased in 2006 (full of PE1950 and PE2950 servers) Replace Dell Racks

115,000 KwH/Y per rack for power
14,375 $/Y per rack for power

So a great improvement in energy efficiency. We probably grew from near 5 teraflops to near 60 teraflops in this time period (double precision, floating point, theoretical).

Metered

R4R04 rack: three L6-30P supplies

All dual power supply nodes
- n33-n37 one gpu/node, 94/120 cpu jobs = 5 amps
- n33-n37 above with 3 network switches 8 amps
- m33-n37 need to get data all gpus working = 7 amps
- n78 with 0 jobs = 0 amps, all gpus working = 2 amps
- mindstore0 and ringtail = 1 amps
- cottontail and shartptail (no rsyncs) = 1 amps
- cottontail and shartpail (rsyncs) = ?? amps ←– DO
- under modest load 8+2+1+1=12, observed 14 amps on left PDUs (front/back)
  - split n78 to right PDU (right side 2 amps plus all single power)
  - all storage 50% on UPS in R4R05, minuscule reduction
All single power supply nodes add to right PDU
- n38-n41 idle load = 3 amps, so 6 amps for all nodes
- n38-n41 full load = 7 amps, so 14 for all nodes
TOTAL IDLE load (14)+(6+2)= 22 amps
TOTAL PEAK load (14+7)+(14+2)= 37 amps …37/90 or 41%

R4R05 rack: four L6-30P supplies

All dual power supplies
- greentail, sanscratch storage, idle = 2 amps
- greentail, 1TB write to sanscratch = 2 amps
- above + 3 switches, petal/swallow/cotton2 tails, hpcmon = 5 amps
- n1-n32 (25 bodes), idle = 5 amps
- gaussian jobs on n1-n32 = 12 amps
TOTAL IDLE load 2+5= 7 amps x2 = 17 amps
TOTAL PEAK load 5+12= 17 amps x 2 = 34 amps … 34/120 or 28%

R4R06 rack: two L6-30P supplies

Mostly empty rack
- AC unit working = 8 amps (120V)
- two switches = 2 amps (208V)
- TOTAL 6 amps

R4R07 rack: two L6-30P supplies

All single power supplies
- mw128 10 nodes, idle = 4 amps, for all nodes 8 amps
- mw128 10 nodes, peak = 8 amps, for all nodes 16 amps
  - 11/28/2018 17 amps observed (with 6/18 nodes at load 36)
  - 03/05/2019 23 amps observed (…wengang jobs, mw128 nodes only! 23+16=39of60 possible)
- two switches = 2 amps
- tinymem 7 nodes, idle = 2 amps, for all nodes 4 amps
- tinymem 7 nodes, peak = 8 amps, for all nodes 16 amps (DOM,no hard disk)
TOTAL IDLE load 8+2+4= 14 amps
TOTAL PEAK load 16+2+16= 34 amps …34/60 is 51%

R4R08 rack: two L6-30P supplies

Mostly empty rack
All dual power supplies
- ringtail, mindstoresrv1 storage = 2 amps
- two switches = 2 amps
TOTAL 4 amps

Back