DokuWiki

A Novella: Replace the Dell Racks with new hardware
Subtitle: A win-win solution proposed by Physical Plant and ITS

Once upon a time, back in 2013, two Dell racks full of compute nodes, sat noisily chewing away energy on the 5th floor of Science Tower. They sucked in nicely cooled air from the floor spewing it out the back of the racks at 105-110 degrees (F). They were giving the three Liebert cooling towers a run for their BTUs. So much so that if one failed the Dell racks needed to be powered down to avoid the data center reaching temperatures beyond 95 degrees (F). The Dell racks were in a foul mood since that last event not too long ago. And so, day after day, they consumed lots of BTUs, and with the ample supply of Watts coming from their L6-30 roots, converted it all into heat. Tons of heat, making life lousy for the Liebert family. Oh, and they performed some computational work too, but if even they did not, the energy consumption remained the same. That's a fact. They were 6 years old and determined to make it to 12. So the story goes.

The Dell racks contain 30 compute nodes, two UPS units, two disks arrays and two switches. We have measured 19 nodes power consumption (pulling one the dual power units out) with a Kill-A-Watt meter for over 775+ hours. The mean power consumption rate is 418.4 watts. That totals to 109,956 KwH/year in power consumption ((watts/1000 Kw per hour) * 24 hours * 365 days * 30 servers). This is a low water mark, it only takes into account thwe compute nodes but that will be the majorioty of energy consumption. We also measured one rack's consumption at the utility panel and Peter's caculation yields 126,000 KwH/year which can be considered a high water mark.

Next we need to convert to a dollar value. A residential electric bill contains a KwH cost as well as generation, distribution, transmission and other cost factors. Typically the total KwH cost is 50% of the total bill. Utility power in the data center comes from our cogen plant as well as CP&L. The model for cogen attempts to balance both; 15% of CP&L utility and 85% cogen generated. If the cogen is down, like for maintenance this jumps to 100% CP&L. One also needs to factor in natural gas costs, heat reclamation costs, etc. If we set the cogen KwH cost at 6.25 cents we'll need to double that to 12.5 cents and use that.

Based on 12.5 cents per Kwh (all costs included) the Dell compuet nodes consumed $13,744.50 per year in power. Best guess is cooling costs are at least that (another possible low water mark) so the total cost for both power and colling consumption is $27,489 per year.

Next step was to collect quotes for a target budget of $82K, 3 years of Dell energy consumption, an arbitrary length of time. That's so we can downscale from there because the new racks ofcourse still consume energy. Four quotes were obtained and they show a similar pattern. Here is the comparison on key features.

Old hardware: 30 nodes, 2.66 ghz, 4 mb L-cache (for cpu), 240 cores (job slots), 80 gb local drive, 340 gb total ram, 12,555 watts (power no cooling), 670 gigaflops (actual measure)

New hardware: 14 nodes, 2.60 ghz, 20 mb L-cache (for cpu), 224 cores (job slots), 1TB local drive, 1,792 gb total ram, 5,400 watts (power no cooling), 4,659 gigaflops (theoretical)

In the representative example above we almost match Dell's job slots total (224 vs 240 However the power consumption would be 2,700 watts and the gigaflops 2,329 (we're doing more work per clock cycle but not increasing performance of the CPUs). That means the new hardware consumes 20% of the Dell's power consumption (does not include not cooling). The total KwH cost of running the new hardware is 20,402 KwH/year or $5,100.

The end.

Back