• 2009 Dell measurements put power consumption at 100,740 KwH per year (how?)
• 2013 Dell measurements (see bottom table) put power consumption at 109,956 KwH per year (for 30 servers)

All dollar figures based on spreadsheet using March 2009 values

• This will overestimate actual costs because now we have cogen generation
  • 64% off peak, 36% on peak consumption
  • includes distribution and transmission charges

• one L6-30 connector can sustain 7,500 watts at 250 volts
  • we run at 208V which can sustain 6,240 watts per connector
• wattometer: example 7 Kwh consumed in 19 hours = 368 watts per hour on average
• 1,000 Btus equal 293 watts
• assume cost of $ 0.10 per Kwh (kilo watt hour)
• annual Kwh = watts/1000 (from wattometer) * 24 * 365 * 30! (6 servers died)
• power costs per year = $0.10/Kwh * watts/1,000 * 24 hours * 365 days

• 1.5 years old
• 32 blade servers in 8 2U-shelves
• 4 L6-30
  • 10,602 watts = 92,856 kwh annual = $21,465
  • 36,175 btus = 10,600 watts = 92,856 kwh annual = $21,465
• total costs: $42,930/year
• “roughly” $4,770 per 2U-shelves (8 2U-shelves + 1 head node of 2U)

• 5.5 years old
• 36 1U rack servers in 18 2U-shelves
• 8 L6-30
  • 11,500 watts = 100,740 kwh annual = $23,098
  • 39,240 btus = 11,497 watts = 100,713 kwh annual = $23,093
• total costs: $46,091/year
• “roughly” $2,310 per 2U-shelves (18 2U-shelves + 2 head nodes of 2U)

• new acquisition
• 13 1U rack servers in 6.5 2U-shelves
• 2 L6-30
  • 5,250 watts = 45,990 kwh annual = $11,754
  • 17,913 btus = 5,249 watts = 45,981 kwh annual= $11,752
• total costs: $23,506/year
• “roughly” $3,134 per 2U-shelves (6.5 2U-shelves + 1 head node of 2U)

• new acquisition
• 4 rack servers in 4 2U-shelves
• 2 L6-30
  • 5,900 watts = 51,684 kwh annual = $12,934
  • 20,171 btus = 5,898 watts = 51,667 kwn annual = $12,930
• total costs: $25,864
• “roughly” $2,874 per 2U-shelves (8 2U-shelves + 1 head node of 2U)

This is what the picture would look like using the 2009 spreadsheet data

What we should look at is teraflops replacement …

• 0.66 teraflops, measured with Linpack (actual)
• E5-2660 is rated at 140 Gflops so one node provides 280 Gflops (theoretical)
• 2 and 1/3rd 1U nodes then replace the cluster
• acquisition costs
  • 3 x $5,782 = $17,346 + PDUs, ethernet switch, cables
  • 48 cores, 840 Gflops
  • power/AC costs: 3($23,506/13) x3 years = $16,273
• add both costs of new hardware and divide by dell total annual costs power+cooling
  • ROI: 3/4 year - but this is on a teraflop by teraflop basis

And if we double the teraflops…1.6 Tflops (actually + 1 Tflops), 1.5 years ROI for 96 cores.

• Dell
  • left rack (from rear) LCBR 01, 09, 08, 02 and UPS on 01
  • right rack (from rear) LCBR 03, 04 and UPS on 07
• BSS
  • LCBR CIR 25/27, 21/23, 22/24, 26/28
• HP
  • LCBR CIR 29/31, 28/30
  • LCBR 06
  • (#3 on Enterprise UPS) PDI1 CIR 31,33

Actual data obtained April - May, 2013

Kill-A-Watt Meter Stats; pulled one power supply … mean/1000 * 24 * 365 * 30!

• Average for “still running” nodes is 418.4 watts or 109,956 KwH/year
• (watts/1000 Kw per hour) * 24 hours * 365 days * 30 servers
• In 2009 dollars that is $25,000 in power (plus that in cooling) or $50,000 per year
• What is that in 2013 dollars (with cogen online in Pine Street)?

We have measured 19 nodes power consumption (pulling one unit out) with a Kill-A-Watt meter over 775+ hours to arrive at a mean consumption rate of 418.4 watts. That totals to 109,956 KwH/year in power consumption ((watts/1000 Kw per hour) * 24 hours * 365 days * 30 servers), which is a low water mark as the racks in question also contain switches, two UPS units, and a disk array. (Note: Peter did a side calculation using the 4 watt measurement on panel directly and came up with 126,000 KwH/year which can be considered the high water mark estimate).

Based on 12.5 cents per Kwh (this is an all inclusive cost including natural gas cost, heat recoup costs, distribution, maintenance etc) the hardware burns away $13,744.50 per year. Best guess is cooling costs are at least that (another possible low water mark) so the total cost for all power consumption is $27,489 per year. If we run the hardware for another 3 years that total cost is $82,467.

If we could replace, or approximate, the 30 compute nodes' computational power (0.6 teraflops) and job slots (240 cores) with new hardware that consumes 50% less in power, our ROI is 6 years based on the low water mark numbers. When using the high water mark the ROI is 5 years.

For $82K one could get CPU-HPC wise …

• Intel Xeon Metrics
  • E5-2650v2 “ivy” 128 gflops
  • E5-2680v2 172.8 gflops

Back