This is an old revision of the document!
Back
All campus utilities are within the physical plant chart of accounts. Departments are not charged.
Overview
Cluster | Blue Sky | Dell | HP | GPU1) | CPU2) | Comment |
| | 12/2006 | 11/2010 | 04/2013 | 04/2013 | |
Age (yrs) | 11 | 5.5 | 1.5 | 0 | 0 | |
Nodes (Nr) | 45 | 36 | 32 | 4 | 13 | |
Cores (Nr) | 90 | 256 | 256 | 3)16 gpu, 64 cpu | 208 | |
Ram (GB) | 1,080 | 384 | 340 | 80 gpu, 512 cpu | 1,664 | |
Teraflops | 4)0.5-0.7 | 5)0.66 | 6)1.5 | 7)18.71 + 8)1.1 | 9)3.6 | |
L6-30 (Nr) | 4 | 8 | 4 | 2 | 2 | |
Capacity10) (Watts) | 24,960 | 49,920 | 24,960 | 12,480 | 12,480 | |
Power (Watts11)) | N/A12) | 13)11,500 | 14)10,602 | 15)5,900 | 16)5,250 | |
Cooling (Btus17)) | | 18)39,240 | 19)36,175 | 20)20,171 | 21)17,913 | |
Annual Costs22) ($) | | 46,091 | 42,930 | 25,864 | 23,506 | 23)using 2009 data |
Details
All dollar figures based on spreadsheet using March 2009 values
Notes
one L6-30 connector can sustain 7,500 watts at 250 volts
wattometer: example 7 Kwh consumed in 19 hours = 368 watts per hour on average
1,000 Btus equal 293 watts
assume cost of $ 0.10 per Kwh (kilo watt hour)
annual Kwh = watts/1000 (from wattometer) * 24 * 365 * 30! (6 servers died)
power costs per year = $0.10/Kwh * watts/1,000 * 24 hours * 365 days
HP Nov-2010
1.5 years old
32 blade servers in 8 2U-shelves
4 L6-30
10,602 watts = 92,856 kwh annual = $21,465
36,175 btus = 10,600 watts = 92,856 kwh annual = $21,465
total costs: $42,930/year
“roughly” $4,770 per 2U-shelves (8 2U-shelves + 1 head node of 2U)
Dell Jan-2007
5.5 years old
36 1U rack servers in 18 2U-shelves
8 L6-30
11,500 watts = 100,740 kwh annual = $23,098
39,240 btus = 11,497 watts = 100,713 kwh annual = $23,093
total costs: $46,091/year
“roughly” $2,310 per 2U-shelves (18 2U-shelves + 2 head nodes of 2U)
Dell Jan-2013
Actual data obtained 11 April - 18 April, 2013
Watt-o-Meter Stats mean/1000 * 24 * 365 * 30!
Node | threads | kwh | time hh:mm | mean | comment | annual kwh | power annual | +cooling annual x2 |
c04 | 8 | 0.49 | 01:24 | 333 | idle, no jobs | 87,512 | $20,357 | $40,714 |
c32 | 8 | 6.94 | 19:35 | 356 | jobs finished | 93,556 | $21,610 | $43,220 |
c27 | 8 | 7.42 | 19:30 | 381 | jobs finished | 100,127 | $22,971 | $45,942 |
c10 | 8 | 10.21 | 23.55 | 425 | still running | 111,690 | $25,367 | $50,734 |
CPU HPC 2012
new acquisition
13 1U rack servers in 6.5 2U-shelves
2 L6-30
5,250 watts = 45,990 kwh annual = $11,754
17,913 btus = 5,249 watts = 45,981 kwh annual= $11,752
total costs: $23,506/year
“roughly” $3,134 per 2U-shelves (6.5 2U-shelves + 1 head node of 2U)
GPU HPC 2012
Replace Dell
This is what the picture would look like using the 2009 spreadsheet data
240 cores, make them all ethernet based
1U yields 16 cores, 15U yields 240 cores with a total of 1,920
GB memory
major change, Dell offers 388
GB
acquisition costs
15 x $5,782 = $86,730 + PDUs, ethernet switch, cables
power/AC costs: $23,506/year x3 years = $70,518
add both costs of new hardware and divide by dell total annual costs power+cooling
What we should look at is teraflops…
0.66 teraflops, measure with Linpack
E5-2660 is rated at 140 Gflops so one node provides 280 Gflops (theoretically)
2 and 1/3rd 1U nodes then replace the cluster
acquisition costs
3 x $5,782 = $17,346 + PDUs, ethernet switch, cables
48 cores, 840 Gflops
power/AC costs: 3($23,506/13) x3 years = $16,273
add both costs of new hardware and divide by dell total annual costs power+cooling
And if we double the teraflops…1.6 Tflops (actually +1Tflops), 1.5 years ROI for 96 cores.
L6-30 Connections
Dell
left rack (from rear) LCBR 01, 09, 08, 02 and UPS on 01
right rack (from rear) LCBR 03, 04 and UPS on 07
BSS
HP
Back