This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:123 [2013/10/10 14:08] hmeij [Replace Dell Racks] |
cluster:123 [2013/10/10 15:21] hmeij |
||
---|---|---|---|
Line 9: | Line 9: | ||
Once upon a time, back in 2013, two Dell racks full of compute nodes, sat noisily chewing away energy on the 5th floor of Science Tower. | Once upon a time, back in 2013, two Dell racks full of compute nodes, sat noisily chewing away energy on the 5th floor of Science Tower. | ||
- | The Dell racks contain 30 compute nodes, two UPS units, two disks arrays and two switches. We have measured 19 nodes power consumption (pulling one the dual power units out) with a Kill-A-Watt meter for over 775+ hours. The mean power consumption rate is 418.4 watts. That totals to 109,956 KwH/year in power consumption ((watts/ | + | The Dell racks contain 30 compute nodes, two UPS units, two disks arrays and two switches. We have measured 19 nodes power consumption (pulling one the dual power units out) with a Kill-A-Watt meter for over 775+ hours. The mean power consumption rate is 418.4 watts. That totals to 109,956 KwH/year in power consumption ((watts/ |
- | + | ||
- | Based on 12.5 cents per Kwh (this is an all inclusive cost including natural gas cost, heat recoup costs, distribution, | + | Next we need to convert to a dollar value. A residential electric bill contains a KwH cost as well as generation, distribution, |
- | + | Based on 12.5 cents per Kwh (all costs included) the Dell compuet nodes consumed $13,744.50 per year in power. Best guess is cooling costs are at least that (another possible low water mark) so the total cost for both power and colling consumption is $27,489 per year. | |
- | If we could replace, or approximate, | ||
- | |||
- | |||
- | |||
- | So, I've collected quotes and they are rolling in. I've taken $82K target as a budget so I can downscale from there. Here is a comparison | ||
+ | Next step was to collect quotes for a target budget of $82K, 3 years of Dell energy consumption, | ||
- | + | Old hardware: | |
- | old racks: 30 nodes, | + | 30 nodes, |
- | + | 80 gb local drive, 340 gb total ram, 12,555 watts (power no cooling), 670 gigaflops | |
- | vendor 1: 14 nodes, 224 job slots, | + | |
- | + | ||
- | vendor 2: 10 nodes, 200 job slots, 4,597 watts, | + | |
- | + | New hardware: | |
- | So we can do it with about 40% of the energy consumption. And I can do it with half the number of nodes listed because with hyperthreading the number of job slots can be doubled (just need more memory). Our conclusion was " all power consumption is $27,489 per year". That means $55K is enough and our ROI is 2 years. | + | 14 nodes, 2.60 ghz, 20 mb L-cache (for cpu), 224 cores (job slots), |
+ | 1TB local drive, 1,792 gb total ram, 5,400 watts (power no cooling), 4,659 gigaflops (theoretical) | ||
+ | In the representative example total energy consumption would be 10,800 watts. | ||
- | My intention is to convince | + | The total cost of running |
- | + | Within two years, the new hardware | |
- | + | ||
- | -Henk | + | |
- | + | ||
- | + | ||
- | From: Meij, Henk | + | |
- | Sent: Wednesday, October 02, 2013 12:10 PM | + | |
- | To: Staye, Peter | + | |
- | Subject: RE: Meeting with Peter 01Oct13 Summary | + | |
- | + | ||
- | oh, btw the nodes are on 208V utility power, makes no difference in our calculations though. | + | |
- | + | ||
- | -Henk | + | |
- | + | ||
- | + | ||
- | From: Meij, Henk | + | |
- | Sent: Wednesday, October 02, 2013 11:42 AM | + | |
- | To: Staye, Peter | + | |
- | Subject: Meeting with Peter 01Oct13 Summary | + | |
- | + | ||
- | My attempt at summarizing our conversation. Make corrections if I got things wrong. | + | |
- | + | ||
- | + | ||
- | + | ||
- | -Henk | + | |
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | will save y'll the details but after many measurements and meeting with Peter Staye (physical plant) we are both in agreement that the dell 30 compute nodes "total cost for all energy consumption is $27,489 per year" | + | |
- | + | ||
- | Next step was to collect quotes | + | |
- | + | ||
- | dell: | + | |
- | 30 nodes, 2.66 ghz, 4 mb L-cache (for cpu), 240 cores, | + | |
- | 80 gb local drive, 340 gb total ram, 12,555 watts (power no cooling), 670 gigaflops (actual measure) | + | |
- | + | ||
- | microway: | + | |
- | 14 nodes, 2.60 ghz, 20 mb L-cache (for cpu), 224 cores, | + | |
- | 1TB local drive, 1,792 gb total ram, 5,400 watts (power no cooling), 4,659 gigaflops (theoretical) | + | |
- | + | ||
- | That means microway hardware burns but 40% of the dell' | + | |
- | + | ||
- | If we take half the microway nodes, turn hyperthreading on (like on queue mw256, doubling the core count), raise the memory to 256gb/node (8gb/core) we would be very close to dell's job slot offering (224 vs 240), have 5x more memory, and 7x more gigaflops. | + | |
- | + | ||
- | That will easily fit within | + | |
- | + | ||
- | Peter Staye agrees. | + | |
+ | * there is enough Infiniband ports available to put all new nodes on such a switch (add cards and cables cost) | ||
+ | * the internal disks need to be of higher speed (10K or better) of a larger size (300 GB or larger) mimicking the Dell disk arrays (add costs) | ||
+ | But it is all very doable within budget of $45-$55K | ||
The end. | The end. |