This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:123 [2013/10/10 14:08] hmeij [Replace Dell Racks] |
cluster:123 [2013/10/18 13:55] hmeij [Update] |
||
---|---|---|---|
Line 7: | Line 7: | ||
Subtitle: A win-win solution proposed by Physical Plant and ITS | Subtitle: A win-win solution proposed by Physical Plant and ITS | ||
- | Once upon a time, back in 2013, two Dell racks full of compute nodes, sat noisily chewing away energy on the 5th floor of Science Tower. | + | Once upon a time, back in 2013, two Dell racks full of compute nodes, sat noisily chewing away energy on the 5th floor of Science Tower. |
- | The Dell racks contain 30 compute nodes, two UPS units, two disks arrays and two switches. We have measured 19 nodes power consumption (pulling one the dual power units out) with a Kill-A-Watt meter for over 775+ hours. The mean power consumption rate is 418.4 watts. That totals to 109,956 KwH/year in power consumption ((watts/ | + | The Dell racks contain 30 compute nodes, two UPS units, two disks arrays and two switches. We have measured 19 nodes power consumption (pulling one of the dual power units out) with a Kill-A-Watt meter for over 775+ total hours. The mean power consumption rate is 418.4 watts. That totals to 109,956 KwH/year in power consumption ((watts/ |
- | + | ||
- | Based on 12.5 cents per Kwh (this is an all inclusive cost including natural gas cost, heat recoup costs, distribution, | + | Next we need to convert to a dollar value. A residential electric bill contains a KwH cost as well as generation, distribution, |
- | + | Based on 12.5 cents the Dell compute nodes consume $13,744.50 per year in power. Best guess is cooling costs are at least that (another possible | |
- | + | ||
- | If we could replace, or approximate, | + | |
- | + | ||
- | + | ||
- | + | ||
- | So, I've collected quotes | + | |
+ | Next step was to collect vendor quotes for a target budget of $82K, 3 years of Dell energy consumption, | ||
- | + | Old hardware: 109,956 KwH/year for power\\ | |
- | old racks: 30 nodes, | + | 30 nodes, |
- | + | 80 gb local drive, 340 gb total ram, 12,555 watts (power no cooling), 670 gigaflops | |
- | vendor 1: 14 nodes, 224 job slots, | + | |
- | + | ||
- | vendor 2: 10 nodes, 200 job slots, 4,597 watts, | + | |
+ | New hardware v1: 47,304 KwH/year for power or 43% of old hardware\\ | ||
+ | 14 nodes, 2.60 ghz, 20 mb L-cache (for cpu), 224 cores (job slots),\\ | ||
+ | 1TB local drive, 1,792 gb total ram, 5,400 watts (power no cooling), 4,659 gigaflops (theoretical) | ||
- | So we can do it with about 40% of the energy consumption. And I can do it with half the number | + | New hardware v2 (half of v1): 23,652 KwH/year for cooling or 22% of Old hardware\\ |
+ | 7 nodes, 2.60 ghz, 20 mb L-cache (for cpu), 112 cores (job slots),\\ | ||
+ | 1TB local drive, 1,792 gb total ram, 2,700 watts (power no cooling), 2,329 gigaflops (theoretical) | ||
- | + | If we reduced the node count to 7 (the minimum configuration to meet the job slot count of the Dell hardware), the total energy consumption (power plus cooling) would be 5,400 watts. | |
- | My intention is to convince | + | In two years, |
- | + | * There are enough Infiniband ports available to put all new hardware nodes on such a switch (add cards and cables cost for each node) | |
+ | * The internal disks on each node need to be of a high speed (10K or better) and of a certain size (300 GB or larger) mimicking the Dell disk arrays (adds costs) | ||
+ | * we maybe able to add two more nodes by switching to a more exapansive lower wattage CPU (and remain within budget as well as below the 50% energy consumption threshold as compared with Dell's consumption. | ||
+ | * accomplished by switching from 8 core 2650v2 (130 watt) 2.6 ghz CPU to 10 core 2660v2 (95 watt) 2.2 ghz CPU | ||
- | -Henk | + | But it is all very doable within a budget of $45-$50K. And it can be the solution for: |
- | + | * replace Dell's racks functions and match or exceed its performance | |
- | From: Meij, Henk | + | * seriously reduce energy consumption benefiting Physical Plant' |
- | Sent: Wednesday, October 02, 2013 12:10 PM | + | * allow ITS to treat the third Liebert cooling tower as backup/ |
- | To: Staye, Peter | + | * being way green |
- | Subject: RE: Meeting with Peter 01Oct13 Summary | + | |
- | oh, btw the nodes are on 208V utility power, makes no difference in our calculations though. | + | The Libert family rejoices. |
- | -Henk | + | ==== Update ==== |
- | + | The table below contains data for a cluster whose nodes are all on the Infiniband switch | |
- | From: Meij, Henk | + | |
- | Sent: Wednesday, October 02, 2013 11:42 AM | + | |
- | To: Staye, Peter | + | |
- | Subject: Meeting with Peter 01Oct13 Summary | + | |
- | + | ||
- | My attempt at summarizing our conversation. Make corrections if I got things wrong. | + | |
- | + | ||
- | + | ||
- | + | ||
- | -Henk | + | |
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | will save y'll the details but after many measurements and meeting with Peter Staye (physical plant) we are both in agreement that the dell 30 compute nodes "total cost for all energy consumption is $27,489 per year" | + | |
- | + | ||
- | Next step was to collect quotes | + | |
- | + | ||
- | dell: | + | |
- | 30 nodes, 2.66 ghz, 4 mb L-cache | + | |
- | 80 gb local drive, 340 gb total ram, 12,555 watts (power no cooling), 670 gigaflops (actual measure) | + | |
- | + | ||
- | microway: | + | |
- | 14 nodes, 2.60 ghz, 20 mb L-cache (for cpu), 224 cores, | + | |
- | 1TB local drive, 1,792 gb total ram, 5,400 watts (power no cooling), 4,659 gigaflops (theoretical) | + | |
- | + | ||
- | That means microway hardware burns but 40% of the dell's power (not cooling). | + | |
- | + | ||
- | If we take half the microway nodes, turn hyperthreading on (like on queue mw256, doubling | + | |
- | + | ||
- | That will easily fit within $55K (two year of Dell power costs) and obtain a ROI of 2 years. | + | |
- | + | ||
- | Peter Staye agrees. | + | |
+ | ^Tnodes^Tcores^THcores^Tmem gb^Watts^%of Dell^TEnergy^TEnergy $/ | ||
+ | |10|160|320|2, | ||
- | The end. | ||
\\ | \\ | ||
**[[cluster: | **[[cluster: |