We have 4 racks of which 3 are powered up. All on utility power including head/login node. Racks are surprisingly cool. Soem digging revealed that the AMD Opteron chip cycles down to 1 Ghz if not used instead of running at 2.4 Ghz all the time.

If you want to use the switches you need to power up the top two shelves within each rack or use an alternate source of power.

We want to separate the data traffic (NFS) from the software management traffic so will be leveraging both ethernet ports on each blade. In order to do that we changed the cabling. In our setup the top procurve switch is always the provision switch (192.168.1.y/255.255.255.0) and the bottom switch is the data switch (10.10.x.y/255.255.0.0). Port 48 of each switch cascades into the next switch so that all 3 procurve switches become one network.

We bought 52 three feet CAT6 ethernet cables for each rack. The purple cables connecting blade to rack in top two shelves within a rack connect to bottom ethernet port (eth0). For bottom two racks, the purple cable connect to top ethernet port (eth1). then the rest of the ehternet ports were connected with the three feet cables. This results in each blade being connected to top and bottom switch. Now the math does not work out smoothly; 4 shelves with 13 blades is 52 eth[0|1] connections but the switches have 48 ports (minus the uplink port). So you have some blades not connected in each rack.

Back