With the second NSF proposal in a “recommended for funding” state, I'm preparing this page so we can address some looming issues and make decisions on our potential new acquisition. In general, these are the main topics:

• Data Center/ITS Items
  • Increased cooling requirements
  • More electric L6-30 connectors
  • 192.168.1.xxx subnet
  • FTE requirments

• Cluster/Committee Items
  • Increase Home Directory Storage
  • More infiniband nodes
  • More nodes with small memory footprint
  • Database server

• New ventures
  • Should we investigate Platform ISF, private cloud for HPC, will do both windows and linux compute nodes in adaptive mode (meaning in virtualized environment).

Below presents some detail regarding these points to get the discussion started.

April 27th, 2010 Update: So i had this bright idea. Spurred on by the following. The private networks on both clusters have a netmask that yields 255 IPs per subnet. Currently the clusters are chewing up about 175 of those. Now, with a new cluster potentially needing 512 more, we have a netmask and routing problem. Could be very messy.

Light bulb! So if we got a huge cluster, why not make it the primary. For example, we could run CentOS/Kusu/Lava on it or Redhat/OCS/LSF (costs involved) and adjust the netmasks to something yielding tons of IPs, like 255.255.0.0 … once running, we ingest into this cluster the petal/swallow-tail and sharptail cluster. Much cleaner and easier to do, and much more flexible. We can then logically segment the physical hardware in “clusters”.

The breaker box panel currently provides 16 L6-30 connectors and can provide 4 more. We need to assess how much of a draw dawn there currently is, and then size out if a new subpanel can provide enough connectors for the new cluster.

Cooling requirements of the new cluster probably will be met after the data center's environment work project is finished. We probably will still be in the scenario that if a cooling unit dies, part or all of the cluster would need to be shut down preventing overheating of the entire data center.

Unfortunately, the netmask of the Dell cluster was set to 255.255.255.0 resulting in 255 IPs in subnet 192.168.1.xxx. After switches, and both clusters, about 75 IPs are available in this range. If the new cluster comes with few multi-cored nodes this may not be a problem. However, if many IPs are needed, the new cluster would reside in another subnet and the routing tables would have to change, if the clusters need to communicate with each other (as in an expansion of the Dell clusters).

Regarding the 0.5 existing FTE allocation, the new cluster would have serious impact on this. Especially if the new cluster introduces a new operating system (for example x86 solaris) and a new scheduler (for example grid engine or PBS). This will also impact the SCIC for additional documentation and user support.

Home directories are currently served from a NetApp filer via NFS (5 TB). This disk space was bought with the Dell cluster from ITS (sidebar: i think this is correct). Since the filer provides lots of functionality that we do not use, we could entertain the idea of giving this space back. We would then configure the new cluster to house a 25-50 TB disk array serving up the home directories via NFS. Would require a UPS for the disk array, or put it on Enterprise UPS. Also would require all nodes on all clusters to communicate with each other, unless we kept the NetApp space and added the new disk array to new cluster only.

If the new cluster expands the existing Dell cluster we would not need another head/login node. That may add more compute nodes. No more than 75 nodes could be added before we run into routing problems.

All quotes obtained with the proposal contain compute nodes all on infiniband and ethernet switches. This may the simplest way to go configuration wise. But the need for many more compute nodes with a small memory footprint probably does not need infiniband connectivity.

The proposal, if i recall correctly, proposes the addition of a database server. Specifics as to what metrics this database server must need is unknown. None of the quotes contain a database server. Probably should be a MySQL server to be compliant with any commercial software purchased in the future. Would also require a UPS and the ability of all compute nodes to communicate with the database server. Potentially we could turn host swallowtail into a database server by adding disks and memory.

Another item of concern, is that if we move away from RHEL to for example Solaris, all the software needs to be recompiled. The new cluster will by far be the most preferred. Recompiling the software base on petaltail was an enormous task. A migration to CentOS is not a huge task as demonstrated by sharptail.

<!-- | quote | $ 246,237 | $ 202,175 | $ 199,528 | -->

Instead of adopting the vendor's suggested storage device, we could insert one of these. This would require a server in front of these devices listed below which would then provide the file systems to all nodes via NFS. However, the storage devices of the vendors are capable of native NFS, so essentially add that server.

All solutions need a UPS or potentially be powered by the ITS enterprise UPS.

Also, as the size increases, we probably can not back up to the VTL anymore. In such a case, the snapshot capability is important (point in time restore). Also we could design an “archive” present on these devices (raid 1 for example). Or perform disk2disk backup copies, still on same device.

<!-- | quote | $ 48,862 | $ 81,750 | $ 49,892 | -->

If we obtained for example the Nexsan 48 TB (raw) block level device, we could entertain some new ideas. Create a 10 TB Lustre file system for fast scratch space. Create a 5-10 TB data archive to avoid data to be duplicated. Enlarge home directory space to 10-15 TB. Perform D2D backup.

<!-- | quote | $ 152,283.77 (+$2K one day, on site) | $ 149,380.49 | $ 176,856.91 (on site included? check, perhaps +2.5K) | $ 149,996.49 (one week on site included) -->

• For point-to-point throughput, IP over InfiniBand (Connected Mode) is comparable to a native InfiniBand
• When a disk is a bottleneck, NFS can not benefit from IPoIB
• When a disk is not a bottleneck, NFS benefits significantly from both IPoIB

HP states

[HP Engineer] That DL380 has a P812 SAS controller, which has support for 6Gb SAS, and four 4x connectors to the MDS60 JBODs, and the RAID controller has eight 6Gb/s SAS physical links. That's a heavy-weight controller (with a 1GB cache), and the 8x PCIe bus (~4GB/s) and InfiniBand (~4GB/s) will be able to keep up with whatever the storage can provide. While the drives are listed as SAS, at that density, they are SATA internal with a SAS interface. As a rule of thumb, you might get as much as 100MB/s per drive, but realistically under normal workload (and across all parts of the platters, not just the fast outer edges), it's safer to assume ~50MB/s. If you implement RAID6 with one parity and one spare, that leaves 10 drives per MSA, or around 500MB/s per. So, it's possible that the storage could provide up to 2GB/s. To me, that seems like a very high number, which would require xfs local file system to see the local bandwidth. And I've not seen a Linux NFS file system generate more than around 6-700MB/s per server, so when we layer that on top, the storage won't be the limit. I think your question was “where will the bottleneck be?”, and I'd speculate that it's the NFS file system. If you are being asked to commit to a specific performance number, then I believe we will need to run actual benchmarks on your configuration because the software environment is subject to a lot of variability.

[Barry]The performance you can expect on IPoIB is between 200mb/s and 300mb/s based on the information provided by the HP HPC engineering teams strings of emails.

Advance Clustering states:

We have used IPoIB on several occasions with great success. Using IPoIB should take away any network bottleneck and leave the drives as the bottleneck. With NFS over IPoIB you will have a theoretical 10Gb link to your NFS storage which still should leave plenty of bandwidth for your calculations.

But, engineer warns, that IPoIB and FSS software might be unstable. Hence, the FSS shelf would be replaced with another expansion shelf, presented as a block device to head node, which now becomes an NFS server. Then IPoIB could be deployed cluster wide. Not an attractive option.

Attached is the updated quote with 10Gb connectivity on the primary storage node without IPoIB but with 10 Gb. The switch we are using has one 10Gb port. ($152,283.77) or an additional $1,070 to quote detailed above). This means traffic between storage node and switch is 10Gb and between switch and nodes is 1Gb.

1. What size was the cluster (storage, nodes)?
   1. WSU 48 nodes, 384 cores, 5 tb, all infiniband, management over ethernet
   2. UCLA 64 ACT nodes, 512 cores, added to 600 node cluster, all infiniband, 100 tb storage
   3. FSU in top 500 size wise, public and owner owned clusters, two ACT clusters 30 & 10 nodes, enterprise panasas storabe 196 tb
   4. KTZT 9 nodes, 40 tb, in two purchases
2. What OS, scheduler? Familiarity with that in-house?
   1. WSU Torgue,
   2. UCLA CentOS, Sun gridengine (goes commercial via oracle?)
   3. FSU CentOS, Rocks, MOAB
   4. KTZT CentOS, Sun gridengine
3. Experiences in the setup and final configuration?
   1. WSU ACT send folks for 1st cluster, second cluster was easy
   2. UCLA very experienced in house and wipe everything delivered
   3. FSU wipes all hardware
   4. KTZT on site visit ($1900), useful, no experiences before hand
4. Was everything the way it was quoted?
   1. WSU Yes
   2. UCLA Yes
   3. FSU Yes
   4. KTZT yes
5. What software is primarily used?
   1. WSU Amber, Gaussian
   2. UCLA All kinds, very similar to us, chem/physics/engineering/bio
   3. FSU All kinds
   4. KTZT NOAA, climate modeling
6. What does the cluster user base look like? Curriculum usage?
   1. WSU Few active accounts (<10), single faculty member + post docs
   2. UCLA 350 users across many dept, primarily grads/faculty, no class usage
   3. FSU hundreds of users
   4. KTZT one faculty, four students
7. How many administrators are involved? IT staff or students?
   1. WSU 15% of post doc system admin
   2. UCLA many IT persons, estimates that the ACT we have spe'ed requires one experienced person
   3. FSU large IT dept
   4. KTZT faculty
8. Experiences with their support, hardware and software related?
   1. WSU ACT support has been excellent, prompt replacement parts, solid phone support, considers this the best part of their operation
   2. UCLA IT does not use it, but the depts folks use it, however when involved on those cases ACT support is very knowledgeable and quick, same person on all contacts/tickets (same as platform)
   3. FSU none, expecpt support, rated excellent, very knowlegeable
   4. KTZT Beyond believable, excellent, grand, both hardware and software
9. Any items you would have done differently with them?
   1. WSU None
   2. UCLA be attentive to quote details, for example infiniband SDR vs DDR
   3. FSU NA
   4. KTZT storage is slow, probably NFS related, get more memory for cpu

UMICH

• Dell complacent, HP proactive (buys $1M in 3 years, 2852 cores)
• No experience with software
• Good HP support experience
• HP more efficient than Dell, supports that
• Torque/PBS shop
• No IPoIB experience (yet)
• Also a Dell shop

If we purchased just one rack via ACT or Dell (HP unlikely at this time), and we reserve $25K for database server, we could still invest in old Dell rack. For instance:

• collapse ehw and ehwfd
  • buy 2 more md1000 with even larger 15k disks
  • makes a single queue of 8 nodes
• collapse elw, emw
  • universal 16 gb memory footprint per node
  • makes a single queue of 12 nodes
• imw
  • double memory to 16 gb/node

Refurbished materials from Dell. But I'm not sure this is worth the effort. Unsupported hardware. The better investment would be to increase the size of the “primary” new cluster, that is go with Advanced Clustering. But worth investigating.

Back