• HP cluster off support 11/30/2013
• We need greentail/disk array support maybe 2 more years?
  • Karen added to budget, Dave to approve ($2200/year)
• We need another disk array
  • For robust D2D backup
• Pressed HP Procurve ethernet backup switch into production
  • Dell Force 10 switch failing or traffic overwhelmed it
  • Need a file server away from the login node
• We need a new cluster with support
  • power consumption versus computational power
  • gpu versus cpu
  • 6 of 36 dell compute nodes have failed

This is what we ended up buying May 2013.

• 16U - estimated draw 6,900 Watts and 23,713 BTUs cooling - $30K/year
• 5 GPU shelves
• 2 PDUs
• 42 TB raw
• FDR interconnects
• 120GB SSD drives on nodes
• 256 gb ram on nodes, 16gb/core
• Areca hardware raid
• Lifetime technical support

• 20U - estimated draw 7,400 Watts - $30K/year for cooling and power
• 5 GPU shelves
• 1 CPU shelf
• 4 PDU - this could be a problem!
• 56TB raw
• QDR interconnects
• 1 TB disk on node, makes for a large /localscratch
• LSI hardware raid card

• 5,900 Watts and 20,131 BTUs/Hour
• smaller infiniband switch (8 port) and ethernet switch (24 port)
  • the 18 port switch has been included, swap out for $2K spare parts
• sandy bridge chip E2660 and larger memory footprint (128gb node, 256gb head node)
• 120GB SSD drives on nodes
• storage: 42TB usable Raid 6
• Lifetime technical support

• Spare parts
  • ?
• Expand Storage
  • upgrade to 56TB usable Raid 6 ($5.3K using 16x4TB disks)
  • upgrade to 90TB usable Raid 60 ($10.3K using 34x3TB disks)

• Alternate storage:
  • add storage server of 2.4 TB Usable 15K fast speed SAS disk ($9K-1K of 4U chassis)
  • leave 18TB local storage on head node

• 5,250 Watts and 17,913 BTUs/Hour
• infiniband switch (18 port needed for IPoIB) and ethernet switch (24 port)
• sandy bridge chip E2660 and larger memory footprint (128gb node, 256gb head node)
• 120GB SSD drives on nodes
• storage: 42TB usable Raid 6
• Lifetime technical support
• Drop software install ($3.5K savings)

• Spare parts
  • ?
• Expand Storage
  • upgrade to 56TB usable Raid 6 ($5.3K using 16x4TB disks)
  • upgrade to 90TB usable Raid 60 ($10.3K using 34x3TB disks)

• Alternate storage:
  • add storage server of 2.4 TB Usable 15K fast speed SAS disk ($9K-1K of 4U chassis)
  • leave 18TB local storage on head node

• 16TB Raid6 storage (14 TB usable - tight for /home)
• full height rack

• 16TB Raid6 storage (14 TB usable - tight for /home)
• 1TB on nodes is wasted (unless we make fast local /localscratch at 7.2K)

09nov12:

• /home and /apps mounted on CPU side. How does GPU access these? Or is job on CPU responsible for this?
• Single versus double precision? Both needed I assume.
• Unit above is Nvidia “Fermi” series, being phased out. “Kepler” K10 and K20 series coming out. Get an earlybird unit, Jim will find out.
• Lava compatibility (almost certain but need to check) AC uses SGE.
• We do not really “know” if our current jobs would experience a boost in speed (hence one unit first - but there is a software problem here)
• Intel Xeon Phi Co-Processors: Intel compilers will work on this platform (which is huge!) and no programming learning curve. (HP Proliant servers with 50+ cores), Jim will find out.
• Vendor states scheduler sees GPUs directly (but how does it then get access to home dirs, check this out) … update: this is not true, CPU job offloads to GPU

AC Specs

• Early 2013 product line up
  
  • http://www.advancedclustering.com/products/pinnacle-flex-system.html Kansas City, KS
• Quote coming for single 4U unit, which could be a one off test unit (compare to HP)

• In order to match the “benchmark option” we need 5 units
  • 8100 Watts, would still fit power wise but not rack wise (we'd need 20U)

• Single rack, 21 TB of disk space (Raid 5/6)
• The IB switch (plus 4 spare cards/cables) is roughly 1/3rd of the price
  • If we remove it, we need QDR Voltaire compliant HCAs and cables (3 ports free)
• The config does not pack as much teraflops for the dollars; we'll see

12nov12:

• GPU hardware only
• scheduler never sees gpus just cpus
• cpu to gpu is one-to-one when using westmere chips
• bright cluster management (image based) - we can front end with lava
• what's the memory connection cpu/gpu???
• home dirs - cascade via voltaire 4036, need to make sure this is compatible!
• software on local disk? home dirs via infiniband ipoib, yes, but self install
• amber (charge for this) and lammps preinstalled - must be no problem, will be confirmed
• 2 K20 per 2 CPUs per rack 900-1000W, 1200 W power supply on each node
• PDU on simcluster, each node has power connection
• quote coming for 4 node simcluster
• testing periods can be staged so you are testing exactly what we're buying if simcluster if within budget (see K20 above)

EC Specs

• http://exxactcorp.com/index.php/solution/solu_detail/119 Fremont, CA

• Lets make this the “benchmark option” based on double precision
• In order to match this with Xeon Phis we'd need 18 of them (probably 5 4U trays)

• This is the (newest) simcluster design (that can be tested starting Jan 2013)
  • 24U cabinet
• We could deprecate 50% of bss24 queue freeing two L6-30 connectors
• Spare parts:
  • Add another HCA card to greentail and connect to Mellanox switch (long cable)
    • also isolates GPU traffic from other clusters
  • 1 8-port switch, 4 HCA cards, 4 long cables (for petal/swallow tails plus spare)
• New head node
  • First let EC install Bright/Openlava (64 CPU cores implies 64 job slots)
    • 16 GPUs implies 16×2,500 or 40,000 cuda cores (625 per job slot on average)
  • Use as standalone cluster or move GPU queue to greentail
  • If so, turn this head node into a 16 job slot ram heavy compute node?
    • 256-512gb (Order?)
    • add local storage? (up to 10 1or2 TB disks)
• Compute nodes
  • add local storage? (up to 10 1or2 TB disks)
• Bright supports openlava and GPU monitoring (get installed)
  • http://www.brightcomputing.com/Linux-Cluster-Workload-Management.php
  • http://www.brightcomputing.com/NVIDIA-GPU-Cluster-Management-Monitoring.php
• EC software install
  • sander, sander.MPI, pmemd, pmemd.cuda (single GPU version), pmemd.cuda.MPI (the multi-GPU version)
  • NVIDIA Toolkit v4.2. Please note that v5.0 is NOT currently supported
  • MVAPICH2 V1.8 or later / MPICH2 v1.4p1 or later recommended, OpenMPI is NOT recommended.)
  • make sure they do not clean source, analyze how they compiled
  • which compiler will they use? which MPI (prefer OpenMPI, have wrapper script for that)

HP 19nov12: meeting notes

• HP ProLiant SL270s Generation 8 (Gen8); 4U half width with 2 CPUs + 8 (max) GPUs
  • The s6500 Chassis is 4U tray holding two S270s servers
• max 8 GPUs (20,000 cuda cores) + 2 CPUs (total 16 cores), dual drives, 256gb max
  • K20 availability will be confirmed by Charlie
• power
  • Charlie will crunch numbers of existing HPC and assess if we can use the current rack
  • otherwise a stand alone half rack solution
• one IB cable to Voltaire per chassis? get new FDR infiniband switch, period.
  • connect greentail with additional HCA card, or voltaire to voltaire?
• our software compilation problem, huge
  • but they have great connections with Nvidia for compilation help (how to qualify that?)
• CMU for GPU monitoring, 3-rendering of what GPU is doing
• This SL270s can also support up to 8 Xeon Phi coprocessors
  • but expect very lengthy delays, Intel is not ready for delivery (1 Phi = 1 double teraflop)

HP Specs

http://h18004.www1.hp.com/products/quickspecs/14405_div/14405_div.HTML

• First unit, single tray in chassis
• This hardware can be tested at ExxactCorp so single tray purchase for testing not a requirement

• 2 chassis in 8U + 4 SL250s + each with 8 GPUs would be a massive GPU cruncher
  • 8 CPUs, 32 GPUs = 64 cpu cores and 80,000 cuda cores (avg 1,250cuda/core)
  • peak performance: 37.44 double, 112.64 single precision (twice the “benchmark option”)
• 1 chassis in 4U + 2 Sl250s + each with * GPUs would the “benchmark option”

• To compare with “benchmark option” price wise; 37% higher (25% less CPU cores)
• To compare with “benchmark option” performance; 12.5% higher (double precision peak)

• When quote is reduced to 1x s6500 chassis and 2x SL250s:
  • To compare with “benchmark option” price wise; 1.6% higher (50% less CPU cores)
  • To compare with “benchmark option” performance; 25% lower (double precision peak)

• HP on site install
• we have 9U in HP rack available (1U for new switch)
  • L6-30 7,500 Watts x3 PDUs (non-UPS) = 22,500 Watts - HP cluster 10,600 Watts
  • leaves 11,898 Watts, should be sufficient for 4 SL270s(redundant power supplies)
• new infiniband switch, isolates GPU cluster traffic from rest of HPC
  • 36 port IB switch overkill
  • still need IB connection greentail to new switch (home dirs IPoIB)
• 1 TB local storage per node
• our software install problem, so is the 12.5% worth it? (with 3 trays)

• Cluster management is ROCKS (we'll pass)
• No scheduler (that's OK, we'll use OpenLava)
• They do not install software, only operating system and
• CUDA driver setup and installation

AX Specs

http://www.amax.com/hpc/productdetail.asp?product_id=simcluster Fremont, CA

• 22U cabinet
• Insurance during shipping is our problem (non-returnable)
• To compare with “benchmark option” price wise; 21% lower (25% less CPU cores)
• To compare with “benchmark option” performance; 22% lower (double precision peak)
• If we go turnkey systems having software installed is huge

• sells both individual racks and turn-key systems
  • racks are 4U with 2 CPUs and 8 GPUs, 2200 Watts, K20X GPUs
  • turn-key units are per customer specifications
• they will install all software components (if license keys are provided)
  • includes CUDA drivers and setup, Amber (pmemd.cuda & pmemd.cuda.MPI, check) and Lammps
  • but also Matlab and Mathematica if needed (wow!)
• standard 2 year warranty though (no biggie)

MW Specs

http://www.microway.com/tesla/clusters.html Plymouth, MA

• At full load 5,900 Watts and 20,131 BTUs/hour

• 2% more expansive than “benchmark option” (as described above with Upgrades), else identical
  • But a new rack (advantageous for data center)
  • With lifetime technical support
  • solid state drives on compute nodes
  • 12 TB local storage (8TB usable)

Then

• 36 port FDR switch replace with 8 port QDR switch for savings (40 vs 56 Gbps)
  • and all server adapter cards to QDR (with one hook up to existing Voltaire switch)
• Expand memory footprint
  • Go to 124 GB memory/noe to beef up the CPU HPC side of things
  • 16 cpu cores/nodes minus 4 cpu/gpu cores/node = 12 cpu cores using 104gb which is about 8 GB/cpu core
• Online testing available (K20, do this)
  • then decide on PGI compiler at purchase time
  • maybe all Lapack libraries too
• Make the head node a compute node (in/for the future and beef it up too, 256 GB ram?)
• Leave the 6x2TB disk space (for backup)
  • 2U, 8 drives up to 6×4=24 TB, possible?
• Add an entry level Infiniband/Lustre solution
  • for parallel file locking

• Spare parts
  • 8 port switch, HCAs and cables, drives …
  • or get 5 years total warranty

• Testing notes
  • Amber, LAMMPS, NAMD
  • cuda v4&5
  • install/config dirs
  • use gnu … with openmpi
  • make deviceQuery

Back