DokuWiki

We will be bringing online 280 more physical cores (560 hyper threads) with Haswell-EP E2650v3 chips 2.3 Ghz with a turbo boost speed of 3.0 Ghz.

That's an 85% increase in job slots. Yea.

• ExacctCorp: Quantum IXR110-512N E5-2600 v3 family
  • 1U Server, Intel Dual socket R (LGA 2011)
  • Dual port Gigabit Ethernet,
  • 480W High efficiency 1x
  • Intel® Xeon® processor E5-2650 v3, 10C, 2.3 GHz 25M, 2x (total 20 cores)
  • 16GB DDR4-2133MHz ECC Registered 1.2V Memory Module 2x (total 32gb ram)
  • 120GB 2.5 SATA III Internal Solid State Drive (SSD) OS Drive and Scratch Drive 2x
  • CentOS 6 Installation
  • 3-Year Warranty on Parts and Labor with Perpetual Email and Telephone Support
• 20 cores/U, 1.6 gb ram/core
• expansion: 14 nodes, 14U, 280 cores,
• expansion: 15 nodes, 15U, 300 cores,

• CDW/Supermicro: 813MFTQ-441CB
  • 1U Server, Intel Dual socket R (LGA 2011)
  • Dual port Gigabit Ethernet,
  • 440W High efficiency 1x
  • Intel® Xeon® processor Haswell-EP E5-2650 v3, 10C, 2.3 GHz 25M, 2x (total 20 cores)
  • 8GB DDR4-2133MHz ECC Registered 1.2V Memory Module 4x (total 32gb ram)
  • 32GB SATAdomSolid State Drive (SSD) 1x ←– new
  • CentOS 6 Installation
  • 3-Year Warranty on Parts and Labor
• 20 cores/U, 1.6 gb ram/core
• expansion: 14 nodes, 14U, 280 cores,
• expansion: 15 nodes, 15U, 300 cores,

• Microway: NumberSmasher
  • 1U Server, Intel Dual socket R (LGA 2011)
  • Dual port Gigabit Ethernet,
  • 440/480W High efficiency 1x
  • Intel® Xeon® processor Haswell-EP E5-2650 v3, 10C, 2.3 GHz 25M, 2x (total 20 cores)
  • 8GB DDR4-2133MHz ECC Registered 1.2V Memory Module 4x (total 32gb ram)
  • 16GB SATAdomSolid State Drive (SSD) 1x ←– new, tiny
  • CentOS 6, MCMS, OpenMPI, SLURM, GNU Installation
  • 3-Year Warranty on Parts and Labor, Lifetime Phone & Email support
• 20 cores/U, 1.6 gb ram/core
• expansion: 14 nodes, 14U, 280 cores,

We are in need to address the problem of tens of thousands of small serial jobs swarming across our larger servers. In doing so these jobs tie up large chunks of memory they do not use and interfere with the scheduling of large parallel jobs (small serial jobs satisfy job prerequisites easily).

So the idea is to assess what we could buy in terms of large core density hardware (max cpu cores per U rack space) with small memory footprints (defined as 1 gb per physical core or less). Nodes can have tiny local disks for OS and local scratch (say 16-120 GB). /home may not be mounted on these systems so input and output files need to be managed by the jobs and copied back and forth using scp. The scheduler will be SLURM. The OS CentOS 6.x latest version.

Some testing results can be found here:

• High Core Count - Low Memory Footprint
• Slurm

The expansion: lines below give an estimation of nr_nodes = int(expansion_budget/node_cost)

• Node option A: Quantum IXR110-512N E5-2600 v2 family
  • 1U Server, Intel Dual socket R (LGA 2011)
  • Dual port Gigabit Ethernet
  • 350W High efficiency 1x
  • Intel® Xeon® processor E5-2620v2, 6C, 2.10 GHz 15M 2x (total 12 cores)
  • 8GB 240-Pin DDR3 1866 MHz ECC/Registered Server Memory 2x (total 16gb ram)
  • 120GB 2.5 SATA III Internal Solid State Drive (SSD) OS Drive and Scratch Drive 2x
  • CentOS 6 Installation
  • 3-Year Warranty on Parts and Labor with Perpetual Email and Telephone Support
• 12 cores/U, 1.3 gb ram/core
• expansion: 26 nodes, 26U, 312 cores

• Node Option B: Quantum IXR110-512N E5-2600 v3 family
  • 1U Server, Intel Dual socket R (LGA 2011)
  • Dual port Gigabit Ethernet,
  • 480W High efficiency 1x
  • Intel® Xeon® processor E5-2650 v3, 10C, 2.3 GHz 25M, 2x (total 20 cores)
  • 16GB DDR4-2133MHz ECC Registered 1.2V Memory Module 2x (total 32gb ram)
  • 120GB 2.5 SATA III Internal Solid State Drive (SSD) OS Drive and Scratch Drive 2x
  • CentOS 6 Installation
  • 3-Year Warranty on Parts and Labor with Perpetual Email and Telephone Support
• 20 cores/U, 1.6 gb ram/core
• expansion: 13 nodes, 13U, 260 cores

• Node option: Pinnacle 1FX3601
  • 2U Server Enclosure, 4 nodes per enclosure 4x, each node with:
  • Dual port Gigabit Ethernet
  • 500W High efficiency 1x
  • Intel® Xeon® processor E5-2630v3, 8C, 2.40 GHz 20M 2x (total 16 cores)
  • 4GB DDR4 2133 MHz ECC/Registered Server Memory 8x (total 32gb ram)
  • 128GB SATA Solid State Drive (SSD) 1x
  • CentOS 6 Installation
  • 3-Year Warranty on Parts and Labor with Perpetual Email and Telephone Support
• 64 cores/2U, 2.0 gb ram/core
• expansion: 16 nodes, 8U, 256 cores

• Node option: Pinnacle 1FX3601
  • 2U Server Enclosure, 4 nodes per enclosure 3x, each node with:
  • Dual port Gigabit Ethernet
  • 500W High efficiency 1x
  • Intel® Xeon® processor E5-2650v3, 10C, 2.30 GHz 25M 2x (total 20 cores)
  • 4GB DDR4 2133 MHz ECC/Registered Server Memory 8x (total 32gb ram)
  • 128GB SATA Solid State Drive (SSD) 1x
  • CentOS 6 Installation
  • 3-Year Warranty on Parts and Labor with Perpetual Email and Telephone Support
• 80 cores/2U, 1.6 gb ram/core
• expansion: 12 nodes, 6U, 240 cores

• Node option: Twin NumberSmasher4x
  • 2U Server Enclosure, 4 nodes per enclosure 3x, each node with:
  • Dual port Gigabit Ethernet
  • 1620W High efficiency 1x
  • Intel® Xeon® processor E5-2630v3, 10C, 2.30 GHz 25M 2x (total 20 cores)
  • 4GB DDR4 2133 MHz ECC/Registered Server Memory 8x (total 32gb ram)
  • 16GB SATADom (SSD) 1x ←– important, with 20 cores might be too small
  • CentOS 6.x, MCMS (provisioning), OpenMPI, SLurm (scheduler) Installation ←– rapid deployment
  • 3-Year Warranty on Parts and Labor with Perpetual Email and Telephone Support
• 80 cores/2U, 1.6 gb ram/core
• expansion: 12 nodes, 6U, 240 cores (3% over budget)

• Node option: Avoton MicroBlade (up to 28 MicroBlades)
  • 6U Enclosure, :
  • Two one port Gigabit Ethernet,internal 2×2.5 Ghz ehternet module
  • 1600W High efficiency 200-240V
  • 19 MicroBlades, each with 4 independent nodes and, each
  • Intel® Atom processor C2750, 8C, 2.40 GHz 1x (total 32 cores/blade) ←– new, see below
  • 8GB DDR3 1600 MHz ECC/Unbuffered 1x (total 32gb ram/blade)
  • 16GB SATA Solid State Drive (SSD) 1x ←– important
  • CentOS 6, Slurm, OpenMPI, GNU compilers Installation (MCMS - Mircroway provisioning tool)
  • MicroBlade Chassis Management Module (virtual media - usb/cdrom)
  • 3-Year Warranty on Parts and Labor with Perpetual Email and Telephone Support
• 608 cores/6U, 1.0 gb ram/core
• expansion: 76 nodes, 6U, 608 cores

• This is the only option with NO hyperthreading.
• http://www.servethehome.com/Server-detail/intel-atom-c2750-8-core-avoton-rangeley-benchmarks-fast-power/
  • Low overall power consumption, performance generally around half of the Intel Xeon E3-1200 V3 series
• http://ark.intel.com/compare/65732,83356,75789,77987
  • compare xeon E5 v2, v3 and atom c2750

• Node option: Twin NumberSmasher4x
  • 2U Server Enclosure, 4 nodes per enclosure 3x, each node with:
  • Dual port Gigabit Ethernet
  • 1620W High efficiency 1x
  • Intel® Xeon® processor E5-2630v3, 10C, 2.30 GHz 25M 2x (total 20 cores)
  • 4GB DDR4 2133 MHz ECC/Registered Server Memory 8x (total 32gb ram)
  • 16GB SATADom (SSD) 1x ←– small, but /home will be mounted
  • CentOS 6.x, MCMS (provisioning), OpenMPI, SLurm (scheduler) Installation ←– rapid deployment
  • 3-Year Warranty on Parts and Labor with Perpetual Email and Telephone Support
• 80 cores/2U, 1.6 gb ram/core
• expansion: 12 nodes, 6U, 240 cores

Customer shall be responsible for shipping charges (this is normal) and shall have its own insurance to cover risk during transit (this is definitely not!).

• Node option A: ServMax E5-2600 v2 family
  • 1U Server, Intel Dual socket (LGA 2011)
  • Dual port Gigabit Ethernet?
  • 95W High efficiency, redundant power supplies
  • Intel® Xeon® processor E5-2660v2, 10C, 2.2 GHz 2x (total 20 cores)
  • 4GB DDR3 1866 MHz E/R Memory 8x (total 32gb ram)
  • 500GB 3.5“ SATA Hard Drive 1x
  • CentOS 6 Installation
  • 1-Year Warranty on Parts and Labor ←– grh
• 20 cores/U, 1.6 gb ram/core
• expansion: 10 nodes, 10U, 200 cores

• Node option B: ServMax E5-2600 v3 family
  • 1U Server, Intel Dual socket (LGA 2011)
  • Dual port Gigabit Ethernet?
  • 120W High efficiency, redundant power supplies
  • Intel® Xeon® processor E5-2600v3, 14C, 2.0 GHz 2x (total 28 cores)
  • 4GB DDR4 2133 MHz E/R Memory 8x (total 32gb ram)
  • 500GB 3.5” SATA Hard Drive 1x
  • CentOS 6 Installation
  • 1-Year Warranty on Parts and Labor ←– grh
• 28 cores/U, 1.1 gb ram/core
• expansion: 8 nodes, 8U, 224 cores

• Node option C: ServMax E5-2600 v2 family
  • 2U Server Enclosure, 4 nodes per enclosure 4x, each node with:
  • Dual port Gigabit Ethernet?
  • 95W High efficiency, redundant power supplies
  • Intel® Xeon® processor E5-2660v2, 10C, 2.2 GHz 8x (total 80 cores)
  • 4GB DDR3 1866 MHz E/R Memory 32x (total 128gb ram)
  • 500GB SATA Hard Drive 4x
  • CentOS 6 Installation
  • 1-Year Warranty on Parts and Labor ←– grh
• 80 cores/2U, 1.6 gb ram/core
• expansion: 2 nodes, 4U, 160 cores
• expansion: 3 nodes, 6U, 240 cores ←– 10% over budget

• Node option D: ServMax E5-2600 v3 family
  • 2U Server Enclosure, 4 nodes per enclosure 4x, each node with:
  • Dual port Gigabit Ethernet?
  • 120W High efficiency, redundant power supplies
  • Intel® Xeon® processor E5-2600v3, 14C, 2.0 GHz 8x (total 112 cores)
  • 4GB DDR4 2133 MHz E/R Memory 32x (total 128gb ram)
  • 500GB SATA Hard Drive 4x
  • CentOS 6 Installation
  • 1-Year Warranty on Parts and Labor ←– grh
• 112 cores/2U, 1.1 gb ram/core
• expansion: 2 nodes, 4U, 224 cores

• Node option: X10DRU E5-2600 v3 family
  • 1U Server, Intel Dual socket
  • Dual port Gigabit Ethernet
  • 750W High efficiency 1x
  • Intel® Xeon® processor E5-2650v3, 10C, 2.30 GHz 25M 2x (total 20 cores)
  • 8GB 240-Pin DDR4 2133 MHz ECC/Registered Server Memory 2x (total 16gb ram)
  • 32GB 2.5 SATA DOM ←– new
  • 3-Year Warranty on Parts
• 20 cores/U, 1.6 gb ram/core
• expansion: 1 nodes, 11U, 220 cores

• Node option: Enterprise MicroBlade (up to 28 MicroBlades)
  • 6U Enclosure, :
  • One port Gigabit Ethernet,internal 1×2.5 Ghz ethernet module
  • 1600W High efficiency 200-240V
  • 12 MicroBlades, each with 1 independent node, each
  • Intel Haswell-EP R3 E5-2650, 10C, 2.30 GHz 2x (total 20 cores/blade)
  • 8GB DDR3 1600 MHz ECC/Unbuffered 4x (total 32gb ram/blade)
  • 32GB SATA DOM 1x ←– new, important
  • No software ←- lots of work
  • MicroBlade Chassis Management Module (virtual media - usb/cdrom) ←– with KVM
  • 3-Year Warranty on Parts and Labor
• cores/6U, 1.6 gb ram/core
• expansion: 12 nodes, 6U, 240 cores

Back