User Tools

Site Tools



Brief Guide to HPCC

This page will be maintained and provide information to get users started using the compute cluster. It is a merger of the old “brief description” page and the “queue description” page.


The High Performance Compute Cluster (HPCC) is comprised of several login nodes (all are on our internal network (vlan 52) so VPN is required for off campus access)

  • primary login node cottontail (Supermicro 4U), primary scheduler (old snapshots of /home on local disk array)
  • secondary login node cottontail2 (HP Proliant G380 2U), backup scheduler
  • secondary login node swallowtail (Dell PowerEdge 2950 2U), backup scheduler, databases
  • sandbox petaltail (Dell PowerEdge 2950 2U), test box, Warewulf provisioning CentOS6
  • rebuild whitetail:/lvhomes (from old /home)(HP Proliant G380 2U), Warewulf OpenHPC provisioning CentOS7
  • zenoss monitoring and alerting server hpcmon (supermicro 1U, centos6)
  • server greentail52 (SuperMicro 36+2, 2U), serving out /sanscratch
  • server sharptail (Supermicro 4U), old /home NFS (defunct), will be rebuild for zfshomes replication
  • server sharptail2 (Supermicro 2U), disaster recovery for off site (active users only)
  • storage servers rstore0 and rstore2 (Supermicro 4U), NFS mounts and Samba shares (2x 120T)
  • storage servers rstore4 and rstore6 (Supermicro 4U), NFS mounts and Samba shares (2x 220T)
  • storage servers mstore0/mindstorsrv1 (Supermicro 4U), mounted on all HPC nodes (2x 110T)

Several types of compute nodes are available via the scheduler:

  • All are running CentOS 6.10 or CentOS 7.7
  • All are x86_64, Intel Xeon chips from 2006 onwards
  • All are on private networks (192.168.x.x and/or 10.10.x.x, no internet)
  • All mount /zfshomes (190T FreeNAS/ZFS appliance 2020) and /sanscratch (xfs, 55T)
  • All have local disks providing varying amounts of /localscratch (usually Raid0, no backup!)
  • Hyperthreading is on but only 50% of logical cores allocated via scheduler

Compute node categories which usually align with queues:

  • 32 nodes with dual quad core chips (Xeon 5620, 2.4 Ghz) in HP blade 4U enclosures (SL2x170z G6) with memory footprint of 12 GB each (384 GB). This cluster has a compute capacity of 1.5 teraflops (measured using Linpack). Known as the HP cluster, or the nodes n1-n32, queue hp12, 256 job slots.
  • 5 nodes with dual eight core chips (Xeon E5-2660, 2.2 Ghz) in ASUS 2U rack servers with a memory footprint of 256 GB each (1,280 GB). Nodes also contain four K20 Telsa GPU each, 2,500 cores/gpu (10,000 gpu cores per node) with GPU memory footprint of 5 GB (20 GB). This cluster has a compute capacity of 23.40 teraflops double precision or 70.40 teraflops single precision on GPU side and 2.9 teraflops on cpu side. Known as the Microway GPU cluster, or the nodes n33-n37, mwgpu (120 job slots). Old queue mw256 merged in.
  • 8 nodes with dual eight core chips (Xeon E5-2660, 2.2 Ghz) in Supermicro 1U rack servers with a memory footprint of 256 GB each (2,048 GB). This cluster has a compute capacity of 5.3 teraflops (estimated). Known as the Microway CPU cluster, or nodes n38-n45, queue mw256fd, 192 job slots.
  • 14 nodes with dual ten core chips (Xeon E5-2550 v3, 2.3 Ghz) in Supermicro 1U rack servers with a memory footprint of 32 GB each (256 GB). This cluster has a compute capacity of 12 teraflops (estimated). Known as the Microway tinymem cluster, or n46-n59, queue tinymem, 448 job slots.
  • 18 nodes with dual twelve core chips (Xeon E5-2650 v4, 2.2 Ghz) in Supermicro 1U rack servers with a memory footprint of 128 GB each (2,304 GB). This cluster has a compute capacity of 14.3 teraflops (estimated). Known as the Microway “Carlos” CPU cluster, or nodes n60-n77, queue mw128, 648 job slots.
  • 1 node with dual eight core chips (Xeon E5-2620 v4, 2.10 Ghz) in Supermicro 1U rack server with a memory footprint of 128 GB. This node has four GTX1080i (32 GB memory footprint) gpus. Known as the “amber128” queue, 24 job slots.
  • 12 nodes with dual twelve core chips (Xeon Silver 4214, 2.20 Ghz) in ASUS ESC4000G4 2U rack servers with a memory footprint of 96 GB ( 1,152 GB, about 20 teraflops dpfp). These node each have four RTX1080S (32 GB memory footprint) gpus providing 702 teraflops (mixed mode). Known as the “rtx2080” rack, nodes n79-n90, queue exx96, 432 job slots.

All queues are available for job submissions via all login nodes. Some nodes on Infiniband switches for parallel computational jobs (queues: mw256fd, hp12). Our total job slot count is roughly 2,144 with our physical core count 1,480. Our total teraflops compute capacity is about 58 cpu side, 25 gpu side (double precision floating point) and 702 gpu side (mixed mode). Our total memory footprint is about 528 GB gpu side, 8,532 GB cpu side.

Home directory file system are provided (via NFS or IPoIB) by the node hpcstore (our file server) from a direct attached disk array. In total, 190 TB of /zfshomes disk space is accessible to the users. Node greentail52 makes available 55 TB of scratch space at /sanscratch via NFS. In addition all nodes provide local scratch space at /localscratch (excludes queue tinymem). The scheduler automatically makes directories in both these scratch areas for each job (named after JOBPID). Backup services for /zfshomes are provided via disk-to-disk replication from node hpcstore to node sharptail disk arrays. The TrueNAs/ZFS appliance performs daily snapshots with a retention window of 365 days. Some faculty have their home directories on node ringtail which provides 33 TB via /home33. Some faculty also have their own storage (2x 110 TB via /mindstore). All home directories will migrate to a FreeNAS/ZFS appliance named hpcstore in 2020 (190T usable, scalable to 1.2P, ETA summer 2020).

Two (old) Rstore storage servers each provide about 104 TB of usable backup space which is not mounted on the compute nodes. Each Rstore server's content is replicated to a dedicated passive standby server of same size, located in same data center but in different racks. As of Spring 2019 we have added two new Rstore servers of 220 T each, fully backed up with replication. Faculty may request shares for their labs to off load static content from the HPCC.

Our Queues

Commercial software has their own queue limited by available licenses. There are no scheduler license resources, just queue jobs up in appropriate queue. Commercial software jobs are processed on the nodes of mw256fd and mw128.

QueueNr Of NodesTotal GB Mem Per NodeTotal Cores In QueueSwitchHostsNotes
stata na na na QDR Infiniband any host 6 licenses

Note: Matlab and Mathematica now have “unlimited licenses”.

QueueNr Of NodesTotal GB Mem Per NodeJob SLots In QueueSwitchHostsNotes
hp12 32 12 256 QDR infiniband n1-n32 CPU
mwgpu 5 256 120 QDR infiniband n33-n37 GPU & CPU
mw256fd 8 256 192 QDR infiniband n38-n45 CPU
tinymem 14 32 448 gigabit ethernet n39-n59 CPU
mw128 18 128 648 gigabit ethernet n60-n77 CPU
amber128 1 128 24 gigabit ethernet n78 GPU & CPU
exx96 12 96 432 gigabit ethernet n79-n90 GPU & CPU

Some guidelines for appropriate queue usage with detailed page links:

  • hp12 is the default queue
    • for processing lots of small to medium memory footprint jobs
  • mwgpu is for GPU (K20) enabled software primarily (Amber, Lammps, NAMD, Gromacs, Matlab, Mathematica)
    • be sure to reserve one or more job slot for each GPU used EXX96
    • be sure to use the correct wrapper script to set up mpirun from mvapich2, mpich3 or openmpi
  • mw256fd are for jobs requiring large memory access (up to 24 jobs slots per node)
    • or requiring lots of threads (job slots) confined to single node (Gaussian, Autodeck)
    • or requiring access to fast /localscratch which is 175 GB on a 15K disk.
    • or requiring larger /localscratch5tb, which is a Raid 0 file system of 5 TB
      • stage temporary data in /localscratch5tb/username/ and it will not be removed
  • tinymem are for small serial jobs with small memory requirements
    • nodes have a sataDOM (non spinning 16G USB device on motherboard) for operating system
    • do not use /localscratch on these nodes, no no
  • mw128 (bought with faculty startup funds) tailored for Gaussian jobs
    • About 2TB /localscratch (Raid 10) on each node
    • Priority access for Carlos' group till 07/01/2020
  • amber128 (donated hardware) tailored for Amber16 jobs
    • Be sure to use mpich3 for Amber
    • Priority access for Amber jobs till 10/01/2020
  • test (swallowtail, petaltail, cottontail2, n29, n33)
    • wall time of 8 hours of CPU usage

There are no wall time limits in our HPCC environment except for queue test. You are responsible for checkpointing though. Consult these pages, all nodes in all queues are DMTCP enabled (read DMTCP. Login nodes and storage nodes are on UPS but all compute nodes are on utility power. Crashes do happen, be prepared to restart your long running jobs.

Other Stuff

Home directory policy and Rstore storage options HomeDir and Storage Options

Checkpointing is supported in all queues, how it works DMTCP page

For a list of software installed consult Software List page, endless…

Details on all scratch spaces consult Scratch Spaces page

For HPCC acknowledgements consult Acknowledgement page

Sample scripts for job submissions (serial, array, parallel, forked and gpu) can be found at /zfshomes/hmeij/jobs/ and /zfshomes/hmeij/k20redo

From off-campus you need to VPN in first, download GlobalProtect client at


cluster/126.txt · Last modified: 2020/07/28 13:17 by hmeij07