This page will be maintained and provide information to get users started using the compute cluster. It is a merger of the old “brief description” page and the “queue description” page.

HPCC maintains and regularly updates an extensive software stack. Including provisioning tools, resource management, file transfer clients, development tools, a variety of scientific libraries, a variety of compilers (e.g. gcc/g++, OneAPI) and communication libraries (e.g., OpenMPI). Primarily provided by OpenHPC (https://openhpc.community/). Many opensource applications are custom compiled (about 100 or so used by many academic disciplines). The HPCC website offers documentation to help users resolve technical issues they may encounter (https://dokuwiki.wesleyan.edu/doku.php?id=cluster:0). Additional technical support (and tutors) is provided by the Scientific Computing and Informatics Center (https://www.wesleyan.edu/scic/) and the Quantitative Analysis Center (https://www.wesleyan.edu/qac/).

ITS funds the system administrative support (0.75 FTE) of one ITS employee. Power and cooling are funded and maintained by Physical Plant. On an annual basis Academic Affairs contributes $25K and the HPCC users contribute $15K. This sets up a 4 year refresh cycle of $160K. On an annual basis Finance contributes up to $10K for maintenance (failed disks etc, monies do not roll over, use it or loose it).

All HPCC hardware is located inside Wesleyan's ITS data center. Physical access is limited by swipe cards to certain ITS personnel. All head/login nodes (2 to 4 or so) are located on an internal subnet protected by Wesleyan's enterprise wide firewall. VPN is required from off campus. The internal HPCC network consists of two private subnets for the 100 or so compute nodes (one for job scheduler and monitoring tools, one for data transfers and NFS mounts). Access to HPCC resources is based on local Linux accounts and groups.

The High Performance Compute Cluster (HPCC) is comprised of several login nodes (all are on our internal network (vlan 52) wesleyan.edu so VPN is required for off campus access as well as for students on campus)

• server cottontail (Supermicro 4U), old scheduler Openlava, CentOS6
• primary login server cottontail2 (Supermicro 1U), new Slurm scheduler, Rocky8, Warewulf, OpenHPC
• zabbix and ganglia monitoring and alerting servers hpcmon (supermicro 1U), CentOS8
• secondary login servers petaltal, swallowtail (HP blades, Rocky8)
• scratch server greentail522(Suoermicro 36+2) serving out /sanscratch, CentOS7, sandbox
• backup Slurm test server sharptail2 (Supermicro 2U), CentOS8, OpenHPC
• storage servers rstore0 and rstore1 (Supermicro 4U), replicated, Samba shares (2x 440T)
• storage servers rstore2 and rstore3 (Supermicro 4U), replicated, Samba shares (2x 440T)
• storage servers mstore0/mstore1 (Supermicro 4U), replicated, mounted on all HPC nodes (2x 110T)
• storage server M40HA TrueNAS, storage appliance for home directories, /zfshomes (500T)
• storage server X20HA TrueNAS, replication target for M40HA (300T)

Several types of compute nodes are available via the scheduler:

• All are running CentOS 8.10 (except some older hardware on CentOS 6 or 7)
• All are x86_64, Intel Xeon chips with OpenHPC compile environment 2.x
• All are on private networks (192.168.x.x and/or 10.10.x.x, no internet)
• All mount /zfshomes (500T TrueNAS/ZFS appliance 2024) and /sanscratch (xfs, 55T)
• All have local disks providing varying amounts of /localscratch (usually Raid0, no backup!)
• Hyperthreading is on (can be allocated via scheduler)

Compute node categories which usually align with queues:

• 32 nodes with dual quad core chips (Xeon 5620, 2.4 Ghz) in HP blade 4U enclosures (SL2x170z G6) with memory footprint of 12 GB each (384 GB). This cluster has a compute capacity of 1.5 teraflops (measured using Linpack). Known as the HP cluster, or the nodes n1-n32, queue hp12, 256 job slots.

• 5 nodes with dual eight core chips (Xeon E5-2660, 2.2 Ghz) in ASUS 2U rack servers with a memory footprint of 256 GB each (1,280 GB). Nodes also contain four K20 Telsa GPU each, 2,500 cores/gpu (10,000 gpu cores per node) with GPU memory footprint of 5 GB (20 GB). This cluster has a compute capacity of 23.40 teraflops double precision or 70.40 teraflops single precision on GPU side and 2.9 teraflops on cpu side. Known as the Microway GPU cluster, or the nodes n33-n37, mwgpu (120 job slots). Old queue mw256 merged in.

• 8 nodes with dual eight core chips (Xeon E5-2660, 2.2 Ghz) in Supermicro 1U rack servers with a memory footprint of 256 GB each (2,048 GB). This cluster has a compute capacity of 5.3 teraflops (estimated). Known as the Microway CPU cluster, or nodes n38-n45, queue mw256fd, 192 job slots.

• 14 nodes with dual ten core chips (Xeon E5-2550 v3, 2.3 Ghz) in Supermicro 1U rack servers with a memory footprint of 32 GB each (256 GB). This cluster has a compute capacity of 12 teraflops (estimated). Known as the Microway tinymem cluster, or n46-n59, queue tinymem, 448 job slots.

• 18 nodes with dual twelve core chips (Xeon E5-2650 v4, 2.2 Ghz) in Supermicro 1U rack servers with a memory footprint of 128 GB each (2,304 GB). This cluster has a compute capacity of 14.3 teraflops (estimated). Known as the Microway “Carlos” CPU cluster, or nodes n60-n77, queue mw128, 648 job slots.

• 1 node with dual eight core chips (Xeon E5-2620 v4, 2.10 Ghz) in Supermicro 1U rack server with a memory footprint of 128 GB. This node has four GTX1080i (32 GB memory footprint) gpus. Known as the “amber128” queue, 24 job slots.

• 12 nodes with dual twelve core chips (Xeon Silver 4214, 2.20 Ghz) in ASUS ESC4000G4 2U rack servers with a memory footprint of 96 GB ( 1,152 GB, about 20 teraflops dpfp). These node each have four RTX1080S (32 GB memory footprint) gpus providing 702 teraflops (mixed mode). Known as the “rtx2080” rack, nodes n79-n90, queue exx96, 432 job slots.

• 2 nodes with dual twelve core chips (Xeon 4214R “Cascade Lake Refresh” 2.4 GHz), Supermicro 1U servers with a memory footprint of 192 GB. These nodes hold four RTX5000 gpus each. Known as the “slurm test” nodes. Service by cottontail2, dual Xeon 5222 “Cascade Lake-SP” 3.8 GHz 4-core with a memory footprint of 96 GB. test queue, nodes n100-n101.

• 6 nodes with dual 28 core chips (Xeon Gold 'Ice Lake-SP' 6330 CPU @ 2.00GHz), Supermicro 1U servers with a memory footprint of 256 GB (1,536 GB, about 27 teraflops dpfp). Know as “astro” rack. mw256 queue nodes n102-n107. Storage server “astrostore” serves the nodes NFSoRDMA (EDR Infiniband). About 164 TB.

• 10 nodes with dual 12 core chips (Xeon Silver 4410Y CPU @ 3.9 Ghz), Emerald Rapids Microway servers with a memory footprint of 256 GB (2,560 GB, about 90 teraflops dpfp). These nodes hold four RTX4070Ti-Super gpus each. Known as the “desktop” racks. mwgpu256, nodes 108-n117.

• 1 node with dual twelve core chips (Xeon Silver 4520, 2.40 Ghz) in ASUS ESC4000-E11 2U rack server with a memory footprint of 512 GB ( about 4 teraflops dpfp). This node has four NVIDIA RTX PRO™ 6000 Blackwell Max-Q Workstation Edition (96 GB memory footprint) gpus providing 440 teraflops (mixed mode). Known as the “Blackwell” gpu server, node n91, queue exx512, 48 job slots. https://www.techpowerup.com/gpu-specs/rtx-pro-6000-blackwell-max-q.c4273

All queues are available for job submissions via cottontail2 login node. Some nodes on Infiniband switches for parallel computational jobs (queues: mw256fd, hp12, mw256). Our total job slot count is roughly 2,672 with our physical core count 1,336. Our total teraflops compute capacity is about 92 cpu side and 2,902 gpu side (mixed mode). Our total memory footprint is about 1,584 GB gpu side and 13,524 GB cpu side.

Home directory file system are provided (via NFS or IPoIB) by the node M40ha (our file server) from a direct attached disk array. In total, 500 TB of /zfshomes disk space is accessible to the users. Node greentail52 makes available 55 TB of scratch space at /sanscratch via NFS. In addition all nodes provide local scratch space at /localscratch (excludes queue tinymem). The scheduler automatically makes directories in both these scratch areas for each job (named after JOBPID). Backup services for /zfshomes are provided via replication to an older X20HA TrueNAS/ZFS appliance. The M40HA TrueNAS/ZFS appliance performs daily snapshots with a retention window of 180 days. Some faculty&students have their home directories on node ringtail which provides 33 TB via /home33. Some faculty&students have their home directories on node ringtail2 which provides 66 TB via /home66. Some faculty&students also have their own storage (2x 110 TB via /mindstore). Static content should be migrated to the Rstore platform.

There are no scheduler commercial software license resources. Only stata has a limited 6 user license. Matlab and Mathematica now have “unlimited licenses”.

Some guidelines for appropriate queue usage with detailed page links:

• hp12 is the default queue
  • for processing lots of small to medium memory footprint jobs
• mwgpu is for GPU (K20) enabled software primarily (Amber, Lammps, NAMD, Gromacs, Matlab, Mathematica)
  • be sure to reserve one or more job slot for each GPU used EXX96
  • be sure to use the correct wrapper script to set up mpirun from mvapich2, mpich3 or openmpi
• mw256fd are for jobs requiring large memory access (up to 24 jobs slots per node)
  • or requiring lots of threads (job slots) confined to single node (Gaussian, Autodeck)
  • or requiring access to fast /localscratch which is 175 GB on a 15K disk.
    • you must stage and save results, for an example read https://dokuwiki.wesleyan.edu * /doku.php?id=cluster:103#submit_2
  • or requiring larger /localscratch5tb, which is a Raid 0 file system of 5 TB
    • stage temporary data in /localscratch5tb/username/ and it will not be removed
• tinymem are for small serial jobs with small memory requirements
  • nodes have a sataDOM (non spinning 16G USB device on motherboard) for operating system
  • do not use /localscratch on these nodes, no no
• mw128 (bought with faculty startup funds) tailored for Gaussian jobs
  • About 2TB /localscratch (Raid 10) on each node
  • Priority access for Carlos' group till 07/01/2020
• amber128 (donated hardware) tailored for Amber16 jobs
  • Be sure to use mpich3 for Amber
  • Priority access for Amber jobs till 10/01/2020
• exx96 contains 4 RTX2080S per node
  • same setup as mwgpu queue
• test contains 8 RTX5000 gpus
  • can be used for production runs
  • beware of preemptive events, checkpoint!
• mw1256, NFSoRDMA, bought with faculty startup monies
  • beware of preemptive events, checkpoint!
  • 6 compute nodes
  • Priority access for Sarah's lab till 4/1/2026
• mwgpu256, contains 40 RTX4070Ti-Super gpus
  • same setup as exx96 queue
• exx512, contains 4 NVIDIA RTX PRO™ 6000 Blackwell Max-Q Workstation Edition GPUs
  • beware of preemptive events, checkpoint!
  • Priority access for Antonio's lab till 10/7/2028

NOTE: we are migrating from Openlava to Slurm during summer 2022. All queues except hp12 and mw256fd will be service by cottontail2 Slurm scheduler.

• New Head Node
• Getting Started with Slurm Guide

There are no wall time limits in our HPCC environment except for queue test. You are responsible for checkpointing though. Consult these pages, all nodes in all queues are DMTCP enabled (read DMTCP. Login nodes and storage nodes are on UPS but all compute nodes are on utility power. Crashes do happen, be prepared to restart your long running jobs.

Home directory policy and Rstore storage options HomeDir and Storage Options

Checkpointing is supported in all queues, how it works DMTCP page

For a list of software installed consult Software List page, endless…

For a list of OpenHPC software installed consult Software List page

Details on all scratch spaces consult Scratch Spaces page

For HPCC acknowledgements consult Acknowledgement page

Sample scripts for job submissions (serial, array, parallel, forked and gpu) can be found at /zfshomes/hmeij/jobs/ and /zfshomes/hmeij/k20redo and /zfshomes/hmeij/slurm

From off-campus you need to VPN in first, download GlobalProtect client at http://vpn.wesleyan.edu

Back