| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
cluster:126 [2022/07/28 14:43]
hmeij07 [Other Stuff] |
cluster:126 [2025/04/18 13:53] (current)
hmeij07 |
| The High Performance Compute Cluster (HPCC) is comprised of several login nodes (all are on our internal network (vlan 52) //wesleyan.edu// so VPN is required for off campus access as well as for students on campus) | The High Performance Compute Cluster (HPCC) is comprised of several login nodes (all are on our internal network (vlan 52) //wesleyan.edu// so VPN is required for off campus access as well as for students on campus) |
| |
| * server ''cottontail'' (Supermicro 4U), old scheduler openlava, CentOS6 | * server ''cottontail'' (Supermicro 4U), old scheduler Openlava, CentOS6 |
| * primary login server ''cottontail2'' (Supermicro 1U), new slurm scheduler, Rocky8 | * primary login server ''cottontail2'' (Supermicro 1U), new Slurm scheduler, Rocky8, Warewulf, OpenHPC |
| * serveer ''swallowtail'' (Dell PowerEdge 2950 2U), sandbox, CentOS6 | * zabbix and ganglia monitoring and alerting servers ''hpcmon'' (supermicro 1U), CentOS8 |
| * server ''petaltail'' (Dell PowerEdge 2950 2U), sandbox, Warewulf v3.6 provisioning, CentOS6 | * secondary login servers ''petaltal, swallowtail'' (HP blades, CentOS7) |
| * zenoss monitoring and alerting server ''hpcmon'' (supermicro 1U), CentOS6 | * scratch server ''greentail522''(Suoermicro 36+2) serving out /sanscratch, CentOS7, sandbox |
| * secondary login server ''greentail52'' (SuperMicro 36+2, 2U), serving out /sanscratch, CentOS7 | * backup Slurm test server ''sharptail2'' (Supermicro 2U), CentOS8, OpenHPC |
| * server ''sharptail'' (Supermicro 4U), /lvm_data (backup), /zfshomes replication, CentOS6 | * storage servers ''rstore0'' and ''rstore1'' (Supermicro 4U), replicated, Samba shares (2x 440T) |
| * server ''sharptail2'' (Supermicro 2U), disaster recovery for off site (active users only), CentOS6 | * storage servers ''rstore2'' and ''rstore3'' (Supermicro 4U), replicated, Samba shares (2x 440T) |
| * storage servers ''rstore4'' and ''rstore5'' (Supermicro 4U), replicated, Samba shares (2x 220T) | |
| * storage servers ''rstore6'' and ''rstore7'' (Supermicro 4U), replicated, Samba shares (2x 220T) | |
| * storage servers ''mstore0/mstore1'' (Supermicro 4U), replicated, mounted on all HPC nodes (2x 110T) | * storage servers ''mstore0/mstore1'' (Supermicro 4U), replicated, mounted on all HPC nodes (2x 110T) |
| * storage server ''hpcstore'' (IXsystems, dual contrtoller shelf and two storage shelves, /zfshomes 235T | * storage server ''M40HA'' TrueNAS, storage appliance for home directories, /zfshomes (500T) |
| | * storage server ''X20HA'' TrueNAS, replication target for ''M40HA'' (300T) |
| |
| Several types of compute nodes are available via the scheduler: | Several types of compute nodes are available via the scheduler: |
| |
| * All are running CentOS 6.10 or CentOS 7.7 (except cottontail2/n100,n101 run rocky8.5) | * All are running CentOS 8.10 (except some older hardware on CentOS 6 or 7) |
| * All are x86_64, Intel Xeon chips from 2006 onwards | * All are x86_64, Intel Xeon chips with OpenHPC compile environment 2.x |
| * All are on private networks (192.168.x.x and/or 10.10.x.x, no internet) | * All are on private networks (192.168.x.x and/or 10.10.x.x, no internet) |
| * All mount /zfshomes (235T TrueNAS/ZFS appliance 2020) and /sanscratch (xfs, 55T) | * All mount /zfshomes (500T TrueNAS/ZFS appliance 2024) and /sanscratch (xfs, 55T) |
| * All have local disks providing varying amounts of /localscratch (usually Raid0, no backup!) | * All have local disks providing varying amounts of /localscratch (usually Raid0, no backup!) |
| * Hyperthreading is on but only 50% of logical cores allocated via scheduler | * Hyperthreading is on (can be allocated via scheduler) |
| |
| Compute node categories which usually align with queues: | Compute node categories which usually align with queues: |
| * 12 nodes with dual twelve core chips (Xeon Silver 4214, 2.20 Ghz) in ASUS ESC4000G4 2U rack servers with a memory footprint of 96 GB ( 1,152 GB, about 20 teraflops dpfp). These node each have four RTX1080S (32 GB memory footprint) gpus providing 702 teraflops (mixed mode). Known as the "rtx2080" rack, nodes n79-n90, queue exx96, 432 job slots. | * 12 nodes with dual twelve core chips (Xeon Silver 4214, 2.20 Ghz) in ASUS ESC4000G4 2U rack servers with a memory footprint of 96 GB ( 1,152 GB, about 20 teraflops dpfp). These node each have four RTX1080S (32 GB memory footprint) gpus providing 702 teraflops (mixed mode). Known as the "rtx2080" rack, nodes n79-n90, queue exx96, 432 job slots. |
| |
| * 2 nodes with dual twelve core chips (Xeon 4214R “Cascade Lake Refresh” 2.4 GHz), Supermicro 1u servers with a memory footprint of 192 GB. These nodes hold four RTX5000 gpus each. Known as the "sslurm test nodes, nodes n100-n101. Service by cottontail2, dual Xeon 5222 “Cascade Lake-SP” 3.8 GHz 4-core with a memory footprint of 96 GB. | * 2 nodes with dual twelve core chips (Xeon 4214R “Cascade Lake Refresh” 2.4 GHz), Supermicro 1U servers with a memory footprint of 192 GB. These nodes hold four RTX5000 gpus each. Known as the "slurm test" nodes. Service by cottontail2, dual Xeon 5222 “Cascade Lake-SP” 3.8 GHz 4-core with a memory footprint of 96 GB. ''test'' queue, nodes n100-n101. |
| |
| All queues are available for job submissions via all login nodes. Some nodes on Infiniband switches for parallel computational jobs (queues: mw256fd, hp12). Our total job slot count is roughly 2,144 with our physical core count 1,480. Our total teraflops compute capacity is about 58 cpu side, 25 gpu side (double precision floating point) and 702 gpu side (mixed mode). Our total memory footprint is about 560 GB gpu side, 8,916 GB cpu side. | * 6 nodes with dual 28 core chips (Xeon Gold 'Ice Lake-SP' 6330 CPU @ 2.00GHz), Supermicro 1U servers with a memory footprint of 256 GB (1,536 GB, about 27 teraflops dpfp). Know as "astro" rack. ''mw256'' queue nodes n102-n107. Storage server "astrostore" serves the nodes NFSoRDMA (EDR Infiniband). About 164 TB. |
| |
| Home directory file system are provided (via NFS or IPoIB) by the node ''hpcstore'' (our file server) from a direct attached disk array. In total, 235 TB of /zfshomes disk space is accessible to the users. Node ''greentail52'' makes available 55 TB of scratch space at /sanscratch via NFS. In addition all nodes provide local scratch space at /localscratch (excludes queue tinymem). The scheduler automatically makes directories in both these scratch areas for each job (named after JOBPID). Backup services for /zfshomes are provided via disk-to-disk replication from node ''hpcstore'' to node ''sharptail'' disk arrays. The TrueNAs/ZFS appliance performs daily snapshots with a retention window of 180 days. Some faculty&students have their home directories on node ''ringtail'' which provides 33 TB via /home33. Some faculty&students have their home directories on node ''ringtail2'' which provides 66 TB via /home33. Some faculty&students also have their own storage (2x 110 TB via /mindstore). Static content should be migrated to the Rstore platform. | * 10 nodes with dual 12 core chips (Xeon Silver 4410Y CPU @ 3.9 Ghz), Emerald Rapids Microway servers with a memory footprint of 256 GB (2,560 GB, about 90 teraflops dpfp). These nodes hold four RTX4070Ti-Super gpus each. Known as the "desktop" racks. ''mwgpu256'', nodes 108-n117. |
| |
| | All queues are available for job submissions via cottontail2 login node. Some nodes on Infiniband switches for parallel computational jobs (queues: mw256fd, hp12, mw256). Our total job slot count is roughly 2,624 with our physical core count 1,312. Our total teraflops compute capacity is about 88 cpu side and 2,462 gpu side (mixed mode). Our total memory footprint is about 1,200 GB gpu side and 13,012 GB cpu side. |
| |
| ===== Our Queues ===== | Home directory file system are provided (via NFS or IPoIB) by the node M40ha (our file server) from a direct attached disk array. In total, 500 TB of /zfshomes disk space is accessible to the users. Node ''greentail52'' makes available 55 TB of scratch space at /sanscratch via NFS. In addition all nodes provide local scratch space at /localscratch (excludes queue tinymem). The scheduler automatically makes directories in both these scratch areas for each job (named after JOBPID). Backup services for /zfshomes are provided via replication to an older X20HA TrueNAS/ZFS appliance. The M40HA TrueNAS/ZFS appliance performs daily snapshots with a retention window of 180 days. Some faculty&students have their home directories on node ''ringtail'' which provides 33 TB via /home33. Some faculty&students have their home directories on node ''ringtail2'' which provides 66 TB via /home66. Some faculty&students also have their own storage (2x 110 TB via /mindstore). Static content should be migrated to the Rstore platform. |
| |
| There are no scheduler commercial software license resources. Only stata has a limited 6 user license. | |
| |
| ^Queue^Nr Of Nodes^Total GB Mem Per Node^Total Cores In Queue^Switch^Hosts^Notes^ | ===== Our Queues ===== |
| | stata | //na// | //na// | //na// | QDR Infiniband | //any host// | 6 licenses | | |
| |
| Note: Matlab and Mathematica now have "unlimited licenses". | There are no scheduler commercial software license resources. Only stata has a limited 6 user license. Matlab and Mathematica now have "unlimited licenses". |
| |
| |
| | amber128 | 1 | 128 | 24 | gigabit ethernet | n78 | GPU & CPU | | | amber128 | 1 | 128 | 24 | gigabit ethernet | n78 | GPU & CPU | |
| | exx96 | 12 | 96 | 432 | gigabit ethernet | n79-n90 | GPU & CPU | | | exx96 | 12 | 96 | 432 | gigabit ethernet | n79-n90 | GPU & CPU | |
| | | test | 2 | 192 | 96 | gigabit ethernet | n100-n101 | GPU & CPU | |
| | | mw256 | 6 | 256 | 672 | EDR infiniband | n102-n102 | CPU | |
| | | mwgpu256 | 10 | 256 | 480 | gigabit ethernet | n108-n117 | GPU & CPU | |
| |
| Some guidelines for appropriate queue usage with detailed page links: | Some guidelines for appropriate queue usage with detailed page links: |
| * exx96 contains 4 RTX2080S per node | * exx96 contains 4 RTX2080S per node |
| * same setup as mwgpu queue | * same setup as mwgpu queue |
| * test (swallowtail, petaltail, cottontail2, n29, n33) | * test contains 8 RTX5000 gpus |
| | * can be used for production runs |
| | * beware of preemptive events, checkpoint! |
| | * mw1256, NFSoRDMA, bought with faculty startup monies |
| | * beware of preemptive events, checkpoint! |
| | * 6 compute nodes |
| | * Priority access for Sarah's lab till 4/1/2026 |
| | * mwgpu256, contains 40 RTX4070Ti-Super gpus |
| | * same setup as exx96 queue |
| | |
| | **NOTE**: we are migrating from Openlava to Slurm during summer 2022. All queues except ''hp12'' and ''mw256fd'' will be service by ''cottontail2'' Slurm scheduler. |
| |
| NOTE: we are migrating from Openlava to Slurm during summer 2022. All queues except ''hp12'' and ''mw256fd'' will be service by ''cottontail2'' Slurm scheduler. | * [[cluster:213|New Head Node]] |
| | * [[cluster:218|Getting Started with Slurm Guide]] |
| |
| **There are no wall time limits in our HPCC environment except for queue ''test''.** You are responsible for checkpointing though. Consult these pages, all nodes in all queues are DMTCP enabled (read [[cluster:190|DMTCP]]. Login nodes and storage nodes are on UPS but all compute nodes are on utility power. Crashes do happen, be prepared to restart your long running jobs. | **There are no wall time limits in our HPCC environment except for queue ''test''.** You are responsible for checkpointing though. Consult these pages, all nodes in all queues are DMTCP enabled (read [[cluster:190|DMTCP]]. Login nodes and storage nodes are on UPS but all compute nodes are on utility power. Crashes do happen, be prepared to restart your long running jobs. |
