User Tools

Site Tools


cluster:126

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
cluster:126 [2025/04/18 13:42]
hmeij07
cluster:126 [2025/04/18 13:53] (current)
hmeij07
Line 53: Line 53:
   * 10 nodes with dual 12 core chips (Xeon Silver 4410Y CPU @ 3.9 Ghz), Emerald Rapids Microway servers with a memory footprint of 256 GB (2,560 GB, about 90 teraflops dpfp). These nodes hold four RTX4070Ti-Super gpus each.  Known as the "desktop" racks. ''mwgpu256'', nodes 108-n117.   * 10 nodes with dual 12 core chips (Xeon Silver 4410Y CPU @ 3.9 Ghz), Emerald Rapids Microway servers with a memory footprint of 256 GB (2,560 GB, about 90 teraflops dpfp). These nodes hold four RTX4070Ti-Super gpus each.  Known as the "desktop" racks. ''mwgpu256'', nodes 108-n117.
  
-All queues are available for job submissions via cottontail2 login node. Some nodes on Infiniband switches for parallel computational jobs (queues: mw256fd, hp12, mw256).  Our total job slot count is roughly 2,624 with our physical core count 1,312. Our total teraflops compute capacity is about 58 cpu side and 2,462 gpu side (mixed mode). Our total memory footprint is about 1,200 GB gpu side,  13,012 GB cpu side.+All queues are available for job submissions via cottontail2 login node. Some nodes on Infiniband switches for parallel computational jobs (queues: mw256fd, hp12, mw256).  Our total job slot count is roughly 2,624 with our physical core count 1,312. Our total teraflops compute capacity is about 88 cpu side and 2,462 gpu side (mixed mode). Our total memory footprint is about 1,200 GB gpu side and 13,012 GB cpu side.
  
-Home directory file system are provided (via NFS or IPoIB) by the node M40ha (our file server) from a direct attached disk array. In total, 500 TB of /zfshomes disk space is accessible to the users. Node ''greentail52'' makes available 55 TB of scratch space at /sanscratch via NFS.  In addition all nodes provide local scratch space at /localscratch (excludes queue tinymem). The scheduler automatically makes directories in both these scratch areas for each job (named after JOBPID). Backup services for /zfshomes are provided via disk-to-disk replication from node ''hpcstore'' to node ''sharptail'' disk arrays. The TrueNAs/ZFS appliance performs daily snapshots with a retention window of 180 days. Some faculty&students have their home directories on node ''ringtail'' which provides 33 TB via /home33. Some faculty&students have their home directories on node ''ringtail2'' which provides 66 TB via /home33. Some faculty&students also have their own storage (2x 110 TB via /mindstore).  Static content should be migrated to the Rstore platform.+Home directory file system are provided (via NFS or IPoIB) by the node M40ha (our file server) from a direct attached disk array. In total, 500 TB of /zfshomes disk space is accessible to the users. Node ''greentail52'' makes available 55 TB of scratch space at /sanscratch via NFS.  In addition all nodes provide local scratch space at /localscratch (excludes queue tinymem). The scheduler automatically makes directories in both these scratch areas for each job (named after JOBPID). Backup services for /zfshomes are provided via replication to an older X20HA TrueNAS/ZFS appliance. The M40HA TrueNAS/ZFS appliance performs daily snapshots with a retention window of 180 days. Some faculty&students have their home directories on node ''ringtail'' which provides 33 TB via /home33. Some faculty&students have their home directories on node ''ringtail2'' which provides 66 TB via /home66. Some faculty&students also have their own storage (2x 110 TB via /mindstore).  Static content should be migrated to the Rstore platform.
  
  
 ===== Our Queues ===== ===== Our Queues =====
  
-There are no scheduler commercial software license resources. Only stata has a limited 6 user license. +There are no scheduler commercial software license resources. Only stata has a limited 6 user license. Matlab and Mathematica now have "unlimited licenses".
- +
-^Queue^Nr Of Nodes^Total GB Mem Per Node^Total Cores In Queue^Switch^Hosts^Notes^ +
-|  stata  |   //na//  |  //na//  |  //na//  |   QDR Infiniband | //any host// |  6 licenses | +
- +
-Note: Matlab and Mathematica now have "unlimited licenses".+
  
  
Line 78: Line 73:
 |  test  |    |  192  |  96  | gigabit ethernet  | n100-n101 |  GPU & CPU  | |  test  |    |  192  |  96  | gigabit ethernet  | n100-n101 |  GPU & CPU  |
 |  mw256  |    |  256  |  672  | EDR infiniband  | n102-n102 |  CPU  | |  mw256  |    |  256  |  672  | EDR infiniband  | n102-n102 |  CPU  |
 +|  mwgpu256  |   10  |  256  |  480  | gigabit ethernet  | n108-n117 |  GPU & CPU  |
  
 Some guidelines for appropriate queue usage with detailed page links: Some guidelines for appropriate queue usage with detailed page links:
Line 107: Line 103:
     * can be used for production runs     * can be used for production runs
     * beware of preemptive events, checkpoint!     * beware of preemptive events, checkpoint!
-  * mw128, NFSoRDMA, bought with faculty startup monies+  * mw1256, NFSoRDMA, bought with faculty startup monies
     * beware of preemptive events, checkpoint!     * beware of preemptive events, checkpoint!
     * 6 compute nodes     * 6 compute nodes
     * Priority access for Sarah's lab till 4/1/2026     * Priority access for Sarah's lab till 4/1/2026
 +  * mwgpu256, contains 40 RTX4070Ti-Super gpus
 +    * same setup as exx96 queue
  
 **NOTE**: we are migrating from Openlava to Slurm during summer 2022. All queues except ''hp12'' and ''mw256fd'' will be service by ''cottontail2'' Slurm scheduler. **NOTE**: we are migrating from Openlava to Slurm during summer 2022. All queues except ''hp12'' and ''mw256fd'' will be service by ''cottontail2'' Slurm scheduler.
cluster/126.txt · Last modified: 2025/04/18 13:53 by hmeij07