DokuWiki

Soon (Feb/2014), we'll have to power down the Dell Racks and grab one L6-30 circuit supplying power to those racks and use it to power up the new Microway servers.

That leaves some spare L6-30 circuits (the Dell racks use 4 each), so we could contemplate grabbing two and powering up two more shelves of the Blue Sky Studio hardware. That would double the Hadoop cluster and the bss24 queue when needed (total of 100 job slots), and offer access to 1.2 TB of memory. This hardware is generally powered off when not in use.

The new Microway hardware is identical to the GPU-HPC hardware we bought minus the GPUs. A total of 8 1U servers will offer

• 256 GB of memory per node (2,048 GB total … that's amazing because if you add the GPU nodes memory footprint, the total for the rack becomes 3,328 GB in 18U of rack space).
• dual 8-core Intel Xeon chips with hyperthreading turned on, so each node presents 32 cores for a toal of 256 cores (job slots). These will be presented as queue mw256f
• Each core is capable of doing 8 instructions per clock cycle and each core will have access to an average of 8 GB of memory
• Each node also has a 300 GB 15K RPM hard disk holding the operating system, swap and provides for a /localscratch of 175 GB, hence the fd of mw256fd queue name. It is to be used just like ehwfd.
• Each node is Infiniband enabled (meaning all our nodes are except the Blue Sky Studio (queue bss24) and /home and /sanscratch are served IPoIB.

Queues:

• elw, emw, ehw, ehwfd and imw disappear (224 job slots)
• mw256fd appears
• on both mw256 and mw256fd exclusive use is disabled (#BSUB -x)
• the max number of jobs slots per node is 32 on mw256fd but 28 on mw256 (because the GPUs also need access to cores (4 per node for now).

Gaussian:

• In order to force your gaussian threads onto the same node (since it is a forked program not a parallel program) you must the following stanza's:

#BSUB -n X (where is equal to or less than the max jobs per node)
#BSUB -R "span[hosts=1]"

MPI:

• You can use the new queue mw256fd just like hp12 or imw
• For parallel programs you may use OpenMPI or MVApich, use the wrapper scripts to set up the environment for mpirun
• On mwgpu you must use MVApich2 when running the GPU enabled software (Amber, Gromacs, Lammps, Namd).
• On mw256 you may run either flavor of MPI with the appropriate binaries.

Scratch:

• On all nodes /sanscratch is always the same and job progress can be viewed from all “tail” login nodes. It is a 5 disk 5TB storage area for large jobs needing much disk space.
• On all nodes /localscratch is a local directory like /tmp. It is tiny (50 GB) and should be used for file locking purposes if you need to do so.
• Only nodes on mw256fd sport a 15K hard disk and /localscratch is 175 GB (replacing the ehwfd functionality.

Back