This shows you the differences between two versions of the page.
Next revision | Previous revision Next revision Both sides next revision | ||
cluster:125 [2014/02/13 18:53] hmeij created |
cluster:125 [2014/02/21 14:55] hmeij |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | Dell Racks power off | + | \\ |
+ | **[[cluster: | ||
+ | |||
+ | Done! | ||
+ | --- // | ||
+ | |||
+ | ==== Dell Racks Power Off ==== | ||
+ | |||
+ | Soon (Feb/2014), we'll have to power down the Dell Racks and grab one L6-30 circuit supplying power to those racks and use it to power up the new Microway servers. | ||
+ | |||
+ | That leaves some spare L6-30 circuits (the Dell racks use 4 each), so we could contemplate grabbing two and powering up two more shelves of the Blue Sky Studio hardware. | ||
+ | |||
+ | The new Microway hardware is identical to the GPU-HPC hardware we bought previously minus the GPUs. A total of 8 1U servers will offer | ||
+ | |||
+ | * 256 GB of memory per node (2,048 GB total ... that's amazing because if you add the GPU nodes memory footprint, the total for the rack becomes 3,328 GB in 18U of rack space). | ||
+ | * dual 8-core Intel Xeon chips with hyperthreading turned on, so each node presents 32 cores for a total of 256 cores (job slots). These will be presented as queue '' | ||
+ | * Each core is capable of doing 8 instructions per clock cycle and each core will have access to an average of 8 GB of memory | ||
+ | * Each node also has a 300 GB 15K RPM hard disk holding the operating system, swap and provides for a / | ||
+ | * Each node is Infiniband enabled (meaning all our nodes are except the Blue Sky Studio, queue '' | ||
+ | |||
+ | ==== What Changes? ==== | ||
+ | |||
+ | Queues: | ||
+ | |||
+ | * elw, emw, ehw, ehwfd and imw disappear (224 job slots) | ||
+ | * mw256fd appears (256 job slots) | ||
+ | * on both mw256 (n33-n37) and mw256fd (n38-n45) exclusive use is disabled (#BSUB -x will not work) | ||
+ | * the max number of jobs slots per node is 32 on '' | ||
+ | |||
+ | Memory: | ||
+ | |||
+ | * Since fewer and fewer nodes are deployed in our cluster with large memory footprints, it becomes important to estimate how much memory you need (add 10-20%) and reserve that via the scheduler so your jobs do not crash. | ||
+ | |||
+ | < | ||
+ | #BSUB -R " | ||
+ | </ | ||
+ | |||
+ | Gaussian: | ||
+ | |||
+ | * In order to force your gaussian threads onto the same node (since it is a forked program not a parallel program), when using any of mw256 queues, you must use the following stanza' | ||
+ | |||
+ | < | ||
+ | |||
+ | #BSUB -n X (where X is equal to or less than the max jobs per node) | ||
+ | #BSUB -R " | ||
+ | |||
+ | </ | ||
+ | |||
+ | MPI: | ||
+ | |||
+ | * You can use the new queue '' | ||
+ | * For parallel programs you may use OpenMPI or MVApich, use the appropriate wrapper scripts to set up the environment for mpirun | ||
+ | * On '' | ||
+ | * On '' | ||
+ | |||
+ | Scratch: | ||
+ | |||
+ | * On all nodes /sanscratch is always the same and job progress can be viewed from all " | ||
+ | * On all nodes / | ||
+ | * Only nodes on '' | ||
+ | |||
+ | Savings: | ||
+ | |||
+ | * 77% less energy is consumed including what's needed for the new hardware, amazing. | ||
+ | |||
+ | Workshop: | ||
+ | |||
+ | * We'll schedule one as soon as '' | ||
+ | |||
+ | ==== What May Also Change? ==== | ||
+ | |||
+ | There is a significant need to run many, many programs that require very little memory (like in the order of 1-5 MB). When such programs run they consume a job slot. When many such programs consume many job slots, like on the large servers in the '' | ||
+ | |||
+ | So we could enable hyperthreading on the nodes of the '' | ||
+ | |||
+ | * if there is no ‘sharing’ required the hyper-threaded node performs the same (that is the operating systems presents 16 cores but only up to 8 jobs are allowed to run, lets say by limiting the JL/H parameter of the queue) | ||
+ | - if there is ‘sharing’ jobs take a 44% speed penalty, however more of them can run, twice as many | ||
+ | |||
+ | So it appears that we could turn hyperthreading on and despite the nodes presenting 16 cores we could limit the number of jobs to 8 until the need arises to run many small jobs and then reset the limit to 16. | ||
+ | |||
+ | |||
+ | \\ | ||
+ | **[[cluster: |