This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:93 [2011/01/08 23:11] hmeij |
cluster:93 [2011/01/10 16:05] hmeij |
||
---|---|---|---|
Line 1: | Line 1: | ||
\\ | \\ | ||
**[[cluster: | **[[cluster: | ||
+ | |||
+ | {{: | ||
====== Greentail ====== | ====== Greentail ====== | ||
Line 18: | Line 20: | ||
The purchase of the HP hardware followed a fierce bidding round in which certain design aspects had to met. | The purchase of the HP hardware followed a fierce bidding round in which certain design aspects had to met. | ||
- | * We continually run of disk space for our home directories. | + | * We continually run out of disk space for our home directories. |
* We wanted more nodes, in fewer queues, with a decent memory footprint. | * We wanted more nodes, in fewer queues, with a decent memory footprint. | ||
* All nodes should be on an Infiniband switch. | * All nodes should be on an Infiniband switch. | ||
Line 24: | Line 26: | ||
* Data (NFS) was to be served up via a secondary gigabit ethernet switch, hence not compete with administrative traffic. | * Data (NFS) was to be served up via a secondary gigabit ethernet switch, hence not compete with administrative traffic. | ||
* (With the HP solution we will actually route data (NFS) traffic over the infiniband switch using OFED/MPI, a practice called [[http:// | * (With the HP solution we will actually route data (NFS) traffic over the infiniband switch using OFED/MPI, a practice called [[http:// | ||
- | * Linux or CentOS as operating system. | + | * Linux (Redhat |
* Flexible on scheduler (options: Lava, LSF, Sun Grid Engine) | * Flexible on scheduler (options: Lava, LSF, Sun Grid Engine) | ||
* The disk array, switches and login node should be backed by some form of UPS (not the compute nodes) | * The disk array, switches and login node should be backed by some form of UPS (not the compute nodes) | ||
Line 35: | Line 37: | ||
In short using linpack (More about [[http:// | In short using linpack (More about [[http:// | ||
- | * greentail' | + | * greentail' |
- | * petaltail/ | + | * petaltail/ |
- | * petaltail/ | + | * petaltail/ |
* so the total is 570 gigaflops, but you'd never want to run across both switches simultaneously | * so the total is 570 gigaflops, but you'd never want to run across both switches simultaneously | ||
- | * sharptail' | + | * sharptail' |
* never got quite all the nodes working together, not sure why | * never got quite all the nodes working together, not sure why | ||
===== Home Dirs ===== | ===== Home Dirs ===== | ||
+ | |||
+ | The home directory disk space (5 TB) on the clusters is served up via NFS from one of our data center NetApp storage servers (named filer3). | ||
+ | |||
+ | In order to do this, your old home directory content will be copied weekly from filer3 to greentail' | ||
+ | |||
+ | To avoid a conflict between home dirs I strongly suggest you create a directory to store the files you will be creating on greentail, for example / | ||
+ | |||
+ | At some point in the future, greentail' | ||
+ | |||
+ | Greentail' | ||
+ | |||
+ | Because of the size of the new home dirs, we will also not be able to provide backup via TSM (Tivoli). | ||
+ | |||
+ | |||
+ | ===== Passwords ===== | ||
+ | |||
+ | The password, shadow and group files of host petaltail were used to populate greentail' | ||
+ | |||
+ | If you change your password, do it on all four hosts (petaltail, swallowtail, | ||
+ | |||
+ | I know, a pain. | ||
+ | |||
===== SSH Keys ===== | ===== SSH Keys ===== | ||
Line 55: | Line 79: | ||
* log into a host, then issue the command '' | * log into a host, then issue the command '' | ||
* supply an empty passphrase (just hit return) | * supply an empty passphrase (just hit return) | ||
- | * then copy the contents of / | + | * then copy the contents of / |
- | * yoiu can have multiple public ssh key entries in this file | + | * you can have multiple public ssh key entries in this file |
- | Note: the software stack on host petaltail/ | + | Note: the software stack on host petaltail/ |
To test if your keys are set up right, simply ssh around the hosts petaltail, swallowtail and greentail. | To test if your keys are set up right, simply ssh around the hosts petaltail, swallowtail and greentail. | ||
===== Rsnapshot ===== | ===== Rsnapshot ===== | ||
+ | |||
+ | [[http:// | ||
+ | |||
+ | Rsnaphot content of all the new home directory content is made available to you at / | ||
+ | |||
+ | Within the snapshot repository you will find directories: | ||
+ | |||
+ | * daily.0 (yesterday), | ||
+ | * weekly.0 (last week), weekly.1 (week before last week) etc ... weekly backups are taken sunday at 10:30 pm | ||
+ | * monthly.0 (last month), monthly.1 (month before last month) etc ... monthly backups are taken on the first day of each month at 10:00 pm | ||
+ | |||
+ | /home and /snapshot are different logical volumes using a different set of disks to protect against loss of data. In addition, both use RAID 6 (double parity) for another layer of protection. | ||
+ | |||
+ | ===== Sanscratch ===== | ||
+ | |||
+ | Previously there were two scratch areas available to your programs: / | ||
+ | |||
+ | However, since our fantastic crash of June 2008 ([[cluster: | ||
+ | |||
+ | On greentail, /sanscratch will be a separate logical volume of 5 TB using a different disk set. SO i urge those that have very large files to stage their files in /sanscratch when running their jobs for best performance. | ||
+ | |||
+ | ===== MPI ===== | ||
+ | |||
===== ... ===== | ===== ... ===== |