DokuWiki

User Tools

Site Tools

Trace:
cluster:93

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
cluster:93 [2011/01/08 23:11]
hmeij 		cluster:93 [2011/01/09 20:56]
hmeij
Line 18: Line 18:
 The purchase of the HP hardware followed a fierce bidding round in which certain design aspects had to met. The purchase of the HP hardware followed a fierce bidding round in which certain design aspects had to met.
  
-  * We continually run of disk space for our home directories.  So the new cluster had to have a large disk array on board.+  * We continually run out of disk space for our home directories.  So the new cluster had to have a large disk array on board.
   * We wanted more nodes, in fewer queues, with a decent memory footprint.   * We wanted more nodes, in fewer queues, with a decent memory footprint.
   * All nodes should be on an Infiniband switch.   * All nodes should be on an Infiniband switch.
Line 24: Line 24:
   * Data (NFS) was to be served up via a secondary gigabit ethernet switch, hence not compete with administrative traffic.   * Data (NFS) was to be served up via a secondary gigabit ethernet switch, hence not compete with administrative traffic.
   * (With the HP solution we will actually route data (NFS) traffic over the infiniband switch using OFED/MPI, a practice called [[http://en.wikipedia.org/wiki/OpenFabrics_Alliance|IPoIB]])   * (With the HP solution we will actually route data (NFS) traffic over the infiniband switch using OFED/MPI, a practice called [[http://en.wikipedia.org/wiki/OpenFabrics_Alliance|IPoIB]])
-  * Linux or CentOS as operating system.+  * Linux (Redhat or CentOS) as operating system.
   * Flexible on scheduler (options: Lava, LSF, Sun Grid Engine)   * Flexible on scheduler (options: Lava, LSF, Sun Grid Engine)
   * The disk array, switches and login node should be backed by some form of UPS (not the compute nodes)   * The disk array, switches and login node should be backed by some form of UPS (not the compute nodes)
Line 35: Line 35:
 In short using linpack (More about [[http://en.wikipedia.org/wiki/LINPACK|Linpack on wikipedia]]) here are the results.  The results are dependent on the combination of total memory, total cores, and speed of processors. In short using linpack (More about [[http://en.wikipedia.org/wiki/LINPACK|Linpack on wikipedia]]) here are the results.  The results are dependent on the combination of total memory, total cores, and speed of processors.
  
-  * greentail's Voltaire infiniband switch is capable of just over 1,500 gigaflops or 1.5 teraflops. +  * greentail's nodes, all on Voltaire infiniband switch, are capable of just over 1,500 gigaflops or 1.5 teraflops. 
-  * petaltail/swallowtail's Cisco infiniband switch is capable of 325 gigaflops +  * petaltail/swallowtail's nodes on Cisco infiniband switch are capable of 325 gigaflops 
-  * petaltail/swallowtail's Force 10 ethernet switch is capable of 245 gigaflops+  * petaltail/swallowtail's nodes on Force 10 ethernet switch are capable of 245 gigaflops
     * so the total is 570 gigaflops, but you'd never want to run across both switches simultaneously     * so the total is 570 gigaflops, but you'd never want to run across both switches simultaneously
-  * sharptail's HP ProCurve ethernet switch is estimated at deleivering between 500-700 gigaflops+  * sharptail's nodes on HP ProCurve ethernet switch are estimated at delivering between 500-700 gigaflops
     * never got quite all the nodes working together, not sure why     * never got quite all the nodes working together, not sure why
  
  
 ===== Home Dirs ===== ===== Home Dirs =====
 +
 +The home directory disk space (5 TB) on the clusters is served up via NFS from one of our data center NetApp storage servers (named filer3).  (Lets refer to those as "old home dirs"). We will be migrating off filer3 to greentail's local disk array.  The path will remain the same on greentail: /home/username. (Lets refer to those as "new home dirs").
 +
 +In order to do this, your old home directory content will be copied weekly from filer3 to greentail's disk array.  When you create new files in your old home dirs they will show up on greentail's new home dirs.  However, if you delete files in old home dirs, and they have already been copied over, the files will remain in your new home dirs.  If you create new files in greentail's new home dirs they will **not** be copied back to your old home dirs.
 +
 +To avoid a conflict between home dirs I strongly suggest you create a directory to store the files you will be creating on greentail, for example /home/username/greentail or /home/username/hp.
 +
 +At some point in the future, greentail's new home dirs will be mounted on the petaltail/swallowtail and sharptail clusters.  Filer3's old home dirs will then disappear permanently.
 +
 +Greentail's new home dirs will provide 10 TB of disk space.  Again, the clusters file system should not be used to archive data. However, doubling the home directory size should provide much needed relief.
 +
 +Because of the size of the new home dirs, we will also not be able to provide backup via TSM (Tivoli).  Backup via TSM to our Virtual Tape Library (VTL) will be replaced with disk to disk backup on greentail's disk array.  That has some serious implications.  Please read the section about RSnapshot.
 +
 +
 +===== Passwords =====
 +
 +The password, shadow and group files of host petaltail were used to populate greentail's equivalent files.
 +
 +If you change your password, do it on all four hosts (petaltail, swallowtail, sharptail and greentail).
 +
 +I know, a pain.
 +
  
 ===== SSH Keys ===== ===== SSH Keys =====
Line 55: Line 77:
   * log into a host, then issue the command ''ssh-keygen -t rsa''   * log into a host, then issue the command ''ssh-keygen -t rsa''
   * supply an empty passphrase (just hit return)   * supply an empty passphrase (just hit return)
-  * then copy the contents of /home/username/.ssh/id_rsa.pub into the file authroized_keys +  * then copy the contents of /home/username/.ssh/id_rsa.pub into the file authorized_keys 
-  * yoiu can have multiple public ssh key entries in this file+  * you can have multiple public ssh key entries in this file
  
-Note: the software stack on host petaltail/swallowtail created ssh keys for you automatically upon your first login, so for most you this is all set.+Note: the software stack on host petaltail/swallowtail created ssh keys for you automatically upon your first login, so for most of you this is all set.
  
 To test if your keys are set up right, simply ssh around the hosts petaltail, swallowtail and greentail. To test if your keys are set up right, simply ssh around the hosts petaltail, swallowtail and greentail.
cluster/93.txt · Last modified: 2011/01/11 20:55 by hmeij

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki