User Tools

Site Tools


cluster:136

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
cluster:136 [2015/08/14 13:47]
hmeij [How do I ..?]
cluster:136 [2020/07/28 13:21] (current)
hmeij07
Line 1: Line 1:
 \\ \\
 **[[cluster:​0|Back]]** **[[cluster:​0|Back]]**
 +
 +''/​home''​ is defunct but remains for compatibility. It has been moved from sharptail to whitetail. New home directories are at ''/​zfshomes''​. Although quotas are in place (starting at 1T for new accounts) users typically get what they need.  Static content should eventually be migrated to our Rstore platform.
 +
 + --- //​[[hmeij@wesleyan.edu|Henk]] 2020/07/28 13:18//
  
 ==== HomeDir & Storage Options ==== ==== HomeDir & Storage Options ====
Line 35: Line 39:
 Do not use any type of copy tool with a GUI or cp/scp or s/ftp. Especially the GUI (drag&​drop) are Verboten! These tools are not smart enough and frequently generated blocked processes that halt everything. Use ''​rsync''​ in a linux/unix environment. Do not use any type of copy tool with a GUI or cp/scp or s/ftp. Especially the GUI (drag&​drop) are Verboten! These tools are not smart enough and frequently generated blocked processes that halt everything. Use ''​rsync''​ in a linux/unix environment.
  
-  +**Check it out:**
-Check it out:+
  
 +  * ''​ssh sharptail.wesleyan.edu''​
   * is the server busy (''​uptime''​ loads < 8 are ok)    * is the server busy (''​uptime''​ loads < 8 are ok) 
   * is there memory available (''​free -m''​ look at free values)   * is there memory available (''​free -m''​ look at free values)
-  * is anybody else using rsync (''​ps -efl | greep rsync''​) +  * is anybody else using rsync (''​ps -efl | grep rsync''​) 
-  * is the server busy writing (''​iotop''​ look at the M/s disk writes(q to quit), values >100-200 M/s bad)+  * is the server busy writing (''​iotop''​ look at the M/s disk writes(q to quit), values >100-200 M/s == busy!)
  
-Three scenarios are depicted below+Three scenarios are depicted below. When crossing the vertical boundaries you are not dealing with local content anymore, thus the content needs to flow over the network. ''​rsync''​ has many features, one of the important one is the use of a remote shell allowing an elegant way to cross these boundaries. ​
  
 <​code>​ <​code>​
  
-                        |                       ​                   |    some lab location ​    +                        |         /home            group share     |    some lab location ​    
 some lab location ​      ​| ​                      ​| ​                   | some lab location ​      ​| ​                      ​| ​                   |
                   <​----------->​ sharptail <​----------->​ Rstore <​-----------> ​                               ​                   <​----------->​ sharptail <​----------->​ Rstore <​-----------> ​                               ​
 some other college ​     |                       ​| ​                   |    ​ some other college ​     |                       ​| ​                   |    ​
-                        |                       ​                   |    some other college ​              +                        |         /​archives ​        lab share      ​|    some other college ​              
  
 </​code>​ </​code>​
  
-  ​* preserve permissions,​ do a checksum between source/destination ​files, observe what will happen+**Some feature examples** 
 + 
 +  ​* preserve permissions,​ do a checksum between source/target ​files, observe what will happen
       * ''​rsync -vac --dry-run''​       * ''​rsync -vac --dry-run''​
-  * delete files on destination not present on source+  * delete files on destination not present on source ​(careful!)
       * ''​rsync --delete''​       * ''​rsync --delete''​
-  * throttle the rate of traffic generated, make your sysadmin happy, use+  * throttle the rate of traffic generated, make your sysadmin happy, use <5000
       * ''​rsync --bwlimit=2500''​       * ''​rsync --bwlimit=2500''​
-  * and much more ...+  * do not look inside files 
 +      * ''​rsync --whole-files''​ 
 +  * use a remote shell from host to host (crossing those vertical boundaries above) 
 +      * ''​rsync ​ -vac /​home/​my/​stuff/ ​ user@somehost.wesleyan.edu:/​home/​my/​stuff/''​
  
-So to put it all togetherfor example move my directory ​in my home directory ​named stuff elsewhere+Note the use of trailing slashesit means update everything inside source ''​stuff/''​ within target ''​stuff/''​. If you left the first trailing slash off the above command it means put source ​directory ​''​stuff/''​ inside target ​directory ​''​stuff/''​ meaning you'll end up with target ''/​home/​my/​stuff/​stuff''​. You've been warned. Use the dry run option if unsure what will happen.
  
-''​rsync --vac --delete --bwlimit=2500 --dry-run /​home/​username/​stuff ​ rstore0:/​data/​2/​somelabgroup/​mydirecotory/''​+** Putting it all together **
  
-Is output ok? Then run again the ''​--dry-run''​ option omitted.+<​code>​
  
-Note the lack of source trailing slash but present destination trailing slash; meaning put source inside destination ​location. If both had a trailing slash it would mean; update source and target at these locations. Beware. ''​--delete''​ may bite.+# copy the dir stuff from lab or remote college to my home on HPCC in tmp area  
 +# (first log in to remote ​location)
  
-Once contents have been migrated ''​rm -rf /home/username/stuff''​+rsync -vac --bwlimit=2500 --whole-files /​home/​user/​stuff user@sharptail.wesleyan.edu:/​home/​user/​tmp/​ 
 + 
 +# sync my HPCC dir stuff folder into /archives locally on sharptail, then clean up 
 +# (first log in to sharptail)  
 + 
 +rsync -vac --bwlimit=2500 /​home/​user/​stuff/ ​ /​archives/​user/​stuff/​ 
 +rm -rf /home/user/stuff/* 
 + 
 +# generate a copy of content on Rstore disk array outside of HPCC but within wesleyan.edu 
 +# (get paths and share names from faculty member, on sharptail do) 
 + 
 +rsync -vac --bwlimit=2500 /​home/​user/​stuff ​ user@rstoresrv0.wesleyan.edu:/​data/​2/​labcontent/​projects/​ 
 + 
 +# you can also do this in reverse, log in to sharptail first 
 + 
 +rsync -vac --bwlimt=2500 user@rstoresrv0.wesleyan.edu:/​data/​2/​labcontent/​projects/​stuff ​ /home/user/  
 + 
 +</​code>​
  
  
cluster/136.1439574464.txt.gz · Last modified: 2015/08/14 13:47 by hmeij