User Tools

Site Tools


cluster:136

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
cluster:136 [2015/08/14 14:10]
hmeij [Moving Content]
cluster:136 [2020/07/28 13:21] (current)
hmeij07
Line 1: Line 1:
 \\ \\
 **[[cluster:​0|Back]]** **[[cluster:​0|Back]]**
 +
 +''/​home''​ is defunct but remains for compatibility. It has been moved from sharptail to whitetail. New home directories are at ''/​zfshomes''​. Although quotas are in place (starting at 1T for new accounts) users typically get what they need.  Static content should eventually be migrated to our Rstore platform.
 +
 + --- //​[[hmeij@wesleyan.edu|Henk]] 2020/07/28 13:18//
  
 ==== HomeDir & Storage Options ==== ==== HomeDir & Storage Options ====
Line 40: Line 44:
   * is the server busy (''​uptime''​ loads < 8 are ok)    * is the server busy (''​uptime''​ loads < 8 are ok) 
   * is there memory available (''​free -m''​ look at free values)   * is there memory available (''​free -m''​ look at free values)
-  * is anybody else using rsync (''​ps -efl | greep rsync''​) +  * is anybody else using rsync (''​ps -efl | grep rsync''​) 
-  * is the server busy writing (''​iotop''​ look at the M/s disk writes(q to quit), values >100-200 M/s bad)+  * is the server busy writing (''​iotop''​ look at the M/s disk writes(q to quit), values >100-200 M/s == busy!)
  
-Three scenarios are depicted below. When crossing the vertical boundaries you are not dealing with local content anymore, thus the content needs to flow over the network. ''​rsync''​ has many features, one of the important one is the use of a remote shell. ​+Three scenarios are depicted below. When crossing the vertical boundaries you are not dealing with local content anymore, thus the content needs to flow over the network. ''​rsync''​ has many features, one of the important one is the use of a remote shell allowing an elegant way to cross these boundaries
  
 <​code>​ <​code>​
Line 57: Line 61:
 **Some feature examples** **Some feature examples**
  
-  * preserve permissions,​ do a checksum between source/destination ​files, observe what will happen+  * preserve permissions,​ do a checksum between source/target ​files, observe what will happen
       * ''​rsync -vac --dry-run''​       * ''​rsync -vac --dry-run''​
   * delete files on destination not present on source (careful!)   * delete files on destination not present on source (careful!)
       * ''​rsync --delete''​       * ''​rsync --delete''​
-  * throttle the rate of traffic generated, make your sysadmin happy, use+  * throttle the rate of traffic generated, make your sysadmin happy, use <5000
       * ''​rsync --bwlimit=2500''​       * ''​rsync --bwlimit=2500''​
-  * use a remote shell form host to host (crossing those vertical boundaries above) +  ​* do not look inside files 
-      * ''​rsync ​ /​home/​my/​stuff/ ​ user@somehost.wesleyan.edu:/​home/​my/​stuff/​''​ +      * ''​rsync --whole-files''​ 
-      * Once contents have been migrated ''​rm -rf /​home/​my/​stuff''​+  ​* use a remote shell from host to host (crossing those vertical boundaries above) 
 +      * ''​rsync  ​-vac /​home/​my/​stuff/ ​ user@somehost.wesleyan.edu:/​home/​my/​stuff/''​
  
-Note the use of trailing slashes, it means update everything inside source ''​stuff/''​ within target ''​stuff/''​. If you left the slash off the above command it means put source directory ''​stuff/''​ inside target directory ''​stuff/'' ​emaning ​you'll end up with target ''/​home/​my/​stuff/​stuff''​. You've been warned. ​USe the dry run option if unsure what will happen.+Note the use of trailing slashes, it means update everything inside source ''​stuff/''​ within target ''​stuff/''​. If you left the first trailing ​slash off the above command it means put source directory ''​stuff/''​ inside target directory ''​stuff/'' ​meaning ​you'll end up with target ''/​home/​my/​stuff/​stuff''​. You've been warned. ​Use the dry run option if unsure what will happen.
  
-** Examples ​**+** Putting it all together ​**
  
 <​code>​ <​code>​
  
-# copy from dir stuff lab or remote college to my home on HPCC in tmp area+# copy the dir stuff from lab or remote college to my home on HPCC in tmp area  
 +# (first log in to remote location)
  
-rsync -vac --bwlimit=2500 /​home/​user/​stuff user@sharptail.wesleyan.edu:/​home/​user/​tmp/​+rsync -vac --bwlimit=2500 ​--whole-files ​/​home/​user/​stuff user@sharptail.wesleyan.edu:/​home/​user/​tmp/​
  
-# sync my stuff folder into /​archives ​inside our HPCC, then clean up +# sync my HPCC dir stuff folder into /​archives ​locally on sharptail, then clean up 
 +# (first log in to sharptail) ​
  
 rsync -vac --bwlimit=2500 /​home/​user/​stuff/ ​ /​archives/​user/​stuff/​ rsync -vac --bwlimit=2500 /​home/​user/​stuff/ ​ /​archives/​user/​stuff/​
Line 83: Line 90:
  
 # generate a copy of content on Rstore disk array outside of HPCC but within wesleyan.edu # generate a copy of content on Rstore disk array outside of HPCC but within wesleyan.edu
 +# (get paths and share names from faculty member, on sharptail do)
 +
 +rsync -vac --bwlimit=2500 /​home/​user/​stuff ​ user@rstoresrv0.wesleyan.edu:/​data/​2/​labcontent/​projects/​
 +
 +# you can also do this in reverse, log in to sharptail first
  
-rsync -vac--bwlimit=2500 /​home/​user/​stuff  ​user@rstoresrv0.wesleyan.edu:/​data/​2/​labcontent/​projects/​+rsync -vac --bwlimt=2500 user@rstoresrv0.wesleyan.edu:/​data/​2/​labcontent/​projects/​stuff ​ /home/user
  
 </​code>​ </​code>​
cluster/136.1439575847.txt.gz · Last modified: 2015/08/14 14:10 by hmeij