DokuWiki

User Tools

Site Tools

Trace:
cluster:136

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
cluster:136 [2015/08/14 18:09]
hmeij [Moving Content] 		cluster:136 [2020/07/28 17:21] (current)
hmeij07
Line 1: Line 1:
 \\ \\
 **[[cluster:0|Back]]** **[[cluster:0|Back]]**
 +
 +''/home'' is defunct but remains for compatibility. It has been moved from sharptail to whitetail. New home directories are at ''/zfshomes''. Although quotas are in place (starting at 1T for new accounts) users typically get what they need.  Static content should eventually be migrated to our Rstore platform.
 +
 + --- //[[hmeij@wesleyan.edu|Henk]] 2020/07/28 13:18//
  
 ==== HomeDir & Storage Options ==== ==== HomeDir & Storage Options ====
Line 37: Line 41:
 **Check it out:** **Check it out:**
  
 +  * ''ssh sharptail.wesleyan.edu''
   * is the server busy (''uptime'' loads < 8 are ok)    * is the server busy (''uptime'' loads < 8 are ok) 
   * is there memory available (''free -m'' look at free values)   * is there memory available (''free -m'' look at free values)
-  * is anybody else using rsync (''ps -efl | greep rsync'') +  * is anybody else using rsync (''ps -efl | grep rsync'') 
-  * is the server busy writing (''iotop'' look at the M/s disk writes(q to quit), values >100-200 M/s bad)+  * is the server busy writing (''iotop'' look at the M/s disk writes(q to quit), values >100-200 M/s == busy!)
  
-Three scenarios are depicted below. When crossing the vertical boundaries you are not dealing with local content anymore, thus the content needs to flow over the network. ''rsync'' has many features, one of the important one is the use of a remote shell. +Three scenarios are depicted below. When crossing the vertical boundaries you are not dealing with local content anymore, thus the content needs to flow over the network. ''rsync'' has many features, one of the important one is the use of a remote shell allowing an elegant way to cross these boundaries. 
  
 <code> <code>
Line 56: Line 61:
 **Some feature examples** **Some feature examples**
  
-  * preserve permissions, do a checksum between source/destination files, observe what will happen+  * preserve permissions, do a checksum between source/target files, observe what will happen
       * ''rsync -vac --dry-run''       * ''rsync -vac --dry-run''
   * delete files on destination not present on source (careful!)   * delete files on destination not present on source (careful!)
       * ''rsync --delete''       * ''rsync --delete''
-  * throttle the rate of traffic generated, make your sysadmin happy, use+  * throttle the rate of traffic generated, make your sysadmin happy, use <5000
       * ''rsync --bwlimit=2500''       * ''rsync --bwlimit=2500''
-  * use a remote shell form host to host (crossing those vertical boundaries above) +  * do not look inside files 
-      * ''rsync  /home/my/stuff/  user@somehost.wesleyan.edu:/home/my/stuff/'' +      * ''rsync --whole-files'' 
-      * Once contents have been migrated ''rm -rf /home/my/stuff''+  * use a remote shell from host to host (crossing those vertical boundaries above) 
 +      * ''rsync  -vac /home/my/stuff/  user@somehost.wesleyan.edu:/home/my/stuff/''
  
-Note the use of trailing slashes, it means update everything inside source ''stuff/'' within target ''stuff/''. If you left the slash off the above command it means put source directory ''stuff/'' inside target directory ''stuff/'' emaning you'll end up with target ''/home/my/stuff/stuff''. You've been warned. USe the dry run option if unsure what will happen.+Note the use of trailing slashes, it means update everything inside source ''stuff/'' within target ''stuff/''. If you left the first trailing slash off the above command it means put source directory ''stuff/'' inside target directory ''stuff/'' meaning you'll end up with target ''/home/my/stuff/stuff''. You've been warned. Use the dry run option if unsure what will happen.
  
-** Examples **+** Putting it all together **
  
 <code> <code>
  
-# copy from dir stuff lab or remote college to my home on HPCC in tmp area+# copy the dir stuff from lab or remote college to my home on HPCC in tmp area  
 +# (first log in to remote location)
  
-rsync -vac --bwlimit=2500 /home/user/stuff user@sharptail.wesleyan.edu:/home/user/tmp/+rsync -vac --bwlimit=2500 --whole-files /home/user/stuff user@sharptail.wesleyan.edu:/home/user/tmp/
  
-# sync my stuff folder into /archives inside our HPCC, then clean up +# sync my HPCC dir stuff folder into /archives locally on sharptail, then clean up 
 +# (first log in to sharptail) 
  
 rsync -vac --bwlimit=2500 /home/user/stuff/  /archives/user/stuff/ rsync -vac --bwlimit=2500 /home/user/stuff/  /archives/user/stuff/
Line 82: Line 90:
  
 # generate a copy of content on Rstore disk array outside of HPCC but within wesleyan.edu # generate a copy of content on Rstore disk array outside of HPCC but within wesleyan.edu
 +# (get paths and share names from faculty member, on sharptail do)
 +
 +rsync -vac --bwlimit=2500 /home/user/stuff  user@rstoresrv0.wesleyan.edu:/data/2/labcontent/projects/
 +
 +# you can also do this in reverse, log in to sharptail first
  
-rsync -vac--bwlimit=2500 /home/user/stuff  user@rstoresrv0.wesleyan.edu:/data/2/labcontent/projects/+rsync -vac --bwlimt=2500 user@rstoresrv0.wesleyan.edu:/data/2/labcontent/projects/stuff  /home/user/ 
  
 </code> </code>
cluster/136.1439575760.txt.gz · Last modified: 2015/08/14 18:09 by hmeij

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki