This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:136 [2015/02/10 22:03] hmeij [HomeDir & Storage Options] |
cluster:136 [2016/07/08 19:03] hmeij07 |
||
---|---|---|---|
Line 11: | Line 11: | ||
* nightly snapshots (point in time backups) are done on sharptail' | * nightly snapshots (point in time backups) are done on sharptail' | ||
- | At this point users need to off load static content to other locations. | + | At this point users need to off load static content to other locations. |
* Keep contents out of /home and migrate it to /archives (7 TB, accessible on all " | * Keep contents out of /home and migrate it to /archives (7 TB, accessible on all " | ||
Line 22: | Line 22: | ||
* these accounts are kept around until we do an account edit and purge (has never happened so far) | * these accounts are kept around until we do an account edit and purge (has never happened so far) | ||
- | The remote option, if your storage needs cannot be supported by /archives, is off-cluster storage. Rstore is our latest storage solution for groups and labs which such needs. | + | The remote |
* ask your lead faculty member if your lab/group has such an area or request one | * ask your lead faculty member if your lab/group has such an area or request one | ||
Line 28: | Line 28: | ||
* details can be found at [[cluster: | * details can be found at [[cluster: | ||
- | ==== Who do I ..? ==== | + | ==== Moving Content |
- | Well, move stuff around? | + | Our file server |
- | With rsync you can: | ||
- | | + | Do not use any type of copy tool with a GUI or cp/scp or s/ftp. Especially the GUI (drag& |
+ | |||
+ | **Check it out:** | ||
+ | |||
+ | * '' | ||
+ | * is the server busy ('' | ||
+ | * is there memory available ('' | ||
+ | * is anybody else using rsync ('' | ||
+ | * is the server busy writing ('' | ||
+ | |||
+ | Three scenarios are depicted below. When crossing the vertical boundaries you are not dealing with local content anymore, thus the content needs to flow over the network. '' | ||
+ | |||
+ | < | ||
+ | |||
+ | | / | ||
+ | some lab location | ||
+ | < | ||
+ | some other college | ||
+ | | / | ||
+ | |||
+ | </ | ||
+ | |||
+ | **Some feature examples** | ||
+ | |||
+ | | ||
* '' | * '' | ||
- | * delete files on destination not present on source | + | * delete files on destination not present on source |
* '' | * '' | ||
- | * throttle the rate of trafic | + | * throttle the rate of traffic |
- | * '' | + | * '' |
- | * and much more ... | + | * use a remote shell from host to host (crossing those vertical boundaries above) |
+ | * '' | ||
+ | |||
+ | Note the use of trailing slashes, it means update everything inside source '' | ||
+ | |||
+ | ** Putting it all together ** | ||
+ | |||
+ | < | ||
+ | |||
+ | # copy the dir stuff from lab or remote college to my home on HPCC in tmp area | ||
+ | # (first log in to remote location) | ||
+ | |||
+ | rsync -vac --bwlimit=2500 / | ||
+ | |||
+ | # sync my HPCC dir stuff folder into /archives locally on sharptail, then clean up | ||
+ | # (first log in to sharptail) | ||
+ | |||
+ | rsync -vac --bwlimit=2500 / | ||
+ | rm -rf / | ||
- | So to put it all together, move my directory in my hom edirectory named stuff elsewhere | + | # generate a copy of content on Rstore disk array outside of HPCC but within wesleyan.edu |
+ | # (get paths and share names from faculty member, on sharptail do) | ||
- | '' | + | rsync -vac --bwlimit=2500 /home/user/ |
- | Note the lack of source trailing slash but present destination trailing slash; meaning put source inside destination location. If both had a trailing slash it would mean; update source and target at these locations. | + | # you can also do this in reverse, log in to sharptail first |
+ | rsync -vac --bwlimt=2500 user@rstoresrv0.wesleyan.edu:/ | ||
+ | </ | ||