This is an old revision of the document!
The HPCC cluster's file server sharptail.wesleyan.edu
serves out all home directories to all nodes at location /home. It is 10 TB in size and it currently takes the nightly process of backup 1-2 hours to churn through. Making it larger would thus generate more traffic on /home. So we've, for now while it works for us, come up with this policy:
At this point users need to off load static content to other locations. Contents like old analyses, results of published papers, etc. Users typically have one local option available:
Users whom are considered inactive have their home directories relocated to /archives/inactive
The remote storage option, if your storage needs cannot be supported by /archives, is off-cluster storage. Rstore is our latest storage solution for groups and labs with such needs.
Well, move stuff around? Try to avoid programs such as cp, sftp/scp for large content migrations. The better bet is rsync
. man rsync
for the manual page.
With rsync you can:
rsync -vac –dry-run
rsync –delete
rsync –bwlimit=10000
So to put it all together, for example move my directory in my home directory named stuff elsewhere
rsync –vac –delete –bwlimit=10000 –dry-run /home/username/stuff rstore0:/data/2/somelabgroup/mydirecotory/
Is output ok? Then run again without the –dry-run
omitted.
Note the lack of source trailing slash but present destination trailing slash; meaning put source inside destination location. If both had a trailing slash it would mean; update source and target at these locations. Beware. –delete
may bite.
Once contents have been migrated rm -rf /home/username/stuff