Differences

This shows you the differences between two versions of the page.

--- cluster:150 [2016/09/16 13:59] – created hmeij07
+++ cluster:150 [2016/11/29 19:36] (current) – hmeij07
@@ Line 2: / Line 2: @@
 **[[cluster:0|Back]]**
+==== Rsync Daemon/Rsnapshot ====
-==== The problem ====
+==== The Problem ====
-Trying to offload heavy read/write traffic from our file server. I also did a deep information dive to assess if we could afford enterprise level storage. That answer basically means a $42K layout at the low end and up to $70K for the high end. I've detailed the result here [[cluster:139|Enterprise Storage]]
+Trying to offload heavy read/write traffic from our file server. I also did a deep information dive to assess if we could afford enterprise level storage. That answer basically means a $42K layout at the low end and up to $70K for the high end. I've detailed the result here [[cluster:149|Enterprise Storage]], lots to be gained by doing that but implementing the **Short Term** plan first as detailed on that page.
+Avoiding NFS as much as possible, I'd like to set up ''rsyncd'' and use it in combination with ''rsnapshot'' generating daily, weekly and monthly point in time snapshots of sharptail:/home to cottontail:/mnt/home/.snapshots (only accessible by root). I'll then remount on localhost read only for self-serve restore actions by users themselves.
+Using version 1.3.1 since I've seriously tested that out, from [[http://rsnapshot.org|rsnapshot.org]]...the newer version uses arbitrary intervals like alpha, beta, gamma but my users will understand daily, weekly, monthly much better.
+Download source, untar, follow simple instructions in file INSTALL on target server.
+Next, and this took some time to dig up, on target server where the snapshots will reside
+<code>
+cpan YAML
+cpan Lchown
+</code>
+On the source server (where /home is local) configure rsync for daemon mode. Create file ''/etc/rsync.conf'' with the following content.  Notice that we expose /home read only which is a nice approach.
+<code>
+syslog facility = local3
+read only = yes list = yes
+auth users = root
+secrets file = /etc/rsyncd.secrets
+hosts allow = 10.11.103.253
+uid = 0
+gid = 0
+[root]
+comment = /
+path = /
+</code>
+Create that ''secrets file'', chmod 600, and put a long string in there, like ''ThisIsVerySecret''. Start the daemons on the source server.
+<code>
+rsync --daemon
+</code>
+Back on target server where snapshots will reside, configure the ''/etc/rsnapshot.conf'' file, here are my settings
+<code>
+[root@cottontail .snapshots]# grep ^[a-z] /etc/rsnapshot.conf
+config_version  1.2
+snapshot_root   /mnt/home/.snapshots/
+cmd_cp          /bin/cp
+cmd_rm          /bin/rm
+cmd_rsync       /usr/bin/rsync
+cmd_ssh /usr/bin/ssh
+cmd_logger      /bin/logger
+cmd_du          /usr/bin/du
+cmd_rsnapshot_diff      /usr/local/bin/rsnapshot-diff
+cmd_preexec     /root/start.sh
+cmd_postexec    /root/stop.sh
+interval        daily   6
+interval        weekly  4
+interval        monthly 3
+interval        yearly  1
+verbose         3
+loglevel        3
+logfile /var/log/rsnapshot
+lockfile        /var/run/rsnapshot.pid
+rsync_short_args        -vac
+rsync_long_args  --delete --numeric-ids --relative --delete-excluded --whole-file --bwlimit=60000 --stats
+du_args -csh
+one_fs          1
+link_dest       1
+backup  sharptail-ib0::root/home/       sharptail/
+</code>
+Couple of notes
+  * cmd_preexec and cmd_postexec just echo the date to console and start/stop the rsync daemons remotely on source server
+  * choose to use whole-file to speed things up
+  * onefs is important
+  * link_dest is important
+  * the backup command points to my private IP of source server using the Infiniband network and then points to the module defined in ''/etc/rsync.conf'' on source server
+Start things up
+<code>
+# on target
+[root@cottontail ~]# /usr/local/bin/rsnapshot daily &
+# on source
+[root@sharptail ~]# lsof -i:873
+COMMAND   PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
+rsync   17814 root    3u  IPv4 261839487      0t0  TCP *:rsync (LISTEN)
+rsync   17814 root    5u  IPv6 261839488      0t0  TCP *:rsync (LISTEN)
+rsync   29717 root    6u  IPv4 261962741      0t0  TCP sharptail-ib0:rsync->cottontail-ib0:54069 (ESTABLISHED)
+# check what rsync is doing
+[root@sharptail ~]# strace -p 29717
+Process 29717 attached - interrupt to quit
+close(5)                                = 0
+lstat("/home/hemamy/data-analysis-for-fcc-lattice-tetra/fcc/data-analysis-for-7-b-spacer/NPs-same-mass/old-data/msd/msd-diff-temp/5.0nmNP/115msd-5.0nmNP-t-115-tetras.dat", {st_mode=S_IFREG|0644, st_size=11125, ...}) = 0
+open("/home/hemamy/data-analysis-for-fcc-lattice-tetra/fcc/data-analysis-for-7-b-spacer/NPs-same-mass/old-data/msd/msd-diff-temp/5.0nmNP/115msd-5.0nmNP-t-115-tetras.dat", O_RDONLY) = 5
+read(5, "1\t43.224064\n2\t79.676036\n3\t111.20"..., 11125) = 11125
+close(5)                                = 0
+lstat("/home/hemamy/data-analysis-for-fcc-lattice-tetra/fcc/data-analysis-for-7-b-spacer/NPs-same-mass/old-data/msd/msd-diff-temp/5.0nmNP/a115ave-msd-5.0nmNP-t-115-tetras.dat", {st_mode=S_IFREG|0644, st_size=8, ...}) = 0
+open("/home/hemamy/data-analysis-for-fcc-lattice-tetra/fcc/data-analysis-for-7-b-spacer/NPs-same-mass/old-data/msd/msd-diff-temp/5.0nmNP/a115ave-msd-5.0nmNP-t-115-tetras.dat", O_RDONLY) = 5
+CTRL-C
+</code>
+Suggest you debug with a small directory first, /home is 10TB in our case with 40+ million files.
+Then remount the user inaccessible area /mnt/home/.snapshots for user access on /snapshots
+<code>
+# /etc/exports content, then exportfs -ra
+/mnt/home/.snapshots       localhost.localdomain(sync,rw,no_all_squash,no_root_squash)
+# /etc/fstab content, then mount /snapshot/daily.0
+/dev/mapper/VG0-lvhomesnaps        /mnt/home/.snapshots     xfs    rw,tcp,intr,bg              0 0
+localhost:/mnt/home/.snapshots/daily.0 /snapshots/daily.0   nfs   ro   0 0
+# test
+[root@cottontail .snapshots]# touch /snapshots/daily.0/sharptail/home/hmeij/tmp/foo
+touch: cannot touch `/snapshots/daily.0/sharptail/home/hmeij/tmp/foo': Read-only file system
+</code>
+Finally, if you get this error, which is hardly informative, you've set num-tries to 0 like I did and fussed over it for some time. Set to 1 or leave uncommented.
+<code>
+-09-15T11:37:06] /usr/local/bin/rsnapshot daily: ERROR: /usr/bin/rsync returned 0.00390625 while processing rsync://sharptail-ib00::root/home/hmeij/python/
+</code>
 \\
 **[[cluster:0|Back]]**