DokuWiki

User Tools

Site Tools

Trace:
cluster:150

Warning: Undefined array key -1 in /usr/share/dokuwiki/inc/html.php on line 1458

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
cluster:150 [2016/09/16 10:30]
hmeij07 [The problem] 		cluster:150 [2016/11/29 14:36] (current)
hmeij07
Line 2: Line 2:
 **[[cluster:0|Back]]** **[[cluster:0|Back]]**
  
 +==== Rsync Daemon/Rsnapshot ====
  
-==== The problem ====+==== The Problem ====
  
 Trying to offload heavy read/write traffic from our file server. I also did a deep information dive to assess if we could afford enterprise level storage. That answer basically means a $42K layout at the low end and up to $70K for the high end. I've detailed the result here [[cluster:149|Enterprise Storage]], lots to be gained by doing that but implementing the **Short Term** plan first as detailed on that page.  Trying to offload heavy read/write traffic from our file server. I also did a deep information dive to assess if we could afford enterprise level storage. That answer basically means a $42K layout at the low end and up to $70K for the high end. I've detailed the result here [[cluster:149|Enterprise Storage]], lots to be gained by doing that but implementing the **Short Term** plan first as detailed on that page. 
Line 48: Line 49:
 </code> </code>
  
-Back on source server, configure the ''/etc/rsnapshot.conf'' file, here are my settings+Back on target server where snapshots will reside, configure the ''/etc/rsnapshot.conf'' file, here are my settings
  
 <code> <code>
Line 86: Line 87:
   * onefs is important   * onefs is important
   * link_dest is important   * link_dest is important
-  * the backup command points to my private IP of source server using the Infiniband network and then points to the module define in ''/etc/rsync.conf'' on source server+  * the backup command points to my private IP of source server using the Infiniband network and then points to the module defined in ''/etc/rsync.conf'' on source server
  
 Start things up Start things up
Line 92: Line 93:
 <code> <code>
  
-# on source +# on target 
-/usr/local/bin/rsnapshot daily &+[root@cottontail ~]# /usr/local/bin/rsnapshot daily &
  
-# watch on target+# on source
 [root@sharptail ~]# lsof -i:873 [root@sharptail ~]# lsof -i:873
 COMMAND   PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME COMMAND   PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
Line 101: Line 102:
 rsync   17814 root    5u  IPv6 261839488      0t0  TCP *:rsync (LISTEN) rsync   17814 root    5u  IPv6 261839488      0t0  TCP *:rsync (LISTEN)
 rsync   29717 root    6u  IPv4 261962741      0t0  TCP sharptail-ib0:rsync->cottontail-ib0:54069 (ESTABLISHED) rsync   29717 root    6u  IPv4 261962741      0t0  TCP sharptail-ib0:rsync->cottontail-ib0:54069 (ESTABLISHED)
 +
 +
 +# check what rsync is doing
 [root@sharptail ~]# strace -p 29717 [root@sharptail ~]# strace -p 29717
 Process 29717 attached - interrupt to quit Process 29717 attached - interrupt to quit
Line 110: Line 114:
 lstat("/home/hemamy/data-analysis-for-fcc-lattice-tetra/fcc/data-analysis-for-7-b-spacer/NPs-same-mass/old-data/msd/msd-diff-temp/5.0nmNP/a115ave-msd-5.0nmNP-t-115-tetras.dat", {st_mode=S_IFREG|0644, st_size=8, ...}) = 0 lstat("/home/hemamy/data-analysis-for-fcc-lattice-tetra/fcc/data-analysis-for-7-b-spacer/NPs-same-mass/old-data/msd/msd-diff-temp/5.0nmNP/a115ave-msd-5.0nmNP-t-115-tetras.dat", {st_mode=S_IFREG|0644, st_size=8, ...}) = 0
 open("/home/hemamy/data-analysis-for-fcc-lattice-tetra/fcc/data-analysis-for-7-b-spacer/NPs-same-mass/old-data/msd/msd-diff-temp/5.0nmNP/a115ave-msd-5.0nmNP-t-115-tetras.dat", O_RDONLY) = 5 open("/home/hemamy/data-analysis-for-fcc-lattice-tetra/fcc/data-analysis-for-7-b-spacer/NPs-same-mass/old-data/msd/msd-diff-temp/5.0nmNP/a115ave-msd-5.0nmNP-t-115-tetras.dat", O_RDONLY) = 5
- +CTRL-C 
 + 
 +</code> 
 + 
 +Suggest you debug with a small directory first, /home is 10TB in our case with 40+ million files. 
 + 
 +Then remount the user inaccessible area /mnt/home/.snapshots for user access on /snapshots 
 + 
 +<code> 
 + 
 +# /etc/exports content, then exportfs -ra 
 +/mnt/home/.snapshots       localhost.localdomain(sync,rw,no_all_squash,no_root_squash) 
 + 
 +# /etc/fstab content, then mount /snapshot/daily.0 
 +/dev/mapper/VG0-lvhomesnaps        /mnt/home/.snapshots     xfs    rw,tcp,intr,bg              0 0 
 +localhost:/mnt/home/.snapshots/daily.0 /snapshots/daily.0   nfs   ro   0 0 
 + 
 +# test 
 +[root@cottontail .snapshots]# touch /snapshots/daily.0/sharptail/home/hmeij/tmp/foo 
 +touch: cannot touch `/snapshots/daily.0/sharptail/home/hmeij/tmp/foo': Read-only file system 
 + 
 +</code> 
 + 
 +Finally, if you get this error, which is hardly informative, you've set num-tries to 0 like I did and fussed over it for some time. Set to 1 or leave uncommented. 
 + 
 +<code> 
 + 
 +2016-09-15T11:37:06] /usr/local/bin/rsnapshot daily: ERROR: /usr/bin/rsync returned 0.00390625 while processing rsync://sharptail-ib00::root/home/hmeij/python/ 
 </code> </code>
  
 \\ \\
 **[[cluster:0|Back]]** **[[cluster:0|Back]]**
cluster/150.1474036250.txt.gz · Last modified: 2016/09/16 10:30 by hmeij07

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki