Differences

This shows you the differences between two versions of the page.

--- cluster:151 [2016/11/29 19:29] – [cluster idea] hmeij07
+++ cluster:151 [2016/12/06 20:14] (current) – [beeGFS cluster idea] hmeij07
@@ Line 10: / Line 10: @@
 This page is best read from the bottom up.
+NOTE:
+I'm reluctantly giving up on beegfs, especially v6.1, it simply works flaky. In the admon gui I can see 2 storage nodes, 4 storage objects, 4 meta servers with clients installed on all meta. /mnt/beegfs is there and content can be created. Then I mirror storage nodes, all is fine. Then I mirror meta servers and the mirrors set up, enabling mirrormd states success. Then the whole environment hangs on /mnt/beegfs.  My sense is helperd is not communication well in a private network environment with no DNS and does not consult /etc/hosts. But I have nothing to back that up with, so I can fix it.
+Back to adding more XFS into my cluster, I'll wait a few more versions.
+ --- //[[hmeij@wesleyan.edu|Henk]] 2016/12/06 15:10//
 ==== beeGFS cluster idea ====
   * Storage servers:
-    * buy 2 with each 12x2TB slow disk, Raid 6, 20T usable
+    * buy 2 with each 12x2TB slow disk, Raid 6, 20T usable (clustered, parallel file system)
       * create 2 6TB volumes on each, quota at 2TB via XFS, 3 users/server
       * only $HOME changes to ''/mnt/beegfs/home[1|2]'' (migrates ~4.5TB away from /home or ~50%)
-      * create 2 buddymirrors, primary on one, secondary on the other server (high availability)
+      * create 2 buddymirrors; each with primary on one, secondary on the other server (high availability)
     * on UPS
     * on Infiniband
@@ Line 29: / Line 35: @@
   * Management and Monitor servers
-    * cottontail
+    * cottontail (on UPS, on Infiniband)
-  * Backups
+  * Backups (rsnapshot.org via rsync daemons [[cluster:150|Rsync Daemon/Rsnapshot]])
     * sharptail:/home --> cottontail
     * serverA:/mnt/beegfs/home1 --> serverB (8TB max)
@@ Line 38: / Line 44: @@
   * Costs (includes 3 year NBD warranty)
     * Microway $12,500
-    * CDW
+    * CDW $14,700
 ==== beegfs-admin-gui ====
   * ''cottontail:/usr/local/bin/beegfs-admin-gui''
+==== upgrade ====
+  * [[http://www.beegfs.com/content/updating-upgrading-and-versioning/|External Link]]
+  * New feature - High Availability for Metadata Servers (self-healing, transparent failover)
+A bit complicated.
+  * Repo base URL baseurl=http://www.beegfs.com/release/beegfs_6/dists/rhel6 via http shows only 6.1-el6
+    * [   ] beegfs-mgmtd-6.1-el6.x86_64.rpm          2016-11-16 16:27  660K
+  * '' yum --disablerepo "*" --enablerepo beegfs repolist'' shows
+    * beegfs-mgmtd.x86_64                            2015.03.r22-el6            beegfs
+  * ''yum install --disablerepo "*" --enablerepo beegfs --downloadonly --downloaddir=/sanscratch/tmp/beegfs/beegfs_6/ *x86_64* -y''
+   * http://www.beegfs.com/release/beegfs_6/dists/rhel6/x86_64/beegfs-mgmtd-2015.03.r22-el6.x86_64.rpm: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found" <-- wrong package version
+So the wget/rpm approach (list all packages present on a particular node else you will get a dependencies failure!)
+<code>
+# get them all
+wget http://www.beegfs.com/release/beegfs_6/dists/rhel6/x86_64/beegfs-mgmtd-6.1-el6.x86_64.rpm
+# client and meta node
+rpm -Uvh ./beegfs-common-6.1-el6.noarch.rpm ./beegfs-utils-6.1-el6.x86_64.rpm ./beegfs-opentk-lib-6.1-el6.x86_64.rpm ./beegfs-helperd-6.1-el6.x86_64.rpm ./beegfs-client-6.1-el6.noarch.rpm ./beegfs-meta-6.1-el6.x86_64.rpm
+# updated?
+[root@cottontail2 beegfs_6]# beegfs-ctl | head -2
+BeeGFS Command-Line Control Tool (http://www.beegfs.com)
+Version: 6.1
+#Sheeesh
+</code>
 ==== Resync Data #2 ====
@@ Line 74: / Line 114: @@
 # define mirrrogroups
-[root@cottontail2 ~]# beegfs-ctl --addmirrorgroup --primary=21701 --secondary=13601 --groupid=1
+[root@cottontail2 ~]# beegfs-ctl --addmirrorgroup [--nodetype=storage] --primary=21701 --secondary=13601 --groupid=1
-[root@cottontail2 ~]# beegfs-ctl --addmirrorgroup --primary=13602 --secondary=21702 --groupid=2
+[root@cottontail2 ~]# beegfs-ctl --addmirrorgroup [--nodetype=storage] --primary=13602 --secondary=21702 --groupid=2
 [root@cottontail2 ~]# beegfs-ctl --listmirrorgroups
@@ Line 346: / Line 386: @@
 Content mirroring will require more disk space. Perhaps snapshots to another node is more useful, also solves backup issue.
+V6 does buddymirror meta mirroring [[http://www.beegfs.com/wiki/MDMirror|External Link]]
 <code>
-# enable meta mirroring, directory based
+# 2015.03 enable meta mirroring, directory based
 # change to 11/04/2016: used --createdir to make this home.
 [root@n7 ~]# beegfs-ctl --mirrormd /mnt/beegfs/home
@@ Line 355: / Line 395: @@
 Mount: '/mnt/beegfs'; Path: '/hmeij-mirror'
 Operation succeeded.
+# V6.1 does it a root level not from a path
+beegfs-ctl --addmirrorgroup --nodetype=meta --primary=38 --secondary=39 --groupid=1
+beegfs-ctl --addmirrorgroup --nodetype=meta --primary=250 --secondary=37 --groupid=2
+beegfs-ctl --mirrromd
 # put some new content in
@@ Line 488: / Line 533: @@
   * made easy [[http://www.beegfs.com/wiki/ManualInstallWalkThrough|External Link]]
   * rpms pulled from repository via petaltail in ''greentail:/sanscratch/tmp/beegfs''
+    * ''yum --disablerepo "*" --enablerepo beegfs list available''
+    * use ''yumdownloader''
 <code>