cluster:151
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
cluster:151 [2016/11/29 16:20] hmeij07 [Resync Data #2] |
cluster:151 [2016/12/06 20:14] (current) hmeij07 [beeGFS cluster idea] |
||
|---|---|---|---|
| Line 6: | Line 6: | ||
| A document for me to recall and make notes of what I read in the manual pages and what needs testing. | A document for me to recall and make notes of what I read in the manual pages and what needs testing. | ||
| - | Basically during the Summer of 2016 I investigated if the HPCC could afford enterprise level storage. I wanted 99.999% uptime, snapshots, high availability and other goodies such as parallel NFS. Netapp came the closest but, eh, still at $42K lots of other options show up. The story is detailed | + | Basically during the Summer of 2016 I investigated if the HPCC could afford enterprise level storage. I wanted 99.999% uptime, snapshots, high availability and other goodies such as parallel NFS. Netapp came the closest but, eh, still at $42K lots of other options show up. That story is detailed at [[cluster: |
| This page is best read from the bottom up. | This page is best read from the bottom up. | ||
| - | ==== cluster idea ==== | + | NOTE: |
| - | * Storage servers: buy 2 now 4k+4k then 3rd in July 4k? | + | I'm reluctantly giving up on beegfs, especially v6.1, it simply works flaky. In the admon gui I can see 2 storage nodes, 4 storage objects, 4 meta servers with clients installed on all meta. /mnt/beegfs is there and content can be created. Then I mirror storage nodes, all is fine. Then I mirror meta servers and the mirrors set up, enabling mirrormd states success. Then the whole environment hangs on / |
| - | * move test users over on 2 nodes, test, only change is $HOME | + | Back to adding more XFS into my cluster, I'll wait a few more versions. |
| + | --- // | ||
| + | ==== beeGFS cluster idea ==== | ||
| - | * Home cluster | + | * Storage servers: |
| - | * cottontail | + | * buy 2 with each 12x2TB slow disk, Raid 6, 20T usable |
| - | * 2-3 new units storage | + | * create |
| - | * cottontail2 meta + n38-n45 | + | * only $HOME changes to ''/ |
| + | * create 2 buddymirrors; | ||
| + | * on UPS | ||
| + | * on Infiniband | ||
| + | |||
| + | * Client servers: | ||
| + | * all compute/ | ||
| + | |||
| + | * Meta servers: | ||
| + | * cottontail2 | ||
| + | * all mirrored | ||
| + | * cottontail2 on UPS | ||
| + | |||
| + | * Management and Monitor servers | ||
| + | * cottontail (on UPS, on Infiniband) | ||
| + | |||
| + | * Backups (rsnapshot.org via rsync daemons [[cluster: | ||
| + | * sharptail:/ | ||
| + | * serverA:/ | ||
| + | * serverB:/ | ||
| + | |||
| + | * Costs (includes 3 year NBD warranty) | ||
| + | * Microway $12,500 | ||
| + | * CDW $14,700 | ||
| ==== beegfs-admin-gui ==== | ==== beegfs-admin-gui ==== | ||
| * '' | * '' | ||
| + | |||
| + | ==== upgrade ==== | ||
| + | |||
| + | * [[http:// | ||
| + | * New feature - High Availability for Metadata Servers (self-healing, | ||
| + | |||
| + | A bit complicated. | ||
| + | |||
| + | * Repo base URL baseurl=http:// | ||
| + | * [ ] beegfs-mgmtd-6.1-el6.x86_64.rpm | ||
| + | * '' | ||
| + | * beegfs-mgmtd.x86_64 | ||
| + | * '' | ||
| + | * http:// | ||
| + | |||
| + | |||
| + | So the wget/rpm approach (list all packages present on a particular node else you will get a dependencies failure!) | ||
| + | |||
| + | < | ||
| + | |||
| + | # get them all | ||
| + | wget http:// | ||
| + | |||
| + | # client and meta node | ||
| + | rpm -Uvh ./ | ||
| + | |||
| + | # updated? | ||
| + | [root@cottontail2 beegfs_6]# beegfs-ctl | head -2 | ||
| + | BeeGFS Command-Line Control Tool (http:// | ||
| + | Version: 6.1 | ||
| + | |||
| + | #Sheeesh | ||
| + | </ | ||
| + | |||
| ==== Resync Data #2 ==== | ==== Resync Data #2 ==== | ||
| Line 55: | Line 114: | ||
| # define mirrrogroups | # define mirrrogroups | ||
| - | [root@cottontail2 ~]# beegfs-ctl --addmirrorgroup --primary=21701 --secondary=13601 --groupid=1 | + | [root@cottontail2 ~]# beegfs-ctl --addmirrorgroup |
| - | [root@cottontail2 ~]# beegfs-ctl --addmirrorgroup --primary=13602 --secondary=21702 --groupid=2 | + | [root@cottontail2 ~]# beegfs-ctl --addmirrorgroup |
| [root@cottontail2 ~]# beegfs-ctl --listmirrorgroups | [root@cottontail2 ~]# beegfs-ctl --listmirrorgroups | ||
| Line 128: | Line 187: | ||
| | | ||
| | | ||
| + | |||
| + | # first rsync drops roughly 5G in both primaries which then get copied to secondaries. | ||
| + | # second rsync does the same so both storage servers loose 20G roughly | ||
| + | # now shut a storage server down and the whole filesystem can still be accessed (HA) | ||
| </ | </ | ||
| Line 323: | Line 386: | ||
| Content mirroring will require more disk space. Perhaps snapshots to another node is more useful, also solves backup issue. | Content mirroring will require more disk space. Perhaps snapshots to another node is more useful, also solves backup issue. | ||
| - | + | V6 does buddymirror meta mirroring [[http:// | |
| < | < | ||
| - | # enable meta mirroring, directory based | + | # 2015.03 |
| # change to 11/04/2016: used --createdir to make this home. | # change to 11/04/2016: used --createdir to make this home. | ||
| [root@n7 ~]# beegfs-ctl --mirrormd / | [root@n7 ~]# beegfs-ctl --mirrormd / | ||
| Line 332: | Line 395: | ||
| Mount: '/ | Mount: '/ | ||
| Operation succeeded. | Operation succeeded. | ||
| + | |||
| + | # V6.1 does it a root level not from a path | ||
| + | beegfs-ctl --addmirrorgroup --nodetype=meta --primary=38 --secondary=39 --groupid=1 | ||
| + | beegfs-ctl --addmirrorgroup --nodetype=meta --primary=250 --secondary=37 --groupid=2 | ||
| + | beegfs-ctl --mirrromd | ||
| # put some new content in | # put some new content in | ||
| Line 465: | Line 533: | ||
| * made easy [[http:// | * made easy [[http:// | ||
| * rpms pulled from repository via petaltail in '' | * rpms pulled from repository via petaltail in '' | ||
| + | * '' | ||
| + | * use '' | ||
| < | < | ||
cluster/151.1480436406.txt.gz · Last modified: 2016/11/29 16:20 by hmeij07
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
