This is an old revision of the document!
A document for me to recall and make notes of what I read in the manual pages and what needs testing.
Basically during the Summer of 2016 I investigated if the HPCC could afford enterprise level storage. I wanted 99.999% uptime, snapshots, high availability and other goodies such as parallel NFS. Netapp came the closest but, eh, still at $42K lots of other options show up. The story is detailed here at The Storage Problem
This page is best read from the bottom up.
cottontail:/usr/local/bin/beegfs-admin-gui
Before
[root@cottontail2 ~]# beegfs-df METADATA SERVERS: TargetID Pool Total Free % ITotal IFree % ======== ==== ===== ==== = ====== ===== = 48 low 29.5GiB 23.3GiB 79% 1.9M 1.5M 82% 49 low 29.5GiB 23.1GiB 78% 1.9M 1.5M 82% 250 low 122.3GiB 116.7GiB 95% 7.8M 7.6M 98% STORAGE TARGETS: TargetID Pool Total Free % ITotal IFree % ======== ==== ===== ==== = ====== ===== = 13601 low 291.4GiB 50.6GiB 17% 18.5M 18.4M 100% 21701 low 291.2GiB 61.8GiB 21% 18.5M 15.8M 85%
Before
# define buddygroup - these are storage target IDs [root@n7 ~]# beegfs-ctl --addmirrorgroup --primary=13601 --secondary=21701 --groupid=101 Mirror buddy group successfully set: groupID 101 -> target IDs 13601, 21701 [root@n7 ~]# beegfs-ctl --listmirrorgroups BuddyGroupID PrimaryTargetID SecondaryTargetID ============ =============== ================= 101 13601 21701 # enable mirroring for data by directory -numTargets needs to be set to max nr of storage servers? [root@n7 ~]# beegfs-ctl --setpattern --buddymirror /mnt/beegfs/hmeij-mirror-data --chunksize=512k --numtargets=2 New chunksize: 524288 New number of storage targets: 2 Path: /hmeij-mirror-data Mount: /mnt/beegfs # copy some contents in (~hmeij is 10G) [root@n7 ~]# rsync -vac --bwlimit /home/hmeij /mnt/beegfs/hmeij-mirror-data/
After
[root@n7 ~]# beegfs-df METADATA SERVERS: (almost no changes...) STORAGE TARGETS: (each target less circa 10G) TargetID Pool Total Free % ITotal IFree % ======== ==== ===== ==== = ====== ===== = 13601 low 291.4GiB 40.7GiB 14% 18.5M 18.4M 99% 21701 low 291.2GiB 51.9GiB 18% 18.5M 15.8M 85% # lets find an object [root@n7 ~]# beegfs-ctl --getentryinfo /mnt/beegfs/hmeij-mirror-data/hmeij/xen/bvm1.img Path: /hmeij-mirror-data/hmeij/xen/bvm1.img Mount: /mnt/beegfs EntryID: 178-581797C8-30 Metadata node: n38 [ID: 48] Stripe pattern details: + Type: Buddy Mirror + Chunksize: 512K + Number of storage targets: desired: 2; actual: 1 + Storage mirror buddy groups: + 101 # original [root@n7 ~]# ls -lh /mnt/beegfs/hmeij-mirror-data/hmeij/xen/bvm1.img -rwxr-xr-x 1 hmeij its 4.9G 2014-04-07 13:39 /mnt/beegfs/hmeij-mirror-data/hmeij/xen/bvm1.img # copy on primary [root@swallowtail ~]# find /var/chroots/data/beegfs_storage -name 178-581797C8-30 /var/chroots/data/beegfs_storage/buddymir/u2018/5817/9/60-58179513-30/178-581797C8-30 [root@petaltail chroots]# ls -lh /var/chroots/data/beegfs_storage/buddymir/u2018/5817/9/60-58179513-30/178-581797C8-30 -rw-rw-rw- 1 root root 4.9G Apr 7 2014 /var/chroots/data/beegfs_storage/buddymir/u2018/5817/9/60-58179513-30/178-581797C8-30 # copy on secondary [root@swallowtail ~]# find /data/beegfs_storage -name 178-581797C8-30 /data/beegfs_storage/buddymir/u2018/5817/9/60-58179513-30/178-581797C8-30 [root@swallowtail ~]# ls -lh /data/beegfs_storage/buddymir/u2018/5817/9/60-58179513-30/178-581797C8-30 -rw-rw-rw- 1 root root 4.9G Apr 7 2014 /data/beegfs_storage/buddymir/u2018/5817/9/60-58179513-30/178-581797C8-30 # seems to work
Definitely want Meta content mirrored, that way I can use the n38-n45 nodes with local 15K disk, plus maybe cottontail2 (raid 1 with hot and cold spare).
Content mirroring will require more disk space. Perhaps snapshots to another node is more useful, also solves backup issue.
# enable meta mirroring, directory based [root@n7 ~]# beegfs-ctl --mirrormd /mnt/beegfs/hmeij-mirror Mount: '/mnt/beegfs'; Path: '/hmeij-mirror' Operation succeeded. # put some new content in [root@n7 ~]# rsync -vac /home/hmeij/iozone-tests /mnt/beegfs/hmeij-mirror/ # lookup meta tag [root@n7 ~]# beegfs-ctl --getentryinfo /mnt/beegfs/hmeij-mirror/iozone-tests/current.tar Path: /hmeij-mirror/iozone-tests/current.tar Mount: /mnt/beegfs EntryID: 3-581392E1-31 # find [root@sharptail ~]# ssh n38 find /data/beegfs_meta -name 3-581392E1-31 /data/beegfs_meta/mirror/49.dentries/54/6C/0-581392F0-30/#fSiDs#/3-581392E1-31 ^^^^^^ # and find [root@sharptail ~]# ssh n39 find /data/beegfs_meta -name 3-581392E1-31 /data/beegfs_meta/dentries/54/6C/0-581392F0-30/#fSiDs#/3-581392E1-31 # seems to work
Writing some initial content to both storage and meta servers; vanilla out of the box beegfs seems to balance the writes across both equally. Here are some stats.
Looks like:
# file content [root@swallowtail ~]# ls -lR /data/beegfs_storage/chunks/u0/57E4/2/169-57E42E75-31 /data/beegfs_storage/chunks/u0/57E4/2/169-57E42E75-31: total 672 -rw-rw-rw- 1 root root 289442 Jun 26 2015 D8-57E42E89-30 -rw-rw-rw- 1 root root 3854 Jun 26 2015 D9-57E42E89-30 -rw-rw-rw- 1 root root 16966 Jun 26 2015 DA-57E42E89-30 -rw-rw-rw- 1 root root 65779 Jun 26 2015 DB-57E42E89-30 -rw-rw-rw- 1 root root 20562 Jun 26 2015 DF-57E42E89-30 -rw-rw-rw- 1 root root 259271 Jun 26 2015 E0-57E42E89-30 -rw-rw-rw- 1 root root 372 Jun 26 2015 E1-57E42E89-30 [root@petaltail ~]# ls -lR /var/chroots/data/beegfs_storage/chunks/u0/57E4/2/169-57E42E75-31 /var/chroots/data/beegfs_storage/chunks/u0/57E4/2/169-57E42E75-31: total 144 -rw-rw-rw- 1 root root 40 Jun 26 2015 DC-57E42E89-30 -rw-rw-rw- 1 root root 40948 Jun 26 2015 DD-57E42E89-30 -rw-rw-rw- 1 root root 100077 Jun 26 2015 DE-57E42E89-30 # meta content [root@sharptail ~]# ssh n38 find /data/beegfs_meta -name 169-57E42E75-31 /data/beegfs_meta/inodes/6A/7E/169-57E42E75-31 /data/beegfs_meta/dentries/6A/7E/169-57E42E75-31 [root@sharptail ~]# ssh n39 find /data/beegfs_meta -name 169-57E42E75-31 (none, no mirror)
[root@n7 ~]# beegfs-ctl --getentryinfo /mnt/beegfs/ Path: Mount: /mnt/beegfs EntryID: root Metadata node: n38 [ID: 48] Stripe pattern details: + Type: RAID0 + Chunksize: 512K + Number of storage targets: desired: 4
greentail:/sanscratch/tmp/beegfs
[root@cottontail ~]# ssh n7 beegfs-net mgmt_nodes ============= cottontail [ID: 1] Connections: TCP: 1 (10.11.103.253:8008); meta_nodes ============= n38 [ID: 48] Connections: TCP: 1 (10.11.103.48:8005); n39 [ID: 49] Connections: TCP: 1 (10.11.103.49:8005); storage_nodes ============= swallowtail [ID: 136] Connections: TCP: 1 (192.168.1.136:8003 [fallback route]); petaltail [ID: 217] Connections: TCP: 1 (192.168.1.217:8003 [fallback route]);