This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:151 [2016/10/20 18:13] hmeij07 |
cluster:151 [2016/11/22 18:46] hmeij07 [Resync Data #2] |
||
---|---|---|---|
Line 2: | Line 2: | ||
**[[cluster: | **[[cluster: | ||
- | ===== SGI Altix 3000 ===== | + | ===== beeGFS |
- | The HPCC community has been offered a SGI Altix 3000 (purchased in 2006), basically a half rack on wheels (20U or so). The Altix has 4 IA-64 processors (family Itanium 2) which aren't particularly fast (1.3 Ghz), but The Altix has 96 GBytes of memory and so is useful | + | A document |
- | * Details about IA-64 [[https:// | + | Basically during the Summer of 2016 I investigated if the HPCC could afford enterprise level storage. I wanted 99.999% uptime, snapshots, high availability and other goodies such as parallel NFS. Netapp came the closest but, eh, still at $42K lots of other options show up. The story is detailed here at [[cluster:149|The Storage Problem]] |
- | * Details about Altix [[https:// | + | |
- | It is running redhat AS2.1 (which definitely ages it), so basic Linux. The node has been configured to fit our environment and will provide | + | This page is best read from the bottom up. |
- | * /home from file server sharptail, over ethernet | + | ==== cluster idea ==== |
- | * /sanscratch from scratch server greentail, over ethernet | + | |
- | * Openlava 2.2 stand alone installation | + | |
- | * icc/ifort version 8.1 on local disk | + | |
- | * Gaussian version ?.?? on local disk | + | |
- | In order to use the local compilers you must " | + | * Storage servers: buy 2 now 4k+4k then 3rd in July 4k? |
+ | |||
+ | * move test users over on 2 nodes, test, only change is $HOME | ||
+ | |||
+ | * Home cluster | ||
+ | * cottontail (mngt+admingiu) | ||
+ | * 2-3 new units storage (+snapshots/ | ||
+ | * cottontail2 meta + n38-n45 meta, all mirrored | ||
+ | |||
+ | ==== beegfs-admin-gui ==== | ||
+ | |||
+ | * '' | ||
+ | |||
+ | ==== Resync Data #2 ==== | ||
+ | |||
+ | If you have 2 buddymirrors and 2 storage servers each with 2 storage objects, beegfs will write to all primary storage targets even if numtargets is to 1 ... it will use all storage objects so best to numtargets' | ||
+ | |||
+ | How does one add a server? | ||
< | < | ||
- | . | + | |
- | | + | # define storage objects, 2 per server |
+ | [root@petaltail ~]# /opt/beegfs/ | ||
+ | [root@petaltail ~]# / | ||
+ | [root@swallowtail data]# / | ||
+ | [root@swallowtail data]# / | ||
+ | |||
+ | |||
+ | [root@cottontail2 ~]# beegfs-df | ||
+ | METADATA SERVERS: | ||
+ | TargetID | ||
+ | ======== | ||
+ | | ||
+ | |||
+ | STORAGE TARGETS: | ||
+ | TargetID | ||
+ | ======== | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | |||
+ | # define mirrrogroups | ||
+ | [root@cottontail2 ~]# beegfs-ctl --addmirrorgroup --primary=21701 --secondary=13601 --groupid=1 | ||
+ | [root@cottontail2 ~]# beegfs-ctl --addmirrorgroup --primary=13602 --secondary=21702 --groupid=2 | ||
+ | |||
+ | [root@cottontail2 ~]# beegfs-ctl --listmirrorgroups | ||
+ | | ||
+ | | ||
+ | 1 | ||
+ | 2 | ||
+ | |||
+ | # define buddygroups, | ||
+ | [root@cottontail2 ~]# beegfs-ctl --setpattern --buddymirror | ||
+ | New chunksize: 524288 | ||
+ | New number of storage targets: 1 | ||
+ | Path: /home1 | ||
+ | Mount: / | ||
+ | |||
+ | [root@cottontail2 ~]# beegfs-ctl --setpattern --buddymirror / | ||
+ | New chunksize: 524288 | ||
+ | New number of storage targets: 1 | ||
+ | Path: /home2 | ||
+ | Mount: / | ||
+ | |||
+ | # drop /home/hmeij in / | ||
+ | [root@petaltail mysql_bak_ptt]# | ||
+ | 3623 | ||
+ | [root@petaltail mysql_bak_ptt]# | ||
+ | 3678 | ||
+ | [root@swallowtail data]# find / | ||
+ | 3623 | ||
+ | [root@swallowtail data]# find / | ||
+ | 3678 | ||
+ | |||
+ | # redefine with numtargets=2, | ||
+ | # resync - no results | ||
+ | # dropping hmeij/ into home2/ yields same amount of files | ||
+ | |||
+ | # rebuild test servers with from scratch with numparts=2 | ||
+ | # drop hmeij/ inot home1/ and slightly more files (couple of 100s), not double the amount | ||
+ | |||
+ | Path: / | ||
+ | Mount: / | ||
+ | EntryID: 1B-5834626F-FA | ||
+ | Metadata node: cottontail2 [ID: 250] | ||
+ | Stripe pattern details: | ||
+ | + Type: Buddy Mirror | ||
+ | + Chunksize: 512K | ||
+ | + Number of storage targets: desired: 2; actual: 2 | ||
+ | + Storage mirror buddy groups: | ||
+ | | ||
+ | + 1 | ||
+ | |||
+ | |||
+ | </ | ||
+ | |||
+ | ==== Resync Data #1 ==== | ||
+ | |||
+ | [[http://www.beegfs.com/ | ||
+ | |||
+ | //If the primary storage target of a buddy group is unreachable, | ||
+ | |||
+ | Testing out fail over and deletion of data on secondary then a full resync process: | ||
+ | |||
+ | |||
+ | * started a full --resyncstorage --mirrorgroupid=101 --timestamp=0 | ||
+ | * got --getentryinfo EntryID for a file in my /mnt/beegfs/ | ||
+ | * did a cat / | ||
+ | * brought primary storage down | ||
+ | * redid the cat above (it hangs for a couple of minutes, then displays the file content) | ||
+ | * while primary down, I ran rm -rf / | ||
+ | * a cat now generates the expected file not found error | ||
+ | * brought up primary and started a full --resyncstorage --mirrorgroupid=101 --timestamp=0 | ||
+ | * the nr of files and dirs discovered is as expected lower by the correct values | ||
+ | * when I now search for the EntryIDs obtained before they are gone from / | ||
+ | |||
+ | Nice that it works. | ||
+ | |||
+ | So you can full storage content mirror. You'll still need rsnapshots to recover lost data or point in time restores. | ||
+ | |||
+ | ==== Mirror Data ==== | ||
+ | |||
+ | When not all storage servers are up, client mounts will fail. This is just an optional " | ||
+ | |||
+ | In order to able able to take a storage server off line without any impact, all content needs to mirrored. | ||
+ | |||
+ | ** Before ** | ||
+ | |||
+ | < | ||
+ | [root@cottontail2 ~]# beegfs-df | ||
+ | METADATA SERVERS: | ||
+ | TargetID | ||
+ | ======== | ||
+ | 48 | ||
+ | 49 | ||
+ | | ||
+ | |||
+ | STORAGE TARGETS: | ||
+ | TargetID | ||
+ | ======== | ||
+ | | ||
+ | | ||
</ | </ | ||
- | You can also find MKL libraries at ''/ | + | ** Before ** |
- | All Openlava commands work the same way as elsewhere in our HPCC environment. In order to use the SGI Altix you must SSH to the head/compute node from any of our " | + | < |
+ | |||
+ | # define buddygroup - these are storage target IDs | ||
+ | [root@n7 ~]# beegfs-ctl --addmirrorgroup --primary=13601 --secondary=21701 --groupid=101 | ||
+ | Mirror buddy group successfully set: groupID 101 -> target IDs 13601, 21701 | ||
+ | |||
+ | [root@n7 ~]# beegfs-ctl --listmirrorgroups | ||
+ | | ||
+ | | ||
+ | 101 | ||
+ | |||
+ | # enable mirroring for data by directory -numTargets needs to be set to max nr of storage servers? | ||
+ | # changed to 11/02/2016: | ||
+ | [root@n7 ~]# beegfs-ctl --setpattern --buddymirror / | ||
+ | [root@n7 ~]# beegfs-ctl --setpattern --buddymirror / | ||
+ | New chunksize: 524288 | ||
+ | New number | ||
+ | Path: / | ||
+ | Mount: / | ||
+ | |||
+ | # copy some contents in (~hmeij is 10G) | ||
+ | [root@n7 ~]# rsync -vac --bwlimit /home/hmeij / | ||
+ | |||
+ | </ | ||
+ | |||
+ | ** After ** | ||
< | < | ||
- | [root@hmeij ~]# ssh hmeij@cottontail | + | [root@n7 ~]# beegfs-df |
- | hmeij@cottontail' | + | |
- | Last login: Thu Oct 20 09:38:40 2016 from 129.133.22.42 | + | |
- | [hmeij@cottontail ~]$ | + | |
- | # then | + | METADATA SERVERS: (almost no changes...) |
+ | STORAGE TARGETS: (each target less circa 10G) | ||
+ | TargetID | ||
+ | ======== | ||
+ | | ||
+ | | ||
- | [hmeij@cottontail | + | # lets find an object |
- | [hmeij@enzo hmeij]$ bqueues | + | [root@n7 ~]# beegfs-ctl --getentryinfo / |
- | QUEUE_NAME | + | Path: /hmeij-mirror-data/ |
- | sgi96 50 | + | Mount: / |
+ | EntryID: 178-581797C8-30 | ||
+ | Metadata node: n38 [ID: 48] | ||
+ | Stripe pattern details: | ||
+ | + Type: Buddy Mirror | ||
+ | + Chunksize: 512K | ||
+ | + Number of storage targets: desired: 2; actual: 1 | ||
+ | + Storage mirror buddy groups: | ||
+ | + 101 | ||
+ | |||
+ | # original | ||
+ | [root@n7 ~]# ls -lh /mnt/beegfs/hmeij-mirror-data/ | ||
+ | -rwxr-xr-x 1 hmeij its 4.9G 2014-04-07 13:39 / | ||
+ | |||
+ | # copy on primary | ||
+ | [root@petaltail chroots]# ls -lh / | ||
+ | -rw-rw-rw- 1 root root 4.9G Apr 7 2014 / | ||
+ | ^^^^^^^^ | ||
+ | |||
+ | # copy on secondary | ||
+ | [root@swallowtail ~]# find / | ||
+ | / | ||
+ | [root@swallowtail ~]# ls -lh / | ||
+ | -rw-rw-rw- | ||
+ | ^^^^^^^^ | ||
+ | |||
+ | # seems to work, notice the '' | ||
</ | </ | ||
- | ===== Gaussian ===== | + | Here is an important note, from community list: |
- | In order to run Gaussian jobs you must be a member of the unix group '' | + | * " |
+ | * so the important line that tells you that this file is mirrored is "Type: Buddy Mirror" | ||
+ | * " | ||
- | (Details | + | Another note: I changed paths for mirrormd and buddymirror to ''/ |
- | ===== Example Job ===== | + | < |
- | (Provide sample job submit script...) | + | [root@cottontail2 ~]# beegfs-net |
+ | meta_nodes | ||
+ | ============= | ||
+ | cottontail2 [ID: 250] | ||
+ | | ||
+ | [root@cottontail2 ~]# beegfs-ctl --listnodes --nodetype=meta --details | ||
+ | cottontail2 [ID: 250] | ||
+ | | ||
+ | | ||
+ | ^^^ | ||
+ | </ | ||
+ | |||
+ | ==== Quota ==== | ||
+ | |||
+ | * [[http:// | ||
+ | * setup XFS | ||
+ | * enable beegfs quota on all clients | ||
+ | * enforce quota | ||
+ | * set quotas using a text file | ||
+ | * seems straightforward | ||
+ | * do BEFORE populating XFS file systems | ||
+ | |||
+ | ==== Meta Backup/ | ||
+ | |||
+ | [[http:// | ||
+ | |||
+ | < | ||
+ | |||
+ | # latest tar | ||
+ | rpm -Uvh / | ||
+ | |||
+ | # backup | ||
+ | cd /data; tar czvf / | ||
+ | |||
+ | # restore | ||
+ | cd / | ||
+ | |||
+ | # test | ||
+ | cd /data; diff -r beegfs_meta beegfs_meta.orig | ||
+ | # no results | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | |||
+ | ==== Resync Meta ==== | ||
+ | |||
+ | [[http:// | ||
+ | |||
+ | * older versions | ||
+ | * new future version will work like storage mirror with HA and self-healing | ||
+ | ==== Mirror Meta ==== | ||
+ | |||
+ | //Metadata mirroring can currently not be disabled after it has been enabled for a certain directory// | ||
+ | |||
+ | Definitely want Meta content mirrored, that way I can use the n38-n45 nodes with local 15K disk, plus maybe cottontail2 (raid 1 with hot and cold spare). | ||
+ | |||
+ | Content mirroring will require more disk space. Perhaps snapshots to another node is more useful, also solves backup issue. | ||
+ | |||
+ | |||
+ | < | ||
+ | |||
+ | # enable meta mirroring, directory based | ||
+ | # change to 11/04/2016: used --createdir to make this home. | ||
+ | [root@n7 ~]# beegfs-ctl --mirrormd / | ||
+ | [root@n7 ~]# beegfs-ctl --mirrormd / | ||
+ | Mount: '/ | ||
+ | Operation succeeded. | ||
+ | |||
+ | # put some new content in | ||
+ | [root@n7 ~]# rsync -vac / | ||
+ | |||
+ | # lookup meta tag | ||
+ | [root@n7 ~]# beegfs-ctl --getentryinfo / | ||
+ | Path: / | ||
+ | Mount: /mnt/beegfs | ||
+ | EntryID: 3-581392E1-31 | ||
+ | |||
+ | # find | ||
+ | [root@sharptail ~]# ssh n38 find / | ||
+ | / | ||
+ | ^^^^^^ ^^ | ||
+ | # and find | ||
+ | [root@sharptail ~]# ssh n39 find / | ||
+ | / | ||
+ | |||
+ | # seems to work | ||
+ | |||
+ | </ | ||
+ | |||
+ | Writing some initial content to both storage and meta servers; vanilla out of the box beegfs seems to balance the writes across both equally. Here are some stats. | ||
+ | |||
+ | |||
+ | ==== / | ||
+ | |||
+ | * Source content 110G in XFS with ~100,000 files in ~2,000 dirs | ||
+ | * /home/hmeij (mix of files, nothing large) plus | ||
+ | * / | ||
+ | | ||
+ | * File content spread across 2 storage servers | ||
+ | * petaltail:/ | ||
+ | * swallowtail:/ | ||
+ | * 56G used in beegfs-storage per storage server | ||
+ | * ~92,400 files per storage server | ||
+ | * ~1,400 dirs per storage server | ||
+ | |||
+ | * Meta content spread across 2 meta servers (n37 and n38) | ||
+ | * 338MB per beegfs-meta server so 0.006% space wise for 2 servers | ||
+ | * ~105,000 files per metadata server | ||
+ | * ~35,000 dirs almost spread evenly across " | ||
+ | |||
+ | * Client (n7 and n8) see 110G in /mnt/beegfs | ||
+ | * 110G in /mnt/beegfs | ||
+ | * ~100,000 files | ||
+ | * ~2,000 dirs | ||
+ | |||
+ | Looks like: | ||
+ | |||
+ | * NOTE: failed to mount /mn/beegfs is the result of out of space storage servers. | ||
+ | |||
+ | < | ||
+ | |||
+ | # file content | ||
+ | |||
+ | [root@swallowtail ~]# ls -lR / | ||
+ | / | ||
+ | total 672 | ||
+ | -rw-rw-rw- 1 root root 289442 Jun 26 2015 D8-57E42E89-30 | ||
+ | -rw-rw-rw- 1 root root 3854 Jun 26 2015 D9-57E42E89-30 | ||
+ | -rw-rw-rw- 1 root root 16966 Jun 26 2015 DA-57E42E89-30 | ||
+ | -rw-rw-rw- 1 root root 65779 Jun 26 2015 DB-57E42E89-30 | ||
+ | -rw-rw-rw- 1 root root 20562 Jun 26 2015 DF-57E42E89-30 | ||
+ | -rw-rw-rw- 1 root root 259271 Jun 26 2015 E0-57E42E89-30 | ||
+ | -rw-rw-rw- 1 root root 372 Jun 26 2015 E1-57E42E89-30 | ||
+ | |||
+ | [root@petaltail ~]# ls -lR / | ||
+ | / | ||
+ | total 144 | ||
+ | -rw-rw-rw- 1 root root 40 Jun 26 2015 DC-57E42E89-30 | ||
+ | -rw-rw-rw- 1 root root 40948 Jun 26 2015 DD-57E42E89-30 | ||
+ | -rw-rw-rw- 1 root root 100077 Jun 26 2015 DE-57E42E89-30 | ||
+ | |||
+ | # meta content | ||
+ | |||
+ | [root@sharptail ~]# ssh n38 find / | ||
+ | / | ||
+ | / | ||
+ | |||
+ | [root@sharptail ~]# ssh n39 find / | ||
+ | (none, no mirror) | ||
+ | |||
+ | </ | ||
+ | ==== Tuning ==== | ||
+ | |||
+ | * global interfaces files ib0-> | ||
+ | * connInterfacesFile = / | ||
+ | * set in / | ||
+ | |||
+ | * backup/ | ||
+ | * see more towards top this page | ||
+ | |||
+ | * storage server tuning | ||
+ | * set on cottontail on sdb, both values were 128 (seems to help -- late summer 2016) | ||
+ | * echo 4096 > / | ||
+ | * echo 4096 > / | ||
+ | * set on cottontail, was 90112 + / | ||
+ | * echo 262144 > / | ||
+ | * do same on greentail? (done late fall 2016) | ||
+ | * all original values same as cottontail (all files) | ||
+ | * set on c1d1 thru c1d6 | ||
+ | * do same on sharptail? | ||
+ | * no such values for sdb1 | ||
+ | * can only find min_free_kbytes, | ||
+ | * stripe and chunk size | ||
+ | |||
+ | < | ||
+ | |||
+ | [root@n7 ~]# beegfs-ctl --getentryinfo / | ||
+ | Path: | ||
+ | Mount: /mnt/beegfs | ||
+ | EntryID: root | ||
+ | Metadata node: n38 [ID: 48] | ||
+ | Stripe pattern details: | ||
+ | + Type: RAID0 | ||
+ | + Chunksize: 512K | ||
+ | + Number of storage targets: desired: 4 | ||
+ | |||
+ | </ | ||
+ | * The cache type can be set in the client config file (/ | ||
+ | * buffered is default, few 100k per file | ||
+ | |||
+ | * tuneNumWorkers in all / | ||
+ | * for meta, storage and clients ... | ||
+ | |||
+ | * metadata server tuning | ||
+ | * read in more detail | ||
+ | |||
+ | ==== Installation ==== | ||
+ | |||
+ | * made easy [[http:// | ||
+ | * rpms pulled from repository via petaltail in '' | ||
+ | |||
+ | < | ||
+ | |||
+ | [root@cottontail ~]# ssh n7 beegfs-net | ||
+ | |||
+ | mgmt_nodes | ||
+ | ============= | ||
+ | cottontail [ID: 1] | ||
+ | | ||
+ | |||
+ | meta_nodes | ||
+ | ============= | ||
+ | n38 [ID: 48] | ||
+ | | ||
+ | n39 [ID: 49] | ||
+ | | ||
+ | |||
+ | storage_nodes | ||
+ | ============= | ||
+ | swallowtail [ID: 136] | ||
+ | | ||
+ | petaltail [ID: 217] | ||
+ | | ||
+ | |||
+ | |||
+ | </ | ||
\\ | \\ | ||
**[[cluster: | **[[cluster: |