Differences

This shows you the differences between two versions of the page.

--- cluster:151 [2016/10/20 14:05]
hmeij07 created
+++ cluster:151 [2016/10/31 13:57]
hmeij07
@@ Line 2: / Line 2: @@
 **[[cluster:0|Back]]**
-===== SGI Altix 3000 =====
+===== beeGFS =====
+A document for me to recall and make notes of what I read in the manual pages and what needs testing.
+Basically during the Summer of 2016 I investigated if the HPCC could afford enterprise level storage. I wanted 99.999% uptime, snapshots, high availability and other goodies such as parallel NFS. Netapp came the closest but, eh, still at $42K lots of other options show up. The story is detailed here at [[cluster:149|The Storage Problem]]
+This page is best read from the bottom up.
+==== cluster idea ====
+  * Storage servers: buy 2 now 4k+4k then 3rd in July 4k?
+  * move test users over on 2 nodes, test, only change is $HOME
+  * Home cluster
+    * cottontail (mngt+admingiu)
+    * 2-3 new units storage (+snapshots/meta backup)
+    * cottontail2 meta + n38-n45 meta, all mirrored
+==== Mirror Meta ====
+Definitely want Meta content mirrored, that way I can use the n38-n45 nodes with local 15K disk, plus maybe cottontail2 (raid 1 with hot and cold spare).
+Content mirroring will require more disk space. Perhaps snapshots to another node is more useful, also solves backup issue.
+<code>
+# enable meta mirroring, directory based
+[root@n7 ~]# beegfs-ctl --mirrormd /mnt/beegfs/hmeij-mirror
+Mount: '/mnt/beegfs'; Path: '/hmeij-mirror'
+Operation succeeded.
+# put some new content in
+[root@n7 ~]# rsync -vac /home/hmeij/iozone-tests /mnt/beegfs/hmeij-mirror/
+# lookup meta tag
+[root@n7 ~]# beegfs-ctl --getentryinfo /mnt/beegfs/hmeij-mirror/iozone-tests/current.tar
+Path: /hmeij-mirror/iozone-tests/current.tar
+Mount: /mnt/beegfs
+EntryID: 3-581392E1-31
+# find
+[root@sharptail ~]# ssh n38 find /data/beegfs_meta -name 3-581392E1-31
+/data/beegfs_meta/mirror/49.dentries/54/6C/0-581392F0-30/#fSiDs#/3-581392E1-31
+                  ^^^^^^
+# and find
+[root@sharptail ~]# ssh n39 find /data/beegfs_meta -name 3-581392E1-31
+/data/beegfs_meta/dentries/54/6C/0-581392F0-30/#fSiDs#/3-581392E1-31
+# seems to work
+</code>
+Writing some initial content to both storage and meta servers; vanilla out of the box beegfs seems to balance the writes across both equally. Here are some stats.
+==== /mnt/beegfs/ ====
+  * Source content 110G in XFS with ~100,000 files in ~2,000 dirs
+    * /home/hmeij (mix of files, nothing large) plus
+    * /home/fstarr/filler (lots of tiny files)
+  * File content spread across 2 storage servers
+    * petaltail:/var/chroot/data/beegfs_storage
+    * swallowtail:/data/beegfs_storage
+    * 56G used in beegfs-storage per storage server
+    * ~92,400 files per storage server
+    * ~1,400 dirs per storage server  mostly in "chunks" dir
+  * Meta content spread across 2 meta servers (n37 and n38)
+    * 338MB per beegfs-meta server so 0.006% space wise for 2 servers
+    * ~105,000 files per metadata server
+    * ~35,000 dirs almost spread evenly across "dentries" and "inodes"
+  * Client (n7 and n8) see 110G in /mnt/beegfs
+    * 110G in /mnt/beegfs
+    * ~100,000 files
+    * ~2,000 dirs
+Looks like:
+<code>
+# file content
+[root@swallowtail ~]# ls -lR /data/beegfs_storage/chunks/u0/57E4/2/169-57E42E75-31
+/data/beegfs_storage/chunks/u0/57E4/2/169-57E42E75-31:
+total 672
+-rw-rw-rw- 1 root root 289442 Jun 26  2015 D8-57E42E89-30
+-rw-rw-rw- 1 root root   3854 Jun 26  2015 D9-57E42E89-30
+-rw-rw-rw- 1 root root  16966 Jun 26  2015 DA-57E42E89-30
+-rw-rw-rw- 1 root root  65779 Jun 26  2015 DB-57E42E89-30
+-rw-rw-rw- 1 root root  20562 Jun 26  2015 DF-57E42E89-30
+-rw-rw-rw- 1 root root 259271 Jun 26  2015 E0-57E42E89-30
+-rw-rw-rw- 1 root root    372 Jun 26  2015 E1-57E42E89-30
+[root@petaltail ~]# ls -lR /var/chroots/data/beegfs_storage/chunks/u0/57E4/2/169-57E42E75-31
+/var/chroots/data/beegfs_storage/chunks/u0/57E4/2/169-57E42E75-31:
+total 144
+-rw-rw-rw- 1 root root     40 Jun 26  2015 DC-57E42E89-30
+-rw-rw-rw- 1 root root  40948 Jun 26  2015 DD-57E42E89-30
+-rw-rw-rw- 1 root root 100077 Jun 26  2015 DE-57E42E89-30
+# meta content
+[root@sharptail ~]# ssh n38 find /data/beegfs_meta -name 169-57E42E75-31
+/data/beegfs_meta/inodes/6A/7E/169-57E42E75-31
+/data/beegfs_meta/dentries/6A/7E/169-57E42E75-31
+[root@sharptail ~]# ssh n39 find /data/beegfs_meta -name 169-57E42E75-31
+(none, no mirror)
+</code>
+==== Tuning ====
+  * global interfaces files ib0->eth1->eth0
+    * connInterfacesFile = /home/tmp/global/beegfs.connInterfacesFile
+    * set in /etc/beegfs-[storage|client|meta|admon|mgmtd].conf and restart services
+  * backup beeGFS EA metadata, see faq
+    * attempt a restore
+    * or just snapshot
+  * storage server tuning
+    * set on cottontail on sdb, both values were 128  (seems to help -- late summer 2016)
+    * echo 4096 > /sys/block/sd?/queue/nr_requests
+    * echo 4096 > /sys/block/sd?/queue/read_ahead_kb
+    * set on cottontail, was 90112 + /etc/rc.local
+    * echo 262144 > /proc/sys/vm/min_free_kbytes
+  * do same on greentail? (done late fall 2016)
+    * all original values same as cottontail (all files)
+    * set on c1d1 thru c1d6
+  * do same on sharptail?
+    * no such values for sdb1
+    * can only find min_free_kbytes, same value as cottontail
+  * stripe and chunk size
+<code>
+[root@n7 ~]# beegfs-ctl --getentryinfo /mnt/beegfs/
+Path:
+Mount: /mnt/beegfs
+EntryID: root
+Metadata node: n38 [ID: 48]
+Stripe pattern details:
++ Type: RAID0
++ Chunksize: 512K
++ Number of storage targets: desired: 4
+</code>
+  * The cache type can be set in the client config file (/etc/beegfs/beegfs-client.conf).
+    * buffered is default, few 100k per file
+  * tuneNumWorkers in all /etc/beegfs/beggfs-C.conf file
+    * for meta, storage and clients ...
+  * metadata server tuning
+    * read in more detail
+==== Installation ====
+  * made easy [[http://www.beegfs.com/wiki/ManualInstallWalkThrough|External Link]]
+  * rpms pulled from repository via petaltail in ''greentail:/sanscratch/tmp/beegfs''
+<code>
+[root@cottontail ~]# ssh n7 beegfs-net
+mgmt_nodes
+=============
+cottontail [ID: 1]
+   Connections: TCP: 1 (10.11.103.253:8008);
+meta_nodes
+=============
+n38 [ID: 48]
+   Connections: TCP: 1 (10.11.103.48:8005);
+n39 [ID: 49]
+   Connections: TCP: 1 (10.11.103.49:8005);
+storage_nodes
+=============
+swallowtail [ID: 136]
+   Connections: TCP: 1 (192.168.1.136:8003 [fallback route]);
+petaltail [ID: 217]
+   Connections: TCP: 1 (192.168.1.217:8003 [fallback route]);
+</code>
 \\
 **[[cluster:0|Back]]**

DokuWiki

User Tools

Site Tools

Differences

Page Tools