This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:151 [2016/10/20 14:31] hmeij07 [SGI Altix 3000] |
cluster:151 [2016/10/28 17:23] hmeij07 [Tuning] |
||
---|---|---|---|
Line 2: | Line 2: | ||
**[[cluster: | **[[cluster: | ||
- | ===== SGI Altix 3000 ===== | + | ===== beeGFS |
- | The HPCC community has been offered a SGI Altix 3000 (purchased in 2006), basically a half rack on wheels (20U or so). The Altix has 4 IA-64 processors (family Itanium 2)which aren't particularly fast (1.3 Ghz), but The Altix has 96 GBytes of memory and so is useful | + | A document |
- | * Details about IA-64 [[https:// | + | Basically during the Summer of 2016 I investigated if the HPCC could afford enterprise level storage. I wanted 99.999% uptime, snapshots, high availability and other goodies such as parallel NFS. Netapp came the closest but, eh, still at $42K lots of other options show up. The story is detailed here at [[cluster:149|The Storage Problem]] |
- | * Details about Altix [[https:// | + | |
- | It is running redhat AS2.1 (which definitely ages it) so basic Linux. The node has been configured to fit our environment and will provide | + | This page is best read from the bottom up. |
- | * /home from file server sharptail, over ethernet | + | ==== cluster idea ==== |
- | * /sanscratch from scratch server greentail, over ethernet | + | |
- | * Openlava 2.2 stand alone installation | + | |
- | * icc/ifort version 8.1 on local disk | + | |
- | * Gaussian version ?.?? on local disk | + | |
- | In order to use the local compilers you must " | + | idea: buy 2 now 4k+4k then 3rd in july 4k? |
+ | move test users over on 2 nodes, test, only change is $HOME | ||
+ | ctt (mngt+admingiu), | ||
- | < | + | make ctt2 master meta node? how? |
- | . / | + | |
- | . / | + | |
- | </ | + | |
- | You can also find MKL libraries at ''/ | ||
- | All Openlava commands work the same way as elsewhere in our HPCC environment. In order to use the SGI Altix you must SSH to the head/compute node from any of our " | + | ==== /mnt/beegfs/ ==== |
- | < | + | source 110G in XFS with ~100,000 files in ~2,000 dirs |
+ | / | ||
+ | |||
+ | | ||
+ | 56G in beegfs-storage per storage server | ||
+ | | ||
+ | | ||
- | [root@hmeij ~]# ssh hmeij@cottontail | + | meta spread across 2 meta servers |
- | hmeij@cottontail' | + | 338MB per beegfs-meta server so 0.006% space wise for 2 servers |
- | Last login: Thu Oct 20 09:38:40 2016 from 129.133.22.42 | + | ~105,000 files per metadata server |
- | [hmeij@cottontail | + | ~35,000 dirs almost spread evenly across " |
- | # then | + | |
+ | 110G in / | ||
+ | | ||
+ | | ||
- | [hmeij@cottontail | + | ==== Tuning ==== |
- | [hmeij@enzo hmeij]$ bqueues | + | |
- | QUEUE_NAME | + | * global interfaces files ib0-> |
- | sgi96 50 | + | * priority order, seems useful |
+ | * set in a file somewhere | ||
+ | |||
+ | * backup beeGFS EA metadata, see faq | ||
+ | * attempt a restore | ||
+ | * or just snapshot | ||
+ | |||
+ | * storage server tuning | ||
+ | * set on cottontail | ||
+ | * echo 4096 > / | ||
+ | * echo 4096 > /sys/block/sd?/ | ||
+ | * set on cottontail, was 90112 + / | ||
+ | * echo 262144 > / | ||
+ | * do same on greentail? (done late fall 2016) | ||
+ | * all original values same as cottontail (all files) | ||
+ | * set on c1d1 thru c1d6 | ||
+ | * do same on sharptail? | ||
+ | | ||
+ | * can only find min_free_kbytes, | ||
+ | * stripe and chunk size | ||
+ | |||
+ | < | ||
+ | |||
+ | [root@n7 ~]# beegfs-ctl --getentryinfo / | ||
+ | Path: | ||
+ | Mount: / | ||
+ | EntryID: root | ||
+ | Metadata node: n38 [ID: 48] | ||
+ | Stripe pattern details: | ||
+ | + Type: RAID0 | ||
+ | + Chunksize: 512K | ||
+ | + Number of storage targets: desired: | ||
</ | </ | ||
+ | * The cache type can be set in the client config file (/ | ||
+ | * buffered is default, few 100k per file | ||
+ | |||
+ | * tuneNumWorkers in all / | ||
+ | * for meta, storage and clients ... | ||
+ | |||
+ | * metadata server tuning | ||
+ | * read in more detail | ||
+ | ==== Installation ==== | ||
+ | * made easy [[http:// | ||
\\ | \\ | ||
**[[cluster: | **[[cluster: |