Solution: TrueNAS, ZFS, 190T usable, RaidZ2 6 spares, read cache, 800G write cache, self healing, snapshots, compression on, deduplication off, encryption off, dual controllers (high availability), 64G ram, 6x 1Gbe RJ45, SAS drives (not SATA), three year warranty, ssh access
— Henk 2019/11/16 12:15
Bought 3 Juniper EX3300-48P switches to support LACP bonding of Ethernet ports for one public and two private subnets serving up /home in the future.
— Henk 2019/12/13 08:34
Looking for an appliance to replace my home directory server which is a Supermicro 4U storage box, using XFS and exporting /home via NFS v3 to compute nodes over a 1 Gbe Ethernet network (which seems fine for our usage, we leverage a lot of local scratch spaces).
We could continue that but our biggest problem is point-in-time restores (snapshots). Using rsync to create snapshots of /home roughly takes 3 days and 3 hours (23 millions files). Add to that our DR copy of active users and you have a backup window of 4.5 days.
FreeNAS would be the answer but I don't have time too build that from scratch. Trying to find an appliance in the $40K budget. Here is my “required” and “desired” functionality:
Appliances probably provide lots more functionality we will not be using (cloud sync, video, music, pics, printing, vmware, etc…). Domain/LDAP maybe…
“Snapshot+Replication” vs HyperBackup
Starting an HPC project to replace our home directory storage server which is now six years old. We have added several projects which bought their own storage and we'd like to consolidate it all onto a single platform and a grow-in-place solution for the future. The current storage footpring for /home would total 40T. Not knowing snapshot storage requirements we would like to start at 100T and grow-in-place to 300T usable.
We only use NFS mounts on a 1Gbe network (which seems fine for our work load) and maybe new purchases will add 10Gbe. Most data is staged on scratch spaces local to the compute nodes so we are phasing out Infiniband (IpoIB). No replication at the current time (may be added in the future). The old server will be expanded to a 100T backup target using rsync deamons.
We would like to have native snapshot functionality for point in time restores and checksumming on read or writes or both. Account management will remain on our login nodes and some automated pathway will be designed to add user accounts to the storage appliance (if needed).
We loosely are targeting a budget of $40K. Open to suggestions. Our sense is that BeeGFS is a little over the top for us and some form of FreeNAS appliance will be sufficient.