We use Zenoss for monitor and alerting the whole HPC. Page can be found here Zenoss

At PEARC20 conference I became aware of Netdata which seems a good tool for our “tails” (login, storage servers for example). Lots of detailed information.

bash <(curl -Ss https://my-netdata.io/kickstart.sh)

Then open port 19999 in firewall for wesleyan.edu.

Gosh, it does alerting too. The original monitor scripts, unsilenced, exist on cottontail:/usr/lib/netadata/conf.d/health.d-orig

• hpcmon Zenoss server
• cottontail Primary log in and scheduler server
• cottontail2 Backup scheduler, centos 6 compile env
• greentail52 NFS server /sanscratch, centos 7 compile env
• ringtail NFS server /home33
• sharptail
• mstore0 NFS server /mindstore
• mindstoresrv1 Replication target for mstore0
• petaltail Sandbox, Warewulf centos 6
• swallowtail Sandbox
• whitetail Openhpc Warewulf centos 7 (powered down)
• sharptail2dr Disaster recovery host for hpcstore (active users) /homesdr

cd /usr/lib/netdata/conf.d/health.d/

for i in `ls`; do \
perl -pi -e "s/to: sysadmin/to: silent/g" $i; \
perl -pi -e "s/to: webmaster/to: silent/g" $i; \
perl -pi -e "s/to: dba/to: silent/g" $i; \
perl -pi -e "s/to: sitemgr/to: silent/g" $i; \
perl -pi -e "s/to: domainadmin/to: silent/g" $i; \
perl -pi -e "s/to: proxyadmin/to: silent/g" $i; \
done

grep -i to: * | grep -v silent

Back