DokuWiki

User Tools

Site Tools

Trace: • 175 • 224 • 134 • 179 • 124 • 153 • 210 • 214 • 172 • 155
cluster:155

This is an old revision of the document!


Back

OpenHPC page 2

Additional tools for the OpenHPC environment. First add these two lines to SMS and all compute nodes. Patch CHROOT as well.

  • /etc/security/limits.conf
# added for RLIMIT_MEMLOCK warnings with libibverbs -hmeij
*                soft    memlock         unlimited
*                hard    memlock         unlimited
  • Skipping SSH restrictions for users
    • Set up password less logins ssh-keygen -t rsa
    • cat ~/.ssh/idrsa.pub » ~/.ssh/authorized_keys
    • Collect all server fingerprints and make a global known_hosts file in ~/.ssh/
  • Skipping Luster installation
  • Nagios monitoring
 yum -y groupinstall ohpc-nagios
 yum -y --installroot=/data/ohpc/images/centos7.2 install nagios-plugins-all-ohpc nrpe-ohpc
 chroot /data/ohpc/images/centos7.2 systemctl enable nrpe
 perl -pi -e "s/^allowed_hosts=/# allowed_hosts=/"  /data/ohpc/images/centos7.2/etc/nagios/nrpe.cfg
 echo "nrpe 5666/tcp # NRPE" >> /data/ohpc/images/centos7.2/etc/services
 echo "nrpe : 192.168.1.249 : ALLOW" >> /data/ohpc/images/centos7.2/etc/hosts.allow
 echo "nrpe : ALL : DENY" >> /data/ohpc/images/centos7.2/etc/hosts.allow
 chroot /data/ohpc/images/centos7.2 /usr/sbin/useradd -c "NRPE user for the NRPE service" \
       -d /var/run/nrpe -r -g nrpe -s /sbin/nologin nrpe
 mv /etc/nagios/conf.d/services.cfg.example /etc/nagios/conf.d/services.cfg
 mv /etc/nagios/conf.d/hosts.cfg.example /etc/nagios/conf.d/hosts.cfg
 perl -pi -e "s/HOSTNAME1/n29/ || s/HOST1_IP/192.168.102.38/" /etc/nagios/conf.d/hosts.cfg
 perl -pi -e "s/HOSTNAME2/n31/ || s/HOST2_IP/192.168.102.40/" /etc/nagios/conf.d/hosts.cfg
 perl -pi -e "s/ \/bin\/mail/\/usr\/bin\/mailx/g" /etc/nagios/objects/commands.cfg
 perl -pi -e "s/nagios\@localhost/root\@ohpc0-test/" /etc/nagios/objects/contacts.cfg
 chkconfig nagios on
 systemctl start nagios
 chmod u+s `which ping`
 echo "relayhost = 192.168.102.42" >> /etc/postfix/main.cf
 echo "root:           hmeij@wes..." >> /etc/aliases
 newaliases
 systemctl restart postfix

# recreate vnfs and reimage nodes, see page1
 wwvnfs -y --chroot /data/ohpc/images/centos7.2
 /root/deploy.sh
  • Page1 OpenHPC
  • Reset nagiosadmin password htpasswd -c /etc/nagios/passwd nagiosadmin
  • Open port 80 in iptables but restrict severely (plain text passwords)
  • http://localhost/nagios
  • On to Ganglia
 yum -y groupinstall ohpc-ganglia
 yum -y --installroot=/data/ohpc/images/centos7.2 install ganglia-gmond-ohpc
# import passwd, shadow and group files for new user account ganglia
 mv /etc/ganglia/gmond.conf /etc/ganglia/gmond.conf-orig
 cp /opt/ohpc/pub/examples/ganglia/gmond.conf /etc/ganglia/
# use provision IP
 perl -pi -e "s/<sms>/192.168.1.249/" /etc/ganglia/gmond.conf
 cp /etc/ganglia/gmond.conf /data/ohpc/images/centos7.2/etc/ganglia/
 echo "gridname MySite" >> /etc/ganglia/gmetad.conf
 systemctl enable gmond
 systemctl enable gmetad
 systemctl start gmond
 systemctl start gmetad
 systemctl restart httpd

 chroot /data/ohpc/images/centos7.2 systemctl enable gmond

# recreate vnfs and reimage nodes, see page1
 wwvnfs -y --chroot /data/ohpc/images/centos7.2
 /root/deploy.sh
  • Not installing ClusterShell, pdsh is already installed
    • add compute hostnames to /etc/hosts.pdsh
    • echo export WCOLL=/etc/hosts.pdsh » /root/.bashrc
[root@ohpc0-test ~]# pdsh uptime
n31:  10:44:25 up 19:14,  1 user,  load average: 0.00, 0.01, 0.05
n29:  10:44:25 up 19:19,  0 users,  load average: 0.00, 0.01, 0.05
  • Skip mrsh installation
  • Skip Genders installation
  • Skip ConMan installation, ipmi serial consoles
  • Skip rsysslog forwarding of compute node logs to SMS
  • Redefine ControlMachine in /etc/slurm.slurm.conf
    • use eth0, not public address eth1
    • and CHROOT/etc/slurm/slurm.conf
    • import file back into database

Ran into a slurm config problem here on compue ndoes. When issuing systemctl status slurm the output revealed a failed start and it reported could not find /var/run/slurmctl.pid … that's the wrong pid process, compute nodes should only start slurmd. So finally got this fixed 

# ON COMPUTE NODES, that is in CHROOT

# Removed file /etc/init.d/slurm
mv /etc/init.d/slurm /root/ 

# Made the following link
[root@n31 ~]# ls -l /etc/systemd/system/multi-user.target.wants/slurmd.service 
lrwxrwxrwx 1 root root 38 Mar 30 14:05 /etc/systemd/system/multi-user.target.wants/slurmd.service -> /usr/lib/systemd/system/slurmd.service  

# now it starts properly
Mar 31 12:41:05 n31.localdomain systemd[1]: Starting Slurm node daemon...
Mar 31 12:41:05 n31.localdomain systemd[1]: PID file /var/run/slurmd.pid not readable (yet?) after start.
Mar 31 12:41:05 n31.localdomain systemd[1]: Started Slurm node daemon.
  • Recreate vnfs, Reimage the whole kaboodle

Link to my previous eval of Slurm and job throughput testing: Slurm. Next submit some test jobs, explained on that page too.

Here are my current settings on slurm.conf in OpenHPC. 

ClusterName=linux                                                                                        
ControlMachine=ohpc0-slurm                                                                               
ControlAddr=192.168.1.249                                                                                
SlurmUser=slurm                                                                                          
SlurmctldPort=6815-6817        <---                                                                          
SlurmdPort=6818                                                                                          
AuthType=auth/munge                                                                                      
StateSaveLocation=/etc/slurm/state    <---                                                                   
SlurmdSpoolDir=/etc/slurm/spool      <---
SwitchType=switch/none
MpiDefault=none
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
ProctrackType=proctrack/pgid
FirstJobId=101               <---
MaxJobCount=999999           <---
SlurmctldTimeout=300
SlurmdTimeout=300
InactiveLimit=0
MinJobAge=300
KillWait=30
Waittime=0
SchedulerType=sched/builtin <---
SchedulerPort=7321          <---
SelectType=select/linear
FastSchedule=1
SlurmctldDebug=3
SlurmdDebug=3
JobCompType=jobcomp/none
PropagateResourceLimitsExcept=MEMLOCK
SlurmdLogFile=/var/log/slurm.log
SlurmctldLogFile=/var/log/slurmctld.log
Epilog=/etc/slurm/slurm.epilog.clean
ReturnToService=1
NodeName=ohpc0-slurm NodeAddr=192.168.1.249 
NodeName=n29 NodeAddr=192.168.102.38
NodeName=n31 NodeAddr=192.168.102.40 
PartitionName=test Nodes=n29,n31 Default=YES MaxTime=INFINITE STATE=UP

Define CPUs, Cores, ThreadsPerCore, etc al later, run with Slurm self-discovered values for now.
Back

cluster/155.1491330722.txt.gz · Last modified: by hmeij07