cluster:139
Warewulf Stateless
- http://warewulf.lbl.gov/trac Warewulf is a scalable systems management suite originally developed to manage large high-performance Linux clusters. My Project Kusu replacement since IBM bought up Platform LSF and dished the hpccommunnity.org web site, grrrh. (old info at https://dokuwiki.wesleyan.edu/doku.php?id=cluster:88)
Get RPMs and install.
[root@petaltail ~]# wget -O /etc/yum.repos.d/warewulf-rhel6.repo http://warewulf.lbl.gov/downloads/repo/warewulf-rhel6.repo --2015-04-07 08:45:56-- http://warewulf.lbl.gov/downloads/repo/warewulf-rhel6.repo Resolving warewulf.lbl.gov... 128.3.7.27 Connecting to warewulf.lbl.gov|128.3.7.27|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 126 [text/plain] Saving to: â/etc/yum.repos.d/warewulf-rhel6.repoâ 100%[======================================================================================>] 126 --.-K/s in 0s 2015-04-07 08:45:56 (22.9 MB/s) - â/etc/yum.repos.d/warewulf-rhel6.repoâsaved [126/126] [root@petaltail ~]# yum install warewulf-common warewulf-cluster warewulf-provision Installed: warewulf-cluster.x86_64 0:3.6-1.el6 warewulf-common.noarch 0:3.6-1.el6 warewulf-icr.x86_64 0:3.6-1.el6 warewulf-provision.x86_64 0:3.6-1.el6 Dependency Installed: dhcp.x86_64 12:4.1.1-43.P1.el6.centos.1 tftp-server.x86_64 0:0.49-7.el6 warewulf-provision-server.x86_64 0:3.6-1.el6 warewulf-vnfs.noarch 0:3.6-1.el6 Complete!
Get and install MySQL and set mysql user root's password.
[root@petaltail warewulf]# vi /etc/warewulf/database-root.conf
[root@petaltail warewulf]# service mysqld status
mysql> set password for 'root'@'localhost' = PASSWORD('some_string');
Query OK, 0 rows affected (0.00 sec)
[root@petaltail warewulf]# chmod o-r /etc/warewulf/database-root.conf
In the provision config file I turned dynamic_hosts, hostfile and localdomain off (I'll manage those manually) and my private network is run over eth0 (192.168.0.0/255.255.0.0).
[root@petaltail warewulf]# vi /etc/warewulf/provision.conf [root@petaltail warewulf]# vi /etc/warewulf/provision.conf
Next comes a piece of mystery. When executing wwinit ALL I ran into a lethal, no less, error. After much digging found the answer here and surprisingly found little in the documentation on this.
- Lethal error thrown by module: /usr/libexec/warewulf/wwinit/91-vnfs.init
- quoting
What it appears you'll need to do initially is build out a chroot
directory. For example, for a Scientific Linux 6.x install, you'd do:
# wwmkchroot sl-6 /var/chroots/sl-6 (or whatever path you want
to store them at -- run wwmkchroot -h for the help)
That will give the base chroot for the VNFS at /var/chroots/sl-6.
You'll then set the environment variable CHROOTDIR when you execute
wwinit.
# CHROOTDIR=/var/chroots/sl-6 wwinit VNFS
I know when you use the 'icr' wwinit functions, it creates a base
chroot based upon the host OS. But none of the other wwinit scripts,
by default, create one; So you will need to build the chroot and
specify where the directory is. Then when the VNFS part of wwinit is
ran, it will build the VNFS and import it into the datastore at that
time, setting the one specified as the default VNFS in the
configuration.
So we start by making the chroot directories, first we'll build a generic centos-6. Then we initialize the warewulf environment.
[root@petaltail ~]# wwmkchroot centos-6 /var/chroots/centos-6 [root@petaltail ~]# CHROOTDIR=/var/chroots/centos-6 wwinit ALL database: Checking /etc/rc.d/init.d/mysqld is installed OK database: Confirming mysqld is configured to start at boot: database: + chkconfig mysqld on OK database: Checking to see if MySQL needs to be started: database: + service mysqld start OK wwsh: Confirming that wwsh accepts some basic commands wwsh: + wwsh quit OK wwsh: + wwsh help OK wwsh: + wwsh node new testnode0000 OK wwsh: + wwsh node list OK wwsh: + wwsh node delete testnode0000 OK domain: Setting default node domain to: "cluster" OK authfiles: Checking to see if /etc/passwd is in the WW Datastore OK authfiles: Checking if /etc/passwd is part of default node configuration authfiles: Checking to see if /etc/group is in the WW Datastore OK authfiles: Checking if /etc/group is part of default node configuration nfsd: Setting domain "cluster" for IDMAPD/NFSv4 OK nfsd: + chkconfig nfs on OK nfsd: + service nfs restart OK nfsd: + exportfs -a OK ntpd: Configured NTP services ntpd: + chkconfig ntpd on OK ntpd: + service ntpd restart OK ssh_keys: Checking ssh keys for root OK ssh_keys: Checking root's ssh config OK ssh_keys: Checking for default RSA1 host key for nodes OK ssh_keys: Checking for default RSA host key for nodes OK ssh_keys: Checking for default DSA host key for nodes OK tftp: + /sbin/chkconfig xinetd on OK tftp: + /sbin/chkconfig tftp on OK tftp: + /sbin/service xinetd restart OK bootstrap: Creating bootstrap for 2.6.32-504.8.1.el6.x86_64: bootstrap: + wwbootstrap 2.6.32-504.8.1.el6.x86_64 OK vnfs: Building the VNFS image and importing into Warewulf: vnfs: + wwvnfs -y --hybridpath=/var/chroots/centos-6 --chroot /var/ OK
Next restart MySQL, httpd and xinetd. Open your firewall on the private network.
- edit /etc/sysconfig/iptables and restart iptables
# local allow -A INPUT -i eth0 -d 192.168.0.0/16 -p tcp --dport 0:65535 -j ACCEPT -A INPUT -i eth0 -d 192.168.0.0/16 -p udp --dport 0:65535 -j ACCEPT
Next restart some warewulf services. I had to edit the /etc/warewulf/dhcpd-template.conf several times and hardcoded the values in for network, netmask and ipaddr because I realized this server is my virtualization KVM test host and eth0 is bridged to br0 and warewulf is looking at the first ethernet and picks up eth1.
[root@petaltail ~]# wwsh dhcp update
Rebuilding the DHCP configuration
Done.
[root@petaltail ~]# wwsh pxe update
No nodes found
[root@petaltail ~]# wwsh
Warewulf> node new b51 --netdev=eth0 --hwaddr=00:00:00:00:00:00 --ipaddr=192.168.1.5 --groups=wwnodes
Warewulf> provision set --lookup groups wwnodes --vnfs=centos-6 --bootstrap=2.6.32-504.8.1.el6.x86_64
Are you sure you want to make the following changes to 1 node(s):
SET: BOOTSTRAP = 2.6.32-504.8.1.el6.x86_64
SET: VNFS = centos-6
Yes/No> y
Warewulf> quit
Booting the node we observe the eth0 MAC address being picked up by DHCPd followed by the tftpboot process. The node boots, et voila, CentOS 6.6 stateless compute node. Next import files we want the same on all nodes and associate them with the nodes.
[root@petaltail ~]# wwsh file import /etc/passwd
[root@petaltail ~]# wwsh file list
dynamic_hosts : -rw-r--r-- 0 root root 10174 /etc/hosts
group : -rw-r--r-- 1 root root 6247 /etc/group
passwd : -rw-r--r-- 1 root root 20094 /etc/passwd
shadow : ---------- 1 root root 23663 /etc/shadow
[root@petaltail ~]# wwsh provision set b[0-51] --fileadd passwd
[root@petaltail ~]# wwsh provision print
#### b51.cluster ##############################################################
b51.cluster: BOOTSTRAP = 2.6.32-504.8.1.el6.x86_64
b51.cluster: VNFS = centos-6
b51.cluster: FILES = dynamic_hosts,group,passwd,shadow
b51.cluster: PRESHELL = FALSE
b51.cluster: POSTSHELL = FALSE
b51.cluster: CONSOLE = UNDEF
b51.cluster: PXELINUX = UNDEF
b51.cluster: SELINUX = DISABLED
b51.cluster: KARGS = "quiet"
b51.cluster: BOOTLOCAL = FALSE
...
Next build a hybrid VNFS, this is the way to add packages to the nodes. For example NTP.
cd /var/chroots/centos-6/ mkdir vnfs vi etc/fstab (inside of chroot area, edit) 192.168.1.217:/var/chroots/centos-6 /vnfs nfs defaults 0 0 wwvnfs --chroot /var/chroots/centos-6 --hybridpath=/vnfs Overwrite original: y # reboot node, observe mount, then add NTP to chgroot yum --tolerant --installroot /var/chroots/centos-6 -y install ntp vi etc/init.d/ntpd (inside of chroot, edit) # chkconfig: 35 58 74 # then make the following links in etc/rc[3|5].d/S58ntpd pointing to ../init.d/ntpd vi etc/ntp.conf (inside of chroot, sole contents are, IP is master on private network) restrict default ignore restrict 127.0.0.0 server 192.168.1.217 restrict 192.168.1.217 nomodify wwvnfs --chroot /var/chroots/centos-6 --hybridpath=/vnfs -y VNFS 'centos-6 has been imported # reboot node
Then build the node up:
- use yum to install the openlava 2.2.1 RPM (pulls in tcl)
- copy the openlava config files into the centos-6 are
- use yum to install postfix
- add links in rc3.d and rc5.d
- remove sendmail links
- yum install perl
- yum install munge
- build RPMs from tar ball
- rebuild the VNFS and reboot
Sometimes you can edit the files in the chroot directly, sometimes you must modify the installtoot directly. It's easiest to just do the latter all the time.
[root@petaltail ~]# cd /var/chroots [root@petaltail chroots]# chroot centos-6 [root@petaltail /]# pwd / <--- inside of installroot [root@petaltail /]# mkdir /var/log/munge [root@petaltail /]# chown munge:munge /var/log/munge [root@petaltail /]# mkdir /var/log/slurm [root@petaltail /]# chown slurm:munge /var/log/slurm chown: invalid user: `slurm:munge' # since the passwd|shadow|group files come from database you need to create the relevant lines [root@petaltail /]# chown slurm:munge /var/log/slurm [root@petaltail /]# exit exit [root@petaltail chroots]# ls centos-6 # outside edit commenting out rc.local directives making these dirs etc [root@petaltail chroots]# vi centos-6/etc/rc.local # and don't forget [root@petaltail chroots]# wwvnfs --chroot /var/chroots/centos-6 --hybridpath=/vnfs -y # reboot node
To build short hostnames you can create a template inside of the chroot environment.
#--- build file CHROOT/root/wwtemplates/network.ww
NETWORKING=yes
HOSTNAME=%{NODENAME}
#--- end
# add that file (using wwsh provision) to the nodes.
[root@]# wwsh file import /var/chroot/centos-6/root/wwtemplates/network.ww \
--path=/etc/sysconfig/network --name=network.ww
[root@]# wwsh provision set n[00-15] --fileadd=network.ww
Second interface: create a template inside of the chroot environment.
wwsh node set b49 --netdev=eth1 \
--hwaddr=00:00:00:00:00:00 --ipaddr=10.10.1.55 \
--netmask=255.255.0.0 --network=255.255.0.0
#--- build file CHROOT/root/wwtemplates/ifcfg-eth1.ww
DEVICE=eth1
BOOTPROTO=static
ONBOOT=yes
HWADDR=%{NETDEVS::ETH1::HWADDR}
IPADDR=%{NETDEVS::ETH1::IPADDR}
NETMASK=%{NETDEVS::ETH1::NETMASK}
NETWORK=%{NETDEVS::ETH1::NETWORK}
#--- end
# add that file (using wwsh provision) to the nodes.
[root@]# wwsh file import /var/chroot/centos-6/root/wwtemplates/ifcfg-eth1.ww \
--path=/etc/sysconfig/network-scripts/ifcfg-eth1 --name=ifcfg-eth1.ww
[root@]# wwsh provision set n[00-15] --fileadd=ifcfg-eth1.ww
Now, lets put it all together which can form the basis for a script.
# make sure it boots across network, alter BIOS settings wwsh node new b6 --netdev=eth0 \ --hwaddr=00:00:00:00:00:00 --ipaddr=192.168.1.12 \ --netmask=255.255.0.0 --network=255.255.0.0 --groups=wwnodes wwsh node set b6 --netdev=eth1 \ --hwaddr=00:00:00:00:00:00 --ipaddr=10.10.100.12 \ --netmask=255.255.0.0 --network=255.255.0.0 wwsh provision set b6 --fileadd passwd,shadow,group wwsh provision set b6 --fileadd hosts,bashrc,profile wwsh provision set b6 --fileadd network.ww,ifcfg-eth1.ww
Useful links
3.6.99
764 cd warewulf/ 765 wget http://warewulf.lbl.gov/downloadds/testing/3.6.99/warewulf-common-3.6.99.tar.gz 766 wget http://warewulf.lbl.gov/downloads/testing/3.6.99/warewulf-common-3.6.99.tar.gz 767 wget http://warewulf.lbl.gov/downloads/testing/3.6.99/warewulf-provision-3.6.99.tar.gz 768 wget http://warewulf.lbl.gov/downloads/testing/3.6.99/warewulf-cluster-3.6.99.tar.gz 769 wget http://warewulf.lbl.gov/downloads/testing/3.6.99/warewulf-vnfs-3.6.99.tar.gz 771 yum groupinstall "Development Tools" 773 man rpmbuild 774 rpmbuild --help 775 rpmbuild -ta warewulf-common-3.6.99.tar.gz 778 rpmbuild -ta warewulf-cluster-3.6.99.tar.gz 779 rpmbuild -ta warewulf-vnfs-3.6.99.tar.gz 782 yum install libselinux-devel 784 yum install libacl-devel libattr-devel 785 rpmbuild -ta warewulf-provision-3.6.99.tar.gz
cluster/139.txt · Last modified: 2018/08/16 12:58 by hmeij07
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
