User Tools

Site Tools



Warewulf Stateless

Get RPMs and install.

[root@petaltail ~]# wget -O /etc/yum.repos.d/warewulf-rhel6.repo
--2015-04-07 08:45:56--                                        
Connecting to||:80... connected.                                                                
HTTP request sent, awaiting response... 200 OK                                                                             
Length: 126 [text/plain]                                                                                                   
Saving to: â/etc/yum.repos.d/warewulf-rhel6.repoâ                                                                         

100%[======================================================================================>] 126         --.-K/s   in 0s      

2015-04-07 08:45:56 (22.9 MB/s) - â/etc/yum.repos.d/warewulf-rhel6.repoâsaved [126/126]

[root@petaltail ~]# yum install warewulf-common warewulf-cluster warewulf-provision 

  warewulf-cluster.x86_64 0:3.6-1.el6          warewulf-common.noarch 0:3.6-1.el6        warewulf-icr.x86_64 0:3.6-1.el6
  warewulf-provision.x86_64 0:3.6-1.el6

Dependency Installed:
  dhcp.x86_64 12:4.1.1-43.P1.el6.centos.1    tftp-server.x86_64 0:0.49-7.el6    warewulf-provision-server.x86_64 0:3.6-1.el6
  warewulf-vnfs.noarch 0:3.6-1.el6


Get and install MySQL and set mysql user root's password.

[root@petaltail warewulf]# vi /etc/warewulf/database-root.conf 
[root@petaltail warewulf]# service mysqld status

mysql> set password for 'root'@'localhost' = PASSWORD('some_string');
Query OK, 0 rows affected (0.00 sec)                                

[root@petaltail warewulf]# chmod o-r /etc/warewulf/database-root.conf                                  

In the provision config file I turned dynamic_hosts, hostfile and localdomain off (I'll manage those manually) and my private network is run over eth0 (

[root@petaltail warewulf]# vi /etc/warewulf/provision.conf 
[root@petaltail warewulf]# vi /etc/warewulf/provision.conf 

Next comes a piece of mystery. When executing wwinit ALL I ran into a lethal, no less, error. After much digging found the answer here and surprisingly found little in the documentation on this.

What it appears you'll need to do initially is build out a chroot
directory. For example, for a Scientific Linux 6.x install, you'd do:

    # wwmkchroot sl-6 /var/chroots/sl-6     (or whatever path you want
to store them at -- run wwmkchroot -h for the help)

  That will give the base chroot for the VNFS at /var/chroots/sl-6.
You'll then set the environment variable CHROOTDIR when you execute

    # CHROOTDIR=/var/chroots/sl-6 wwinit VNFS

  I know when you use the 'icr' wwinit functions, it creates a base
chroot based upon the host OS. But none of the other wwinit scripts,
by default, create one; So you will need to build the chroot and
specify where the directory is. Then when the VNFS part of wwinit is
ran, it will build the VNFS and import it into the datastore at that
time, setting the one specified as the default VNFS in the

So we start by making the chroot directories, first we'll build a generic centos-6. Then we initialize the warewulf environment.

[root@petaltail ~]# wwmkchroot centos-6 /var/chroots/centos-6

[root@petaltail ~]# CHROOTDIR=/var/chroots/centos-6  wwinit ALL
database:     Checking /etc/rc.d/init.d/mysqld is installed                  OK
database:     Confirming mysqld is configured to start at boot:
database:      + chkconfig mysqld on                                         OK
database:     Checking to see if MySQL needs to be started:
database:      + service mysqld start                                        OK
wwsh:         Confirming that wwsh accepts some basic commands
wwsh:          + wwsh quit                                                   OK
wwsh:          + wwsh help                                                   OK
wwsh:          + wwsh node new testnode0000                                  OK
wwsh:          + wwsh node list                                              OK
wwsh:          + wwsh node delete testnode0000                               OK
domain:       Setting default node domain to: "cluster"                      OK
authfiles:    Checking to see if /etc/passwd is in the WW Datastore          OK
authfiles:    Checking if /etc/passwd is part of default node configuration
authfiles:    Checking to see if /etc/group is in the WW Datastore           OK
authfiles:    Checking if /etc/group is part of default node configuration
nfsd:         Setting domain "cluster" for IDMAPD/NFSv4                      OK
nfsd:          + chkconfig nfs on                                            OK
nfsd:          + service nfs restart                                         OK
nfsd:          + exportfs -a                                                 OK
ntpd:         Configured NTP services
ntpd:          + chkconfig ntpd on                                           OK
ntpd:          + service ntpd restart                                        OK
ssh_keys:     Checking ssh keys for root                                     OK
ssh_keys:     Checking root's ssh config                                     OK
ssh_keys:     Checking for default RSA1 host key for nodes                   OK
ssh_keys:     Checking for default RSA host key for nodes                    OK
ssh_keys:     Checking for default DSA host key for nodes                    OK
tftp:          + /sbin/chkconfig xinetd on                                   OK
tftp:          + /sbin/chkconfig tftp on                                     OK
tftp:          + /sbin/service xinetd restart                                OK
bootstrap:    Creating bootstrap for 2.6.32-504.8.1.el6.x86_64:
bootstrap:     + wwbootstrap 2.6.32-504.8.1.el6.x86_64                       OK
vnfs:         Building the VNFS image and importing into Warewulf:
vnfs:          + wwvnfs -y --hybridpath=/var/chroots/centos-6 --chroot /var/ OK

Next restart MySQL, httpd and xinetd. Open your firewall on the private network.

  • edit /etc/sysconfig/iptables and restart iptables
# local allow
-A INPUT -i eth0 -d -p tcp --dport 0:65535 -j ACCEPT
-A INPUT -i eth0 -d -p udp --dport 0:65535 -j ACCEPT

Next restart some warewulf services. I had to edit the /etc/warewulf/dhcpd-template.conf several times and hardcoded the values in for network, netmask and ipaddr because I realized this server is my virtualization KVM test host and eth0 is bridged to br0 and warewulf is looking at the first ethernet and picks up eth1.

[root@petaltail ~]# wwsh dhcp update
Rebuilding the DHCP configuration

[root@petaltail ~]# wwsh pxe update
No nodes found

[root@petaltail ~]# wwsh
Warewulf> node new b51 --netdev=eth0 --hwaddr=00:00:00:00:00:00 --ipaddr= --groups=wwnodes
Warewulf> provision set --lookup groups wwnodes --vnfs=centos-6 --bootstrap=2.6.32-504.8.1.el6.x86_64
Are you sure you want to make the following changes to 1 node(s):

     SET: BOOTSTRAP            = 2.6.32-504.8.1.el6.x86_64
     SET: VNFS                 = centos-6

Yes/No> y
Warewulf> quit

Booting the node we observe the eth0 MAC address being picked up by DHCPd followed by the tftpboot process. The node boots, et voila, CentOS 6.6 stateless compute node. Next import files we want the same on all nodes and associate them with the nodes.

[root@petaltail ~]# wwsh file import /etc/passwd

[root@petaltail ~]# wwsh file list
dynamic_hosts           : -rw-r--r-- 0   root root            10174 /etc/hosts
group                   : -rw-r--r-- 1   root root             6247 /etc/group
passwd                  : -rw-r--r-- 1   root root            20094 /etc/passwd
shadow                  : ---------- 1   root root            23663 /etc/shadow

[root@petaltail ~]# wwsh provision set b[0-51] --fileadd passwd

[root@petaltail ~]# wwsh provision print
#### b51.cluster ##############################################################
    b51.cluster: BOOTSTRAP        = 2.6.32-504.8.1.el6.x86_64
    b51.cluster: VNFS             = centos-6
    b51.cluster: FILES            = dynamic_hosts,group,passwd,shadow
    b51.cluster: PRESHELL         = FALSE
    b51.cluster: POSTSHELL        = FALSE
    b51.cluster: CONSOLE          = UNDEF
    b51.cluster: PXELINUX         = UNDEF
    b51.cluster: SELINUX          = DISABLED
    b51.cluster: KARGS            = "quiet"
    b51.cluster: BOOTLOCAL        = FALSE

Next build a hybrid VNFS, this is the way to add packages to the nodes. For example NTP.

cd /var/chroots/centos-6/
mkdir vnfs

vi etc/fstab (inside of chroot area, edit) /vnfs nfs defaults 0 0

wwvnfs --chroot /var/chroots/centos-6 --hybridpath=/vnfs
Overwrite original: y

# reboot node, observe mount, then add NTP to chgroot

yum --tolerant --installroot /var/chroots/centos-6 -y install ntp

vi etc/init.d/ntpd (inside of chroot, edit)
# chkconfig: 35 58 74

# then make the following links in etc/rc[3|5].d/S58ntpd pointing to ../init.d/ntpd

vi etc/ntp.conf (inside of chroot, sole contents are, IP is master on private network)
restrict default ignore
restrict nomodify

wwvnfs --chroot /var/chroots/centos-6 --hybridpath=/vnfs -y
VNFS 'centos-6 has been imported

# reboot node

Then build the node up:

  • use yum to install the openlava 2.2.1 RPM (pulls in tcl)
    • copy the openlava config files into the centos-6 are
  • use yum to install postfix
    • add links in rc3.d and rc5.d
    • remove sendmail links
  • yum install perl
  • yum install munge
    • build RPMs from tar ball
  • rebuild the VNFS and reboot

Sometimes you can edit the files in the chroot directly, sometimes you must modify the installtoot directly. It's easiest to just do the latter all the time.

[root@petaltail ~]# cd /var/chroots
[root@petaltail chroots]# chroot centos-6
[root@petaltail /]# pwd                  
/                          <--- inside of installroot              
[root@petaltail /]# mkdir /var/log/munge 
[root@petaltail /]# chown munge:munge /var/log/munge
[root@petaltail /]# mkdir /var/log/slurm
[root@petaltail /]# chown slurm:munge /var/log/slurm
chown: invalid user: `slurm:munge'

# since the passwd|shadow|group files come from database you need to create the relevant lines

[root@petaltail /]# chown slurm:munge /var/log/slurm
[root@petaltail /]# exit
[root@petaltail chroots]# ls

# outside edit commenting out rc.local directives making these dirs etc
[root@petaltail chroots]# vi centos-6/etc/rc.local

# and don't forget
[root@petaltail chroots]# wwvnfs --chroot /var/chroots/centos-6 --hybridpath=/vnfs  -y

# reboot node

To build short hostnames you can create a template inside of the chroot environment.

#--- build file CHROOT/root/wwtemplates/network.ww
#--- end 

# add that file (using wwsh provision) to the nodes.

[root@]# wwsh file import /var/chroot/centos-6/root/wwtemplates/network.ww \
 --path=/etc/sysconfig/network --name=network.ww

[root@]# wwsh provision set n[00-15] --fileadd=network.ww

Second interface: create a template inside of the chroot environment.

wwsh node set b49 --netdev=eth1 \
--hwaddr=00:00:00:00:00:00 --ipaddr= \
--netmask=  --network=

#--- build file CHROOT/root/wwtemplates/ifcfg-eth1.ww
#--- end 

# add that file (using wwsh provision) to the nodes.

[root@]# wwsh file import /var/chroot/centos-6/root/wwtemplates/ifcfg-eth1.ww \
 --path=/etc/sysconfig/network-scripts/ifcfg-eth1 --name=ifcfg-eth1.ww

[root@]# wwsh provision set n[00-15] --fileadd=ifcfg-eth1.ww

Now, lets put it all together which can form the basis for a script.

# make sure it boots across network, alter BIOS settings

wwsh node new b6 --netdev=eth0 \
--hwaddr=00:00:00:00:00:00 --ipaddr= \
--netmask=  --network=

wwsh node set b6 --netdev=eth1 \
--hwaddr=00:00:00:00:00:00 --ipaddr= \
--netmask=  --network=

wwsh provision set b6 --fileadd passwd,shadow,group
wwsh provision set b6 --fileadd hosts,bashrc,profile
wwsh provision set b6 --fileadd network.ww,ifcfg-eth1.ww

Warewulf Golden Image

Useful links


  764  cd warewulf/
  765  wget
  766  wget
  767  wget
  768  wget
  769  wget
  771  yum groupinstall "Development Tools"
  773  man rpmbuild
  774  rpmbuild --help
  775  rpmbuild -ta warewulf-common-3.6.99.tar.gz 
  778  rpmbuild -ta warewulf-cluster-3.6.99.tar.gz 
  779  rpmbuild -ta warewulf-vnfs-3.6.99.tar.gz 
  782  yum install libselinux-devel
  784  yum install  libacl-devel  libattr-devel
  785  rpmbuild -ta warewulf-provision-3.6.99.tar.gz 

Warewulf Statefull Warewulf Golden Image

cluster/139.txt · Last modified: 2018/08/16 08:58 by hmeij07