DokuWiki

Also read these pages and this page will make more sense: Warewulf Stateless, Warewulf Statefull.

For some time now I have been looking for a provisioning tool. I've tried along the way …

• Project Kusu, now defunct, but a great, simple template driven system. No fancy gui.
• HP's CMU, also a great tool, golden image approach. The nice feature of CMU is that master node can delegate hundreds of node to be image by a designated compute node relieving the master node.
• Bright Computing, a very complex tool that takes over every config file imaginable. Simple tasks become very burdensome, never achieved traction with this tool.
• xCAT, the behemoth of open source provisioning tools and more. It does it all, which means a huge learning curve.
• http://warewulf.lbl.gov/trac Warewulf. I settled on this for the reasons that it is simple and written in a language that's readable (Perl).

The requirements of the provisioning tool were two fold:

• My HPCC environment is flooded with small jobs that run for weeks to months (no wall time) but have small memory requirements (< 1GB). Thus I want to design stateless compute nodes, or virtual compute nodes, and frequently tailor the config & setup to the scientific needs (mostly non graphical, just CPU compute bound jobs, very little IO).
• The HPCC also encounters very large jobs (for us that is 16-32 cores with memory requirements in the 256 GB range) utilizing X11, OpenGL, Nvidia and other large complex analyses software. In this case one compute node is build up to satisfaction, then we grab a “golden image” and deploy.

So I settled on Warewulf which does these two approaches and sports an active forum for questions.

Not finding much on the “golden image” use of Warewulf, I've written up experiences. Hope it helps somebody. In this write we start with setting up stateless nodes, then we adept them to statefull using the “golden image” approach.

Install Warewulf and poke around the shell wwsh. Consult the stateless page. Lets set up stateless node b6. The files have been loaded in the MySQL database. The “ww” files are template driven files whose contents will be populated by warewulf, like changing IPs and HWADDRes. I keep them in my centos-6 stateless chroot but use them in other chroots as well (/var/chgroot/centos-6/root/wwtemplates/).

wwsh node new b6 --netdev=eth0 \
--hwaddr=00:00:00:00:00:00 --ipaddr=192.168.1.12 \
--netmask=255.255.0.0  --network=255.255.0.0
--groups=wwnodes

wwsh node set b6 --netdev=eth1 \
--hwaddr=00:00:00:00:00:00 --ipaddr=10.10.100.12 \
--netmask=255.255.0.0  --network=255.255.0.0

wwsh provision set b6 --fileadd passwd,shadow,group
wwsh provision set b6 --fileadd hosts,bashrc,profile
wwsh provision set b6 --fileadd network.ww,ifcfg-eth1.ww

As opposed to the stateless, which grabs it's OS content from the master node, in the “golden image” approach we are going to retrieve the content of a selected node (node b0 in this example) to the Warewul master (node petaltail in this case).

Set rsync to work between the two. Unmount any NFS file systems on the node. Adjust the exclusion selections in file (ie after /home is unmounted on node I do want the mountpoint…scratch space, etc).

• /usr/lib/libexec/warewulf/wwmkchroot/golden-ststem.tmpl

# minder: all NFS file systems unmounted?

mkdir /var/chgroots/goldimages; cd /var/chgroots/goldimages

SOURCEADDR=b0 wwmkchroot golden-system /var/chroots/goldimages/b0.chroot 

Next, modify the properties of the node to image. http://warewulf.lbl.gov/trac/wiki/Recipes/StatefulProvisioning

wwsh object modify -s bootloader=sda b6
wwsh object modify -s diskformat=sda1,sda3 b6
wwsh object modify -s filesystems= \ 
"mountpoint=/boot:dev=sda1:type=ext4:size=500, \ 
dev=sda3:type=swap:size=2048, \
mountpoint=/:dev=sda7:type=ext4:size=70000" \ 
b6

More on the filesystems option later. In this I want to set up /, /boot and swap. Sizes in MB. Only one hard drive on this node (sda).

Next we need to get the node booted and trasnfer the VNFS image made from the node b0 contents. At this time look on your master node in /var/lib/mysql and make sure you have enugh disk space (these VNFS images will be around 1 GB I observed). Also, back up that database with mysqldump.

# make the image, takes 10 minutes or so
wwvnfs --chroot=/var/chroots/goldimages/b0.chroot 

# switch node to image VNFS
wwsh provision set b6 --vnfs=b0.chroot

# just to be prudent
wwsh pxe update
wwsh dhcp update
service dhcpd restart

# check the configs
wwsh object print b6 -p :all
wwsh provision list

# next for provisioning (just to sure) on first PXE boot
wwsh provision set --bootlocal=UNDEF b6

# turn the node on

The console of the target node will now show the IP being assigned, the getvnfs process (be patient) and finally the root login. Check around the node for improvements, tweeks, then adjust and recreate the VNFS.

After all that is done, disable provisioning so that the master ignores the PXE boot and the target node botos of local disk.

# ignore PXE boot
wwsh provision set --bootlocal=EXIT b6

Back