DokuWiki

User Tools

Site Tools

Trace:
cluster:144

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
cluster:144 [2015/12/11 15:44]
hmeij [Warewulf Golden Image] 		cluster:144 [2018/07/26 18:52] (current)
hmeij07 [Deploying]
Line 10: Line 10:
   * Project Kusu, now defunct, but a great, simple template driven system. No fancy gui.   * Project Kusu, now defunct, but a great, simple template driven system. No fancy gui.
   * HP's CMU, also a great tool, golden image approach. The nice feature of CMU is that master node can delegate hundreds of node to be image by a designated compute node relieving the master node.   * HP's CMU, also a great tool, golden image approach. The nice feature of CMU is that master node can delegate hundreds of node to be image by a designated compute node relieving the master node.
-  * Bright Computing, a very complex tool that takes over every config file imaginable. Simple tasks become very burdensome, never achived traction with this too. +  * Bright Computing, a very complex tool that takes over every config file imaginable. Simple tasks become very burdensome, never achieved traction with this tool. 
-  * xCAT, the behemoth of open source provisioning tools. It does it all, which means a huge learning curve.  +  * xCAT, the behemoth of open source provisioning tools and more. It does it all, which means a huge learning curve.  
-  * [[http://warewulf.lbl.gov/trac]] Warewulf. I settled on this for the reasons that is simple and written in a language that's readable (Perl).+  * [[http://warewulf.lbl.gov/trac]] Warewulf. I settled on this for the reasons that it is simple and written in a language that's readable (Perl).
  
 The requirements of the provisioning tool were two fold: The requirements of the provisioning tool were two fold:
  
-  * My HPCC is flooded with small jobs that run for weeks to months (no wall time) but have small memory requirements (< 1GB). Thus I want to design stateless compute nodes and frequently tailor the setup to the scientific needs (non graphical).+  * My HPCC environment is flooded with small jobs that run for weeks to months (no wall time) but have small memory requirements (< 1GB). Thus I want to design stateless compute nodes, or virtual compute nodes, and frequently tailor the config & setup to the scientific needs (mostly non graphical, just CPU compute bound jobs, very little IO).
   * The HPCC also encounters very large jobs (for us that is 16-32 cores with memory requirements in the 256 GB range) utilizing X11, OpenGL, Nvidia and other large complex analyses software. In this case one compute node is build up to satisfaction, then we grab a "golden image" and deploy.   * The HPCC also encounters very large jobs (for us that is 16-32 cores with memory requirements in the 256 GB range) utilizing X11, OpenGL, Nvidia and other large complex analyses software. In this case one compute node is build up to satisfaction, then we grab a "golden image" and deploy.
  
-So I settled on WareWulf which does these two approaches and sports an active forum for questions.+So I settled on Warewulf which does these two approaches and sports an active forum for questions.
  
      
 +Not finding much on the "golden image" use of Warewulf, I've written up experiences. Hope it helps somebody. In this write we start with setting up stateless nodes, then we adept them to statefull using the "golden image" approach.
  
 +Install Warewulf and poke around the shell ''wwsh''. Consult the stateless page. Lets set up stateless node ''b6''. The files have been loaded in the MySQL database. The "ww" files are template driven files whose contents will be populated by warewulf, like changing IPs and HWADDRes. I keep them in my centos-6 stateless chroot but use them in other chroots as well (/var/chgroot/centos-6/root/wwtemplates/).
  
 <code> <code>
Line 38: Line 40:
 wwsh provision set b6 --fileadd hosts,bashrc,profile wwsh provision set b6 --fileadd hosts,bashrc,profile
 wwsh provision set b6 --fileadd network.ww,ifcfg-eth1.ww wwsh provision set b6 --fileadd network.ww,ifcfg-eth1.ww
 +
 +</code>
 +
 +As opposed to the stateless, which grabs it's OS content from the master node, in the "golden image" approach we are going to retrieve the content of a selected node (node b0 in this example) to the Warewul master (node petaltail in this case).
 +
 +Set ''rsync'' to work between the two. Unmount any NFS file systems on the node. Adjust the exclusion selections in template file (ie after /home is unmounted on node I do want the mount point...scratch space, etc).
 +
 +
 +
 +<code>
 +
 +# minder: all NFS file systems unmounted?
 +# or add rsync excludes in
 +# /usr/libexec/warewulf/wwmkchroot/golden-system.tmpl
 +
 +mkdir /var/chgroots/goldimages; cd /var/chgroots/goldimages
  
 SOURCEADDR=b0 wwmkchroot golden-system /var/chroots/goldimages/b0.chroot  SOURCEADDR=b0 wwmkchroot golden-system /var/chroots/goldimages/b0.chroot 
 +
 +</code>
 +
 +Next, modify the properties of the node to image.
 +[[http://warewulf.lbl.gov/trac/wiki/Recipes/StatefulProvisioning]]
 +
 +<code>
  
 wwsh object modify -s bootloader=sda b6 wwsh object modify -s bootloader=sda b6
-wwsh object modify -s diskformat=sda1,sda3 b6 +wwsh object modify -s diskformat=sda1,sda7 b6 
-wwsh object modify -s filesystems="mountpoint=/boot:dev=sda1:type=ext4:size=500,dev=sda3:type=swap:size=2048,mountpoint=/:dev=sda7:type=ext4:size=70000" b6+wwsh object modify -s filesystems= \  
 +"mountpoint=/boot:dev=sda1:type=ext4:size=500, \  
 +dev=sda3:type=swap:size=2048, \ 
 +mountpoint=/:dev=sda7:type=ext4:size=fill" \  
 +b6
  
 +# see note below on filesystems....
 +
 +</code>
 +
 +More on the ''filesystems'' option later. In this example I want to set up /, /boot and swap. Sizes in MB. Only one hard drive on this node (sda).
 +
 +Next we need to get the node booted and trasnfer the VNFS image made from the node b0 contents. At this time look on your master node in /var/lib/mysql and make sure you have enough disk space (these VNFS images will be around 1 GB as observed). Also, back up that database with ''mysqldump''.
 +
 +<code>
 +
 +# make the image, takes 10 minutes or so
 wwvnfs --chroot=/var/chroots/goldimages/b0.chroot  wwvnfs --chroot=/var/chroots/goldimages/b0.chroot 
 +
 +# switch node to image VNFS
 wwsh provision set b6 --vnfs=b0.chroot wwsh provision set b6 --vnfs=b0.chroot
  
 +# Rajil adds for GPU
 +# Keep in mind that nouveau should be disabled,
 +wwsh provision set c038 --kargs=\
 +"acpi_pad.disable=1 console=ttyS0,115200 rd.driver.blacklist=nouveau nouveau.modeset=0" 
 +
 +# just to be prudent
 wwsh pxe update wwsh pxe update
 wwsh dhcp update wwsh dhcp update
 service dhcpd restart service dhcpd restart
  
 +# check the configs
 wwsh object print b6 -p :all wwsh object print b6 -p :all
 wwsh provision list wwsh provision list
  
-PXE boot node, then turn provisioning off (or on UNDEF)+# next for provisioning (just to sure) on first PXE boot 
 +wwsh provision set --bootlocal=UNDEF b6 
 + 
 +# turn the node on 
 + 
 +</code> 
 + 
 +The console of the target node will now show the IP being assigned, the ''getvnfs'' process (be patient) and finally the root login. Check around the node for improvements, tweeks, then adjust and recreate the VNFS. 
 + 
 +After all that is done, disable provisioning so that the master ignores the PXE boot and the target node boots of local disk. 
 + 
 +<code> 
 + 
 +# ignore PXE boot
 wwsh provision set --bootlocal=EXIT b6 wwsh provision set --bootlocal=EXIT b6
  
 +</code>
 +
 +**filesystems**
 +
 +This is currently not working as expected. In my first attempts I'd specify sda1 (size=500), sda2 (size=2048, type=swap) and sda3 (size=fill) but what I end up with is a standard layout it looks like.  Any sizes are also ignored. So for now I just pick the ones I want (sda1, sda3, sda7).
 +
 +Note: this problem turns out to be hardware related, it is not appearing on newer hardware
 + --- //[[hmeij@wesleyan.edu|Henk]] 2017/09/13 08:11//
 +
 +
 +This also happens after I remove any UUID references in /etc/fstab, clean up /etc/mtab and clean any and all files in /dev/disk/by-uuid and rebuild VNFS.
 +
 +<code>
 +
 +fdisk -l
 +
 +Disk /dev/sda: 80.0 GB, 80026361856 bytes
 + 255 heads, 63 sectors/track, 9729 cylinders
 + Units = cylinders of 16065 * 512 = 8225280 bytes
 + Sector size (logical/physical): 512 bytes / 512 bytes
 + I/O size (minimum/optimal): 512 bytes / 512 bytes
 + Disk identifier: 0x000ce092
 +
 +    Device Boot      Start         End      Blocks   Id  System
 + /dev/sda1   *           1          13      104391   83  Linux
 + /dev/sda2              14        1543    12289725   83  Linux
 + /dev/sda3            1544        1798     2048287+  82  Linux swap / Solaris
 + /dev/sda4            1799        9729    63705757+   5  Extended
 + /dev/sda5            1799        2053     2048256   83  Linux
 + /dev/sda6            2054        2184     1052226   83  Linux
 + /dev/sda7            2185        9729    60605181   83  Linux
 +
 +df -h
 +
 + Filesystem      Size  Used Avail Use% Mounted on
 + /dev/sda7        57G  1.9G   53G   4% /
 + tmpfs            12G     0   12G   0% /dev/shm
 + /dev/sda1        95M   49M   42M  54% /boot
  
 </code> </code>
 +
 +Warewulf 3.6.99 and CentOS 6.5
 +
 +==== Part 2 ====
 +
 +To avoid the problems detailed above I started over with a fresh node.  First I installed CentOS 6.7 vanilla on the hard disk with partitions sda1 (/boot, 500 MB), sd2 (2048 MB, swap) and sda3 (/, size=fill). Since I had a new kernel now we need to make a new bootstrap image and provision it.
 +
 +<code>
 +
 +  wwbootstrap --chroot=/var/chroots/goldimages/n0.chroot 2.6.32-573.12.1.el6.x86_64
 +  wwsh provision set n22 --vnfs=n0.chroot --bootstrap=2.6.32-573.12.1.el6.x86_64 
 +
 +</code>
 +
 +Before we create the n0.chroot VNFS image edit the file and comment out these lines, with a golden image approach we do not have a hybridization path but we want these file systems included.
 +
 +<code>
 +
 +# /etc/warewul/vnfs.conf
 +
 +# comment out for golden image
 +#hybridize += /usr/X11R6
 +#hybridize += /usr/lib/locale
 +#hybridize += /usr/lib64/locale
 +#hybridize += /usr/include
 +
 +</code>
 +
 +I also cleaned the following files inside the chroot environment:
 +
 +<code>
 +
 +# /etc/fstab, edited (clean also mtab and remove files /dev/disk/by-uuid/*
 +tmpfs                   /dev/shm                tmpfs   defaults        0 0
 +devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
 +sysfs                   /sys                    sysfs   defaults        0 0
 +proc                    /proc                   proc    defaults        0 0
 +10.11.103.42:/home /home nfs tcp,soft,intr,bg
 +10.11.103.42:/sanscratch /sanscratch nfs tcp,soft,intr,bg
 +
 +# /boot/grub/device.map
 +# This file was written by Warewulf bootstrap (capability setup-filesystems)
 +(hd0) /dev/sda
 +
 +# /etc/grub.conf
 +# This file was written by Warewulf bootstrap (capability setup-filesystems)
 +serial --speed= --unit= --word= --parity=
 +terminal_input console serial; terminal_output console serial
 +default 0
 +timeout 10
 +root (hd0,0)
 +
 +title CentOS release 6.7 - 2.6.32-573.12.1.el6.x86_64
 +    kernel /vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=/dev/sda3 rhgb quiet
 +    initrd /initramfs-2.6.32-573.12.1.el6.x86_64.img
 +
 +</code>
 +
 +Now when first provisioning happens the right partitions are created and the node imaged. With bootload=EXIT or simply shutting down the warewulf master dhcpd process, the node is now booting from local disk.
 +
 +Yea.
 +
 +==== Deploying ====
 +
 +As part of my deployment I edited out any device information in file ''/etc/udev/rules.d/70-persistent-net.rules'', that information will be filled in on boot. Remade my VNFS. The reason for this is that in migrating from RHEL 5.5 to CentOS 6.7 I observed the order of the NICs to reverse themselves (but not always). Not sure why but will tell me what NIC eth0 will be.
 +
 +Then we build a template file with node specs in it like so:
 +
 +<code>
 +# HP blades reverses HWADDR (don't ask)
 + # use nic port bottom (no changes)
 + # set both ipaddr/hwaddr then PXEboot, wait...ww times out but images
 + # post edit
 + #   /etc/udev/rules.d fix eth? entries per below, then startup scripts
 + #   eth0 down CC 10.10.x.x onboot=no
 + #   eth1 up   CD 192.168.x.x onboot=yes
 + # reboot (with bootload=EXIT)
 + n1  1C:C1:DE:1C:82:D5 192.168.102.10 1C:C1:DE:1C:82:D4 10.10.102.10 10.11.103.10
 + n2  1C:C1:DE:1C:81:01 192.168.102.11 1C:C1:DE:1C:81:00 10.10.102.11 10.11.103.11
 +...
 +</code>
 +
 +My deploy script (give an entire line from above as arguments)
 +
 +<code>
 +
 +#!/bin/bash
 +
 + # deploy a n0.chroot node via PXE golden image transfer
 + # dynamic files are always in stateless CHROOT/centos-6/root/wwtemplates
 + node=$1
 + hwaddr0=$2
 + ipaddr0=$3
 + hwaddr1=$4
 + ipaddr1=$5
 + ipaddri=$6
 +
 + if [ $# != 6 ]; then
 +         echo "missing args: node hwaddr0 ipaddr0 hwaddr1 ipaddr1 ipaddri"
 +         exit
 + fi
 +
 + wwsh object delete $node -y 
 + sleep 3
 +
 + wwsh node new $node --netdev=eth0 \
 + --hwaddr=$hwaddr0 --ipaddr=$ipaddr0 \
 + --netmask=255.255.0.0  --network=255.255.0.0 -y
 +
 + wwsh node set $node --netdev=eth1 \
 + --hwaddr=$hwaddr1 --ipaddr=$ipaddr1 \
 + --netmask=255.255.0.0  --network=255.255.0.0 -y
 +
 + wwsh node set $node --netdev=ib0 \
 + --ipaddr=$ipaddri \
 + --netmask=255.255.0.0  --network=255.255.0.0 -y
 +
 + wwsh provision set $node --fileadd passwd,shadow,group -y
 + wwsh provision set $node --fileadd hosts,bashrc,profile -y
 + wwsh provision set $node --fileadd network.ww,ifcfg-eth1.ww,ifcfg-ib0.ww -y
 +
 +# note: no diskpartition, already exists and fails on this hardware,
 +# otherwise add diskpartion=sda so that "mkbootable" succeeds
 +
 + wwsh object modify -s bootloader=sda $node -y
 + wwsh object modify -s diskformat=sda1,sda4 $node -y
 +
 + if [ "$node" == 'n22' ]; then
 + # golden images with 3 partitions
 + wwsh object modify -s filesystems="mountpoint=/boot:dev=sda1:type=ext4:size=1024,dev=sda2:type=swap:size=6144,mountpoint=/:dev=sda3:type=ext4:size=+" $node -y
 + else
 + # hp blade 4 partitions
 + wwsh object modify -s filesystems="mountpoint=/boot:dev=sda1:type=ext4:size=1024,dev=sda2:type=swap:size=6144,mountpoint=/:dev=sda4:type=ext4:size=+" $node -y
 + fi
 +
 + wwsh provision set $node --vnfs=n0.chroot -y
 + wwsh provision set $node --bootstrap=2.6.32-573.12.1.el6.x86_64 -y
 +
 + wwsh provision set --bootlocal=UNDEF $node -y
 +
 + wwsh pxe update
 + wwsh dhcp update
 + service dhcpd restart
 + service httpd restart
 +
 + echo "now reboot: $node"
 +
 + echo "wwsh provision set --bootlocal=EXIT $node -y"
 +
 +
 +</code>
 +
 +
 \\ \\
 **[[cluster:0|Back]]** **[[cluster:0|Back]]**
cluster/144.1449848674.txt.gz ยท Last modified: 2015/12/11 15:44 by hmeij

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki