This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
cluster:127 [2014/04/03 21:31] hmeij [Testing Xen] |
cluster:127 [2014/04/07 13:48] hmeij |
||
---|---|---|---|
Line 4: | Line 4: | ||
==== Virtual HPCC services ==== | ==== Virtual HPCC services ==== | ||
- | Thoughts on how to create virtual compute nodes in the HPCC stack. Specifically, | + | Thoughts on how to create virtual compute nodes in the HPCC stack. Specifically, |
==== Building Ganeti ==== | ==== Building Ganeti ==== | ||
Line 10: | Line 10: | ||
I'll select Ganeti to start with as it appears the simplest to setup. | I'll select Ganeti to start with as it appears the simplest to setup. | ||
- | On hold, these Xen tools are very nice, no need for Ganeti up front (virt-manager, | + | Update: |
==== Building Xen ==== | ==== Building Xen ==== | ||
- | To come. | + | < |
+ | |||
+ | yum install -y --nogpgcheck xen kernel-xen \ | ||
+ | virt-manager libvirt libvirt-python python-virtinst | ||
+ | |||
+ | chkconfig xend on | ||
+ | |||
+ | # disable selinux / | ||
+ | |||
+ | # add to xen kernel grub line some stuff | ||
+ | title CentOS (2.6.18-371.6.1.el5xen) | ||
+ | root (hd0,0) | ||
+ | kernel / | ||
+ | module / | ||
+ | module / | ||
+ | |||
+ | # hostname fully qualified (in case we go to Ganeti later) | ||
+ | |||
+ | # edit some settings in / | ||
+ | |||
+ | # turn off some services with chkconfig | ||
+ | # cups, iptables, ip6tables, autofs, crond, sysstat, bluetooth | ||
+ | # firstboot, ipmi, iscsi, iscsid | ||
+ | |||
+ | reboot | ||
+ | |||
+ | </ | ||
+ | |||
+ | * [[http:// | ||
+ | * [[http:// | ||
+ | * [[https:// | ||
+ | |||
+ | Then launch '' | ||
+ | |||
+ | Build a base clone and install what you need (like the Lava scheduler files). Give it a static IP via / | ||
+ | |||
+ | Now you build a script to clone and post prep new clones build from it. | ||
+ | |||
+ | < | ||
+ | |||
+ | #/ | ||
+ | |||
+ | # automate this | ||
+ | |||
+ | virt-clone --original vmdemo --name bvm7 --file / | ||
+ | # wait 3 mins | ||
+ | sleep 180 | ||
+ | |||
+ | virsh start bvm7 | ||
+ | # wait 2 mins | ||
+ | sleep 120 | ||
+ | |||
+ | # clobber vmdemo rc.local that sets up a static ip that you know rather than dhcp for automation | ||
+ | cd /root | ||
+ | scp rc.local vmdemo:/ | ||
+ | |||
+ | ssh vmdemo "cat / | ||
+ | ssh vmdemo "scp / | ||
+ | |||
+ | ssh vmdemo "cat / | ||
+ | ssh vmdemo "scp / | ||
+ | |||
+ | # this is the clone | ||
+ | ssh vmdemo reboot | ||
+ | # wait 2 mins | ||
+ | sleep 120 | ||
+ | |||
+ | </ | ||
==== Testing Xen ==== | ==== Testing Xen ==== | ||
- | ^3d Lennard-Jones melt: for 10,000 steps with 32,000 atoms^^^ | + | ^ 3d Lennard-Jones melt: for 10,000 steps with 32,000 atoms ^^^ |
^ Queue, node, HT ^ Jobs/Node, Loop Time ^ Comment | ^ Queue, node, HT ^ Jobs/Node, Loop Time ^ Comment | ||
| hp12, n15, no-HT | 01 jobs, 481 | | | | hp12, n15, no-HT | 01 jobs, 481 | | | ||
| hp12, n15, no-HT | 07 jobs, 482 | | | | hp12, n15, no-HT | 07 jobs, 482 | | | ||
| hp12, n2, yes-HT | | hp12, n2, yes-HT | ||
- | | hp12, n2, yes-HT | + | | hp12, n2, yes-HT |
- | | bss24, no-HT | 01 jobs, 844 | | + | | bss24, many, no-HT | 01 jobs, 844 | |
- | | bss24, no HT | + | | bss24vm, bvm1, VM |
- | | bss24vm, bvm1, no-HT | + | | bss24vm, bvm1, VM |
- | | bss24vm, bvm1, no-HT | + | | bss24vm, bvm1, VM |
- | | bss24vm, bvm1, no-HT | + | | bss24vm, bvm2, VM |
- | | bss24vm, bvm2, no-HT | + | | bss24vm, bvm1, VM | 32 jobs, 13661 | 32 vcpus, 1024 ram | |
+ | | bss24vm, bvm1, VM | 03 jobs, 1273 | 3 vcpus, 256 ram, make it 128 | | ||
+ | | bss24vm, bvm1, VM | 05 jobs, 2108 | 5 vcpus, 256 ram | | ||
+ | | bss24vm, bvm1, VM | 31 jobs, 13844 | 31 vcpus, 21504 ram | | ||
+ | ^ multiple vms ^^^ | ||
+ | | bss24vm, bvm1-4, VM | 4x01jobs, 1818 | 4x1vcpu, 4x128 ram | | ||
+ | | bss24vm, bvm1-8, VM | 8x01jobs, 3745 (1 hour) | 8x1vcpu, 8x128 ram | | ||
+ | | bss24vm, bvm1-6, VM | 6x32jobs, 82457 (22hrs) | ||
+ | ^ optimal physical to virtual cpu ratio for best performance according to Xen ^^^ | ||
+ | | bss24, many, no HT | 02 jobs, 826 | equivalent to hp12, yes-HT | ||
+ | | bss24vm, bvm2-3, VM | 2x02jobs, 1708 | 2x2vcpus, 2x128 ram, | | ||
+ | | bss24vm, bvm2-5, VM | 4x02jobs, 3497 | 4x2vcpus, 4x128 ram, optimal physical to virtual cpus ratio | | ||
+ | | bss24vm, bvm1-8, VM | 8x02jobs, XXXX | 8x2vcpus, 8x128 ram, optimal physical to virtual cpus ratio | | ||
+ | ==== Hmmm... ==== | ||
+ | We have some very old hardware here that does not support virtualization in the BIOS (hardware level). | ||
+ | The 6vms with 32vcpus finished work after 22 hours. That would imply one of these nodes could run say 20 VMs with 20x32=640 job slots. And we have 20 servers (12,800 job slots) that could be virtualized but there is no performance gain. Those servers will finish the 12,800 jobs in the same time not being virtualized. Too bad really. | ||
+ | Following that thought, we could virtualize (KVM at hardware level) one of the new Asus servers; at 256 gb of memory this would yield 8,192 job slots for one node 128 mb per vcpu). That might be an option. | ||
\\ | \\ |