Differences

This shows you the differences between two versions of the page.

--- cluster:217 [2022/06/20 14:04] – hmeij07
+++ cluster:217 [2022/06/24 13:53] (current) – [Slurm #3] hmeij07
@@ Line 4: / Line 4: @@
 ==== Slurm entangles ====
-So, vaguely I remember when redoing my K20 gpu nodes I had troubles with that ASUS hardware and Warewulf 3.6. Now I have deployed a production cluster using OpenHPC 2.4, Rocky 8.5 and Warewulf 3.9 version. Same deal. Do not know what is going on but just documenting.
+So, vaguely I remember when redoing our K20 gpu nodes I had troubles with that ASUS hardware and Warewulf 3.6. Now I have deployed a production cluster using OpenHPC 2.4, Rocky 8.5 and Warewulf 3.9 version. Same deal. Do not know what is going on but just documenting.
 That's too bad as I was hoping to have a single operating system cluster. But now I will have to think about what to do with our CentOS 7 hardware which is running the old scheduler. The hope was to migrate everything to Slurm scheduler.
@@ Line 17: / Line 17: @@
   - Advanced > Network Stack – Enable
   - within that tab enable PXE4 and PXE6 support
-  - Boot -> boot order, network first then hard drive
+  - Boot > Boot order; network first then hard drive
   - Save & Exit > Yes & Reset
-Next we create the warewulf object and boot (see deploy script, at bottom).
+Next we create the warewulf node object and boot (see deploy script, at bottom).
 When this ASUS hardware boots, it sends over the correct mac address. We observe....
@@ Line 33: / Line 33: @@
 # in /etc/httpd/logs/access_log
-Jun  10 09:13:57 cottontail2 in.tftpd[388239]: Client ::ffff:192.168.102.100 finished /warewulf/ipxe/bin-i386-pcbios/undionly.kpxe
+Jun  10 09:13:57 cottontail2 in.tftpd[388239]: Client ::ffff:192.168.102.100 \
+finished /warewulf/ipxe/bin-i386-pcbios/undionly.kpxe
 </code>
@@ Line 58: / Line 59: @@
 ==== Slurm #1 ====
-First thought was to install OHPC v1.3 CentOS7 slurmd client on the node, then join that to OHPC v2.4 slurmctld. To do that first ''yum install'' the ohpc-release from this location
+First Idea:  install OHPC v1.3 CentOS7 slurmd client on the node, then join that to OHPC v2.4 slurmctld. To do that first ''yum install'' the ohpc-release from this location
   * http://repos.openhpc.community/ohpc-1.3/1.3.9/base/CentOS_7/x86_64/ohpc-release-1.3-1.el7.x86_64.rpm
@@ Line 82: / Line 83: @@
 Make sure munge/unmunge work between 1.3/2.4, that date is in sync (else you get error #16), and startup slurmd with 2.4 config files in place. This works but slurmd client of 1.3 fails to register. This appears to be an error in that the slurm versions are too far apart, 2018 vs 2020. Hmm, why is ophc v2.4 running such an old slurm version?
+Had to uncomment this for slurmd to start (seems ok because they are slurmctld settings not used by slurmd client...according to slurm list)
+<code>
+#SelectType=select/cons_tres
+#SelectTypeParameters=CR_CPU_Memory
+</code>
 <code>
@@ Line 104: / Line 114: @@
 ==== Slurm #2 ====
-I downloaded Slurm source the closest version just above ohpc v2.4 version. Next compile 20.11.9 slurm and see if it is accepted on ohpc v2.4 slurm 20.11.8
+Second Idea: download Slurm source the closest version just above ohpc v2.4 version. Next compile 20.11.9 slurm and see if it is accepted on ohpc v2.4 slurm 20.11.8 to register ....
 <code>
@@ Line 124: / Line 134: @@
 /usr/local/slurm-20.11.9/lib/slurm/auth_munge.so
-# make the generic /usr/local/slurm -> /usr/local/slurm-20.11.9 link
+#
 </code>
@@ Line 130: / Line 140: @@
 YES! it does register, hurray.
-Finish with startup at boot in ''/etc/rc.local'' and exports envs in ''/etc/bashrc''
+Finish with
+  * make generic /usr/local/slurm -> /usr/local/slurm-20.11.9 link
+  * copy over munge.key, restart munge
+  * startup at boot in ''/etc/rc.local''
+  * export envs in ''/etc/bashrc''
+  * make dirs /var/log/slurm /var/spool/slurm
 Do **NOT** mount ''/opt/intel'' and ''/opt/ohpc/pub'' from SMS, that's all Rocky8.5 stuff.
 ==== Slurm #3 ====
-Just a mental note. There is a warning on Slurm web page re the older versions archives page
+There is a warning on Slurm web page re the older versions archives page
   * https://www.schedmd.com/archives.php
-Due to a security vulnerability (CVE-2022-29500), all versions of Slurm prior to 21.08.8 or 20.11.9 are no longer available for download...so why is openhpc v2.4 running such an old slurm version?
+"Due to a security vulnerability (CVE-2022-29500), all versions of Slurm prior to 21.08.8 or 20.11.9 are no longer available for download"  ... so why is openhpc v2.4 running such an old slurm version?
+Third Idea: once we're fully deployed I may go to latest Slurm version, run on different ports with maybe newer munge version (although that should not matter, why does this scheduler even need munge?)
+Run Slurm outside of openhpc via local compile in ''/usr/local/''. A standalone version, the most recent version.
-Once we're fully deployed I may go to latest Slurm version, run on different ports with maybe newer munge version (although that should not matter, why does this scheduler even need munge?) and run Slurm outside of openhpc. A standalone version.
+Decided to go this route v22.05.02 standalone version (with ohpc v1.3 or v2.4 munge packages). You need all three packages (munge, munge-libs and munge-devel) on host where you compile slurm (note: cottontail2 for rocky8.5, node n90 for centos7). Then just copy.
+ --- //[[hmeij@wesleyan.edu|Henk]] 2022/06/23 19:07//
 ==== Deploy ====