Differences

This shows you the differences between two versions of the page.

--- cluster:134 [2014/08/14 14:53]
hmeij [Slurm]
+++ cluster:134 [2014/08/14 20:04]
hmeij [Slurm]
@@ Line 12: / Line 12: @@
       * copied the munge.key from head node to all compute nodes
     * slum installed from source code with
-      * \-\-prefix=/opt/slurm-14  \-\-sysconfdir=/opt/slurm-14/etc
+      * --prefix=/opt/slurm-14  --sysconfdir=/opt/slurm-14/etc
       * launched the configurator web page and set up a simple setup
         * created the openssl key and cert (see slurm web pages)
@@ Line 34: / Line 34: @@
 echo "$SLURMD_NODENAME JOB_PID=$SLURM_JOB_ID"
-echo DONE
+date
 </code>
-Slurm is installed on a PE2950 with dual quad cores and 16 GB. It is part of my high priority queue and allocated to Openlava (v2.2).
+<code>
+# I then submit it like so
+for i in `seq 1 8`; do cat run | sed "s/NUMBER/$i/g" | sbatch; done
+# and observe
+sinfo
+PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
+test*        up   infinite      2   comp v[2-3]
+test*        up   infinite      5  alloc v[1,4-5,7-8]
+test*        up   infinite      1   idle v6
+# and when I raised the total number of jobs to 25,000 the distribution is
+^v1  ^v2  ^v3  ^v4  ^v5  ^v6  ^v7  ^v8 ^
+|3138|3130|3149|3133|3108|3119|3110|3113
+# time to process queues of different sizes (I stage them with the --begin parameter)
+# jobs do have to open the output files though, just some crude testing of slurm
+# scheduling prowness
+^NrJobs^1,000^10,000^25,000^
+|mm:ss | 0:33|  6:32| 19:37|
+# 20 mins for 25,000 jobs via sbatch
+</code>
+Slurm is installed on a PE2950 with dual quad cores and 16 GB of memory. It is part of my high priority queue and allocated to Openlava (v2.2).
+My slurm compute nodes (v1-v8) are created in a virtual KVM environment on another PE2950 (2.6 Ghz, 16 GB ram) dual quad core with hyperthreading and virtualization turned on in the BIOS. Comments on how to build that KVM environment are here [[cluster:132|LXC Linux Containers]]. The nodes are single core, 1 GB or ram.
+From slurm.conf
+<code>
+# COMPUTE NODES
+NodeName=v[1-8] NodeAddr=192.168.150.[1-8] CPUs=1 RealMemory=1 \
+  Sockets=1 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN
+NodeName=swallowtail NodeAddr=192.168.1.136 CPUs=8 RealMemory=16 \
+  Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 State=UNKNOWN
+PartitionName=test Nodes=v[1-8] Default=YES MaxTime=INFINITE State=UP
+</code>
+==== IO error ====
+At around 32K jobs I ran into IO problems.
+<code>
+sbatch: error: Batch job submission failed: I/O error writing script/environment to file
+</code>
+Oh, this is OS error from ext3 file system, max files and dirs exceeded.
+Switching "StateSaveLocation=/sanscratch_sharptail/tmp/slurmctld" to XFS file system.
+==== High Throughput ====
+[[https://computing.llnl.gov/linux/slurm/high_throughput.html]]
+  * MaxJobCount=100000
+  * SlurmctldPort=6820-6825
+  * SchedulerParameters=max_job_bf=100,interval=30  (NOTE:  unset these, pages states default values are fine)
+ WARNING: We will use a much slower algorithm with proctrack/pgid, use Proctracktype=proctrack/linuxproc or some other proctrack when using jobacct_gather/linux
-My nodes are created in a virtual KVM environment also on a PE2950 (2.6 Ghz, 16 GB ram) dual quad core with hyperthreading and virtualization turned on in the BIOS. Comments on how to build that KVM environment are here [[cluster:132|LXC Linux Containers]]
+Debug Level is 3. Maybe go to 1.
 \\
 **[[cluster:0|Back]]**

DokuWiki

User Tools

Site Tools

Differences

Page Tools