This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:134 [2014/08/14 14:53] hmeij [Slurm] |
cluster:134 [2014/08/19 15:30] hmeij [High Throughput] |
||
---|---|---|---|
Line 34: | Line 34: | ||
echo " | echo " | ||
- | echo DONE | + | date |
</ | </ | ||
- | Slurm is installed on a PE2950 with dual quad cores and 16 GB. It is part of my high priority queue and allocated to Openlava (v2.2). | + | < |
- | My nodes are created in a virtual KVM environment | + | # I then submit it like so |
+ | for i in `seq 1 8`; do cat run | sed " | ||
+ | |||
+ | # and observe | ||
+ | sinfo | ||
+ | PARTITION AVAIL TIMELIMIT | ||
+ | test* up | ||
+ | test* up | ||
+ | test* up | ||
+ | |||
+ | # and when I raised the total number of jobs to 25,000 the distribution is | ||
+ | ^v1 ^v2 ^v3 ^v4 ^v5 ^v6 ^v7 ^v8 ^ | ||
+ | |3138|3130|3149|3133|3108|3119|3110|3113 | ||
+ | |||
+ | # time to process queues of different sizes (I stage them with the --begin parameter) | ||
+ | # jobs do have to open the output files though, just some crude testing of slurm | ||
+ | # scheduling prowness | ||
+ | |||
+ | ^NrJobs^1, | ||
+ | |mm:ss | 0:33| 6:32| 19:37| | ||
+ | |||
+ | # 20 mins for 25,000 jobs via sbatch | ||
+ | |||
+ | </ | ||
+ | |||
+ | Slurm is installed on a PE2950 with dual quad cores and 16 GB of memory. It is part of my high priority queue and allocated to Openlava (v2.2). | ||
+ | |||
+ | My slurm compute | ||
+ | |||
+ | From slurm.conf | ||
+ | |||
+ | < | ||
+ | |||
+ | # COMPUTE NODES | ||
+ | NodeName=v[1-8] NodeAddr=192.168.150.[1-8] CPUs=1 RealMemory=1 \ | ||
+ | Sockets=1 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN | ||
+ | NodeName=swallowtail NodeAddr=192.168.1.136 CPUs=8 RealMemory=16 \ | ||
+ | Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 State=UNKNOWN | ||
+ | |||
+ | PartitionName=test Nodes=v[1-8] Default=YES MaxTime=INFINITE State=UP | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | ==== IO error ==== | ||
+ | |||
+ | At around 32K jobs I ran into IO problems. | ||
+ | |||
+ | < | ||
+ | |||
+ | sbatch: error: Batch job submission failed: I/O error writing script/ | ||
+ | |||
+ | </ | ||
+ | |||
+ | Oh, this is OS error from ext3 file system, max files and dirs exceeded. | ||
+ | |||
+ | Switching " | ||
+ | |||
+ | |||
+ | ==== High Throughput ==== | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | * MaxJobCount=120000 | ||
+ | * SlurmctldPort=6820-6825 | ||
+ | * SchedulerParameters=max_job_bf=100, | ||
+ | |||
+ | ^NrJobs^N^hh: | ||
+ | |50, | ||
+ | |75, | ||
+ | |100, | ||
+ | |||
+ | The N=16 is 8 one core VMs and swallowtail itself (8 cores). So we're sxheduling bound. But 50,000 jobs per hour is grand! Certainly would exceed what we throw at it. And that's on old hardware. | ||
+ | |||
+ | |||
+ | < | ||
+ | WARNING: We will use a much slower algorithm with proctrack/ | ||
+ | Proctracktype=proctrack/ | ||
+ | </ | ||
+ | |||
+ | After fixing that. | ||
+ | |||
+ | |||
+ | ^NrJobs^N^hh: | ||
+ | |50, | ||
+ | |||
+ | |||
+ | Debug Level is 3. Maybe go to 1. | ||
\\ | \\ | ||
**[[cluster: | **[[cluster: |