cluster:208
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
cluster:208 [2021/10/18 18:55] hmeij07 [Changes] |
cluster:208 [2022/05/31 12:25] hmeij07 [gpu testing] |
||
|---|---|---|---|
| Line 383: | Line 383: | ||
| --- // | --- // | ||
| + | ===== gpu testing ===== | ||
| + | |||
| + | * test slurm v 21.08.1 | ||
| + | * n33-n37 each: 4 gpus, 16 cores, 16 threads, 32 cpus | ||
| + | * submit one at a time, observe | ||
| + | * part=test, n 1, B 1:1:1, cuda_visible=0, | ||
| + | * " | ||
| + | * all on same gpu | ||
| + | * part=test, n 1, B 1:1:1, cuda_visible not set, no node specified, n33 only | ||
| + | * " | ||
| + | * all gpus used? nope, all on the same one 0 | ||
| + | * redoing above with a '' | ||
| + | * even distribution across all gpus, 17th submit reason too | ||
| + | * part=test, n 1, B 1:1:1, cuda_visible not set, no node specified, n[33-34] avail | ||
| + | * while submitting 34 jobs, one at a time (30s delay), slurm fills up n33 first (all on gpu 0) | ||
| + | * 17th submit goes to n34, gpu 1 (weird), n33 state=alloc, | ||
| + | * 33th job, " | ||
| + | * 34th job, " | ||
| + | * all n33 and n34 jobs on single gpu without cuda_visible set | ||
| + | * how that works with gpu util at 100% with one jobs is beyond me | ||
| + | * do all 16 jobs log the same wall time? Yes, between 10.10 and 10.70 hours. | ||
| + | |||
| + | * ohpc slurm v | ||
| ===== Changes ===== | ===== Changes ===== | ||
| Line 430: | Line 453: | ||
| </ | </ | ||
| - | ** Weight | + | ** Partition |
| - | Weight nodes by the memory per logical core: jobs will be allocated the nodes with the lowest weight which satisfies their requirements. So CPU jobs will be routed last to gpu queues because they have the highest weight (=lowest priority). | + | If set you can list more than one queue... |
| + | < | ||
| + | srun --partition=exx96, | ||
| + | </ | ||
| + | |||
| + | The above will fill up n79 first, then n78, then n36... | ||
| + | |||
| + | ** Node Weight Priority ** | ||
| + | |||
| + | Weight nodes by the memory per logical core: jobs will be allocated the nodes with the lowest weight which satisfies their requirements. So CPU jobs will be routed last to gpu queues because they have the highest weight (=lowest priority). | ||
| < | < | ||
| hp12: 12/8 = 1.5 | hp12: 12/8 = 1.5 | ||
| Line 460: | Line 492: | ||
| Makes for a better 1-1 relationship of physical core to '' | Makes for a better 1-1 relationship of physical core to '' | ||
| - | Deployed. My need to set threads=1 and cpus=(quantity of physical cores) | + | Deployed. My need to set threads=1 and cpus=(quantity of physical cores)...this went horribly wrong it resaulted in sockets=1 setting and threads=1 for each node. |
| --- // | --- // | ||
| + | |||
| + | We did set number of cpus per gpu (12 for n79) and minimum memory settings. Now we experience 5th job pending with 48 cpus consumed. When using sbatch set -n 8 because sbatch will override defaults. | ||
| + | |||
| + | < | ||
| + | srun --partition=test | ||
| + | </ | ||
| \\ | \\ | ||
| **[[cluster: | **[[cluster: | ||
cluster/208.txt ยท Last modified: 2022/11/02 17:28 by hmeij07
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
