This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:208 [2022/05/26 17:23] hmeij07 [Feedback] |
cluster:208 [2022/06/01 18:28] hmeij07 [gpu testing] |
||
---|---|---|---|
Line 385: | Line 385: | ||
===== gpu testing ===== | ===== gpu testing ===== | ||
- | * n33 only, free of jobs, 4 gpus, 16 cores, 16 threads, 32 cpus | + | |
+ | | ||
+ | * submit one at a time, observe | ||
+ | * part=test, n 1, B 1:1:1, cuda_visible=0, | ||
+ | * " | ||
+ | * all on same gpu | ||
+ | * part=test, n 1, B 1:1:1, cuda_visible not set, no node specified, n33 only | ||
+ | * " | ||
+ | * all gpus used? nope, all on the same one 0 | ||
+ | * redoing above with a '' | ||
+ | * even distribution across all gpus, 17th submit reason too | ||
+ | * part=test, n 1, B 1:1:1, cuda_visible not set, no node specified, n[33-34] avail | ||
+ | * while submitting 34 jobs, one at a time (30s delay), slurm fills up n33 first (all on gpu 0) | ||
+ | * 17th submit goes to n34, gpu 1 (weird), n33 state=alloc, | ||
+ | * 33th job, " | ||
+ | * 34th job, " | ||
+ | * all n33 and n34 jobs on single gpu without cuda_visible set | ||
+ | * how that works with gpu util at 100% with one jobs is beyond me | ||
+ | * do all 16 jobs log the same wall time? Yes, between 10.10 and 10.70 hours. | ||
- | * submit one at a time, observe where pmemd.cuda ends up | + | * ohpc v2.4 slurm v 20.11.8 |
+ | * part=test, n 1, B 1:1:1, cuda_visible=0, | ||
+ | * hit a bug, you must specify cpus-per-gpu **and** mem-per-gpu | ||
+ | * then slurm detects 4 gpus on allocated node and allows 4 jobs on a single allocated gpu | ||
+ | * twisted logic | ||
+ | * so recent openhpc version but old slurm version in software stack | ||
+ | * trying standalone install on openhpc prod cluster - auth/munge error, no go | ||
+ | * do all 4 jobs have similar wall time? Yes on n100 varies from 0.6 to 0.7 hours | ||
+ | |||
+ | * ohpc v2.4 slurm v 20.11.8 | ||
+ | * part=test, n 1, B 1:1:1, cuda_visible=0, | ||
+ | * same as above but all 16 jobs run on gpu 0 | ||
+ | * so the limit to 4 jobs on rtx5000 gpu is a hardware phenomenon? | ||
* | * | ||
+ | |||
+ | |||
+ | |||
===== Changes ===== | ===== Changes ===== | ||