User Tools

Site Tools


cluster:142

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
cluster:142 [2015/08/03 14:36]
hmeij
cluster:142 [2020/02/27 08:59] (current)
hmeij07
Line 1: Line 1:
 \\ \\
 **[[cluster:​0|Back]]** **[[cluster:​0|Back]]**
 + 
 ===== Scratch Spaces ===== ===== Scratch Spaces =====
  
-We have different ... blah, blah, to come+We have different ​locations for scratch spaceSome local to the nodes, some mounted across the networkHere is the current setup as of August 2019.
  
   * **/​localscratch**   * **/​localscratch**
-    * local to each node, off different sizes+    * Local to each node, different sizes roughly around 50-80 GB 
 +    * Warning: on nodes n46-n59 there is no hard disk but a SataDOM (usb device plugged directly into system board, 16 GB in size, holds just the OS). Do not use /​localscratch on these nodes.
  
   * **/​sanscratch** ​   * **/​sanscratch** ​
-    * two 5 TB file systems ​mounted IpoIB  using NFS+    * 55 TB file system ​mounted IpoIB using NFS or plain Ethernet 
 +      * greentail52 is the file server 
 +      * /​sanscratch/​username/​ can be used for staging (this is not backed up!) 
 +      * /​sanscratch/​checkpoints/​JOBPID is for checkpoint files (you need to create this in your job)
  
  
   * **/​localscratch5tb**   * **/​localscratch5tb**
 +    * 5 TB file system provided by local drives (3x2TB, Raid 0) on each node in the ''​mw256fd''​ queue
 +    * The list of nodes done: n38-n45, all are done (10sep15)
  
-48 TB of local scratch space will be available in 6 TB chunks on the nodes in the queue ''​mw256fd''​. That yields 5TB of local scratch space per node. Everybody may use this but it has specifically been put in place for the Guassian ​jobs yielding massive RWF files (application scratch files).+  * **/​localscratch** 
 +    * 2 TB file system on nodes in queue ''​mw128''​ (n60-n77) 
 + 
 +  * **/​localscratch** 
 +    * ~800GB file system on nodes in queue ''​exx96''​ (n79-n90) on SSD NVMe 
 + 
 + 
 +48 TB of local scratch space will be made available in 6 TB chunks on the nodes in the queue ''​mw256fd''​. That yields 5TB of local scratch space per node using Raid 0 and file type ''​ext4'',​ mounted at /​localscratch5tb. Everybody may use this but it has specifically been put in place for Gaussian ​jobs yielding massive RWF files (application scratch files)
 + 
 +**Note: Everybody is welcome to store content in ''/​localscratch5tb/​username/''​ for easy job access of large data files unless it interferes with jobs. However be warned that a) it's local storage, b) it's raid 0 (one disk failures and all data is lost), c) it's like /tmp read and write permission for all (do ''​chmod go-rwx /​localscratch5tb/​username''​ for some protection, and d) this file system is not backed up. In addition, ''/​sanscratch/​username/''​ will also be allowed.** 
 +  
 + 
 +You need to change your working directory to the location the scheduler has made for you. Also save your output before the job terminates, the scheduler will remove that working directory. Here is the workflow...
  
 <​code>​ <​code>​
 +
 #!/bin/bash #!/bin/bash
 # submit like so: bsub < run.forked # submit like so: bsub < run.forked
  
-# if writing large checkpoint files uncommnet ​next lines+# if writing large checkpoint files uncomment ​next lines
 #ionice -c 2 -n 7 -p $$ #ionice -c 2 -n 7 -p $$
 #ionice -p $$ #ionice -p $$
- 
-rm -rf err* out* output.* 
  
 #BSUB -q mw256fd #BSUB -q mw256fd
Line 40: Line 57:
 MYSANSCRATCH=/​sanscratch/​$LSB_JOBID MYSANSCRATCH=/​sanscratch/​$LSB_JOBID
 MYLOCALSCRATCH=/​localscratch/​$LSB_JOBID MYLOCALSCRATCH=/​localscratch/​$LSB_JOBID
-export MYSANSCRATCH MYLOCALSCRATCH+MYLOCALSCRATCH5TB=/​localscratch5tb/​$LSB_JOBID 
 +export MYSANSCRATCH MYLOCALSCRATCH ​MYLOCALSCRATCH5TB
  
-# cd to remote working ​dir +# cd to remote working ​directory 
-cd $MYSANSCRATCH+cd $MYLOCALSCRATCH5TB
 pwd pwd
  
 # environment # environment
-export GAUSS_SCRDIR="​$MYSANSCRATCH"+export GAUSS_SCRDIR="​$MYLOCALSCRATCH5TB"
  
 export g09root="/​share/​apps/​gaussian/​g09root"​ export g09root="/​share/​apps/​gaussian/​g09root"​
Line 55: Line 73:
 #. $gdvroot/​gdv/​bsd/​gdv.profile #. $gdvroot/​gdv/​bsd/​gdv.profile
  
-copy input data to fast disk+stage input data to localscratch5tb
 cp ~/​jobs/​forked/​gaussian.com . cp ~/​jobs/​forked/​gaussian.com .
 touch gaussian.log touch gaussian.log
Line 65: Line 83:
 #gdv < gaussian.com > gaussian.log #gdv < gaussian.com > gaussian.log
  
-# save results back to homedir+# save results back to homedir ​!!!
 cp gaussian.log ~/​jobs/​forked/​output.$LSB_JOBID cp gaussian.log ~/​jobs/​forked/​output.$LSB_JOBID
 +
 </​code>​ </​code>​
  
cluster/142.1438627000.txt.gz · Last modified: 2015/08/03 14:36 by hmeij