User Tools

Site Tools


cluster:147

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
cluster:147 [2020/01/24 13:37]
hmeij07
cluster:147 [2020/02/27 13:06] (current)
hmeij07
Line 14: Line 14:
   * Users Guide [[https://​upc-bugs.lbl.gov/​blcr/​doc/​html/​BLCR_Users_Guide.html]]   * Users Guide [[https://​upc-bugs.lbl.gov/​blcr/​doc/​html/​BLCR_Users_Guide.html]]
  
-When we move to Openlava 3.x all queues will support checkpointing,​ which means you can run your job in a "​wrapper"​ and if the job or cluster crashes you can restart your job from last checkpoint file.+All queues will support checkpointing,​ which means you can run your job in a "​wrapper"​ and if the job or cluster crashes you can restart your job from last checkpoint file.
  
-Checkpointing is an expensive operation so do not checkpoint under 6 hours. For example, if your job runs for a month checkpoint once a day, if your job runs for a week checkpoint every 12 hours. From this point on I expect all users to checkpoint. Some software does this internally (Amber, Gaussian). For applications or home grown code you can use BLCR. (Too bad it does not work out of box within Openlava).+Checkpointing is an expensive operation so do not checkpoint under 6 hours. For example, if your job runs for a month checkpoint once a day, if your job runs for a week checkpoint every 12 hours. From this point on I expect all users to checkpoint. Some software does this internally (Amber, Gaussian). For applications or home grown code you can use BLCR.
  
 You need to test out checkpointing before you rely on it. I've notice that some local code, when opening files for output, BLCR does not notice it. The code below has such an example (file fid.txt). Hopefully future versions of BLCR will fix this. Or maybe we should open files differently,​ this needs investigating further. You need to test out checkpointing before you rely on it. I've notice that some local code, when opening files for output, BLCR does not notice it. The code below has such an example (file fid.txt). Hopefully future versions of BLCR will fix this. Or maybe we should open files differently,​ this needs investigating further.
  
-BLCR, Berkely Lab Checkpoint and Restart, remembers file paths and process ids. The code stages the necessary STDOUT and STDERR files Openlava ​generates and invokes the relocation feature while ignore old process ids. If an application is large, and static, it is advisable to not save the application inside the checkpoint file.+BLCR, Berkely Lab Checkpoint and Restart, remembers file paths and process ids. The code stages the necessary STDOUT and STDERR files scheduler ​generates and invokes the relocation feature while ignore old process ids. If an application is large, and static, it is advisable to not save the application inside the checkpoint file.
  
 At the bottom of this page is the current version of ''​blcr_wrapper''​ program which will hide the complexity for you. ''​blcr_watcher''​ is a program that is in your PATH already and will terminate the wrapper if the application finishes inside of a check point time interval. I will work with any group interested to customize your ''​blcr_wrapper''​ for your lab/group. At the bottom of this page is the current version of ''​blcr_wrapper''​ program which will hide the complexity for you. ''​blcr_watcher''​ is a program that is in your PATH already and will terminate the wrapper if the application finishes inside of a check point time interval. I will work with any group interested to customize your ''​blcr_wrapper''​ for your lab/group.
cluster/147.txt ยท Last modified: 2020/02/27 13:06 by hmeij07