User Tools

Site Tools


cluster:189

This is an old revision of the document!



Back

Structure and History of HPCC

History

In 2006, 4 Wesleyan qfaculty members approached ITS with a proposal to centrally manage a whigh performance computing center (HPCC) seeding the effort with an NSF grant (about $190K). ITS offered 0.5 FTE for a dedicated “hpcadmin”. An Advisory Group was formed by these faculty plus hpcadmin (5 members, not necessarily our current “power users”). Another NSF grant reward was added in 2010 (about $105K). An alumni donation followed in 2016 (about $10K). In 2018 the first instance of “faculty startup monies” was contribute to the HPCC (about $92K, see “Priority Policy” below. In 2019, a TrueNAS/ZFS appliance was purchased (about $40K) followed in 2020 by a GPU expansion project (about $96K). The latter two were self-funded expenditures, see “Funding Policy” below. To view the NSF grants visit Acknowledgement

The Advisory Group meets with the user base yearly in reading week of the Spring semester (early May) before everybody scatters for the summer. At this meeting the hpcadmin reviews the past year, previews the coming year and users are contributing feedback on progress and problems.

Structure

The Wesleyan HPCC is part of the Scientific Computing and Informatics Center ( SCIC ). The SCIC project leader is appointed by the Director of the Quantitative Analysis Center QAC . The Director of the QAC reports to the Associate Provost. The hpcadmin has a direct report with ITS Deputy Director and an indirect report with QAC Director.

The QAC has an Apprenticeship Program in which students are trained in Linux and several program languagues of their choice and other options. From this pool of students some become the QAC and SCIC helpdesk and tutors.

Funding Policy

After an 8 year run of the HPCC, and a drying up of grant opportunities at NSF, it was decided to explore self-funding so the HPCC effort could continue without external dependence on funds. A report was made of the HPCC progress including topics such as Publications, Citations, Honors Theses, Growth in Jobs Submitted, Pattern of Pending Jobs, and General Inventory. The report summary can be viewed at this page Provost Report . This report was discussed between Provost and HPC Advisory Group.

Several months later a pattern emerged. The Provost would annually contribute $25K if the HPC user base raised $15K annually. That would amount to $160K in 4 years enough for a hardware refresh or new hardware acquisition. Finances also contributed $10K for maintenance such as failed disks, network switches, etc, but these funds do not “roll over”. Use it or loose it. All funds start July 1st.

In order for the HPC user base to raise $15K annually, CPU and GPU hourly usage was deployed. A dictionary is maintained listing PIs and their members (students majors, lab students, grads, phd candidates, collaborators, etc). Each PI then quarterly contributes to the user fund based on a scheme yieldingq $15K annually.

Here is 2019's queue usage 2019 Queue Usage and 2019 contribution scheme.

Contribution Scheme for 01 July 2019 onwards
Hours (K) - Rate ($/CPU Hour)

  • 0-5 = Free
  • >5-25 = 0.03
  • >25-125 = 0.006
  • >125-625 = 0.0012
  • >625-3125 = 0.00024
  • >3125 = 0.000048

A cpu usage of 3,125,000 hours/year would cost $ 2,400.00
A gpu hour of usage is 3x the cpu hourly rate.

We currently have about 1,450 physical cpu cores, 60 gpus, 520 gb of gpu memory and 8,560 gb cpu memory provided by about 120 compute nodes and login nodes. Scratch spaces are provide local to compute nodes (2-5 tb) or over the network via NFS (55 tb). Home directories are under quota (10 tb) but these will disappear in the future with the TrueNAS/ZFS appliance (190 tb, 475 tb effective assuming a compression rate of 2.5x). a guide can be found here Brief Guide to HPCC and the software is located here Software Page

Priority Policy

This policy was put in place about 3 years ago to deal with the issues surrounding new monies infusions from for example new faculty “startup monies”, new grant monies, or donations to the HPCC.

There are few Principles in this Priority Access Policy

  1. Contributions, of any kind and from any source, immediately becomes a community wide resource.
  2. Priority access is granted for 3 years starting at the date of deployment (user access).
  3. Only applies to newly purchased resources which should be under warranty in the priority period.

The main objective is to build an HPCC community resource for all users with no (permanent) special treatment of any subgroup.

The first principle implies that all users have access to the new resources immidiately when deployed. Root privilege is for hpcadmin only, sudo privilge may be used if/when necessary to achieve some purpose. The hpcadmin will maintain the new resource(s) while configuration(s) of new resource(s) will be done by consent of all parties involved. Final approval by the Advisory Group initiates deployment activities.

The second principle grants priority access to certain resource(s) for a limited time to a limited group. The same PI/users relationship will be used as is used in the CPU Usage Contribution scheme. Priority access means if during the priority period the priority members jobs go into pending mode for more than 24 hours the hpcadmin will clear compute nodes of running jobs and force those pending jobs to run.

All users should be aware this may happen so please checkpoint your jobs with a checkpoint interval of 24 hours. Please consult BLCR Checkpoint in OL3 (serial jobs) and BLCR Checkpoint in OL3 (parallel jobs).


Back

cluster/189.1578766325.txt.gz · Last modified: 2020/01/11 13:12 by hmeij07