This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:73 [2021/06/22 19:47] hmeij07 [Galario] |
cluster:73 [2023/12/14 14:27] hmeij07 [Miniconda3-py311] |
||
---|---|---|---|
Line 3: | Line 3: | ||
====== Software ====== | ====== Software ====== | ||
+ | |||
+ | [[cluster: | ||
+ | |||
+ | Below is all centos 6 and/or 7, which may or may not run in 8. | ||
+ | |||
IMPORTANT NOTE: **Since moving to the TrueNAS/ZFS appliance all references to** ''/ | IMPORTANT NOTE: **Since moving to the TrueNAS/ZFS appliance all references to** ''/ | ||
Line 18: | Line 23: | ||
on either cottontail2 or node n33 located at / | on either cottontail2 or node n33 located at / | ||
Some compilations require 7.x and are in / | Some compilations require 7.x and are in / | ||
+ | |||
+ | |||
+ | |||
+ | ====== Miniconda3-py311 ====== | ||
+ | |||
+ | * Starting a new branch, the other Miniconda3 section listed on this is quite loaded | ||
+ | * Python 3.11 | ||
+ | |||
+ | < | ||
+ | |||
+ | source / | ||
+ | or | ||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | which mpirun python conda | ||
+ | conda list | ||
+ | |||
+ | export CUDA_HOME=/ | ||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | |||
+ | |||
+ | </ | ||
+ | |||
+ | * conda install (same as what is listed on openhpc [[https:// | ||
+ | * cudatoolkit=10.2 cudnn=7 for exx96 queue | ||
+ | * same error, module cudatoolkit not found, but I found Cupy and it does load | ||
+ | * https:// | ||
+ | * thayerlab | ||
+ | |||
+ | < | ||
+ | |||
+ | # pollacklab | ||
+ | |||
+ | [hmeij@greentail52 ~]$ source / | ||
+ | |||
+ | [hmeij@greentail52 ~]$ conda activate sage | ||
+ | |||
+ | (sage) [hmeij@greentail52 ~]$ sage --version | ||
+ | SageMath version 10.0, Release Date: 2023-05-20 | ||
+ | |||
+ | (sage) [hmeij@greentail52 ~]$ sage | ||
+ | ┌────────────────────────────────────────────────────────────────────┐ | ||
+ | │ SageMath version 10.0, Release Date: 2023-05-20 | ||
+ | │ Using Python 3.11.4. Type " | ||
+ | └────────────────────────────────────────────────────────────────────┘ | ||
+ | |||
+ | sage: exit | ||
+ | (sage) [hmeij@greentail52 ~]$ conda deactivate | ||
+ | [hmeij@greentail52 ~]$ | ||
+ | |||
+ | </ | ||
+ | |||
+ | * **mdtraj, matplotlib, pandas** | ||
+ | |||
+ | < | ||
+ | |||
+ | [hmeij@greentail52 ~]$ which python | ||
+ | / | ||
+ | | ||
+ | [hmeij@greentail52 ~]$ python | ||
+ | Python 3.11.4 (main, Jul 5 2023, 13:45:01) [GCC 11.2.0] on linux | ||
+ | Type " | ||
+ | >>> | ||
+ | >>> | ||
+ | |||
+ | </ | ||
+ | |||
+ | * ** seaborn** | ||
+ | * https:// | ||
+ | * did not install on miniconda3/ | ||
+ | |||
+ | |||
+ | ====== Masurca ====== | ||
+ | |||
+ | |||
+ | * module: masurca/ | ||
+ | * https:// | ||
+ | * supports openmp (maybe) | ||
+ | * queues mwgpu, exx96 (centos 7) | ||
+ | |||
+ | |||
+ | < | ||
+ | |||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | |||
+ | |||
+ | [hmeij@cottontail2 ~]$ masurca --version | ||
+ | version 4.1.0 | ||
+ | |||
+ | |||
+ | </ | ||
+ | |||
+ | ====== Trimmomatic ====== | ||
+ | |||
+ | * http:// | ||
+ | * binary download and stage | ||
+ | * java -jar trimmomatic-0.39.jar ... | ||
+ | |||
+ | < | ||
+ | |||
+ | / | ||
+ | [root@cottontail Trimmomatic-0.39]# | ||
+ | total 176 | ||
+ | drwxr-xr-x 2 root root 8 Apr 8 2019 adapters | ||
+ | -rw-r--r-- 1 root root 35147 May 16 2018 LICENSE | ||
+ | -rw-r--r-- 1 root root 128502 Apr 8 2019 trimmomatic-0.39.jar | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | ====== HPC SDK ====== | ||
+ | |||
+ | * https:// | ||
+ | * Nvidia SDK v 22.2 | ||
+ | |||
+ | ====== Magenta ====== | ||
+ | |||
+ | An open source research project exploring the role of machine learning as a tool in the creative process. | ||
+ | |||
+ | * https:// | ||
+ | * installed in miniconda3' | ||
+ | * queues: mwgpu, amber128, exx96 | ||
+ | * (MUSC/ | ||
+ | |||
+ | ====== EasyBuild ====== | ||
+ | |||
+ | EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way. | ||
+ | |||
+ | * supports 2469 different software packages (incl. toolchains, bundles) | ||
+ | * local page with lots of details | ||
+ | |||
+ | Easybuild uses '' | ||
+ | |||
+ | **Installed modules** and their dependencies | ||
+ | |||
+ | * astropy/ | ||
+ | * https:// | ||
+ | * contains package mpi4py, multithreading | ||
+ | * import erfa (needs to be loaded too), added later (pycuda, h5py for helios) | ||
+ | * to find '' | ||
+ | * emcee/ | ||
+ | * https:// | ||
+ | * contains package mpi4py, multithreading | ||
+ | * PyCUDA/ | ||
+ | * | ||
+ | |||
+ | To use module, discover the modules and load desired module. On '' | ||
+ | |||
+ | < | ||
+ | |||
+ | # cuda libs location | ||
+ | export LD_LIBRARY_PATH=\ | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | $LD_LIBRARY_PATH | ||
+ | |||
+ | |||
+ | # discover all modules | ||
+ | module use / | ||
+ | |||
+ | # view availability | ||
+ | module avail | ||
+ | |||
+ | # load | ||
+ | module load module_name | ||
+ | |||
+ | # list modules loaded | ||
+ | module list | ||
+ | |||
+ | # check, start app, import package (by full module name) | ||
+ | # note that these modules use different python versions | ||
+ | which python | ||
+ | python | ||
+ | >>> | ||
+ | |||
+ | # unload all modules (better to exit shell and start new one, | ||
+ | # purge will also unload system modules... | ||
+ | module purge | ||
+ | |||
+ | </ | ||
+ | |||
+ | ====== Hoomd ====== | ||
+ | |||
+ | HOOMD-blue is a Python package that runs simulations of particle systems on CPUs and GPUs. | ||
+ | |||
+ | * https:// | ||
+ | * requires both minconda3 (centos7) **and** cuda-10.2 (so queue exx96 only) | ||
+ | * python3.x included | ||
+ | * starrlab | ||
+ | |||
+ | Set up env for miniconda and cuda | ||
+ | |||
+ | < | ||
+ | |||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | |||
+ | | ||
+ | | ||
+ | | ||
+ | |||
+ | which mpirun python conda | ||
+ | which nvcc | ||
+ | |||
+ | # installation (not sure how this works, does the * not get expanded in shell?) | ||
+ | # seems to have worked forcing gpu package install on greentail52 | ||
+ | conda install -c conda-forge hoomd=*=*gpu* | ||
+ | |||
+ | |||
+ | $ conda list | egrep -i " | ||
+ | cudatoolkit | ||
+ | hoomd | ||
+ | |||
+ | |||
+ | # ahh, hamed found an error | ||
+ | conda install -c conda-forge cudatoolkit=10.2 | ||
+ | |||
+ | The following packages will be DOWNGRADED: | ||
+ | |||
+ | cudatoolkit | ||
+ | hoomd 2.9.4-gpu_py38h02d30ca_1 --> 2.9.4-gpu_py38h5bdc439_1 | ||
+ | |||
+ | |||
+ | </ | ||
+ | |||
+ | * installed/ | ||
+ | |||
+ | < | ||
+ | |||
+ | ca-certificates-2021.10.8 | ||
+ | certifi-2021.10.8 | ||
+ | conda-4.10.3 | ||
+ | cudatoolkit-11.0.221 | ||
+ | hoomd-2.9.4 | ||
+ | openssl-1.1.1h | ||
+ | tbb-2020.2 | ||
+ | |||
+ | </ | ||
+ | |||
+ | * and | ||
+ | |||
+ | < | ||
+ | |||
+ | conda install -c conda-forge gsd | ||
+ | |||
+ | |||
+ | The following NEW packages will be INSTALLED: | ||
+ | |||
+ | _openmp_mutex | ||
+ | gsd conda-forge/ | ||
+ | libgomp | ||
+ | |||
+ | The following packages will be UPDATED: | ||
+ | |||
+ | certifi | ||
+ | conda | ||
+ | libgcc-ng | ||
+ | openssl | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | |||
+ | ====== Miniconda2 ====== | ||
+ | |||
+ | * miniconda2 (python 2.7) is hiding in section | ||
+ | * " | ||
+ | * some packages require centos 6 (like cufflinks) | ||
+ | * many packages are also available in miniconda3 section | ||
+ | * requires centos 7 (python 3.9) | ||
+ | |||
+ | ====== Sequencing Tools #4 ====== | ||
+ | |||
+ | Added to miniconda3, see section " | ||
+ | |||
+ | * Requires centos7 so queues mwgpu, amber128, exx96 | ||
+ | * debug server greentail52 | ||
+ | * PREFIX=/ | ||
+ | * python 3.9 | ||
+ | * jcoolon lab | ||
+ | | ||
+ | |||
+ | < | ||
+ | |||
+ | # environment | ||
+ | source / | ||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | |||
+ | |||
+ | libgcc-7.2.0 | ||
+ | samtools-1.7 | ||
+ | bowtie2-2.2.5 | ||
+ | perl-5.26.2 | ||
+ | bedtools-2.30.0 | ||
+ | |||
+ | # cufflinks requires centos 6, weird, so also added to miniconda2, | ||
+ | # see section " | ||
+ | |||
+ | </ | ||
+ | |||
+ | ====== Sequencing Tools #3 ====== | ||
+ | |||
+ | Suite of tools for high-accuracy basecaling, assembly, polishing, and alignment. Installed in a specific miniconda3 environment (see below). | ||
+ | |||
+ | * Requires centos7 so queues mwgpu, < | ||
+ | * debug server greentail52 | ||
+ | * PREFIX=/ | ||
+ | * python 3.9 | ||
+ | * gcc 9.2 | ||
+ | * cmake 3.8.5 | ||
+ | * chernofflab | ||
+ | |||
+ | < | ||
+ | |||
+ | # environment for all modules | ||
+ | source / | ||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | export PATH=/ | ||
+ | |||
+ | which conda python pip mpirun gcc cmake | ||
+ | python --version | ||
+ | |||
+ | / | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | / | ||
+ | |||
+ | |||
+ | </ | ||
+ | |||
+ | ** Guppy Server ** | ||
+ | |||
+ | * https:// | ||
+ | * standalone build | ||
+ | * Local accelerated basecalling for Nanopore data | ||
+ | * v5.0.11 | ||
+ | |||
+ | < | ||
+ | |||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | ldd `which guppy_basecall_server` | ||
+ | |||
+ | # cpu; note as user start server on scratch server greentail52 | ||
+ | guppy_basecall_server --config dna_r9.4.1_450bps_fast.cfg -p 5555 -l /tmp/guppy & | ||
+ | |||
+ | # check if running, use '' | ||
+ | lsof -i:5555 | ||
+ | COMMAND | ||
+ | guppy_bas 80002 bchernoff | ||
+ | |||
+ | # nodes can reach this server at this IP or start their own on localhost: | ||
+ | 192.168.102.251 greentail52-eth0 | ||
+ | 10.10.102.251 | ||
+ | |||
+ | </ | ||
+ | |||
+ | ** Guppy Client ** | ||
+ | |||
+ | * https:// | ||
+ | * miniconda3 python3.9 | ||
+ | * Python client library for Guppy. | ||
+ | * Full Python client library for communicating with guppy_basecall_server. | ||
+ | * v0.1.0 | ||
+ | |||
+ | < | ||
+ | |||
+ | pip install cython PyUnicode timer | ||
+ | pip install pyguppyclient | ||
+ | Successfully installed flatbuffers-1.11 ont-fast5-api-3.3.0 pyguppyclient-0.0.6 pyzmq-17.1.2 | ||
+ | |||
+ | note: from pyguppyclient import GuppyBasecallerClient, | ||
+ | success, but do not know how to execute the example | ||
+ | |||
+ | </ | ||
+ | |||
+ | ** Medaka ** | ||
+ | |||
+ | * https:// | ||
+ | * miniconda3 | ||
+ | * Medaka is a tool to create consensus sequences and variant calls from nanopore sequencing data. | ||
+ | * v1.4.3 | ||
+ | |||
+ | < | ||
+ | |||
+ | # To activate this environment, | ||
+ | # $ conda activate medaka | ||
+ | # To deactivate an active environment, | ||
+ | # $ conda deactivate | ||
+ | |||
+ | # requires | ||
+ | (medaka)$ conda list | egrep -i " | ||
+ | minimap2 | ||
+ | pbgzip | ||
+ | py-bgzip | ||
+ | samtools | ||
+ | tabix | ||
+ | |||
+ | </ | ||
+ | |||
+ | ** Flye ** | ||
+ | |||
+ | Somewhat annoying, no miniconda3 package, (python 3.9, gcc 9.2). Local building (without installation) on centos7 greentail52. | ||
+ | |||
+ | * setup env as for all packages in #3, see top of section | ||
+ | * v2.9.2 (June 2023) | ||
+ | * python bin/flye | ||
+ | * change the PATHS accordingly | ||
+ | |||
+ | < | ||
+ | |||
+ | / | ||
+ | 2.9.2-b1794 | ||
+ | |||
+ | </ | ||
+ | |||
+ | * v2.9.1 (October 2022) | ||
+ | * python bin/flye | ||
+ | * change the PATHs accordingly | ||
+ | |||
+ | < | ||
+ | |||
+ | / | ||
+ | 2.9.1-b1780 | ||
+ | |||
+ | </ | ||
+ | |||
+ | * v2.9 (August 2021) | ||
+ | * python bin/flye | ||
+ | |||
+ | < | ||
+ | |||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | which flye | ||
+ | / | ||
+ | flye --version | ||
+ | 2.9-b1774 | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | |||
+ | ** Filtlong ** | ||
+ | |||
+ | * https:// | ||
+ | * binary | ||
+ | * Filtlong is a tool for filtering long reads by quality. | ||
+ | * v0.2.1 | ||
+ | |||
+ | < | ||
+ | |||
+ | # note: fails with gcc 9.2, native gcc ok 4.8.5, UNSET gcc92/ | ||
+ | export PATH=/ | ||
+ | ldd `which filtlong` | ||
+ | |||
+ | </ | ||
+ | |||
+ | ** Bamtools** | ||
+ | |||
+ | * https:// | ||
+ | * standalone build | ||
+ | * C++ API & command-line toolkit for working with BAM data | ||
+ | * v2.5.2 | ||
+ | |||
+ | < | ||
+ | |||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | ldd `which bamtools` | ||
+ | |||
+ | </ | ||
+ | |||
+ | ** BBmap ** | ||
+ | |||
+ | * location: / | ||
+ | * java programs | ||
+ | * https:// | ||
+ | |||
+ | ** Pilon ** | ||
+ | |||
+ | * location: / | ||
+ | * just one big jar file | ||
+ | * latest java is in $PATH | ||
+ | |||
+ | < | ||
+ | |||
+ | [hmeij@greentail52 pilon]$ java -jar pilon-1.24.jar --help | ||
+ | Pilon version 1.24 Thu Jan 28 13:00:45 2021 -0500 | ||
+ | |||
+ | Usage: pilon --genome genome.fasta [--frags frags.bam] [--jumps jumps.bam] [--unpaired unpaired.bam] | ||
+ | | ||
+ | pilon --help for option details | ||
+ | < | ||
+ | |||
+ | </ | ||
+ | |||
+ | ** haslr ** | ||
+ | |||
+ | * miniconda3 env | ||
+ | * https:// | ||
+ | * conda install -c bioconda haslr | ||
+ | |||
+ | < | ||
+ | |||
+ | The following NEW packages will be INSTALLED: | ||
+ | |||
+ | _openmp_mutex | ||
+ | fastutils | ||
+ | haslr bioconda/ | ||
+ | k8 | ||
+ | libgomp | ||
+ | libzlib | ||
+ | minia bioconda/ | ||
+ | minimap2 | ||
+ | pluggy | ||
+ | ruamel.yaml | ||
+ | ruamel.yaml.clib | ||
+ | toolz conda-forge/ | ||
+ | |||
+ | </ | ||
+ | |||
+ | Compiled from source, do **not** source miniconda3, we are using standalone pyhton3 | ||
+ | |||
+ | < | ||
+ | |||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | [hmeij@greentail52 20231006]$ which python | ||
+ | / | ||
+ | |||
+ | [hmeij@greentail52 20231006]$ pwd | ||
+ | / | ||
+ | |||
+ | [hmeij@greentail52 20231006]$ ll bin | ||
+ | total 11448 | ||
+ | -rwxr-xr-x 1 hmeij its | ||
+ | -rwxr-xr-x 1 hmeij its 3823032 Oct 6 15:04 haslr_assemble | ||
+ | -rwxr-xr-x 1 hmeij its 19949 Oct 6 14:48 haslr.py | ||
+ | -rwxr-xr-x 1 hmeij its 20969840 Oct 6 14:59 minia | ||
+ | -rwxr-xr-x 1 hmeij its 23856 Oct 6 15:01 minia_nooverlap | ||
+ | -rwxr-xr-x 1 hmeij its | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | ** stLFR ** | ||
+ | |||
+ | * same centos 7 environment | ||
+ | * https:// | ||
+ | |||
+ | < | ||
+ | |||
+ | cd / | ||
+ | |||
+ | [hmeij@greentail52 20230929]$ ./ | ||
+ | Version: | ||
+ | 1.00 | ||
+ | |||
+ | Contact: | ||
+ | dengli1@genomics.cn && guolidong@genomics.cn | ||
+ | |||
+ | Usage: | ||
+ | stLFR_GapCloser [options] | ||
+ | ... | ||
+ | |||
+ | |||
+ | # and another request by Luis | ||
+ | # https:// | ||
+ | |||
+ | [hmeij@greentail52 stLFR]$ pwd | ||
+ | / | ||
+ | [hmeij@greentail52 stLFR]$ ll GapCloser-bin-v1.12-r6/ | ||
+ | total 175 | ||
+ | -rwxr-xr-x 1 hmeij its 184713 Jan 11 2013 GapCloser | ||
+ | -rw-r--r-- 1 hmeij its 44533 Jan 11 2013 GapCloser_Manual.pdf | ||
+ | |||
+ | </ | ||
+ | |||
+ | ** BUSCO ** | ||
+ | |||
+ | * https:// | ||
+ | * conda package fails on glib 2.17, weird | ||
+ | |||
+ | < | ||
+ | |||
+ | [hmeij@n79 ~]$ docker images | ||
+ | REPOSITORY | ||
+ | ezlabgva/ | ||
+ | |||
+ | |||
+ | [hmeij@n79 busco_wd]$ docker run --rm -v $(pwd):/ | ||
+ | 2023-10-28 20:30:00 INFO: ***** Start a BUSCO v5.5.0 analysis, current time: 10/28/2023 20:30:00 ***** | ||
+ | 2023-10-28 20:30:00 INFO: | ||
+ | 2023-10-28 20:30:00 INFO: Mode is genome | ||
+ | 2023-10-28 20:30:00 INFO: ' | ||
+ | 2023-10-28 20:30:00 INFO: | ||
+ | |||
+ | that's a problem, trying to reach the internet ... | ||
+ | |||
+ | </ | ||
+ | |||
+ | ** picard ** | ||
+ | |||
+ | * https:// | ||
+ | * Warning: using Java 20 but only Java 17 has been tested. | ||
+ | |||
+ | < | ||
+ | |||
+ | picard centos7, weird | ||
+ | > Failed to load native library ' | ||
+ | |||
+ | picard centos8 | ||
+ | |||
+ | [hmeij@cottontail2 20231108]$ pwd | ||
+ | / | ||
+ | |||
+ | [hmeij@cottontail2 20231108]$ ls -l build/libs/ | ||
+ | total 119872 | ||
+ | -rw-r--r-- 1 hmeij its 62520451 Nov 8 11:46 picard-3.1.0-3-ga9194bd97-SNAPSHOT-all.jar | ||
+ | -rw-r--r-- 1 hmeij its 1856980 Nov 8 11:01 picard-3.1.0-3-ga9194bd97-SNAPSHOT.jar | ||
+ | -rw-r--r-- 1 hmeij its 62520451 Nov 8 11:46 picard.jar | ||
+ | [hmeij@cottontail2 20231108]$ | ||
+ | |||
+ | |||
+ | </ | ||
+ | ====== Galario CPU+GPU ====== | ||
+ | |||
+ | * c/c++ compiled library with FFTW3 | ||
+ | * should be gpu and cpu able (not sure how to work it yet) | ||
+ | * install notes at / | ||
+ | * sample program at / | ||
+ | * requires centos7 | ||
+ | * queues: mwgpu, amber128, exx96 | ||
+ | * debug node greentail52 | ||
+ | |||
+ | **GPU is nonfunctional** | ||
+ | |||
+ | Conversion of python code to C binaries was successful \\ | ||
+ | read the history.txt file | ||
+ | --- // | ||
+ | |||
+ | < | ||
+ | |||
+ | # sample scripts in / | ||
+ | |||
+ | # read README, it's complicated | ||
+ | |||
+ | </ | ||
====== VSEARCH ====== | ====== VSEARCH ====== | ||
Line 84: | Line 754: | ||
* queues: mwgpu, amber128, exx96 | * queues: mwgpu, amber128, exx96 | ||
* for debug host use greentail52 | * for debug host use greentail52 | ||
+ | |||
+ | ** Due to technical limitations, | ||
+ | |||
+ | < | ||
+ | |||
+ | source / | ||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | which mpirun python conda | ||
+ | |||
+ | </ | ||
+ | |||
< | < | ||
- | # https:// | + | # https:// |
conda install -c conda-forge galario | conda install -c conda-forge galario | ||
Line 104: | Line 786: | ||
mpi4py | mpi4py | ||
openmpi | openmpi | ||
+ | | ||
+ | pandas | ||
+ | python-dateutil | ||
+ | pytz | ||
+ | schwimmbad | ||
+ | |||
| | ||
</ | </ | ||
- | For astronomy group added (' | + | For astronomy group (amhughes and students) |
< | < | ||
Line 117: | Line 805: | ||
numpy | numpy | ||
time 1.8 h516909a_0 | time 1.8 h516909a_0 | ||
+ | astropy | ||
+ | attrs conda-forge/ | ||
+ | click conda-forge/ | ||
+ | coverage | ||
+ | hypothesis | ||
+ | iniconfig | ||
+ | more-itertools | ||
+ | packaging | ||
+ | pluggy | ||
+ | psutil | ||
+ | py | ||
+ | pyparsing | ||
+ | pytest | ||
+ | pytest-arraydiff | ||
+ | pytest-astropy | ||
+ | pytest-astropy-he~ conda-forge/ | ||
+ | pytest-cov | ||
+ | pytest-doctestplus conda-forge/ | ||
+ | pytest-filter-sub~ conda-forge/ | ||
+ | pytest-mock | ||
+ | pytest-openfiles | ||
+ | pytest-remotedata | ||
+ | sortedcontainers | ||
+ | toml | ||
+ | |||
</ | </ | ||
+ | |||
+ | For nwells/ | ||
+ | |||
+ | < | ||
+ | |||
+ | conda install -c schrodinger pymol-bundle | ||
+ | |||
+ | package | ||
+ | ---------------------------|----------------- | ||
+ | apbs-1.5 | ||
+ | biopython-1.78 | ||
+ | bzip2-1.0.8 | ||
+ | collada2gltf-2.1.4 | ||
+ | conda-4.10.3 | ||
+ | curl-7.71.1 | ||
+ | dbus-1.13.18 | ||
+ | expat-2.4.1 | ||
+ | fontconfig-2.13.1 | ||
+ | freemol-1.158 | ||
+ | freetype-2.10.4 | ||
+ | glew-2.0.0 | ||
+ | glib-2.69.0 | ||
+ | gst-plugins-base-1.14.0 | ||
+ | gstreamer-1.14.0 | ||
+ | h5py-2.10.0 | ||
+ | hdf4-4.2.13 | ||
+ | hdf5-1.10.6 | ||
+ | icu-58.2 | ||
+ | jpeg-9d | ||
+ | krb5-1.18.2 | ||
+ | libcurl-7.71.1 | ||
+ | libglu-9.0.0 | ||
+ | libholoplaycore-0.1.0_rc4 | ||
+ | libnetcdf-4.7.4 | ||
+ | libpng-1.6.37 | ||
+ | libssh2-1.9.0 | ||
+ | libtiff-4.0.10 | ||
+ | libuuid-2.32.1 | ||
+ | libxcb-1.13 | ||
+ | libxml2-2.9.10 | ||
+ | lz4-c-1.9.2 | ||
+ | mengine-1 | ||
+ | mpeg_encode-1 | ||
+ | mtz2ccp4_px-1.0 | ||
+ | olefile-0.46 | ||
+ | pcre-8.45 | ||
+ | pdb2pqr-2.1.2+pymol | ||
+ | pillow-6.2.1 | ||
+ | pmw-2.0.1+3 | ||
+ | pthread-stubs-0.4 | ||
+ | pycollada-0.7.1+bdf414c7 | ||
+ | pykerberos-1.2.1 | ||
+ | pymol-2.4.1 | ||
+ | pymol-bundle-2.4.1 | ||
+ | pymol-web-examples-2.4 | ||
+ | pyqt-5.9.2 | ||
+ | qt-5.9.7 | ||
+ | rigimol-1.3 | ||
+ | sip-4.19.13 | ||
+ | xorg-libxau-1.0.9 | ||
+ | xorg-libxdmcp-1.1.3 | ||
+ | zstd-1.4.5 | ||
+ | |||
+ | |||
+ | </ | ||
+ | |||
+ | Then | ||
+ | --- // | ||
+ | |||
+ | < | ||
+ | |||
+ | # using pip3 | ||
+ | Successfully installed magenta-2.1.3 python-rtmidi-1.1.2 | ||
+ | |||
+ | Successfully installed keras-2.7.0 libclang-12.0.0 tensorboard-2.7.0 \ | ||
+ | tensorflow-2.7.0 tensorflow-estimator-2.7.0 tensorflow-io-gcs-filesystem-0.22.0 | ||
+ | |||
+ | </ | ||
+ | |||
+ | Added\\ | ||
+ | --- // | ||
+ | # https:// | ||
+ | |||
+ | < | ||
+ | |||
+ | conda install -c bioconda samtools=1.9 --force-reinstall | ||
+ | |||
+ | |||
+ | The following NEW packages will be INSTALLED: | ||
+ | |||
+ | gettext | ||
+ | htslib | ||
+ | libdeflate | ||
+ | libglib | ||
+ | libiconv | ||
+ | |||
+ | The following packages will be UPDATED: | ||
+ | |||
+ | cffi pkgs/ | ||
+ | libedit | ||
+ | samtools | ||
+ | |||
+ | The following packages will be SUPERSEDED by a higher-priority channel: | ||
+ | |||
+ | glib pkgs/ | ||
+ | libffi | ||
+ | ncurses | ||
+ | python | ||
+ | sqlite | ||
+ | |||
+ | $ samtools --version | ||
+ | samtools 1.9 | ||
+ | Using htslib 1.9 | ||
+ | Copyright (C) 2018 Genome Research Ltd. | ||
+ | |||
+ | </ | ||
+ | |||
+ | * **iqtree** fcohanlab/ | ||
+ | |||
+ | < | ||
+ | |||
+ | [hmeij@greentail52 ~]$ iqtree --version | ||
+ | IQ-TREE multicore version 2.0.3 for Linux 64-bit built Dec 20 2020 | ||
+ | |||
+ | |||
+ | </ | ||
+ | |||
+ | * https:// | ||
+ | * calterlab, annika (used pip to install) | ||
+ | |||
+ | < | ||
+ | |||
+ | Successfully installed | ||
+ | GridDataFormats-1.0.1 MDAnalysis-2.3.0 | ||
+ | fasteners-0.18 mmtf-python-1.1.3 | ||
+ | mrcfile-1.4.3 msgpack-1.0.4 numpy-1.23.4 | ||
+ | |||
+ | </ | ||
+ | |||
+ | For **Lammps** (starrlab) 23April2023, | ||
+ | |||
+ | * / | ||
+ | * make yes-ml-pace | ||
+ | * make yes-gpu | ||
+ | * https:// | ||
+ | |||
+ | < | ||
+ | |||
+ | # Note: you must add this line to you env | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | |||
+ | # Note2: had to install 'conda install cudnn' and make two links | ||
+ | to higher level libraries (a must not do) | ||
+ | for libcudnn.so.8 and libcusolver.so.11 | ||
+ | |||
+ | [hmeij@greentail52 ~]$ which python | ||
+ | / | ||
+ | [hmeij@greentail52 ~]$ pip list | grep tensorflow | ||
+ | mesh-tensorflow | ||
+ | tensorflow | ||
+ | tensorflow-addons | ||
+ | tensorflow-datasets | ||
+ | tensorflow-estimator | ||
+ | tensorflow-gan | ||
+ | tensorflow-hub | ||
+ | tensorflow-io-gcs-filesystem 0.32.0 | ||
+ | tensorflow-metadata | ||
+ | tensorflow-probability | ||
+ | [hmeij@greentail52 ~]$ which pacemaker | ||
+ | / | ||
+ | |||
+ | </ | ||
+ | |||
====== pytraj ====== | ====== pytraj ====== | ||
Line 129: | Line 1015: | ||
* git clone https:// | * git clone https:// | ||
* python ./setup.py install | * python ./setup.py install | ||
- | * requires CentOS7 so queues mwgpu, amber128 and exx96 | + | * requires CentOS7 so queues mwgpu, |
* or for debugging use greentail52 | * or for debugging use greentail52 | ||
* / | * / | ||
Line 169: | Line 1055: | ||
====== OpenHPC ====== | ====== OpenHPC ====== | ||
- | So I remember where I put the repos ''/ | + | So I remember where I put the tar repo file |
- | * whitetail tar ball for centos7/ | + | * OpenHPC v2.4 |
- | * http://repos.openhpc.community/OpenHPC/2.0RC1/CentOS_8/x86_64/ | + | * Rocky 8.5 |
- | | + | * Slurm 20.11.8 |
- | * https://github.com/ | + | * Warewulf 3.9 |
+ | * x86_64 | ||
+ | |||
+ | < | ||
+ | |||
+ | [root@cottontail2 ~]# ll /share/ | ||
+ | total 8290421 | ||
+ | -rw------- 1 root root | ||
+ | -rw-r--r-- 1 root root 10720 Mar 17 09:35 ohpc-release-2-1.el8.x86_64.rpm | ||
+ | -rw-r--r-- 1 root root 8531681280 Mar 17 09:33 OpenHPC-2.4.EL_8.x86_64.tar | ||
+ | |||
+ | |||
+ | |||
+ | # this is all that is needed for ohpc base compute client | ||
+ | # see the recipe for cct2 and follow compute node only ingredients | ||
+ | |||
+ | [root@cottontail2 ~]# ll /share/apps/ | ||
+ | total 95196 | ||
+ | -rw-r--r-- 1 root root 2696360 Feb 10 09:21 hwloc-ohpc-2.7.0-3.9.ohpc.2.6.x86_64.rpm | ||
+ | -rw-r--r-- 1 root root | ||
+ | -rw-r--r-- | ||
+ | -rw-r--r-- 1 root root 8204 Feb 10 09:23 ohpc-filesystem-2.6-2.3.ohpc.2.6.noarch.rpm | ||
+ | -rw-r--r-- 1 root root 10720 Feb 10 09:23 ohpc-release-2-1.el8.x86_64.rpm | ||
+ | -rw-r--r-- 1 root root 7124 Feb 10 09:27 ohpc-slurm-client-2.6.1-5.1.ohpc.2.6.1.x86_64.rpm | ||
+ | -rw-r--r-- 1 root root 74326532 Feb 10 09:23 singularity-ohpc-3.7.1-5.1.ohpc.2.1.x86_64.rpm | ||
+ | -rw-r--r-- 1 root root 22168 Feb 10 09:24 slurm-contribs-ohpc-22.05.2-14.1.ohpc.2.6.x86_64.rpm | ||
+ | -rw-r--r-- 1 root root | ||
+ | -rw-r--r-- 1 root root 18764472 Feb 10 09:24 slurm-ohpc-22.05.2-14.1.ohpc.2.6.x86_64.rpm | ||
+ | -rw-r--r-- 1 root root | ||
+ | -rw-r--r-- 1 root root | ||
+ | |||
+ | </code> | ||
====== OneAPI ====== | ====== OneAPI ====== | ||
* Requires CentOS 7 so queues mwgpu, amber128, exx96 | * Requires CentOS 7 so queues mwgpu, amber128, exx96 | ||
+ | * debug node greentail52 | ||
* Production version is installed in / | * Production version is installed in / | ||
- | * source / | + | * source / |
< | < | ||
Line 203: | Line 1121: | ||
</ | </ | ||
- | * Test version is installed on local disk on host **greentail52** | + | * __Test__ |
* source / | * source / | ||
* Read [[cluster: | * Read [[cluster: | ||
Line 220: | Line 1138: | ||
* centos7 R v 3.6.0 | * centos7 R v 3.6.0 | ||
* use your HPC username and credentials | * use your HPC username and credentials | ||
+ | |||
+ | < | ||
+ | |||
+ | # native OS installation for rstudio | ||
+ | /bin/R | ||
+ | |||
+ | </ | ||
====== DMTCP ===== | ====== DMTCP ===== | ||
Line 343: | Line 1268: | ||
</ | </ | ||
+ | |||
+ | * **sratools** https:// | ||
+ | * The SRA Toolkit and SDK from NCBI is a collection of tools and libraries for using data in the INSDC Sequence Read Archives. | ||
+ | * requires centos7 (glibc) so queues mwgpu, amber128, exx96 | ||
+ | |||
+ | < | ||
+ | |||
+ | export PATH=/ | ||
+ | |||
+ | </ | ||
+ | |||
+ | * **adapterremoval** https:// | ||
+ | * **bwa** https:// | ||
+ | * **bowtie2** http:// | ||
+ | |||
+ | These are all part of Miniconda3 (centos7), to setup the environment consult | ||
+ | |||
+ | * https:// | ||
+ | * queues: mwgpu, amber128, exx96, debug server greentail52 | ||
+ | |||
+ | They are also available via Miniconda2 (centos6), consult | ||
+ | |||
+ | * https:// | ||
+ | * queues: hp12, mw256fd, tinymem, debug server swallowtail | ||
+ | |||
+ | < | ||
+ | |||
+ | [hmeij@greentail52 ~]$ conda list | egrep " | ||
+ | adapterremoval | ||
+ | bowtie2 | ||
+ | bwa | ||
+ | |||
+ | </ | ||
+ | |||
+ | * **mapDamage2** | ||
+ | * https:// | ||
+ | * docker container is broke | ||
+ | * requires centos6 so queues hp12, mw128 .. may run elsewhere? | ||
+ | * conda install -c ' | ||
+ | |||
+ | |||
+ | < | ||
+ | |||
+ | # env | ||
+ | export PATH="/ | ||
+ | export LD_LIBRARY_PATH="/ | ||
+ | |||
+ | The following NEW packages will be INSTALLED: | ||
+ | |||
+ | mapdamage2 | ||
+ | pysam bioconda/ | ||
+ | seqtk bioconda/ | ||
+ | |||
+ | [hmeij@petaltail ~]$ mapDamage --version | ||
+ | 2.0.6 | ||
+ | |||
+ | </ | ||
+ | |||
+ | * **PCAngsd**, | ||
+ | * https:// | ||
+ | * apparently the repo is needed ''/ | ||
+ | |||
+ | < | ||
+ | |||
+ | source / | ||
+ | export PATH=/ | ||
+ | export LD_LIBRARY_PATH=/ | ||
+ | |||
+ | which pcangsd | ||
+ | / | ||
+ | pcangsd --version | ||
+ | pcangsd 1.10 | ||
+ | |||
+ | </ | ||
+ | |||
+ | * **angsd**, requires miniconda3, same environment as pcangsd | ||
+ | * http:// | ||
+ | * add location of angsd to PATH env | ||
+ | |||
+ | < | ||
+ | |||
+ | $ pwd | ||
+ | / | ||
+ | |||
+ | $ ls -l angsd | ||
+ | -rwxr-xr-x 1 hmeij its 5517248 Jul 6 14:11 angsd | ||
+ | |||
+ | $ ./ | ||
+ | -> angsd version: 0.940-dirty (htslib: 1.16) build(Jul | ||
+ | |||
+ | </ | ||
+ | |||
+ | * **Standard RAxML version** | ||
+ | * https:// | ||
+ | |||
+ | < | ||
+ | |||
+ | # add path to $PATH | ||
+ | |||
+ | [hmeij@greentail52 tmp]$ ll / | ||
+ | total 3202 | ||
+ | -rwxr-xr-x 1 hmeij its 1177632 Apr 25 14:49 raxmlHPC-AVX | ||
+ | -rwxr-xr-x 1 hmeij its 1177632 Apr 25 14:49 raxmlHPC-AVX2 | ||
+ | -rwxr-xr-x 1 hmeij its 1152040 Apr 25 14:49 raxmlHPC_serial | ||
+ | -rwxr-xr-x 1 hmeij its 1185704 Apr 25 14:49 raxmlHPC-SSE3 | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | * **qualimap ** | ||
+ | * http:// | ||
+ | * java & R app, probably will work in centos7 and rocky8 | ||
+ | * / | ||
+ | |||
+ | * **gatk** | ||
+ | * https:// | ||
+ | * java & R app. probably will work in centos7 and rocky8 | ||
+ | * / | ||
+ | |||
+ | For both apps above java is v20, developers want and tested v17 | ||
+ | |||
+ | For both apps you may find R local in operating system or a recent versions at | ||
+ | |||
+ | * (rocky 8) module load R/4.1.2 | ||
+ | |||
+ | |||
+ | |||
Line 664: | Line 1716: | ||
* location: / | * location: / | ||
* https:// | * https:// | ||
- | * biolab (jcoolon, tearley) | + | |
+ | * queues: hp12, tinymem, mw256fd mw128 | ||
+ | * debug server swallowtail | ||
+ | | ||
Line 674: | Line 1729: | ||
< | < | ||
+ | # env | ||
export PATH="/ | export PATH="/ | ||
export LD_LIBRARY_PATH="/ | export LD_LIBRARY_PATH="/ | ||
Line 687: | Line 1743: | ||
hdf5 | hdf5 | ||
kallisto | kallisto | ||
+ | |||
+ | |||
+ | # --- // | ||
+ | # Added packages below for Prof Coolon' | ||
+ | |||
+ | |||
+ | conda list | egrep -i " | ||
+ | bedtools | ||
+ | bowtie2 | ||
+ | cufflinks | ||
+ | samtools | ||
+ | |||
+ | # fcohan lab | ||
+ | [hmeij@petaltail ~]$ iqtree --version | ||
+ | IQ-TREE multicore version 2.1.4-beta COVID-edition for Linux 64-bit built Jun 24 2021 | ||
</ | </ | ||
+ | |||
+ | |||
**Trinity** assembles transcript sequences from Illumina RNA-Seq data. | **Trinity** assembles transcript sequences from Illumina RNA-Seq data. | ||
Line 879: | Line 1952: | ||
====== Galario ====== | ====== Galario ====== | ||
- | Newer GPU enabled | + | Newer CPU+GPU standalone library |
- | --- // | + | --- // |
+ | Newer CPU only version installed in Miniconda3 (top of page)\\ | ||
+ | Miniconda3 is using python 3.8\\ | ||
+ | --- // | ||
- | Accelerated Library for Analysing Radio Interferometer Observations | + | |
+ | Accelerated Library for Analysing Radio Interferometer Observations | ||
* / | * / | ||
* contains python 2.7.15 | * contains python 2.7.15 | ||
- | * packages galario, numpy, scipy, others (' | + | * packages galario, numpy, scipy, |
* packages mpi4py, emcee, astropy | * packages mpi4py, emcee, astropy | ||
Line 897: | Line 1974: | ||
# added by Miniconda2 installer | # added by Miniconda2 installer | ||
export PATH="/ | export PATH="/ | ||
+ | |||
+ | package | ||
+ | ---------------------------|----------------- | ||
+ | certifi-2021.5.30 | ||
+ | libblas-3.8.0 | ||
+ | libcblas-3.8.0 | ||
+ | liblapack-3.8.0 | ||
+ | libopenblas-0.3.10 | ||
+ | libstdcxx-ng-9.3.0 | ||
+ | numpy-1.21.0 | ||
+ | pandas-1.2.5 | ||
+ | pip-21.1.3 | ||
+ | python-3.7.3 | ||
+ | python-dateutil-2.8.1 | ||
+ | python_abi-3.7 | ||
+ | pytz-2021.1 | ||
+ | setuptools-49.6.0 | ||
+ | six-1.16.0 | ||
+ | wheel-0.36.2 | ||
+ | |||
</ | </ | ||
Line 1300: | Line 2397: | ||
[[https:// | [[https:// | ||
+ | |||
+ | * program v13,2.1 installed | ||
+ | |||
+ | * program v13.2 installed | ||
* program: mcc, math, mathematica (v 12.2) | * program: mcc, math, mathematica (v 12.2) | ||
Line 1497: | Line 2598: | ||
====== LAMMPS CPU/GPU ====== | ====== LAMMPS CPU/GPU ====== | ||
+ | |||
+ | ** 25Apr2023** | ||
+ | |||
+ | * / | ||
+ | * lmp_serial and lmp_mpi | ||
+ | * 3 cuda versions with 3 different precision settings | ||
+ | * notes: / | ||
+ | * does **not** support openmp, look for the OpenHPC version for that | ||
+ | * 05/24/2023 remade with package CREATION added | ||
+ | * also read miniconda3 section for tensorflow/ | ||
+ | |||
+ | < | ||
+ | |||
+ | # CPU | ||
+ | |||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | which mpirun mpicc python gcc | ||
+ | |||
+ | |||
+ | # CPU + GPU (exx96 queue, may run on test and amber128 queues) | ||
+ | |||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | which nvcc | ||
+ | |||
+ | # related for starrlab | ||
+ | # https:// | ||
+ | |||
+ | [hmeij@cottontail2 ~]$ module load miniconda3/ | ||
+ | [hmeij@cottontail2 ~]$ which pacemaker | ||
+ | / | ||
+ | [hmeij@cottontail2 ~]$ pacemaker --version | ||
+ | pacemaker/ | ||
+ | ace_evaluator | ||
+ | |||
+ | [hmeij@cottontail2 ~]$ python | ||
+ | Python 3.9.10 | packaged by conda-forge | (main, Feb 1 2022, 21: | ||
+ | [GCC 9.4.0] on linux | ||
+ | Type " | ||
+ | >>> | ||
+ | >>> | ||
+ | |||
+ | |||
+ | </ | ||
** USER_QUIP ** | ** USER_QUIP ** | ||
Line 2398: | Line 3550: | ||
* https:// | * https:// | ||
* C11++ on CentOS6, two version of gcc | * C11++ on CentOS6, two version of gcc | ||
- | * read the R-3.6.1_configure file in ''/ | + | * read the R-3.6.1_configure file in ''/ |
< | < | ||
Line 2554: | Line 3706: | ||
====== Amber ====== | ====== Amber ====== | ||
+ | |||
+ | ** Amber 22 ** | ||
+ | |||
+ | Details on how to run on CentOS7 can be found at | ||
+ | |||
+ | * https:// | ||
** Amber 20 ** | ** Amber 20 ** | ||
Line 2981: | Line 4139: | ||
< | < | ||
+ | |||
+ | # update | ||
+ | [hmeij@greentail52 ~]$ java --version | ||
+ | java 20.0.2 2023-07-18 | ||
+ | |||
+ | |||
export JAVAHOME=/ | export JAVAHOME=/ | ||
export CLASSPATH=$JAVAHOME/ | export CLASSPATH=$JAVAHOME/ | ||
Line 3335: | Line 4499: | ||
====== Matlab ====== | ====== Matlab ====== | ||
+ | * installed R2023a | ||
+ | * installed R2022a | ||
+ | |||
+ | * installed R2021a | ||
* program: Matlab, link to binary is ' | * program: Matlab, link to binary is ' | ||
Line 3355: | Line 4523: | ||
====== Stata ====== | ====== Stata ====== | ||
+ | |||
+ | * program stata[-mp|-se] in your $PATH, version 18 | ||
+ | * v17 will still run so add ' | ||
+ | * so that makes for 12 licenses 8-) | ||
* program: Stata v 17, 6-user network license, includes stata, stata-mp, stata-se | * program: Stata v 17, 6-user network license, includes stata, stata-mp, stata-se |