Third-Party Software

Installing Your Own Software

Individuals and research groups may install third party applications and libraries for their own use in the following locations:

  • $HOME
  • /usr/local/unsupported

Packages built for personal use only should be installed in $HOME.

The /usr/local/unsupported directory is intended to host user-installed and maintained software packages and datasets that are shared with a group of users on the system. Users who add content to /usr/local/unsupported are fully responsible for the maintenance of the files and software versions. Please read the /usr/local/unsupported/README.RCS file for more information.

To request a new subdirectory within /usr/local/unsupported, please contact RCS with the following information:

  • The name of the requested subdirectory, which can be your project's name (e.g., UAFCLMIT) or the type of software you intend to install in the directory (e.g., "ClimateModels")
  • A general description of what you intend to install
  • A rough estimate of the amount of storage you will need (e.g., 100 MB)

Using The Software Stack

Chinook already has builds of many third-party software packages (see below for a listing). There are often multiple builds of a particular software package - different versions, different compilers used to build the software, different compile-time flags, et cetera. To avoid conflicts between these many disparate package builds, Chinook employs an environment module system you can use to load and unload different combinations of software packages into your environment.

What are Environment Modules?

The environment modules found on Chinook (often referred to simply as "modules") are Tcl script files that are used to update shell environment variables such as PATH, MANPATH, and LD_LIBRARY_PATH. These variables allow your shell to discover the particular application or library as specified by the module. Some environment modules set additional variables (such as PYTHONPATH or PERL5LIB), while others simply load a suite of other modules.

Common module commands

Command Result
module avail list all available modules
module avail pkg list all available modules beginning with the string pkg
module load pkg load a module named pkg
module swap old new attempt to replace loaded module named old with one named new
module unload pkg unload a module named pkg
module list list all currently-loaded modules
module purge unload all modules
module show pkg summarize environment changes made by module named pkg (sometimes incomplete)

Searching for modules

Because module avail will search for the provided string only at the beginning of a module's fully-qualified name, it can be difficult to use module avail to search for modules nested in any kind of hierarchy. This is the case on Chinook - modules are categorized, then named. Here are some examples:

  • compiler/GCC/version
  • devel/CMake/version
  • math/GMP/version

To find modules for GCC using a pure module avail command, you would need to run module avail compiler/GCC. This is difficult, because you must already know that the module is in the compiler category.

To make things more complicated, module avail is also case-sensitive. Running module avail devel/cmake will not find the module named devel/CMake/version.

Better module searching

One workaround for these impediments is to combine module avail output with grep's full-text case-insensitive string matching ability. The example below additionally uses Bash file descriptor redirection syntax to redirect stderr to stdout because module avail outputs to stderr.

module avail --terse 2>&1 | grep -i pkg

replacing pkg with the string you are searching for.

RCS is currently evaluating Lmod as a replacement for Chinook's current environment modules framework. Lmod has many desirable features, including but not limited to a more user-friendly module avail behavior.

For more information on Chinook's module framework, please visit http://modules.sourceforge.net/index.html.


Compiler Toolchains

Compiler toolchains are modules that bundle together a set of compiler, MPI, and numerical library modules. To use a compiler toolchain, load the compiler toolchain module and all the submodules will be loaded. This will set variables such as PATH, CPATH, LIBRARY_PATH, LD_LIBRARY_PATH, and others. Other variable conventions such as CC and CXX are not automatically defined.

Since Chinook is an Intel-based HPC cluster, RCS defaults to compiling software using Intel-based compiler toolchains.

Toolchain Name Version Comprises
foss 2016b GNU Compiler Collection 5.4.0, Penguin-modified OpenMPI 1.10.2, OpenBLAS 0.2.18, FFTW 3.3.4, ScaLAPACK 2.0.2
pic-intel 2016b Intel Compiler Collection 2016.3.210 (2016 update 3), Penguin-modified OpenMPI 1.10.6, Intel Math Kernel Library (MKL) 11.3.3.210




MPI Libraries

RCS defaults to compiling software against OpenMPI.

Name Version Compiled by Notes
Intel GCC
MPICH2 1.5
MVAPICH2 2.1
MVAPICH2-PSM 2.1
OpenMPI 1.10.2 Included in pic-intel, pic-foss compiler toolchains
OpenMPI 1.6.5
OpenMPI 1.7.5
OpenMPI 1.8.8




Maintained Software Installations

As of 2016-10-19.

Name Version Compiled by Notes
Intel GCC
Anaconda 2 4.0.0
Anaconda 3 2.5.0
Autoconf 2.69
Automake 1.15
Autotools 20150215
BamTools 2.4.0
BayeScan 2.1
BCFtools 1.3.1
binutils 2.26 Included in pic-foss compiler toolchain
Bison 3.0.4
Boost 1.61.0
BWA 0.7.15
bzip2 1.0.6
cairo 1.14.6
CMake 3.5.2
cURL 7.49.1
Doxygen 1.8.11
ESMF 7.0.0
expat 2.2.0
FASTX-Toolkit 0.0.14
FFTW 3.3.4 Included in pic-foss compiler toolchain
flex 2.6.0
fontconfig 2.12.1
freetype 2.6.5
g2clib 1.4.0
g2lib 1.4.0
GCC 5.4.0 Included in pic-foss compiler toolchain
GDAL 2.1.0
gettext 0.19.8
GLib 2.49.5
GMP 6.1.1
GSL 2.1
gzip 1.6
HDF 4.2.11
HDF5 1.8.17
HTSlib 1.3.1
icc 2016.3.210 Included in pic-intel compiler toolchain
idl 8.4.1
ifort 2016.3.210 Included in pic-intel compiler toolchain
imkl 11.3.3.210 Included in pic-intel compiler toolchain
JasPer 1.900.1
libffi 3.2.1
libgtextutils 0.7
libjpeg-turbo 1.5.0
libpng 1.6.24
libreadline 6.3
libtool 2.4.6
libxml2 2.9.4
M4 1.4.17
makedepend 1.0.5
MATLAB R2015a
MATLAB R2015b
MATLAB R2016a
Mothur 1.38.1.1
NASM 2.12.02
NCL 6.3.0 Binary distribution
ncurses 6.0
netCDF 4.4.1
netCDF-C++4 4.3.0
netCDF-Fortran 4.4.4
OpenBLAS 0.2.18 Included in pic-foss compiler toolchain
PCRE 8.39
Perl 5.22.1
pixman 0.34.0
pkg-config 0.29.1
Python 2.7.12
Rosetta 3.8
SAMtools 1.3.1
ScaLAPACK 2.0.2 Included in pic-foss compiler toolchain
Singularity 2.2
SQLite 3.13.0
Szip 2.1
Tcl 8.6.5
Tk 8.6.5
UDUNITS 2.2.20
VCFtools 0.1.14
X11 20160819
XZ 5.2.2
zlib 1.2.8




Software requests

RCS evaluates third-party software installation requests for widely-used HPC software on a case-by-case basis. Some factors that affect request eligibility are:

  • Applicability to multiple research groups
  • Complexity of the installation process
  • Software licensing

If a third-party software installation request is found to be a viable candidate for installation, RCS may elect to install the software through one of several means:

  • RPM
  • Binary (pre-built) distribution
  • Source build

If an application or library is available through standard RPM repositories (Penguin Computing, CentOS, EPEL, ...) then the RPM may be installed. Users should test the installed software to determine if it meets requirements. If the RPM version does not meet needs, please contact RCS to have alternate installation methods evaluated.

Software that is not installed as an RPM will be installed in a publicly-available location and be accessible via Linux environment modules. If the software is built from source, then RCS will default to using the Intel compiler suite.

MATLAB Usage

Policy

Due to the shared nature of the Chinook login nodes RCS requests that MATLAB be run on the RCS Linux workstations for data processing and on the Chinook compute nodes for computationally intensive tasks. Please see the Remote Login page for more information on running MATLAB on the workstations and the MATLAB parallel computing section for information on running MATLAB on the compute nodes.

MATLAB sessions that are launched on the Chinook login nodes may be killed at RCS discretion due to these jobs affecting all users on the login nodes.

MATLAB Parallel Computing

MATLAB jobs can execute tasks in parallel either using 24 cores on a single node, which requires no additional setup, or multiple cores on multiple nodes which may require some settings to be configured. Jobs on a single node can be submitted through Slurm, while jobs that use multiple nodes must be submitted through a MATLAB session.

Parallel Jobs

Parallel jobs in MATLAB run on "parallel pools" which is a collection of MATLAB workers that will run tasks on the Chinook cluster. Some useful commands and things to keep in mind are the following:

  • MATLAB workers do not have access to any graphical capabilities so data cannot be plotted in parallel
  • Parallel jobs make use of the the parfor, parfeval, and/or the spmd MATLAB directives
  • parfor is used for loops that are not interdependent, and each iteration of the loop will be run on a separate core
  • pareval is used for functions that can be run asynchronously
  • spmd stands for Single Program Multiple Data, and is for functions that operate on different data, but use the same code. This can spread each instance to its own worker, doing work in parallel. This is most often accomplished with the use of the labindex, the ID of an individual worker

For more information please see the MATLAB Parallel Computing documentation

Single Node

Jobs that use only a single node (up to 23 workers or threads in the debug or small queue) may be submitted directly to the Slurm batch scheduler. When submitting a job directly to Slurm the parpool profile must be set to 'local' to use the CPUs available to the compute node. Do not use the 'local' profile when submitting jobs through MATLAB or on the login nodes or Workstations as this will use as many cores as specified and will affect all users on a system.

Submitting a MATLAB script to be run on compute nodes requires using the Batch Submission System. Your MATLAB script must initially set up a parpool which will launch the set of threads to be used for a parallel process. For example the following MATLAB script will create a parpool to generate a set of random numbers:

    
%============================================================================
% parallelTest.m - Parallel Random Number - A Trivial Example
%============================================================================
% this will read in the "ntasks" provided by the Slurm batch submission script
parpool('local', str2num(getenv('SLURM_NTASKS')))
size = 1e7;
endMatrix = [];
tic % Start timing
parfor i = 1:size
    % Work that is independent of each task in the loop goes here
    endMatrix = [endMatrix; rand]
end
timing = toc; % end timing
fprintf('Timing: %8.72f seconds.n', timing);
delete(gcp); % Clean up the parallel pool of tasks
exit;

To submit this to the Chinook compute nodes a batch script must be created:

    
#!/bin/bash
#SBATCH --partition=$PARTITION
#SBATCH --ntasks=24
#SBATCH --tasks-per-node=24
#SBATCH --time=DD:HH:MM:SS
#SBATCH --job-name=$JOBNAME
module purge
module load slurm
module load matlab/R2016b
cd $SLURM_SUBMIT_DIR
matlab -nosplash -nodesktop -r "/path/to/parallelTest"

If this file were named parallelMatlab.slurm you would then submit it with:
chinook00 % sbatch parallelMatlab.slurm

Multiple Nodes

MATLAB jobs that use multiple nodes must be run in an interactive session. Some settings may need to be set by a user to allow MATLAB to interactively submit jobs to the Slurm batch scheduler.

Add the Slurm folder to MATLAB's PATH

MATLAB GUI

  • Click the Set Path button in the toolbar
  • Check if /import/usrlocal/pkg/matlab/matlab-R2016b/toolbox/local/slurm is in your path
  • If not
    • Click Add Folder
    • Type in or copy /usr/local/pkg/matlab/matlab-R2016b/toolbox/local/slurm to the Folder name field
    • Click Save, save pathdef.m to $HOME/.matlab/R2016b/pathdef.m

MATLAB Command Line

  • Run the path function
  • Check to see if /import/usrlocal/pkg/matlab/matlab-R2016b/toolbox/local/slurm is in your path
  • If not run addpath('/usr/local/pkg/matlab/matlab-R2016b/toolbox/local/slurm');

Import the MATLAB Parallel Profile

To use the MATLAB DCS you will need to import the Parallel Profile for Chinook. This can be done through the MATLAB GUI or the command line.

MATLAB GUI

  • Click the Parallel drop down menu in the toolbar
  • Click Manage Cluster Profiles
  • Click Import
  • Navigate to /usr/local/pkg/matlab/slurm_scripts/
  • Select chinook.settings and click Open

You should now have debug, t1small, t1standard, t2small, and t2standard in your Cluster Profile Window

Command Line

                
>> clusterProfile = parallel.importProfile('/usr/local/pkg/matlab/slurm_scripts/chinook');
>> clusterProfile
            
            

You should see a 1x5 cell array containing 'debug','t1small','t1standard','t2small', and 't2standard'

NumWorkers and NumThreads

Two other variables need to be set before running a job on the MATLAB DCS: NumWorkers and NumThreads.
NumWorkers is the number of MATLAB Workers available to the job. By default each Worker is assigned to a core on a node. Our license allows 64 Workers to be used across all users. You will need one more Worker than the number that your job requires because one Worker is dedicated to managing the others. For a two node job you can use up to 47 Workers for computations, and for three node jobs, 63 Workers.

Multiple threads can be assigned to a worker, if a Worker can benefit from multiple threads. NumWorkers * NumThreads should be less than the total number of cores that are available. If not the job will use every core and may have unexpected behavior. There are two ways to set NumWorkers and NumThreads:

MATLAB GUI
  • Click on the Parallel button in the toolbar
  • Click on Manage Cluster Profiles
  • Click on the Profile you wish to edit
  • Click on Edit in the bottom right corner of the window
  • In Number of workers available to cluster NumWorkers enter the number of workers (must be less than 64)
  • In Number of computation threads to use on each worker NumThreads enter the number of threads
Command Line
                
                myCluster = parcluster('$PARTITION');
                myCluster.NumWorkers = $NumWorkers; %less than 64
                myCluster.NumThreads = $NumThreads
            
            

Running Jobs

Parallel jobs can be run during an interactive session of MATLAB by first setting up a Parallel pool on a partition. The Parallel pool will only start if there are nodes available. Starting a parallel pool can be started with the following commands in MATLAB:
myCluster = parcluster('$PROFILE');
where $PROFILE is the name of one of the queues on Chinook (debug, t1/t2standard, t1/t2small). This will start a parallel pool where can be done. Scripts or functions that are run in the interactive session of MATLAB that use parfor, parfeval, or spmd will run that code in the parallel pool that you have open.

When you are done with the parallel pool you will have to close it using the following command in MATLAB:
delete(gcp);

Submitting Jobs

You can also submit jobs to the queue using MATLAB. The profiles that can be submitted to in this way are debug, t1small, t1standard, t2small, and t2standard, where each profile corresponds to a partition available on Chinook. For more information about the partitions on Chinook please see our partition overview.

To submit the job you will need to call the MATLAB batch function. The example below shows how to submit a function to be run on the compute nodes:

            
%Leaving this option blank will use the chosen default profile%
myCluster = parcluster('$PROFILE');

%Run the script and pass its arguments. 'pool' sets the number workers%
%numWorkers must be at most 1 less than the total number of workers allocated in the profile%
%because MATLAB uses one additional worker to manage all the others%
myJob = myCluster.batch(@$FUNCTION,NumberOfOutputs,{input1,input2....,inputN},'pool',$NUMWORKERS);

%Wait for the job to finish before doing anything else%
wait(myJob);

%If submitted interactively do work on the output matrices%
WORK HERE

%Clean up the job%
destroy(myJob);
delete(gcp); 
            
        

where $PROFILE is the partition you wish to use, $FUNCTION is the function, in a .m file, that you wish to run, and $NUMWORKERS is the number of Workers you wish to use.

Customizing sbatch Parameters for MATLAB Job Submission

For multi-node jobs you may have to modify the CommunicatingSubmitFunction may need to be modified to shorten the Walltime on a job. The steps to do this are the following:

  • Create a location for your custom MATLAB CommunicatingSubmitFunction mkdir ~/matlab_scripts for example
  • Copy communicatingSubmitFcn.m from /usr/local/pkg/matlab/slurm_scripts/communicatingSubmitFcn.m ~/matlab_scripts/customCommunicatingSubmitFcn.m
  • For clarity name your personal copy of communicatingSubmitFcn.m to something other than communicatingSubmitFcn.m to insure your profile calls the correct function.

  • Modify the name of the function in customCommunicatingSubmitFcn.m as well
  • Modify the following line in your customCommunicatingSubmitFcn.m:
    
    additionalSubmitArgs = sprintf('--partition=t1standard --ntasks=%d', props.NumberOfTasks);
                    
                

    and add in --time=D-HH:MM:SS to the section in sprintf like so:

    
    additionalSubmitArgs = sprintf('--partition=t1standard --time=1-12:00:00 --ntasks=%d', props.NumberOfTasks);
                
            
  • After you've made your parcluster run the following command: set(myCluster, 'CommunicatingSubmitFcn', @customCommunicatingSubmitFcn); and run your job as normal