Purdue University

Software

Link to section 'Module System' of 'Software' Module System

The Anvil cluster uses Lmod to manage the user environment, so users have access to the necessary software packages and versions to conduct their research activities. The associated module command can be used to load applications and compilers, making the corresponding libraries and environment variables automatically available in the user environment. The module software used on Anvil, Lmod, is a hierarchical module system, meaning a module can only be loaded after loading the necessary compilers and MPI libraries that it depends on. This helps avoid conflicting libraries and dependencies being loaded at the same time. A list of all available modules on the system can be found with the “module spider” command. The “module spider” command can also be used to search for specific module names. When users log into Anvil, a default compiler (GCC), MPI libraries (OpenMPI), and runtime environments (e.g., Cuda on GPU-nodes) are automatically loaded into the user environment. It is recommended that users explicitly specify which modules and which versions are needed to run their codes in their job scripts via the “module load” command. Users are advised not to insert “module load” commands in their bash profiles, as this can cause issues during initialization of certain software (e.g. Thinlinc).

Most modules on Anvil include extensive help messages, so users can take advantage of the “module help” command to find information about a particular application or module. Every module also contains two environment variables named $RCAC_APPNAME_ROOT and $RCAC_APPNAME_VERSION identifying its installation prefix and its version. Users are encouraged to use generic environment variables such as CC, CXX, FC, MPICC, MPICXX etc. available though the compiler and MPI modules while compiling their code.

Link to section 'Compiling, performance, and optimization on Anvil' of 'Software' Compiling, performance, and optimization on Anvil

Anvil CPU nodes have GNU, Intel, and AOCC (AMD) compilers available along with multiple MPI implementations (OpenMPI and Intel MPI). Anvil GPU nodes will also provide the PGI compiler. Users may want to note the following AMD Milan specific optimization options that can help improve the performance of your code on Anvil:

1. The majority of the applications on Anvil will be built using gcc/10.2.0 which features an AMD Milan specific optimization flag (-march=znver2).
2. AMD Milan CPUs support the Advanced Vector Extensions 2 (AVX2) vector instructions set. GNU, Intel, and AOCC compilers all have flags to support AVX2. Using AVX2, up to eight floating point operations can be executed per cycle per core, potentially doubling the performance relative to non-AVX2 processors running at the same clock speed.
3. In order to enable AVX2 support, when compiling your code, use the -march=znver2 flag (for GCC 10.2, Clang and AOCC compilers) or -march=core-avx2 (for Intel compilers and GCC prior to 9.3).

Other Software Usage Notes:

1. Use the same environment that you compile the code to run your executables. When switching between compilers for different applications, make sure that you load the appropriate modules before running your executables.
2. Explicitly set the optimization level in your makefiles or compilation scripts. Most well written codes can safely use the highest optimization level (-O3), but many compilers set lower default levels (e.g. GNU compilers use the default -O0, which turns off all optimizations).
3. Turn off debugging, profiling, and bounds checking when building executables intended for production runs as these can seriously impact performance. These options are all disabled by default. The flag used for bounds checking is compiler dependent, but the debugging (-g) and profiling (-pg) flags tend to be the same for all major compilers.
4. Some compiler options are the same for all available compilers on Anvil (e.g. "-o"), while others are different. Many options are available in one compiler suite but not the other. For example, Intel, PGI, and GNU compilers use the -qopenmp, -mp, and -fopenmp flags, respectively, for building OpenMP applications.
5. MPI compiler wrappers (e.g. mpicc, mpif90) all call the appropriate compilers and load the correct MPI libraries depending on the loaded modules. While the same names may be used for different compilers, keep in mind that these are completely independent scripts.

For Python users, Anvil provides two Python distributions: 1) a natively compiled Python module with a small subset of essential numerical libraries which are optimized for the AMD Milan architecture and 2) binaries distributed through Anaconda. Users are recommended to use virtual environments for installing and using additional Python packages.

A broad range of application modules from various science and engineering domains will be installed on Anvil, including mathematics and statistical modeling tools, visualization software, computational fluid dynamics codes, molecular modeling packages, and debugging tools. Some of these packages are listed in the table below:

In addition, Singularity will be supported on Anvil and Nvidia GPU Cloud containers are available on Anvil GPU nodes.