Purdue University

Compiling Serial Programs

A serial program is a single process which executes as a sequential stream of instructions on one processor core. Compilers capable of serial programming are available for C, C++, and versions of Fortran.

Here are a few sample serial programs:

• serial_hello.f
• serial_hello.f90
• serial_hello.f95
• serial_hello.c

• serial_hello.cpp

  To load a compiler, enter one of the following:

$ module load intel
$ module load gcc

$ ifort myprogram.f -o myprogram

$ gfortran myprogram.f -o myprogram

$ ifort myprogram.f90 -o myprogram

$ gfortran myprogram.f90 -o myprogram

$ ifort myprogram.f90 -o myprogram

$ gfortran myprogram.f95 -o myprogram

$ icc myprogram.c -o myprogram

$ gcc myprogram.c -o myprogram

$ icc myprogram.cpp -o myprogram

$ g++ myprogram.cpp -o myprogram

The Intel and GNU compilers will not output anything for a successful compilation. Also, the Intel compiler does not recognize the suffix ".f95".

Compiling MPI Programs

OpenMPI and Intel MPI (IMPI) are implementations of the Message-Passing Interface (MPI) standard. Libraries for these MPI implementations and compilers for C, C++, and Fortran are available on all clusters.

INCLUDE 'mpif.h'

INCLUDE 'mpif.h'

INCLUDE 'mpif.h'

#include <mpi.h>

#include <mpi.h>

Here are a few sample programs using MPI:

• mpi_hello.f
• mpi_hello.f90
• mpi_hello.f95
• mpi_hello.c
• mpi_hello.cpp

To see the available MPI libraries:

$ module avail openmpi 
$ module avail impi

$ mpiifort program.f -o program

$ mpif77 program.f -o program

$ mpiifort program.f90 -o program

$ mpif90 program.f90 -o program

$ mpiifort program.f95 -o program

$ mpif90 program.f95 -o program

$ mpiicc program.c -o program

$ mpicc program.c -o program

$ mpiicpc program.C -o program

$ mpiCC program.C -o program

The Intel and GNU compilers will not output anything for a successful compilation. Also, the Intel compiler does not recognize the suffix ".f95".

Here is some more documentation from other sources on the MPI libraries:

• Message Passing Interface Forum
• Open MPI Home
• Open MPI Documentation

Compiling OpenMP Programs

All compilers installed on Brown include OpenMP functionality for C, C++, and Fortran. An OpenMP program is a single process that takes advantage of a multi-core processor and its shared memory to achieve a form of parallel computing called multithreading. It distributes the work of a process over processor cores in a single compute node without the need for MPI communications.

INCLUDE 'omp_lib.h'

use omp_lib

use omp_lib

#include <omp.h>

#include <omp.h>

Sample programs illustrate task parallelism of OpenMP:

• omp_hello.f
• omp_hello.f90
• omp_hello.f95
• omp_hello.c
• omp_hello.cpp

A sample program illustrates loop-level (data) parallelism of OpenMP:

• omp_loop.c

To load a compiler, enter one of the following:

$ module load intel
$ module load gcc

$ ifort -openmp myprogram.f -o myprogram

$ gfortran -fopenmp myprogram.f -o myprogram

$ ifort -openmp myprogram.f90 -o myprogram

$ gfortran -fopenmp myprogram.f90 -o myprogram

$ ifort -openmp myprogram.f90 -o myprogram

$ gfortran -fopenmp myprogram.f95 -o myprogram

$ icc -openmp myprogram.c -o myprogram

$ gcc -fopenmp myprogram.c -o myprogram

$ icc -openmp myprogram.cpp -o myprogram

$ g++ -fopenmp myprogram.cpp -o myprogram

The Intel and GNU compilers will not output anything for a successful compilation. Also, the Intel compiler does not recognize the suffix ".f95".

Here is some more documentation from other sources on OpenMP:

• OpenMP Home
• Community of OpenMP Users
• Intel OpenMP
• GCC OpenMP

Compiling Hybrid Programs

A hybrid program combines both MPI and shared-memory to take advantage of compute clusters with multi-core compute nodes. Libraries for OpenMPI and Intel MPI (IMPI) and compilers which include OpenMP for C, C++, and Fortran are available.

INCLUDE 'omp_lib.h'
INCLUDE 'mpif.h'

use omp_lib
INCLUDE 'mpif.h'

use omp_lib
INCLUDE 'mpif.h'

#include <mpi.h>
#include <omp.h>

#include <mpi.h>
#include <omp.h>

A few examples illustrate hybrid programs with task parallelism of OpenMP:

• hybrid_hello.f
• hybrid_hello.f90
• hybrid_hello.f95
• hybrid_hello.c
• hybrid_hello.cpp

This example illustrates a hybrid program with loop-level (data) parallelism of OpenMP:

• hybrid_loop.c

To see the available MPI libraries:

$ module avail impi
$ module avail openmpi

The following tables illustrate how to compile your hybrid (MPI/OpenMP) program. Any compiler flags accepted by Intel ifort/icc compilers are compatible with their respective MPI compiler.

$ mpiifort -openmp myprogram.f -o myprogram

$ mpiifort -openmp myprogram.f90 -o myprogram

$ mpiifort -openmp myprogram.f90 -o myprogram

$ mpiicc -openmp myprogram.c -o myprogram

$ mpiicpc -openmp myprogram.C -o myprogram

$ mpif77 -openmp myprogram.f -o myprogram

$ mpif90 -openmp myprogram.f90 -o myprogram

$ mpif90 -openmp myprogram.f90 -o myprogram

$ mpicc -openmp myprogram.c -o myprogram

$ mpiCC -openmp myprogram.C -o myprogram

$ mpif77 -fopenmp myprogram.f -o myprogram

$ mpif90 -fopenmp myprogram.f90 -o myprogram

$ mpif90 -fopenmp myprogram.f95 -o myprogram

$ mpicc -fopenmp myprogram.c -o myprogram

$ mpiCC -fopenmp myprogram.C -o myprogram

The Intel and GNU compilers will not output anything for a successful compilation. Also, the Intel compiler does not recognize the suffix .f95.

Intel MKL Library

Intel Math Kernel Library (MKL) contains ScaLAPACK, LAPACK, Sparse Solver, BLAS, Sparse BLAS, CBLAS, GMP, FFTs, DFTs, VSL, VML, and Interval Arithmetic routines. MKL resides in the directory stored in the environment variable MKL_HOME, after loading a version of the Intel compiler with module.

By using module load to load an Intel compiler your environment will have several variables set up to help link applications with MKL. Here are some example combinations of simplified linking options:

$ module load intel
$ echo $LINK_LAPACK
-L${MKL_HOME}/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread

$ echo $LINK_LAPACK95
-L${MKL_HOME}/lib/intel64 -lmkl_lapack95_lp64 -lmkl_blas95_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread

RCAC recommends you use the provided variables to define MKL linking options in your compiling procedures. The Intel compiler modules also provide two other environment variables, LINK_LAPACK_STATIC and LINK_LAPACK95_STATIC that you may use if you need to link MKL statically.

RCAC recommends that you use dynamic linking of libguide. If so, define LD_LIBRARY_PATH such that you are using the correct version of libguide at run time. If you use static linking of libguide, then:

• If you use the Intel compilers, link in the libguide version that comes with the compiler (use the -openmp option).
• If you do not use the Intel compilers, link in the libguide version that comes with the Intel MKL above.

Here are some more documentation from other sources on the Intel MKL:

• Intel MKL Documentation

Provided Compilers

Compilers are available on Bell for Fortran, C, and C++. Compiler sets from Intel and GNU are installed.

Detailed documentation on each compiler set available on Bell follows.

On Bell, the following set of compiler and libraries for building code are recommended:

• GCC 9.3.0
• OpenMPI

To load the recommended set:

$ module load rcac
$ module list

More information about using these compilers:

Compiling GPU Programs on AMD GPUs

The Bell cluster nodes contain 2 AMD GPUs that support ROCm, HIP, CUDA and OpenCL. See the detailed hardware overview for the specifics on the GPUs in Bell. This section focuses on using HIP and CUDA with the ecosystem of ROCm drivers, libraries and compiler tools (including conversion tools that can transform existing CUDA codes to run on both Nvidia and AMD GPU hardware).

A simple HIP program has a basic workflow:

• Initialize an array on the host (CPU).
• Copy array from host memory to GPU memory.
• Apply an operation to array on GPU.
• Copy array from GPU memory to host memory.

Here is a sample HIP program:

• gpu_hello_hip.cpp

Both front-ends and GPU-enabled compute nodes have the ROCm HIP tools and libraries available to compile HIP programs. To compile a HIP program, load ROCm module, and use hipcc to compile the program:

$ module load rocm
$ hipcc gpu_hello.cpp -o gpu_hello
./gpu_hello
No GPU specified, using first GPUhello, world

The above example illustrates only how to copy an array between a CPU and its GPU but does not perform a serious computation.

The following example illustrates conversion of an existing CUDA-based code to HIP programming model so that it could then be compiled and executed on AMD GPUs. The program times three square matrix multiplications on a CPU and on the global and shared memory of a GPU:

• mm.cu

$ module load rocm
# Convert CUDA to HIP
$ hipify-perl --inplace mm.cu

# Compile with HIP compiler and run!
$ hipcc mm.cu -o mm
$ ./mm 0
                                                            speedup
                                                            -------
Elapsed time in CPU:                    7900.3 milliseconds
Elapsed time in GPU (global memory):      13.9 milliseconds  568.7
Elapsed time in GPU (shared memory):       6.4 milliseconds  1230.8

For best performance, the input array or matrix must be sufficiently large to overcome the overhead in copying the input and output data to and from the GPU.

For more information about AMD, ROCm, HIP, and GPUs:

• ROCm and HIP Learning Center (Webinars and Presentations)
• ROCm Platform Documentation
• Introduction to HIP
• AMD