Originally published on 2024/09/12
Republished on luojied.org on 2025/08/26

A walkthrough of building MPAS 8.2.1 on the NTU Wildfly cluster.

If you are interested not just in “how to do” but “how do I come up with this way to do” as well, you might be interested in sections that begins with ‘🤔’ emoji.

1. Preparation

Log into Wildfly in your preferred terminal.

Submit a job (preferably on devqueue):

qsub -I -l walltime=2:00:00 -l select=1:ncpus=8 -N Build_MPAS -q dev -P <PROJECT_ID>

Wait for the job to run and you get the interactive shell.

Load the necessary modules:

module avail # this is to check all the modules
#==============================================#
module load intel/2024.1
module load netcdf-c/4.9.2/intel2024.1
netcdf-fortran/4.6.1/intel2024.1
icc/latest

🤔 How do I know which modules to load…?

The rationale is, MPAS depends on PIOor SMIOL for I/O. And these libraries depend on netcdf and parallel-netcdf (pnetcdf for short), which further depends on hdf5-parallel. So to make life easier, we look for higher level libs first, if they are not there, we fall back to a lower level and manually build the higher level.

After checking with module avail, we know that PIO isn’t there, but we can use the built-in SMIOL offered by MPAS version 8.0 onwards, it’s actually less of a headache. But seems there is no parallel-netcdf. So we have to build it.

Here intel/2024.1 is chosen instead of GNU’sgcc. This is because we know that we have to build pnetcdf, but that is built with the intel suite (notice the name netcdf-c/4.9.2/intel2024.1). If we use a different suite to build pnetcdf, say gcc and openmpi, because netcdfwasn’t built with them, when we build pnetcdf , the compiler would complain.

Check the loaded modules by module list . Make sure the output looks like the following:

[luojie@hpc-wfly-a010 pnetcdf-1.12.3]$ module list
Currently Loaded Modulefiles:
 1) intel/2024.1   2) netcdf-c/4.9.2/intel2024.1   3) netcdf-fortran/4.6.1/intel2024.1   4) compiler-rt/latest   5) icc/latest

Key:
auto-loaded

2. Build pnetcdf

Find a suitable place to download the pnetcdf’s source code.

Download using wget https://parallel-netcdf.github.io/Release/pnetcdf-1.12.3.tar.gz and then use tar -xvf pnetcdf-1.12.3.tar.gz to e[X]tract [V]erbosely the downloaded tarball [F]ile.

Go to the extracted directory, type the following line to configure . Don’t forget to change the path after the —-prefix flag to the desired install location!

CC=mpiicc FC=mpiifort CFLAGS=-fPIC ./configure --prefix=/home/luojie/MPAS-8.2.1/lib/pnetcdf --enable-relax-coord-bound

🤔 What does the command above mean…?

make is a tool to compile a program. Some programs comes with configure executable tool to allow customising before they are compiled. Because when it’s compiled, it is not as flexible anymore. This tool is usually generated by GNU’s autoconf tool, hence they usually have the same name configure .

We use these configurations because the make configuration of MPAS when build target = intel-mpi are the following:

"FC_PARALLEL = mpiifort" \
"CC_PARALLEL = mpiicc" \
"CXX_PARALLEL = mpiicpc" \
"FC_SERIAL = ifort" \
"CC_SERIAL = icc" \
"CXX_SERIAL = icpc" .

And you can check the presence of mpiicc by typing which mpiicc . For more information on configure, check the INSTALL file.

The -fPIC flag is to generate [P]osition [I]ndependent [C]ode.

Change the --prefix <PATH> flag to somewhere suitable for you.

And then use the following lines to install pnetcdf:

make -j8 # eight tasks in parallel, faster speed
make install

And then update the ~/.bash_profile file so that when MPAS is being built, it can find pnetcdf. Put the following lines into the bottom of ~/.bash_profile. And then copy and paste them to run in the terminal, for the current session.

# pnetCDF
export PATH=/home/luojie/MPAS-8.2.1/lib/pnetcdf/bin:$PATH
export LD_LIBRARY_PATH=/home/luojie/MPAS-8.2.1/lib/pnetcdf/lib:$LD_LIBRARY_PATH
export PNETCDF=/home/luojie/MPAS-8.2.1/lib/pnetcdf # required for PIO installation later

3. Build MPAS

It should be pretty trivial from here. Download the source code from https://github.com/MPAS-Dev/MPAS-Model/archive/refs/tags/v8.2.1.tar.gz using wget, use tarto extract, go to the extracted directory, and issue the following commands:

#MPAS
make clean CORE=init_atmosphere
make -j8 intel-mpi CORE=init_atmosphere PRECISION=double

make clean CORE=atmosphere
make -j8 intel-mpi CORE=atmosphere PRECISION=double

🤔 How to figure out that …?

You can run make first without any flags/arguments first. There will be a help menu with valuable information.

As a general rule of thumb for building MPAS at different locations with different compiler suites, when you are not sure with target you should choose, for example intel-mpi here, you can refer to the Makefile file to see which compilers each target use.

For example, I loaded intel/2024.1 previously. As a compiler suite, it must have some compiler executables (usually located /<PATH_TO_LIB>/bin, because they are [BIN]aries), the path of which, most of the time will be put into the $PATH variable (otherwise, what’s the point of loading this module?). I checked PATH by echo $PATH , and then immediately noticed /usr/local/intel/2024.1/vtune/2024.1/bin64:/usr/local/intel/2024.1/mpi/2021.12/bin .

Checking the contents of those directories reveals mpicc, mpiicc, mpifc, mpiifort and such. So I knew which compilers executables that I have. And comparing with MPAS’s Makefile I figured out I should use intel-mpi .

When you see this,

*******************************************************************************
MPAS was built with default double-precision reals.
Debugging is off.
Parallel version is on.
Using the mpi_f08 module.
Papi libraries are off.
TAU Hooks are off.
MPAS was built without OpenMP support.
MPAS was built without OpenMP-offload GPU support.
MPAS was built without OpenACC accelerator support.
Position-dependent code was generated.
MPAS was built with .F files.
The native timer interface is being used
Using the SMIOL library.
*******************************************************************************
make[1]: Leaving directory '/home/luojie/MPAS-8.2.1/MPAS-Model-8.2.1'

congratulations! You have built MPAS on the Wildfly cluster.