Articles under category Default Category

Published on 2025/08/25
Originally finished on 2024/10/03

Note: This method is not the most efficient, unless you would need large chunks of data. For downloading anything shorter than a whole month, I would actually recommend downloading using the cdsapi module in python as a script or actually use the CDS portal. You can download about 11 days of data on pressure levels in one go, and even more for data on the surface level. (I am planning to write more on that later.) This article is thus provided as-is and for your reference only.

Although NCAR’s repository for ERA5 data in grib was phased out, the data can still be somehow obtained here as for now: TDS Catalog.

ERA5 has different categories like surface (SFC) and pressure level (PL). They do not need to be separated to two streams of initial conditions.

Also, the variable LANDSEA which is a binary mask, is located in ERA5’s invariant category. If it is not downloaded and ungrib-ed, MPAS will complain:

ERROR: ********************************************************************************
ERROR: LANDSEA field not found in meteorological data file ERA5PL:2023-04-22_00
CRITICAL ERROR: ********************************************************************************
Logging complete.  Closing file at 2024/10/03 14:33:39

Check the Vtable for ERA5:

GRIB | Level| Level| Level| metgrid  |  metgrid | metgrid                                  |
Code | Code |   1  |   2  | Name     |  Units   | Description                              |
-----+------+------+------+----------+----------+------------------------------------------+
 129 | 100  |   *  |      | GEOPT    | m2 s-2   |                                          |
     | 100  |   *  |      | HGT      | m        | Height                                   |
 130 | 100  |   *  |      | TT       | K        | Temperature                              |
 131 | 100  |   *  |      | UU       | m s-1    | U                                        |
 132 | 100  |   *  |      | VV       | m s-1    | V                                        |
 157 | 100  |   *  |      | RH       | %        | Relative Humidity                        |
 165 |  1   |   0  |      | UU       | m s-1    | U                                        | At 10 m
 166 |  1   |   0  |      | VV       | m s-1    | V                                        | At 10 m
 167 |  1   |   0  |      | TT       | K        | Temperature                              | At 2 m
 168 |  1   |   0  |      | DEWPT    | K        |                                          | At 2 m
     |  1   |   0  |      | RH       | %        | Relative Humidity                        | At 2 m
 172 |  1   |   0  |      | LANDSEA  | 0/1 Flag | Land/Sea flag                            |
 129 |  1   |   0  |      | SOILGEO  | m2 s-2   |                                          |
...

GRIB Code for Land/Sea flag is 172.

{A87F2431-62FD-4C68-843F-DFCF7B561B92}.png

(https://thredds.rda.ucar.edu/thredds/catalog/files/g/d633000/e5.oper.invariant/197901/catalog.html)

It is in the invariant field.

Published on 2025/08/25
Originally finished on 2024/10/01

TL;DR: Use the existing modules on NSCC.

Note: NSCC is a local supercomputer. If your supercomputer does not have wps out-of-the-box or as a module, you might need to download and build it.

Get WPS

Don’t rebuild the wheel.

module load wps
module load wrf
cp -r /app/apps/wps <RELEVANT_PATH>

Then edit the namelist file.

For Vtable, refer to this website: WRF - Free Data.

Use WPS

First, use link_grib followed by grib files. It will generate files like GRIBFILE.AAA.

Then, edit the namelist.wps accordingly. Mainly, look for the start and end datetime and the file interval, and also the filename prefix in the &ungrib section.

Then, use ungrib.exe. If it complains, fix according to the error prompt.

ungrib.exe will look into GRIBFILES one by one and try to extract useful fields. Since not all GRIBFILE might be relevant, it is normal to see empty lines, something like

Inventory for date = 20XX-XX-XX XX:00:00

PRES   GEOPT    HGT      TT       UU       VV       RH       DEWPT    LANDSEA  SOILGEO  SOILHGT  PSFC     PMSL     SKINTEMP SEAICE   SST      SNOW_DEN SNOW_EC  SNOW     SNOWH    ST000007 ST007028 ST028100 ST100289 SM000007 SM007028 SM028100 SM100289
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------

Published on 2025/08/25
Originally finished on 2024/09/27

Problem:

problem screenshot

Appeared when attempting to run plot_delta_sst.ncl file in the MPAS tutorial:

fatal:ContourPlotPreDraw: Workspace reallocation would exceed maximum size 100000000

Fix:

https://web.archive.org/web/20250825061156/https://www.ncl.ucar.edu/Support/talk_archives/2010/2264.html

The easiest way to increase the size is to put a line like the following into your ~/.hluresfile:
*wsMaximumSize : 500000000

Originally published on 2024/09/12
Republished on luojied.org on 2025/08/26

A walkthrough of building MPAS 8.2.1 on the NTU Wildfly cluster.

If you are interested not just in “how to do” but “how do I come up with this way to do” as well, you might be interested in sections that begins with ‘🤔’ emoji.

1. Preparation

Log into Wildfly in your preferred terminal.

Submit a job (preferably on devqueue):

qsub -I -l walltime=2:00:00 -l select=1:ncpus=8 -N Build_MPAS -q dev -P <PROJECT_ID>

Wait for the job to run and you get the interactive shell.

Load the necessary modules:

module avail # this is to check all the modules
#==============================================#
module load intel/2024.1
module load netcdf-c/4.9.2/intel2024.1
netcdf-fortran/4.6.1/intel2024.1
icc/latest

🤔 How do I know which modules to load…?

The rationale is, MPAS depends on PIOor SMIOL for I/O. And these libraries depend on netcdf and parallel-netcdf (pnetcdf for short), which further depends on hdf5-parallel. So to make life easier, we look for higher level libs first, if they are not there, we fall back to a lower level and manually build the higher level.

After checking with module avail, we know that PIO isn’t there, but we can use the built-in SMIOL offered by MPAS version 8.0 onwards, it’s actually less of a headache. But seems there is no parallel-netcdf. So we have to build it.

Here intel/2024.1 is chosen instead of GNU’sgcc. This is because we know that we have to build pnetcdf, but that is built with the intel suite (notice the name netcdf-c/4.9.2/intel2024.1). If we use a different suite to build pnetcdf, say gcc and openmpi, because netcdfwasn’t built with them, when we build pnetcdf , the compiler would complain.

Check the loaded modules by module list . Make sure the output looks like the following:

[luojie@hpc-wfly-a010 pnetcdf-1.12.3]$ module list
Currently Loaded Modulefiles:
 1) intel/2024.1   2) netcdf-c/4.9.2/intel2024.1   3) netcdf-fortran/4.6.1/intel2024.1   4) compiler-rt/latest   5) icc/latest

Key:
auto-loaded

2. Build pnetcdf

Find a suitable place to download the pnetcdf’s source code.

Download using wget https://parallel-netcdf.github.io/Release/pnetcdf-1.12.3.tar.gz and then use tar -xvf pnetcdf-1.12.3.tar.gz to e[X]tract [V]erbosely the downloaded tarball [F]ile.

Go to the extracted directory, type the following line to configure . Don’t forget to change the path after the —-prefix flag to the desired install location!

CC=mpiicc FC=mpiifort CFLAGS=-fPIC ./configure --prefix=/home/luojie/MPAS-8.2.1/lib/pnetcdf --enable-relax-coord-bound

🤔 What does the command above mean…?

make is a tool to compile a program. Some programs comes with configure executable tool to allow customising before they are compiled. Because when it’s compiled, it is not as flexible anymore. This tool is usually generated by GNU’s autoconf tool, hence they usually have the same name configure .

We use these configurations because the make configuration of MPAS when build target = intel-mpi are the following:

"FC_PARALLEL = mpiifort" \
"CC_PARALLEL = mpiicc" \
"CXX_PARALLEL = mpiicpc" \
"FC_SERIAL = ifort" \
"CC_SERIAL = icc" \
"CXX_SERIAL = icpc" .

And you can check the presence of mpiicc by typing which mpiicc . For more information on configure, check the INSTALL file.

The -fPIC flag is to generate [P]osition [I]ndependent [C]ode.

Change the --prefix <PATH> flag to somewhere suitable for you.

And then use the following lines to install pnetcdf:

make -j8 # eight tasks in parallel, faster speed
make install

And then update the ~/.bash_profile file so that when MPAS is being built, it can find pnetcdf. Put the following lines into the bottom of ~/.bash_profile. And then copy and paste them to run in the terminal, for the current session.

# pnetCDF
export PATH=/home/luojie/MPAS-8.2.1/lib/pnetcdf/bin:$PATH
export LD_LIBRARY_PATH=/home/luojie/MPAS-8.2.1/lib/pnetcdf/lib:$LD_LIBRARY_PATH
export PNETCDF=/home/luojie/MPAS-8.2.1/lib/pnetcdf # required for PIO installation later

3. Build MPAS

It should be pretty trivial from here. Download the source code from https://github.com/MPAS-Dev/MPAS-Model/archive/refs/tags/v8.2.1.tar.gz using wget, use tarto extract, go to the extracted directory, and issue the following commands:

#MPAS
make clean CORE=init_atmosphere
make -j8 intel-mpi CORE=init_atmosphere PRECISION=double

make clean CORE=atmosphere
make -j8 intel-mpi CORE=atmosphere PRECISION=double

🤔 How to figure out that …?

You can run make first without any flags/arguments first. There will be a help menu with valuable information.

As a general rule of thumb for building MPAS at different locations with different compiler suites, when you are not sure with target you should choose, for example intel-mpi here, you can refer to the Makefile file to see which compilers each target use.

For example, I loaded intel/2024.1 previously. As a compiler suite, it must have some compiler executables (usually located /<PATH_TO_LIB>/bin, because they are [BIN]aries), the path of which, most of the time will be put into the $PATH variable (otherwise, what’s the point of loading this module?). I checked PATH by echo $PATH , and then immediately noticed /usr/local/intel/2024.1/vtune/2024.1/bin64:/usr/local/intel/2024.1/mpi/2021.12/bin .

Checking the contents of those directories reveals mpicc, mpiicc, mpifc, mpiifort and such. So I knew which compilers executables that I have. And comparing with MPAS’s Makefile I figured out I should use intel-mpi .

When you see this,

*******************************************************************************
MPAS was built with default double-precision reals.
Debugging is off.
Parallel version is on.
Using the mpi_f08 module.
Papi libraries are off.
TAU Hooks are off.
MPAS was built without OpenMP support.
MPAS was built without OpenMP-offload GPU support.
MPAS was built without OpenACC accelerator support.
Position-dependent code was generated.
MPAS was built with .F files.
The native timer interface is being used
Using the SMIOL library.
*******************************************************************************
make[1]: Leaving directory '/home/luojie/MPAS-8.2.1/MPAS-Model-8.2.1'

congratulations! You have built MPAS on the Wildfly cluster.