Esm4.5 public release candidate

.github/workflows/docker-build-deploy.yml

@@ -0,0 +1,54 @@
+name: Build and Deploy Container to GHCR
+
+on:
+  push:
+    branches: [ "main" ]
+    tags: [ '*' ] # Triggers the workflow when any Git tag is pushed
+  workflow_dispatch: # Allows manual triggering from the UI
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository }}
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write # Crucial: Allows the action to push to GHCR
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log in to the Container registry
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          tags: |
+            type=raw,value=latest,enable={{is_default_branch}}
+            type=sha
+            type=ref,event=tag # Extracts the Git tag and applies it to Docker
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          file: exec/Dockerfile
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          # Recommended for large builds: Utilize GitHub Actions cache
+          cache-from: type=gha
+          cache-to: type=gha,mode=max

.gitmodules

@@ -6,22 +6,31 @@
 	url = http://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere.git
 [submodule "src/atmos_drivers"]
 	path = src/atmos_drivers
-	url = http://github.com/NOAA-GFDL/atmos_drivers.git
+	url = https://github.com/NOAA-GFDL/atmos_drivers.git
 [submodule "src/atmos_phys"]
 	path = src/atmos_phys
 	url = http://github.com/NOAA-GFDL/atmos_phys.git
-[submodule "src/coupler"]
-	path = src/coupler
-	url = http://github.com/NOAA-GFDL/coupler.git
+[submodule "src/FMScoupler"]
+	path = src/FMScoupler
+	url = http://github.com/NOAA-GFDL/FMScoupler.git
 [submodule "src/ice_param"]
 	path = src/ice_param
 	url = http://github.com/NOAA-GFDL/ice_param.git
-[submodule "src/mom6"]
-	path = src/mom6
-	url = https://github.com/NOAA-GFDL/MOM6-examples.git
-[submodule "src/ocean_shared"]
-	path = src/ocean_shared
-	url = https://github.com/NOAA-GFDL/ocean_BGC
+[submodule "src/MOM6"]
+	path = src/MOM6
+	url = https://github.com/NOAA-GFDL/MOM6.git
+[submodule "src/SIS2"]
+	path = src/SIS2
+	url = https://github.com/NOAA-GFDL/SIS2.git
+[submodule "src/icebergs"]
+	path = src/icebergs
+	url = https://github.com/NOAA-GFDL/icebergs.git
+[submodule "src/ocean_BGC"]
+	path = src/ocean_BGC
+	url = https://github.com/NOAA-CEFI-Regional-Ocean-Modeling/ocean_BGC.git
 [submodule "src/lm4p"]
 	path = src/lm4p
 	url = https://github.com/NOAA-GFDL/lm4.git
+[submodule "src/mkmf"]
+	path = src/mkmf
+	url = https://github.com/NOAA-GFDL/mkmf.git

README.md

@@ -2,7 +2,7 @@
 
 ## What Is Included
 * [src]((https://github.com/NOAA-GFDL/ESM4/tree/master/src) source code for the ESM4 model (all code is in submodules)
-* [exec]((https://github.com/NOAA-GFDL/ESM4/tree/master/exec) Makefiles to compile the code 
+* [exec]((https://github.com/NOAA-GFDL/ESM4/tree/master/exec) Files to compile the code in a container
 * [run]((https://github.com/NOAA-GFDL/ESM4/tree/master/run) Simple run script
 
 ## Cloning
@@ -15,81 +15,108 @@ or
 git clone --recursive https://github.com/NOAA-GFDL/ESM4.git
 ```
 
-## Compiling
+## Container
 
-### Building the container
-The [container folder](container) provides example Dockerfiles and Signularity
-definition files to use to build AM4 containers using either GCC/GFORTAN or
-Intel oneAPI. There is a script that can be used to build the intel
-singularity containers, and the first step of this script can be used with the
-other GFDL climate models.
+### Pulling the container
+There is a container provided that has the application used to 
+run the ESM4.5 model. The container for this model can be pulled 
+using a container tool. Here is an example using apptainer
+```
+apptainer pull docker://ghcr.io/noaa-gfdl/esm4:esm4.5
+```
+This is an MPI application which is built using Intel ifx and icx
+compilers with intel-mpi for an x86 system.  The container *should*
+be able to run on an x86 system with avx instructions, an ABI 
+compatible MPI (mpich), and a compatible glib-C installation. The
+[Dockerfile](exec/Dockerfile) contains the details of the dependencies,
+compilers, and application build.
+
+### Running the container
+When this model is run as a container, it needs to be executed using
+an MPI-bind strategy with apptainer or whatever container launching
+tool you are using. You will need to ensure that the system MPI and
+communication library paths are included in the bind paths at execution.
+
+Additionally, it is recommended that when updating the 
+`LD_LIBRARY_PATH` in the container, you preserve the container's
+`LD_LIBRARY_PATH`.  Here is an example using Apptainer
+``` bash
+module unload netcdf-c netcdf-fortran hdf5
+module load mpich
+export APPTAINERENV_LD_LIBRARY_PATH=/path/to/mpi/lib:/path/to/libfabric/lib:\$LD_LIBRARY_PATH
+```
+By including the '\' before the '$', the system MPI will be found
+and the container `LD_LIBRARY_PATH` will be preserved inside the container.
 
+### MPICH
+This container will only run on X86 systems with mpich.  If you are
+using openMPI, you will need to modify the 
+[Dockerfile](exec/Dockerfile) to use openMPI instead of Intel-mpi.
 
-### Building from source
-This model was originally compiled and run with the intel16 compiler.
-It is recommended that you compile with an intel compiler.
-
-Compiling assumes that you have an intel compiler, MPI (impi, mpich,
-openmpi, etc), netcdf, and hdf5 in your LD_LIBRARY_PATH and LIBRARY_PATH.
-It is also assumed that nf-config and nc-config are in your path. 
-If you work on a machine with modules, you may need to load these 
-packages into your environment.
-
-Makefiles have been included in the 
-[exec/](https://github.com/NOAA-GFDL/ESM4/tree/master/exec) folder.
-There are several option for compiling, which can be found in the 
-[template/intel.mk](https://github.com/NOAA-GFDL/ESM4/blob/master/exec/templates/intel.mk).  
-You may need to edit the template/intel.mk to update the compiler names
-or add any CPPDEF options specific for your system.
-The most common compile with optimizations on and with openmp would be 
-```bash
+## Compiling
+
+### Building the container
+The [exec/](https://github.com/NOAA-GFDL/ESM4/tree/main/exec) 
+folder contains a Dockerfile and Makefile that can be used to 
+build the model container.  To build the container using podman:
+```
 cd exec
-make OPENMP=on
+podman build -t esm4:esm4.5 -f Dockerfile
 ```
-If you would like to compile with *-O2* instead of *-O3* do
-```bash
-make REPRO=on OPENMP=on
+*NOTE*: you may need to bind a filesystem using the `--volume` 
+option in podman
+
+If the intended use is to run the container using apptainer, you
+will need to convert the OCI podman container to a docker archive
+and then to a SIF file:
 ```
-To compile with *-O0* and debug flags do
-```bash
-make BLD_TYPE=DEBUG OPENMP=on
+podman save -o esm4_esm4.5.tar localhost/esm4:esm4.5
+apptainer build --disable-cache esm4_esm4.5.sif docker-archive://esm4_esm4.5.tar
 ```
-Compiling with openMP is optional.
+You can now copy the esm4_esm4.5.sif to your run directory and
+use apptainer to launch.
 
-
-Here are examples of how to compile the model on various systems:
-
-gaea (NOAA RDHPCS cray system)
-```bash
-module load intel
-module load cray-netcdf
-module load cray-hdf5
-git clone --recursive git@github.com:NOAA-GFDL/ESM4.git
-cd ESM4/exec
-make MKL_LIBS="none" OPENMP=y
 ```
-Compiling on orion (MSU)
-```bash
-module load intel impi netcdf hdf5
-export LIBRARY_PATH=${LIBRARY_PATH}:${LD_LIBRARY_PATH}
-git clone --recursive git@github.com:NOAA-GFDL/ESM4.git
-cd ESM4/exec
-make OPENMP=on
+cd $run_directory
+module unload netcdf-c netcdf-fortran hdf5
+module load mpich
+export APPTAINER_BINDPATH=/path/to/mpi/lib,/path/to/libfabric/lib
+export APPTAINERENV_LD_LIBRARY_PATH=/path/to/mpi/lib:/path/to/libfabric/lib:\$LD_LIBRARY_PATH
+
+srun --ntasks=768 --cpus-per-task=4 --export=ALL,OMP_NUM_THREADS=4 apptainer exec --writable-tmpfs --bind ${PWD} ${PWD}/esm4_esm4.5.sif /apps/ESM4/exec/esm45.x : --ntasks=5796 --cpus-per-task=1 --export=ALL,OMP_NUM_THREADS=1 apptainer exec --writable-tmpfs --bind ${PWD} ${PWD}/esm4.5_compile-prod-openmp.sif /apps/ESM45/exec/esm45.x
+
 ```
+### Building from source
+It is possible to build this code from source by following the
+workflow of the Dockerfile and adapting the Makefile for your
+system. However, the preferred method for building and running the
+ESM4.5 is in a container.
 
 ## Model running
 A work directory needed for running the model can be obtained from
-ftp://data1.gfdl.noaa.gov/users/ESM4/ESM4Documentation/GFDL-ESM4/inputData/ESM4_rundir.tar.gz
+# This needs to be filled in
 
 The directory contains input.nml as the namelist, various input tables needed
 for running the model, and model input files in a folder called INPUT/.  There
 is also a directory named RESTART/ that should be empty at the beginning of
 each run. 
 
-There is a skeleton of a run script named [run/ESM4_run.sh](https://github.com/NOAA-GFDL/ESM4/blob/master/run/ESM4_run.sh).  You must update this
-script to run the model.  Include a path to the work directory and the executable.
-You should also update the program you need to run the model on your system.  The
-default for this script is `srun`.
+The container needs to be run on an x86 system with mpich-flavor MPI.
+If your system has openMPI, you will need to use the Dockerfile to build
+your own container.  Below is an example of how to set up your run using
+slurm:
+```
+cd $run_directory
+module unload netcdf-c netcdf-fortran hdf5
+module load mpich
+export APPTAINER_BINDPATH=/path/to/mpi/lib,/path/to/libfabric/lib
+export APPTAINERENV_LD_LIBRARY_PATH=/path/to/mpi/lib:/path/to/libfabric/lib:\$LD_LIBRARY_PATH
+
+srun --ntasks=768 --cpus-per-task=4 --export=ALL,OMP_NUM_THREADS=4 apptainer exec --writable-tmpfs --bind ${PWD} ${PWD}/esm4_esm4.5.sif /apps/ESM4/exec/esm45.x : --ntasks=5796 --cpus-per-task=1 --export=ALL,OMP_NUM_THREADS=1 apptainer exec --writable-tmpfs --bind ${PWD} ${PWD}/esm4.5_compile-prod-openmp.sif /apps/ESM45/exec/esm45.x
+```
+This set up requires a total of 8864 cores to execute the model.
+For assistance with setting up the binds for MPI on your system, you
+should contact you system adminstrators.
 
 
 ## Disclaimer