ccpp-capgen v1: a unified successor to ccpp-prebuild and ccpp-capgen v0

[climbfuji]

Description

Note. ccpp-capgen v1 was previously called ccpp-capgen-ng. Some of the supporting documents attached below (until 2026-06-28) use this terminology.

ccpp-capgen v1 is a next-generation code generator for the CCPP framework. It is not a criticism of the existing tools — both ccpp-prebuild and ccpp-capgen v0 have served their respective communities well and embody years of careful design. This project attempts to synthesize the strengths of both: the performance of ccpp-prebuild and the metadata richness and feature set of ccpp-capgen v0, while also taking advantage of relaxed code-generator requirements that have emerged over time and lessons learned from operating both tools in development, research, and production. It also aims to resolve several outstanding issues that have been blocking progress toward a long-discussed unification of the two code generators.

Supporting documents — including a redesign analysis, the LLM prompt used to bootstrap development, developer and user documentation, and a migration guide — will be provided as part of the draft pull request.

The code was developed with the assistance of an LLM (Claude, by Anthropic) as a coding partner. Where possible, existing code was reused and repackaged.

A pull request will follow shortly. Feedback and questions are welcome.

Testing

The following tests and features are currently supported by ccpp-capgen v1:

• Former ccpp-capgen v0 tests:
  • advection
  • advection_auto_clone (support legacy constituents auto-cloning from metadata)
  • capgen_ng
  • ddthost
  • nested_suite
  • var_compat
• Former ccpp-prebuild tests:
  • chunked_data
  • blocked-data not supported in capgen or capgen-ng
  • opt_arg
• Newly added features/tests:
  • multi-instances
  • multi-instances + constituents
  • suite_allocate (exercises suite-owned variables with various types of runtime-known dimensions)
  • constituents_dim (suite- and host-owned variables with dimensions from constituents objects)
• ccpp-capgen v1 unit testing framework
  • 1300+ unit and docstring tests
• Host model integration
  • CCPP-SCM all suites compile and all manual tests run (a sample of all permutations)
    • Transition CCPP framework from ccpp-prebuild to ccpp-capgen v1 ccpp-scm#695 and PRs listed in there
    • DRAFT Update to capgen-ng #761
    • Updates for ccpp-capgen v1 ccpp-physics#1221
    • Updates for ccpp-capgen v1 TEMPO#31
  • CAM-SIMA Python unit tests pass, system tests pass with Intel and GNU
    • https://github.com/climbfuji/CAM-SIMA/tree/feature/capgen-ng
    • Feature/capgen ng climbfuji/CAM-SIMA#1
    • Feature/capgen ng climbfuji/atmospheric_physics#1
    • Sll regression tests (test/system) pass with both GNU and Intel, and with b4b-identical results except for two tests that also fail on development (SMS_Ln9.ne3pg3_ne3pg3_mg37.FKESSLER.derecho_intel.cam-outfrq_se_cslam_multitape, SMS_Ln9.ne3pg3_ne3pg3_mg37.FADIAB.derecho_gnu.cam-outfrq_se_cslam)
  • UFS started (@dustinswales)
    • Update 2026/06/26: all suites compile, and for many the results are bit-for-bit identical
  • NEPTUNE complete
    • Pull requests open in NRL Enterprise GitHub
    • All 300+ tests pass with the three mandatory compilers (Intel LLVM, GCC, LLVM native)
    • Regular, mid- and high-altitude physics covered (including proprietary NRL physics code)

Issues

A list of issues that will be addressed with this issue will be provided in the associated pull request.

Addressing this issue will also address/close the following issues if/when the participating host models switch to capgen-ng:

• Rethinking horizontal dimensions #760
• Weird Capgen fortran parsing error? #757
• Bug in capgen (test/capgen_test) ? surface_air_pressure outvar in ddt_suite #755
• Document --kind-type in ccpp_capgen.py CLI/help, and consider reversing the syntax for external kind modules #754
• Remove unused metadata attribute state_variable #753
• capgen: Add capability and test for passing mpi communicator as a flat field #751
• capgen: case-sensitivity when comparing metadata to Fortran #746
• Bug in scripts/file_utils.py / move_modified_files routine #744
• Unallocated fields passed into Group Caps #690
• Capgen performance when using many suites #688
• Add support for MPI_Comm (mpi08) object in capgen #686
• Transition UFS weather model to Capgen #682
• Transition CCPP-SCM to Capgen #680
• Transition NEPTUNE to capgen #679
• Enhanced/Expanded Testing  #595
• capgen's Fortran parser does not allow array bounds which are not Fortran identifiers #588
• Add "number of instances" dimension to module-level variables to support multi-instance usage #561
• capgen: check handling of type attribute for ddts #516

To check/may not be implemented yet:

• capgen Fortran parser does not accept internal DDTs as part of a registered DDT #749
• capgen: Invalid character declaration, 'character(len=nf_file_length) ' #748
• capgen: dimension specifier foo(N) is not the same as foo(1:N) in Fortran code #747
• Errors when integer variable is output from one phase and used as an interstitial dimension in another #681
• CCPP Framework to provide API calls for suite inspection by host #380
• Fortran/metadata comparison incorrectly reporting bug when comparing Fortran variable declarations and metadata for derived types #360

Working towards:

• Simplification of ccpp-framework code #737

Supporting Materials

Date and time are all local U.S. Mountain Time. Subsections are sorted from newest to oldest.

The documents below were produced by Claude during development and have not yet been formally reviewed. They are shared for transparency and to give context for the design decisions made. They should be treated as starting points rather than authoritative references.

If documents were reviewed/edited by a human, it is mentioned explicitly.

LLM-Generated Drafts (20260605T1924/latest)

The documents capgen_compat_layer.md and briefing_pm.md were fully reviewed and edited.

briefing.md
briefing_pm.md
capgen_compat_layer.md
constituents.md
constituents_overhaul.md
migration.md
redesign_analysis.md
redesign_prompt.md

LLM-Generated Drafts (20260601T0935)

The document briefing_pm.md was fully reviewed and edited.

briefing.md
briefing_pm.md
constituents_overhaul.md
migration.md
redesign_analysis.md
redesign_prompt.md

LLM-Generated Drafts (20260521T1542)

The document auto_clone_constituents.md was fully reviewed and edited.

auto_clone_constituents.md

Slides from ccpp-framework meeting 2026/05/21

https://docs.google.com/presentation/d/1oG9o3HDJH2RA4swwCxTIc97JS6MxBRlo8qUAJENBEIY/edit?slide=id.g3e3de55f508_0_0#slide=id.g3e3de55f508_0_0

LLM-Generated Drafts (20260519T0903)

The document briefing_pm.md was fully reviewed and edited.

migration_20260519T0905.md
briefing_20260519T0905.md
briefing_pm_20260519T0905.md
constituents_20260519T0905.md
constituents_overhaul_20260519T0905.md

LLM-Generated Drafts (20260513T0733)

constituents_20260513T0733.md
constituents_overhaul_20260513T0733.md
migration_20260513T0733.md
redesign_analysis_20260513T0733.md
redesign_prompt_20260513T0733.md

Comparison: size of code base:

LLM-Generated Drafts (202060505T2044/initial)

redesign_analysis_original_202060505T2044.md
redesign_prompt_original_20260505T2044.md

[bbakernoaa]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

[climbfuji]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

@bbakernoaa Thank you very much for your offer to help, very much appreciated. There are many moving pieces. So far, this has been tested with CCPP-SCM and with NEPTUNE (with three compilers, Intel LLVM + GCC + LLVM native). The next tests need to cover UFS and CAM-SIMA. For CAM-SIMA, I'll work with the developers @peverwhee and @nusbaume. For UFS, your help would be appreciated, but we should coordinate with @dustinswales.

To prepare for UFS and SCM updates, we need to first fix the Metadata vs Fortran inconsistencies in the NCAR CCPP physics. I created pull requests for the UFS weather model, UFSATM, and CCPP physics ufs/dev for that (b4b identical w/ current baseline). It would be great if we could push these pull requests through quickly. When these changes come back to the authoritative NCAR ccpp-physics main branch, we can update the SCM and NEPTUNE branches. The changes will minimize the differences that we need to make when we transition to the new code generator. Maybe you can help with raising the priority for these PRs. I am also tagging @grantfirl and @dustinswales here.

[bbakernoaa]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

@bbakernoaa Thank you very much for your offer to help, very much appreciated. There are many moving pieces. So far, this has been tested with CCPP-SCM and with NEPTUNE (with three compilers, Intel LLVM + GCC + LLVM native). The next tests need to cover UFS and CAM-SIMA. For CAM-SIMA, I'll work with the developers @peverwhee and @nusbaume. For UFS, your help would be appreciated, but we should coordinate with @dustinswales.

To prepare for UFS and SCM updates, we need to first fix the Metadata vs Fortran inconsistencies in the NCAR CCPP physics. I created pull requests for the UFS weather model, UFSATM, and CCPP physics ufs/dev for that (b4b identical w/ current baseline). It would be great if we could push these pull requests through quickly. When these changes come back to the authoritative NCAR ccpp-physics main branch, we can update the SCM and NEPTUNE branches. The changes will minimize the differences that we need to make when we transition to the new code generator. Maybe you can help with raising the priority for these PRs. I am also tagging @grantfirl and @dustinswales here.

@climbfuji Thanks. I would really like to get this moving forward as fast as possible. @dustinswales how can I help?

[climbfuji]

@climbfuji I'm really interested to get this moving forward. How can I help move this forward?

@bbakernoaa Thank you very much for your offer to help, very much appreciated. There are many moving pieces. So far, this has been tested with CCPP-SCM and with NEPTUNE (with three compilers, Intel LLVM + GCC + LLVM native). The next tests need to cover UFS and CAM-SIMA. For CAM-SIMA, I'll work with the developers @peverwhee and @nusbaume. For UFS, your help would be appreciated, but we should coordinate with @dustinswales.
To prepare for UFS and SCM updates, we need to first fix the Metadata vs Fortran inconsistencies in the NCAR CCPP physics. I created pull requests for the UFS weather model, UFSATM, and CCPP physics ufs/dev for that (b4b identical w/ current baseline). It would be great if we could push these pull requests through quickly. When these changes come back to the authoritative NCAR ccpp-physics main branch, we can update the SCM and NEPTUNE branches. The changes will minimize the differences that we need to make when we transition to the new code generator. Maybe you can help with raising the priority for these PRs. I am also tagging @grantfirl and @dustinswales here.

@climbfuji Thanks. I would really like to get this moving forward as fast as possible. @dustinswales how can I help?

Realized that Dustin is out of the office this week. I suggest we work with EPIC to prioritize the metadata-fortran bug fixes this week and early next week start working on dropping capgen-ng in the UFS.

@ligiabernardet FYI