Skip to content

SBM: conservative coarse/fine AMR coupling for spectral-bin liquid transport (fixed two-level, periodic; qualification build only) - #5010

Open
pressel wants to merge 13 commits into
erf-model:developmentfrom
pressel:erf-sbm-m4b-periodic-amr
Open

pressel wants to merge 13 commits into
erf-model:developmentfrom
pressel:erf-sbm-m4b-periodic-amr

Conversation

@pressel

@pressel pressel commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR adds the numerical machinery to couple ERF's spectral-bin microphysics (SBM) liquid spectrum across a coarse/fine AMR interface: conservative flux correction (reflux) of every bin, time-consistent coarse-to-fine ghost data, mass-conserving restriction, and a fail-closed coarse synchronization step. It is exercised end to end on a fixed, two-level, triply periodic mesh in a separately compiled qualification build.

The normal ERF build is not changed for users. Production SBM runs still require amr.max_level = 0, RemakeLevel still rejects SBM, and no new runtime inputs are exposed. This PR is the tested foundation for the next steps (dynamic regrid/derefinement, multilevel restart, nonperiodic boundaries), which are not included. It builds on #4106, #4127, #4164, #4185 and #4228.

Why bin microphysics needs this

In a bulk scheme, refluxing $q_c$ and $q_r$ at a coarse/fine boundary is enough. In a bin scheme the prognostic state is the size distribution itself: the per-bin liquid mass $M_i$, and in two-moment bins also the per-bin number $C_i$ and any attached extensive properties. Many different spectra share the same $q_c$ and $q_r$, so correcting only bulk water would leave the spectrum inconsistent at the interface.

The coarse/fine coupling must therefore:

  • conserve every bin's mass and number, and each attached property, on the composite (leaf) mesh, without double counting coarse cells under the fine level;
  • form fine-level ghost values from the coarse spectrum and dry-air density at the same time, so refinement does not look like spurious droplet growth or evaporation;
  • keep every two-moment bin physically realizable ($a_i C_i \le M_i \le b_i C_i$, with the bin-ownership convention of SBM: Add reference spectral reconstruction and remapping contracts #4164) without clipping or silent repair.

What the PR adds

1. Time-consistent coarse-to-fine ghost cells

During a fine substep, coarse-only stencil cells are filled from the coarse old/new spectral and density endpoints at the stage's actual time:

$$z_a(t)=\frac{(1-\theta),U_a(t_0)+\theta,U_a(t_1)}{(1-\theta),\rho_d(t_0)+\theta,\rho_d(t_1)},\qquad \theta=\frac{t-t_0}{t_1-t_0}.$$

Here $U_a$ is the density-weighted bin quantity and $\rho_d$ the dry-air density. Interpolating the ratio $U/\rho_d$, or dividing by only the end-of-step density, is rejected by tests.

  • Each of the three stage roles (input, limiter base, old-step anchor) uses its own time.
  • Same-level fine data and periodic images take precedence over coarse data in both ghost layers.
  • Two-moment bins are converted to endpoint-number coordinates after this physical interpolation and before WENO-Z3, as in the single-level scheme from SBM: Add native mapped spectral advection with grouped FCT #4185.

Coarse-only ghost cells are first filled with NaN, and every one is checked before any stencil reads it. A missed write therefore aborts on all ranks, even if a stale finite value from an earlier stage was present.

2. Reflux of the accepted spectral transport

The flux correction uses the accepted face rates after the coupled group limiter, for every mass, number and attached-property component:

$$F^{acc}_{a,f}=F^{L}_{a,f}+\lambda_f,\big(F^{H}_{a,f}-F^{L}_{a,f}\big),\qquad 0\le\lambda_f\le 1,$$

with one $\lambda_f$ per face shared by the whole bin group. It does not use the unlimited high-order proposal or the projected cloud/rain fluxes.

  • Accumulation: stage rates are added chunk by chunk directly into an AMReX YAFluxRegister, so no full-spectrum face ledger is kept in production.
  • Weights: they match ERF's integrators. Compressible RK3 uses $\Delta t$ on the final stage only; anelastic Heun uses $\Delta t/2$ on each of its two stages. The three fine substeps are summed.
  • Correction: it is applied to the mapped amount $H_a=\omega U_a$, with $\omega=\det J/(m_x m_y)$, on uncovered coarse cells next to the fine level:

$$\Delta H_a=\pm\frac{1}{\Delta\xi_c}\Big(I^{c}_{a}-\tfrac14\sum I^{f}_{a}\Big).$$

3. Conservative, fail-closed coarse synchronization

After the fine substeps, a private coarse candidate is built:

  1. covered coarse cells take the average of their children's mapped amount $H$ (not of $U$);
  2. the chunked reflux corrections are applied;
  3. the whole spectrum is checked for finiteness, realizability, attached-property support and representable quotients.

Only if the check passes is the pending flux amount consumed (exactly once), the spectrum published, and compact $q_c/q_r$ regenerated from it. A rejected candidate leaves the published spectrum unchanged.

ERF's own host reflux and average-down run first and cannot be rolled back, so a failure after that point stops the run (fail-stop) rather than claiming a rollback. A device-side admission gate (SBMStateAdmission) performs the check without host copies on success.

4. Level lifecycle pieces

  • When a fine level is created from the coarse level, a carrier-relative (piecewise-constant in $z_a=U_a/\rho_d$) prolongation is used.
  • The SBM interface flux registers are created and bound per level, and cleared or rebound with the level.
  • SBMStateManager::begin_step accepts the bounded round-off between ERF's fine-substep start times without allowing a match to a neighbouring substep.

Scope: what this PR does not do

  • No multilevel SBM in the production executable. The max_level == 0 guard, periodic-only boundaries and the RemakeLevel abort are unchanged.
  • No dynamic regrid, remake or derefinement, no multilevel checkpoint/restart, and no nonperiodic spectral boundaries.
  • No new cloud processes (condensation, activation, collision, sedimentation) and no particle diffusion.
  • GPU execution is not qualified. The code follows the AMReX GPU patterns, but only CPU/MPI runs were made.

Effect on existing runs

  • Non-SBM runs: behaviour is unchanged. The cloud-chamber budget report keeps its original position before the AMR sync.
  • Single-level SBM runs: no numerical change. The new code paths need a second level, and the extra transport bookkeeping is allocated only when more than one level exists.

Qualification build and tests

cmake -S . -B build-sbm-qual -DERF_ENABLE_SBM_AMR_QUALIFICATION=ON \
      -DERF_ENABLE_TESTS=ON -DERF_ENABLE_MPI=ON -DERF_PRECISION=DOUBLE -DERF_DIM=3
ctest --test-dir build-sbm-qual -R '^SBM_AMR_(Qualification|Profile)_' --output-on-failure

ERF_ENABLE_SBM_AMR_QUALIFICATION (default OFF) compiles an executable that accepts only a fixed two-level, periodic, static-grid deck with no checkpointing, regridding or forcing. Its input switches are an error in the normal build. All qualification-only checks, fixtures and log records live in Source/Microphysics/SBM/ERF_SBMAMRQualification.{H,cpp}, which is compiled only in that build; the shared ERF sources carry only short guarded hooks.

Integration tests (Tests/test_files/SBM/) run three fine substeps per coarse step and check:

  • composite leaf inventories for every spectral component, against a round-off bound tied to the operation count and MPI reduction depth;
  • one-moment and two-moment spectra with an attached number-carried property;
  • nonzero interface transfer with an active limiter;
  • an exactly zero-wind control, which must produce zero carrier, zero accepted rates and no limiter change;
  • 1-rank vs 2-rank parity;
  • a split decomposition with spatially unique donor values: received coarse-parent and fine–fine ghost values are compared with the owning ranks' valid data at both ghost depths;
  • an empty-fine-rank layout;
  • a single-rank store fault that must abort all ranks without hanging;
  • spectral-width/rank profiles (N = 32/128/256, 1/2/4 ranks).

Unit tests (ERF_GTestSBMTransport.cpp, ERF_GTestSBMAMRTransfer.cpp) include:

  • coupled WENO/FCT/reflux check (FineStageG1WenoAcceptedRateReachesMappedYAOracle): a manufactured 2M-plus-property state with density varying in space and time, run through three fine RK3 substeps. Its WENO-Z3 reference is computed independently of ERF's routines. It checks the accepted interface rates and the resulting coarse-cell correction, and separates them from wrong-sign, missing/duplicated time weight, swapped component, ratio-interpolation and "WENO on physical M/N" alternatives;
  • flux-register sign and stage-weight checks for RK3 and Heun;
  • restriction/prolongation conservation and failure atomicity;
  • missed ghost writes: separate negative tests for skipping only the store and for skipping only the write count.

Testing performed (local)

macOS arm64, AppleClang 21, Open MPI 5.0.9, Debug, 3D DOUBLE, after merging current development:

Build Result
Qualification 117/117 SBM*/Auxiliary* gtests; 151/151 SBM CTests (all existing single-level SBM_M3_*, restart and regrid-on-restart tests, plus the 17 new AMR qualification and profile tests)
Normal (non-qualification) 117/117 SBM*/Auxiliary* gtests; no new warnings

Sphinx (-n -W) and codespell pass on the changed files. Not run locally: CUDA/HIP/SYCL, SINGLE precision, Windows. The new unit tests keep device kernels outside gtest test bodies, and the coupled check is skipped in SINGLE precision.

Status

Draft, merged with current development. The merge ported the SBM qualification guards and decks to the per-level erf.compressibility_type / erf.timestepping_type inputs (#4996). It also passes the SBM parent stage after the new immersed-wall arguments of erf_slow_rhs_post.

I checked the interactions with #4370 and #4853 (SBM restart onto re-made level-0 grids) and #4867 (anelastic carrier in mapped form):

  • Restart and regridding stay single-level. The qualification build still forbids checkpoint/restart, and RemakeLevel still rejects SBM.
  • SBM_M3_StaticTerrain_RegridOnRestart and the anelastic donor-timestep tests pass on the merged branch.
  • The anelastic zero-flow AMR control is unaffected, because its carrier is exactly zero on a ConstantDz mesh.

Suggested review order

  1. Source/Microphysics/SBM/ERF_SBMAMRFluxRegister.*: register state machine, stage weights, consume-once.
  2. Source/Microphysics/SBM/ERF_SBMTransport.cpp (fill_coarse_parent_ghosts, advance_stage): ghost interpolation and accepted-rate accumulation.
  3. Source/Microphysics/SBM/ERF_SBMAMRTransfer.*: mapped restriction, prolongation, reflux application, admission.
  4. Source/Microphysics/SBM/ERF_SBMAMRCoupling.cpp (sbm_synchronize_levels, init_sbm_level_from_coarse) and Source/TimeIntegration/*: ordering of the host and SBM synchronization. post_timestep calls the SBM synchronization after the host reflux/average-down.
  5. Source/Microphysics/SBM/ERF_SBMAMRQualification.*: qualification-only checks (not built by default).
  6. Docs/sphinx_doc/SpectralBinMicrophysics.rst: user-facing status.

- move device lambdas out of gtest TestBody (CUDA extended-lambda rule)
- skip the coupled WENO/FCT/YA oracle in SINGLE precision
- print inventory lines at full precision for rank-parity comparison
- require the exact-zero anelastic carrier for zero-flow qualification
- poison coarse-only ghosts with one mask read per cell
- add a tally-only negative test for the parent-ghost write count
- accept fixed-notation values in nonzero log regexes
- keep the split ghost-value check per transport, not per process
- use the spatially unique fixture for profile parity runs
- drop the source-text budget-ordering test; ERF.cpp documents the order
- make SBM transport and ERF qualification members unconditional in headers
- move unit-test probes out of the transform and FCT hot paths
- pick the rank-local fault cell from the actual coarse-only ghost mask
- compare the quotient marker by equality to avoid FE_INVALID on NaN
- reset all interval state in accept_reflux; null-check sbm_transport
- remove stray indentation before preprocessor directives
Resolve conflicts with the per-level CompressibilityType/timestepping_type
inputs (erf-model#4996), the immersed-wall arguments of erf_slow_rhs_post, the
boundary-plane writer, and the per-test MPI TMPDIR setup.

- port the SBM qualification guards from anelastic/anelastic_type/
  substepping_type to compressibility_type/timestepping_type
- qualification decks use erf.timestepping_type = RK3_NoSubstepping;
  the zero-flow run uses erf.compressibility_type = Anelastic, RK2
- pass the SBM parent stage after the immersed-wall arguments
- create the per-test TMPDIR in the SBM AMR qualification runner
Keep the shared ERF sources to thin hooks so the SBM multilevel work does
not grow ERF.cpp, the constructors, or level creation.

- ERF_SBMAMRCoupling.cpp holds the SBM flux-register setup, fine-level
  initialization from coarse, and the coarse/fine restriction/reflux
  synchronization formerly inline in post_timestep
- ERF_SBMAMRQualification.{H,cpp} holds every qualification-only check,
  fixture profile, inventory and log record; it is compiled only with
  ERF_ENABLE_SBM_AMR_QUALIFICATION
- replace the ERF_SBM_AMR_TEST_* environment variables with the
  qualification-only inputs erf.sbm_amr_qualification_unique_profile and
  erf.sbm_amr_qualification_parent_store_fault; normal builds reject them
- no numerical or log-format change
The macOS CI build (ERF_ENABLE_ALL_WARNINGS, non-qualification) flagged
the split-ghost arguments that are only read in qualification builds, a
leftover probe component, two unused test locals, an unused test
parameter, and a Long/size_t comparison.
- GCC/CUDA/SYCL: give std::max an explicit std::uint64_t type; ULL and
  uint64_t differ on Linux
- reset the test's reference amrex::YAFluxRegister before CrseAdd. AMReX
  does not zero it on construction; Debug's signaling-NaN fill was masked
  by norm0 skipping NaN, while optimized builds read garbage. Also require
  both reflux candidates to be finite before comparing them
- make the qualification-only ERF members public; they launch device
  lambdas, which CUDA does not allow in private members
Resolve the include conflict in ERF_Constructors.cpp by keeping both the
SBM qualification header and the moving-bodies structure registry.
@pressel
pressel marked this pull request as ready for review November 9, 2026 00:32
- GCC CI timed out on seven EXPECT_DEATH tests: the forked child's
  amrex::Abort hangs in MPI_Abort on Linux. Raise the expected
  amrex::Abort as an exception (amrex::system::throw_exception) and match
  its message instead, so the checks run under MPI on every platform.
- HIP CI: rename a loop-local 'covered' flag that shadowed the covered
  mask in the YA oracle test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant