Workflow

Three workflow families are available:

  • Site learning — optimize learnable parameters against observed site-level fluxes or states

  • Site simulation — run the model forward for site datasets

  • Grid simulation — run the model forward month by month on gridded data

All three share the same top-level config structure:

  • model

  • data

  • parameterization

and then use one workflow-specific block:

  • site_learning

  • site_simulation

  • grid_simulation

The script decides which workflow is executed. The config does not use a top-level workflow selector.

Note

The entry scripts in scripts/ decide which workflow runs. The YAML files in examples/ provide concrete experiment setups built on the shared config structure.

The bundled examples/ files demonstrate each workflow concretely. The shared base config is:

  • examples/example_base_config.yaml

Workflow-specific example configs build on top of it with inherit.

Site Learning

This workflow optimizes learnable parameters against observed site-level flux or state targets.

Targets are configured directly under site_learning.targets. Each entry names the ADELM output to supervise and then defines the observed variable mapping, an optional 1-based layer selector for layered outputs such as soil_moisture, plus per-target sample_loss_weight and site_loss_weight.

Hint

Site learning requires at least one active nn_global or nn_feature_based parameter declaration.

Cross-validation

Cross-validation is configured under site_learning.cross_validation.

No cross-validation

site_learning:
  cross_validation:
    enabled: false

Spatial cross-validation

site_learning:
  cross_validation:
    enabled: true
    scheme: spatial
    n_folds: 5
    cv_seed: 42

Temporal cross-validation

site_learning:
  cross_validation:
    enabled: true
    scheme: temporal
    n_folds: 5
    cv_seed: 42

Common entry points

Site-learning script

python scripts/site_learning.py --config examples/example_single_learning_config.yaml
python scripts/site_learning.py --config examples/example_spatial_learning_config_no_cv.yaml
python scripts/site_learning.py --config examples/example_spatial_learning_config_spatial_cv.yaml
python scripts/site_learning.py --config examples/example_spatial_preload_learning_config.yaml

This script reads epochs, learning rate, chunk size, and related training settings from site_learning.training.

site_learning.training.seed controls training randomness and NN initialisation. It also controls the reproducible train/monitor split when monitor_within_train_enabled: true. site_learning.cross_validation.cv_seed controls only how random CV folds are constructed and normally stays fixed.

For cross-validation workflows, site_learning.py runs fold 0 by default. Use --fold N to run a different fold explicitly.

Example configs

These example configs are bundled for site-learning workflows:

  • examples/example_single_learning_config.yaml

  • examples/example_spatial_learning_config_no_cv.yaml

  • examples/example_spatial_learning_config_spatial_cv.yaml

  • examples/example_spatial_preload_learning_config.yaml

They demonstrate both supported learning modes:

  • nn_global for parameters shared across all selected sites

  • nn_feature_based for parameters predicted from attributes or, for layered soil parameters, attributes plus soil texture

The preload example demonstrates staged learning:

  1. run a spatial-learning experiment and save nn_weights.pt

  2. reuse that file through site_learning.initialization.init_nn_weights_path

  3. freeze selected physical parameters with site_learning.initialization.frozen_parameters

  4. continue training the remaining parameters

They also include example training settings for epochs, learning rate, chunk size, gradient clipping, early stopping, and learning-rate scheduling.

Note

site_learning.output_dir is a container directory; each run is written into a seed- and fold-specific subdirectory such as no_cv_seed_100 or spatial_cv_fold1_seed_100.

Typical outputs

Typical site-learning outputs include:

File

Description

runtime_summary.txt

Parameterization, diagnostics, training, evaluation, and final inference summary

nn_weights.pt

Saved neural-parameterization weights

<target_name>.csv

Per-target tables containing site_id, date, obs, sim, and test/train flags

metrics.csv

Test metrics summarised by target

train_log.csv

Epoch-by-epoch training history

config_snapshot.yaml

Resolved config snapshot for the run

site_simulation.nc

Final full forward export written from the last evaluation when site_learning.save_final_inference: true

Site Simulation

This workflow runs ADELM forward on site datasets without optimisation.

Note

Site simulation runs with either default parameter sources or learned weights from a previous site-learning run.

Common entry points

Site-simulation script

python scripts/site_simulation.py --config examples/example_site_simulation_default_parameters.yaml
python scripts/site_simulation.py --config examples/example_site_simulation_learned_parameters.yaml

Example config

The site-simulation workflow can start from the shared base config:

  • examples/example_base_config.yaml

Two dedicated site-simulation example configs are also bundled:

  • examples/example_site_simulation_default_parameters.yaml

  • examples/example_site_simulation_learned_parameters.yaml

The first uses the default ADELM registry parameter sources. The second loads pretrained NN parameterization weights produced by a site-learning run.

The learned-parameter example is intentionally written with a glob pattern such as:

  • .../outputs/example_spatial_learning/**/nn_weights.pt

This lets one site-simulation launch fan out over multiple learned checkpoints. For example, if the pattern matches:

  • spatial_cv_fold1_seed100/nn_weights.pt

  • spatial_cv_fold1_seed200/nn_weights.pt

and site_simulation.output_dir is:

  • .../outputs/example_site_simulation_learned

then ADELM writes the corresponding forward runs into:

  • .../outputs/example_site_simulation_learned/spatial_cv_fold1_seed100/

  • .../outputs/example_site_simulation_learned/spatial_cv_fold1_seed200/

This workflow is meant to reuse learned parameterizations as they were trained. Changing fixed parameters is fine, but changing the learned NN parameterization scheme or architecture will usually make the checkpoint incompatible.

Warning

Use site_simulation.nn_weights_path to validate or reuse learned weights as they were trained. If you change learned parameter sources or NN architecture, ADELM raises an error when settings are incompatible.

Typical outputs

Typical site-simulation outputs include:

File

Description

runtime_summary.txt

Parameterization and driver summary for the run

site_simulation.nc

Full forward export containing time-varying outputs, static parameters, and final states

Grid Simulation

This workflow is intended for large gridded forward runs. Because the data volume is much larger than in site workflows, the grid runner advances month by month and can write intermediate checkpoints.

Note

Grid simulation is intended for long forward runs after site-scale configuration and parameterization are settled.

Common entry points

Grid-simulation script

python scripts/grid_simulation.py --config examples/example_grid_simulation_default_parameters.yaml
python scripts/grid_simulation.py --config examples/example_grid_simulation_learned_parameters.yaml

Use --resume-from path/to/YYYYMM.npz to warm-start from a saved monthly checkpoint. Use --device-id 0 when you want to force a specific visible GPU.

Example config

The bundled grid examples are:

  • examples/example_grid_simulation_default_parameters.yaml

  • examples/example_grid_simulation_learned_parameters.yaml

It demonstrates:

  • data.grid.drivers_path with per-driver monthly file templates

  • grid bounding-box selection through grid_simulation.domain.selection

  • monthly stepping with a separate spin-up window

  • optional learned-weight reuse through grid_simulation.nn_weights_path

Typical outputs

Typical grid-simulation outputs include:

File

Description

monthly/output_YYYYMM.nc

Monthly-mean values of the user-selected time-varying outputs

daily/output_YYYYMM.nc

Daily values of the user-selected time-varying outputs

checkpoints/YYYYMM.npz

Restart checkpoint used for warm start / resume

checkpoints/spinup_YYYYMM_minusN.npz

Intermediate spin-up restart checkpoints written at the end of each spin-up month

Grid-output selection uses three modes:

  • [] skips that frequency entirely

  • all saves all available time-varying outputs at that frequency

  • [a, b] saves only the named outputs at that frequency

These selections apply to time-varying model outputs, including states if you name them explicitly. Static parameters are not part of the grid time-series exports. Restart states are stored in checkpoint files instead.

When adapting the example configs to your own data:

  1. copy examples/example_base_config.yaml to a new file

  2. update data.grid.*_path entries

  3. update data.mapping

  4. choose the workflow block you need

  5. set the corresponding output directory