- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, the authors focus on the dependency between neural predictor and deterministic physical correctors in hard-constrained climate emulators. They stress that assessing corrected forecasts may hide the role and possible weaknesses of the predictor itself.
In this regard, the authors present an evaluation scheme based on 3 different schemes: corrected rollouts, predictions (before correction) but evaluated on corrected trajectories, and autonomous rollouts (correction turned off). In addition, they cross train/inference correction, where the weights of the predictor are held constant when the inference condition changes. Moreover, the authors add matched non-physical controls to separate physical correction effect from generic perturbation effect.
For ACE2, they run an assessment of forecast accuracy and physical consistency and dependency on correction, over variables and forecasting horizons. They find that correction effect is horizon-dependent. They test with a three-seed pilot which shows that models trained on correction give a worse uncorrected precipitation forecast in short horizon, but it gives better uncorrected forecast in longer horizon. On a different nine-seed experiment, they observe that supervision directly on raw precipitation prediction can help to offset the deficit of uncorrected 72h errors. But keeping the accuracy of corrected forecast on 24h.
Moreover, in other experiments on ACE2S-SHiELD+, and adapted ConCerNet's heat equation system, they try to verify the generality of the dependency between predictor and corrector. In general, they recommend to assess raw predictor accuracy, correction effect and training/inference effect separately.
Critical strengths
- Systematic evaluation of the interplay between predictor training and physical correction. Nice experimental setting to clearly differentiate between the direct effect of the physical corrector application and its impact on the trained predictor during training. The way to differentiate raw predictions on corrected trajectories and autonomous uncorrected rollouts is very good. Especially the latter also reveals differences in the distribution of recurrent inputs.
In particular, the application of matched controls, crossed training-inference experiments, and multi-horizon predictions enable uncovering effects that are otherwise hidden by ordinary evaluation with corrected output.
This same structure is of potential value when assessing other autoregressive models with deterministic correction operators.
- Transparent experiments and informative intervention results.
Positive and negative results (including negative results like failure of preregistered hypothesis that uncorrected precipitation deficit increases as a function of forecast horizon) are presented.
Intervention with 9-seed raw supervision generates relatively strong empirical evidence of the power of directly supervising pre-correction outputs to greatly benefit autonomous prediction of precipitation, with no harm to corrected accuracy for short horizons.
Authors carefully separate statistical significance from practical significance.
Authors explain weaknesses of small seed experiments and evaluation within the pre-training period.
These transparency factors lend confidence to the empirical findings.
Critical weaknesses
1. Insufficient conceptual novelty and mechanistic de-coupling of correction dependency.
The main finding - the fact that hard physical correctors may mask degradation of raw neural predictions - is a topic that has already been studied by Chapman et al. (2026) -- including the topic of unidentifiable directions in correction operators and direct supervision of raw outputs as a potential solution.
This paper builds upon the study with autonomous rollouts, crossed training-inference evaluation, and matched controls.
These are valuable experimental developments but the conceptual progress compared to the existing work is still relatively modest.
In a related issue, for the ConCerNet experiment, the physical projection clearly cancels out the mean of the predicted vector field.
If training only supervises projected outputs, it may not control for errors in the canceled direction.
Therefore, part of the large uncorrected error ratio we see, and much of the benefit from reintroducing physical projection, might be due to this algebraic feature of the projection operator.
The current approach of evaluating vector field RMSE in terms of trajectory error also mixes vector field prediction error and differences in states at which the vector fields are evaluated.
A way to further unpack prediction error into physically allowed and constraint-violating components, using vector field evaluations at fixed state with state RMSE separately, would be useful for determining whether projection training does more to harm the learned physical dynamics or simply to render unidentifiable the discarded components.
Without such an analysis, it's hard to fully mechanistically interpret the ConCerNet results.
2. Not much evidence for generality in the key climate results.
The most interesting result from ACE2 (flipping the uncorrected precipitation error gap on forecast horizon) is only on 3 adaptation seeds. While I think the authors call out this experiment correctly as descriptive, there's a lack of statistical power that limits how much we can say from this.
Also, the months being evaluated are in the pretraining range for the checkpoint for ACE2 we get. While they're excluded from local adaptation, we can't say whether the behavior we're seeing is generic to unseen time periods of climate.
And while the 9 seed matched controls are statistically consistent for each seed, only 8 out of 24 comparisons are statistically above the authors' practical importance thresholds that they've set, and all of those are for temperature. So the benefits of short horizon forecasting for other variables don't seem to be as large as the statistical comparisons suggest.
We do have the direct supervision experiment which adds support. But that experiment focuses on different forecast horizons for the uncorrected and corrected evaluation endpoints. And there are no equivalent longer-horizon results to show whether this intervention is dealing with the wider predictor-corrector dependency that was highlighted in the pilot.
Finally, the additional climate model evidence lies within the ACE2 family and the ConCerNet experiment relies on a simplified heat equation system with a known analytic invariant.
This raises questions about the generality of these results with respect to different physical correction mechanisms and with respect to different forecasting environments.
Questions for authors
Q1. Mechanistic understanding of ConCerNet Results Can the authors provide a breakdown of the uncorrected vector field error into parallel and orthogonal components w.r.t the physically admissible space?
Specifically, how much of the 26.87x error ratio is accounted for by the projection-away component?
Further, could the authors compare the vector fields of both predictors at the same input states and show the RMSE of state trajectories vs vector field (trajectory) RMSE?
These will help to separate out learned dynamics degradation and unconstrained projection nullspace components.
Q2. Robustness of horizon-sensitive ACE2 results Is there a possibility to expand the crossed train-inference experiment beyond the three adaptation seeds, especially for the 10d and 30d horizons where the uncorrected precip error gap flips?
Is it possible to assess the same effects on dates outside of pretraining for the model as released?
Generally speaking, how much of the flip is still observed across different independent climate periods and initializations for forecasts?
Q3. Range of intervention in raw supervision The presented intervention is strong in improving 72hr uncorrected precip forecasts while not hurting 24hr corrected forecasting accuracy.
What are the corrected 72hr results on the withheld evaluation set?
How does the intervention affect 10d and 30d forecasting, other atmospheric variables, as well as the size of physical corrections?
It would also help add value to clarify what extra behaviour is specifically introduced by the current intervention in light of other related raw supervision intervention solutions discussed by Chapman et al. (2026).