- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors design a hybrid modeling mechanism that can enhance a data-driven emulator with a mechanistic pathway to enable counterfactual scenarios, even if, for training, only factual data is available. The proposed mechanism offers a middle path between the two commonly available ways to assess counterfactual scenarios: having counterfactual data to train the emulator or having the full mechanistic model.
The mechanism takes the pretrained emulator and fixes the variable of interest to a reference value. The pathway then describes the change from the reference value to any other value mechanistically, which the pretrained model is then fine-tuned to adapt to. They evaluate their method on an advection-diffusion transport example, showing that it substantially improves the error, and on a realistic climate emulation task asking counterfactual questions under shifts. Formally, the authors derive causal identification guarantees that they machine-checked in Lean.
Critical strengths
The authors present a simple but compelling idea with convincing results in a well-digestible way. I especially appreciate the "intuition" and "example" bits.
-
The approach is a genuinely new and effective way to use hybrid modeling, where usually the ML component is used to correct or improve a mechanistic model. Here, however, the mechanistic part is used to specifically correct the inability of the ML component to extrapolate beyond its training domain into counterfactual regimes.
-
The downstream task of counterfactual scenarios in climate emulation is well chosen and highly compelling. The task of disentangling and SST when no controlled counterfactual data exists is an important challenge in the Earth sciences. The results prove the effectiveness of their approach, and I do believe it holds merit for many other applications.
Critical weaknesses
I do not have strong issues with this work. There are two small things I would highlight.
-
The advection-diffusion experiment appears slightly tautological, as the pathway seems to need the exact full mechanics, which is, in principle, what this approach tries to bypass. I understand that here it serves more like a proof of concept, while the climate emulation is the realistic setup.
-
I am missing an ablation on the reference value. It would be important to know how sensitive the approach is to that choice.
Questions for authors
I am currently leaning toward a clear accept, as the idea is compelling and the results are convincing. Nonetheless, a sensitivity analysis of the reference point would further strengthen my confidence in the robustness of the method.
4 comments
Dear reviewer Fnap:
Thank you again for the positive assessment and your constructive feedback. As the discussion period concludes soon, we kindly ask if the rebuttal has addressed your concerns.
Should any further questions or concerns arise, we are happy to provide clarification during the remaining discussion time.
Response to authors
Dear authors,
Thank you very much for your response and the additional experiments (including the ones in the responses to the other reviewers). The added pathway-scaling and reference-value analysis address both of my raised concerns. It is interesting to see that there seems to be a trade-off between catching extreme tests and maintaining factual skill, and it does make sense to me. All the additions strengthen my confidence in my assessment.
Dear reviewer Fnap:
Thank you so much again for your valuable suggestions and continued positive feedback! We are really glad that the additional analyses addressed your concerns and strengthened your confidence in our work.
We truly appreciate the time and effort you and the other reviewers have put into reviewing our work. Thank you again!
Response to Reviewer Fnap
Thank you very much for the positive assessment. We added the reference-value sensitivity study as suggested: we observe that the gains hold up to 2× CO$_2$, and the default 1× is the best practical choice.
[W1] Advection–diffusion is the proof of concept; climate is the realistic test.
rK5t).Manuscript update:
§§ 4.1, B.4, and F.1.3; Table 5.[W2, Q] Reference value: gains hold up to 2×, and the default 1× is the best practical choice.
T5: Reference value s0 from 1× to 5× CO₂ (ReRoute zero-shot, physical pathway. Baseline (no-RC): released model trained without random-CO₂ data. AMIP hist. is the factual test; the other columns are CO₂ interventions. Training-mix mean: average CO₂ over the baseline's training data. At s0 = 4×, the abrupt-4×CO₂ query equals the reference, so ReRoute reduces to the baseline there. α, lower is better: channel-mean RMSE of normalized time-mean biases; mean ± std over 5 realizations for AMIP, scored 2012–2020, and random-CO₂, 5 years; over 36 initial conditions for abrupt-4×CO₂, 90 days. Bold: best per column and every value whose gap to it is within one standard deviation.)
Manuscript update:
Table 6, § F.1.4.Thank you again for the encouraging review and the helpful suggestions!