- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes comparison-aware active learning for choosing which scenarios to run through an expensive integrated assessment model while a surrogate ensemble predicts the rest. It builds a comparison graph over scenario inputs, estimates uncertainty for single scenarios and for scenario pairs, sends high-risk scenarios back to the real model through a calibrated fallback set, and selects queries that most shrink that set. On a public GCAM energy dataset and a new 5,180-run dietary dataset, it reports the best accuracy against six baselines and 85.71% and 67.29% fewer real-model runs.
Critical strengths
-
The problem is framed around what scientists actually use. The method targets differences between scenarios rather than only pointwise error, and its fallback set states which scenarios still need the real simulator (Section 3).
-
The new dietary dataset is a real contribution. It contains 5,180 solved GCAM runs with a balanced design in which each transition level appears equally often in every region (Appendix A.1).
-
The comparison with baselines is controlled. All six baselines share the surrogate architecture, training procedure, initial labeled set and budget, so differences come from the acquisition rule alone (Appendix B.1).
Critical weaknesses
-
The fallback rule would benefit from direct validation. The paper does not yet report tolerance values or how often scenarios left to the surrogate exceed them, so further experiments would strengthen the reliability claim.
-
The savings and the method are difficult to reconstruct. The 85.71% and 67.29% figures appear only in Figure 2, and the run counts, the baseline stopping rule and parameters such as rho_t and tau are not yet given.
-
The evaluation could report more. The edge metrics appear to use pairs the method itself targets, the gains are modest (Diet sign accuracy 90.22% against 89.33% for Random), and Table 2 does not include variance.
I find the framing and the new dataset valuable, and I appreciate the authors' effort. However, the fallback rule is not yet validated, the savings and the method are hard to reconstruct, and the accuracy margins come without variance. I think to add fallback validation, a full configuration in the appendix and results with variance would make this paper be better.
Questions for authors
- Could the authors share the tolerances the fallback rule uses and how often scenarios left to the surrogate exceed them, compared with the baselines?
- Could the authors list in the appendix the full configuration and the run counts behind Figure 2, including calibration and fallback runs?
- Could the authors report Tables 2 and 3 with standard deviations and evaluate the edge metrics on pairs the method did not choose?
Suggestions that did not affect my score. It would help to put the savings for every method in a table rather than only in Figure 2, and to use parenthetical citations where the text currently uses textual ones.