acceptodds
Withdrawn from ICLR 2027

Comparison-aware Active Learning for Simulation-Efficient Global Change Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Global change analysis investigates how complex systems respond under alternative socioeconomic, technological, and policy scenarios. It relies on computationally expensive global change models and draws conclusions from both individual scenario outcomes and differences between scenarios. To improve the efficiency of analyzing large numbers of scenarios, surrogate models provide a more efficient alternative to direct model simulation. However, their construction still depends on costly original simulator evaluations for training data, and low average prediction error from these surrogate models does not guarantee sufficient accuracy for individual scenarios or present their differences in scientific analysis. To address these limitations, we formulate simulation-efficient global change analysis as reducing original-model evaluations while maintaining accuracy for both scenario outcomes and scientifically relevant comparisons. Particularly, we propose a comparison-aware active learning method, which first constructs an input-derived comparison graph, and then estimates both outcome- and comparison-level uncertainty with an ensemble of surrogate models. Based on these estimates, a calibrated fallback strategy finally identifies scenarios that may require original-model evaluation. We further propose one acquisition strategy, which can select scenarios according to their expected reduction in future original-model evaluations. Experiments on a public energy dataset and a newly constructed dietary dataset show that our method improves performance over existing approaches, reducing the number of evaluations of the original model by 85.71% and 67.29%, respectively, while maintaining high edge sign accuracies of 96.65% and 90.22%. Overall, our method provides an efficient and reliable solution for surrogate modeling in global change analysis.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
active learning, global change analysis, surrogate model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5XZ2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes comparison-aware active learning for choosing which scenarios to run through an expensive integrated assessment model while a surrogate ensemble predicts the rest. It builds a comparison graph over scenario inputs, estimates uncertainty for single scenarios and for scenario pairs, sends high-risk scenarios back to the real model through a calibrated fallback set, and selects queries that most shrink that set. On a public GCAM energy dataset and a new 5,180-run dietary dataset, it reports the best accuracy against six baselines and 85.71% and 67.29% fewer real-model runs.

Critical strengths
  1. The problem is framed around what scientists actually use. The method targets differences between scenarios rather than only pointwise error, and its fallback set states which scenarios still need the real simulator (Section 3).

  2. The new dietary dataset is a real contribution. It contains 5,180 solved GCAM runs with a balanced design in which each transition level appears equally often in every region (Appendix A.1).

  3. The comparison with baselines is controlled. All six baselines share the surrogate architecture, training procedure, initial labeled set and budget, so differences come from the acquisition rule alone (Appendix B.1).

Critical weaknesses
  1. The fallback rule would benefit from direct validation. The paper does not yet report tolerance values or how often scenarios left to the surrogate exceed them, so further experiments would strengthen the reliability claim.

  2. The savings and the method are difficult to reconstruct. The 85.71% and 67.29% figures appear only in Figure 2, and the run counts, the baseline stopping rule and parameters such as rho_t and tau are not yet given.

  3. The evaluation could report more. The edge metrics appear to use pairs the method itself targets, the gains are modest (Diet sign accuracy 90.22% against 89.33% for Random), and Table 2 does not include variance.

I find the framing and the new dataset valuable, and I appreciate the authors' effort. However, the fallback rule is not yet validated, the savings and the method are hard to reconstruct, and the accuracy margins come without variance. I think to add fallback validation, a full configuration in the appendix and results with variance would make this paper be better.

Questions for authors
  1. Could the authors share the tolerances the fallback rule uses and how often scenarios left to the surrogate exceed them, compared with the baselines?
  2. Could the authors list in the appendix the full configuration and the run counts behind Figure 2, including calibration and fallback runs?
  3. Could the authors report Tables 2 and 3 with standard deviations and evaluate the edge metrics on pairs the method did not choose?

Suggestions that did not affect my score. It would help to put the savings for every method in a table rather than only in Figure 2, and to use parenthetical citations where the text currently uses textual ones.

Reviewer_E5hfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a comparison-aware AL method for global change analysis. The main idea is to select scenarios for expensive simulator evaluations while considering both individual predictions and differences between scenarios. The authors use a comparison graph, ensemble uncertainty, and a fallback strategy to reduce the number of simulator calls. Experiments on two GCAM datasets show some improvements over existing AL methods.

Critical strengths
  • I think the problem is interesting and has some practical value, since running large scientific simulations can be very expensive.

  • The idea of considering comparisons between scenarios, instead of only individual predictions, makes sense to me.

  • The paper is generally easy to follow, and the experiments show some improvements in prediction accuracy and simulation efficiency.

Critical weaknesses
  1. I feel the method is mainly a combination of existing ideas, such as ensemble uncertainty, graph-based comparisons, and fallback strategies. I am not very clear about the main difference from previous AL methods, especially STAP.

  2. The experiments are not fully convincing to me. The improvements over the strongest baselines seem relatively small, and the ablations do not clearly separate the contributions of comparison-aware acquisition and fallback. I think more focused experiments would help.

  3. Some important design choices are not well justified. For example, the comparison graph is constructed from input distances, but I am not sure whether these comparisons are really the ones scientists care about. Also, I would like to see more evidence that the calibrated fallback is reliable.

Questions for authors
  • Could the authors explain more clearly what is new compared with previous AL methods, especially STAP? I feel several components are already quite standard.

  • Can the authors separately test comparison-aware acquisition and comparison-aware fallback? The current No-Edges ablation removes both, so it is hard to tell which part brings the improvement.

  • The paper reports large savings compared with running all simulations, but other baselines also seem to save many simulator calls. Could the authors report the additional savings over the strongest baseline more clearly?

  • How well does the fallback strategy actually control prediction errors? Can the authors report how often the accepted surrogate predictions exceed the given error thresholds?

  • How sensitive are the results to the comparison graph? Would the method still work well with different graph constructions or a comparison set defined by domain experts?

Reviewer_8jWSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to get results for every scenario in a fixed set with as few GCAM runs as possible. The authors use an ensemble of neural networks as a surrogate. In each round, they pick a batch of scenarios to run with GCAM and then retrain the surrogate. A scenario is picked by how many final fallback runs it is expected to save, minus its own run. At the end, GCAM is run for the scenarios the surrogate is unsure about. This check looks at both the scenario itself and its differences with other scenarios. On two GCAM datasets, the total number of runs drops by about 86 percent and 67 percent compared with running every scenario.

Critical strengths
  1. The problem setting is practical, and the goal is clearly stated. In integrated assessment studies, researchers usually need results for a whole set of scenarios. The paper counts the total cost as runs during learning plus fallback runs at the end. It also requires both single results and differences between scenarios to be accurate. Conclusions are often drawn from comparing two scenarios, so this focus on differences makes sense.
  2. The method is simple and consistent, and the savings are clear. The same quantity is used to pick scenarios and to decide when to stop, which makes the method easy to follow. On both datasets, the number of GCAM runs drops sharply. The ablations also show that removing the stopping rule leads to many more runs, and removing the difference term leads to larger errors on differences.
Critical weaknesses
  1. The reliability side of the method is not yet fully checked. The paper says the method is both efficient and reliable. But it does not report how many scenarios not sent to fallback have a true error above the threshold. Both datasets have ground truth for every scenario, so this could be computed directly. The calibration uses a median, so the estimate may be optimistic. Key settings are also missing, such as the error thresholds, the size of the calibration set and the reliability discount. A looser threshold naturally means fewer fallback runs, so the savings alone are hard to judge. I suggest reporting this rate and showing how the savings change with the threshold.
  2. The construction of the comparison graph needs more explanation. The graph is built automatically from distances between input parameters. In real studies, the scenarios to compare are usually set by the research question, and they are not always close in input. The paper has no ablation on how the graph is built. The edges used to evaluate differences may also be the edges of this same graph. So it is not clear whether the results carry over to the comparisons researchers actually care about.
  3. The relation to existing active learning methods could be clearer. The selection rule is close to expected error reduction. Counting the total cost as training runs plus fallback runs is similar to the cost accounting in MCAL. Threshold based auto labeling with error control has also been studied, for example in TBAL. The related work section does not discuss these lines of work, so it is hard to see which parts are new.
Questions for authors
  1. Among the scenarios not sent to fallback, what fraction has a true error above the threshold? How do the savings change as the threshold changes?
  2. How should the comparison graph be chosen in real studies? Would the results change if the pairs were set by the research question rather than by input distance?
  3. Compared with expected error reduction, MCAL and TBAL, what is new in this method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.