acceptodds
Under review as a conference paper at ICLR 2027

Threshold-Aware Conformal Routing

Abstract

High-fidelity simulations are essential to scientific and engineering design, but can be expensive to run repeatedly. Learned surrogates offer a faster alternative, yet their higher errors may alter downstream decisions. This accuracy-speed tradeoff creates a need to determine whether a surrogate can be used or the full simulator remains necessary. We study decisions determined by whether a scalar quantity of interest lies above or below a fixed threshold. For each input, we use the surrogate when its conformal interval lies entirely on one side of the threshold and route the input to simulation when the interval intersects it. Standard conformal prediction constructs intervals without reference to the downstream decision threshold: even a narrow interval near the threshold can cross it and trigger simulation, whereas a wider interval farther away can remain entirely on one side and require no simulation. We introduce Threshold-Aware Conformal Routing (TACR), which learns an input-dependent scale using a threshold-aware objective that concentrates interval tightness near the decision boundary. Exact split-conformal calibration on held-out data preserves distribution-free marginal coverage, which also upper-bounds the probability of an incorrect threshold decision that is not routed. Across various scientific and engineering datasets, TACR reduces simulator deferrals by 14-75% relative to standard conformal prediction at the same coverage target. Against a variant without threshold-local weighting but with similar predictor accuracy, TACR further reduces deferrals by 10-24% on four datasets. These results show that optimizing interval allocation for routing can reduce simulator calls without weakening the standard conformal guarantee.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Conformal Prediction, Selective Prediction, Learning to Defer, Scientific Machine Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1yiHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Threshold-Aware Conformal Routing (TACR) to reduce expensive simulator calls when downstream decisions depend on a fixed scalar threshold. TACR learns input-dependent scales through a threshold-weighted objective that encourages tighter intervals near the decision boundary. Standard split-conformal calibration is then applied to preserve marginal coverage, and inputs whose intervals intersect the threshold are routed to simulation. Experiments on six datasets show reduced simulation rates compared with standard CP.

Critical strengths
  1. The paper is generally well organized and easy to follow. The connection between CP and simulation routing is interesting and practically relevant.

  2. The proposed method is well aligned with the problem formulation. The threshold-aware interval construction and routing rule directly correspond to the goal of reducing unnecessary simulator calls, while standard conformal calibration provides a principled coverage guarantee.

  3. The experiments cover six datasets and several surrogate architectures. The threshold and backbone sensitivity analyses further strengthen the empirical evaluation.

Critical weaknesses
  1. Eq. (3) aims to minimize the expected routing rate, whereas Eq. (6) minimizes a threshold-weighted approximation of average interval width. Minimizing the latter does not necessarily imply a lower routing probability. Moreover, the theoretical analysis in Section 3.3 mainly establishes the standard conformal coverage guarantee and its implied decision-error bound, rather than explaining why the proposed objective should improve routing efficiency. A more direct justification of this connection would strengthen the paper's central claim.

  2. Since the proposed TACR fine-tunes the predictor, the improvements over Standard CP may arise from various reasons, e.g., changes in predictor accuracy and the learned interval allocation. Although the Uniform ablation partially addresses this concern, similar g-MAE does not necessarily imply comparable threshold-decision accuracy. Moreover, the improvements over Uniform are less convincing on AhmedML and ShapeNet-Car (Table 3).

  3. Most of the baselines focus on conventional uncertainty estimation or interval construction, rather than directly optimizing the decision of whether to defer to a more expensive model. Given that the paper discusses selective prediction, learning-to-defer, and decision-aware conformal methods in Section 2, comparisons with stronger routing-oriented alternatives, where the settings can be made comparable, would provide a more convincing assessment of the proposed method.

Questions for authors
  1. Can the authors evaluate TACR and Uniform using the same frozen predictor, training only their respective scale heads? This would more directly isolate the contribution of threshold-local interval allocation from changes in prediction quality.

  2. Is there empirical or theoretical evidence that minimizing Eq. (6) consistently decreases the actual routing probability in Eq. (3)? In particular, how sensitive is the training objective to the mini-batch quantile approximation?

  3. Can the authors try more relavant routing-oriented baselines?

Reviewer_jiJEOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper considers incorporating downstream decision-making tasks into the construction of prediction sets to reduce the need for simulations (which I understand as a way of obtaining ground-truth labels). The main idea is to construct smaller prediction sets near the decision boundary while allowing larger prediction sets farther away from the boundary, where larger sets have little impact on the final decisions. Experimental results show that the proposed method requires fewer simulations than the baseline methods.

Critical strengths

This paper incorporates downstream decision-making tasks into the construction of prediction sets.

Critical weaknesses
  1. The paper does not provide a theoretical analysis of the proposed method.
  2. The proposed optimization objective is not aligned with the population-level objective.
Questions for authors
  1. The paper does not provide a theoretical analysis of the reduction in the number of simulations achieved by the proposed method. The theoretical results presented in the paper are based on standard split conformal prediction.
  2. The population objective is to minimize the expected routing rate E[r(X)]\mathbb{E}[r(X)] in Eq. (3). However, the proposed method instead optimizes a weighted prediction interval length. Why not directly optimize a smooth approximation of the routing indicator, such as ∑i∈Bsigmoid⁡(q^Bσi−∣g^i−τ∣)\sum_{i\in B}\operatorname{sigmoid}\left(\hat{q}_B\sigma_i - |\hat{g}_i-\tau|\right)? In this setting, a wider prediction interval may still be preferable if it results in a lower routing rate. I believe directly optimizing this surrogate would be more appropriate. Otherwise, the authors should provide a clear justification for their choice or an empirical comparison between the two objectives.
  3. The introduction does not clearly explain the motivation behind the proposed method or sufficiently highlight its novelty compared with existing work.
  4. Existing studies have already considered optimizing prediction-set losses under coverage constraints and using prediction sets for downstream decisions. Since the routing loss can be expressed as L(C)=1τ∈C\mathcal{L}(C)=\mathbf{1}\\{\tau\in C\\}, what is the key novelty of TACR compared with these existing frameworks? I suggest that the authors clarify this distinction and, if possible, provide empirical comparisons with these methods Cortes-Gomez et al. (2025), Kiyani et al. (2025) and Yeh et al. (2025).

Reference:

Cortes-Gomez et al. (2025). Utility-Directed Conformal Prediction: A Decision-Aware Framework for Actionable Uncertainty Quantification. ICLR 2025.

Kiyani et al. (2025). Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents. ICML 2025.

Yeh et al. (2025). Conformal Risk Training: End-to-End Optimization of Conformal Risk Control. NeurIPS 2025.

Reviewer_ZiqhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a practical surrogate–simulator handoff problem by using threshold-aware conformal intervals to decide when the surrogate estimate can be trusted and when high-fidelity simulation is required. A neural surrogate predicts the simulator field from which a task-specific scalar functional is computed. The authors use a held-out calibration set to construct a conformal interval around this quantity. If the calibrated interval lies on one side of the threshold, the surrogate prediction is accepted. Otherwise it's routed to the simulator.

The proposed Threshold-Aware Conformal Routing (TACR) learns a scale and relies on a local weighting objective to assign interval width around the decision boundary. TACR is compared with several baselines: Standard CP (Conformal Prediction), CQR (Conformalized Quantile Regression), a Gaussian scale-learning approach (TW-UQ), a variation without the local weighting (Uniform), and a five-member Deep Ensemble, which provides an additional uncertainty-based comparison.

The central objective is to reduce high-fidelity simulator calls while preserving the reliability of downstream threshold decisions. TACR often reduces the simulation rate, particularly relative to TW-UQ and Uniform on several datasets. The main question is whether the reported routing improvements come from threshold-aware interval allocation or from the different training budget, and changes to the underlying surrogate output.

Critical strengths
  1. Well-motivated problem. The use of cheap surrogate when it is sufficiently reliable, and of the simulator only when necessary is a major concern in practice.
  2. Clear conformal formulation. The method separates the surrogate output, the derived scalar quantity, the conformal interval, and the routing decision. The coverage argument is standard and correctly applied.
  3. Interesting methodological idea. TACR learns an input-dependent scale and uses threshold-local weighting to allocate interval width according to the relevance of the fixed threshold.
  4. Broad empirical evaluation. The experiments cover six scientific and engineering datasets and include several useful comparisons, including Standard CP, CQR, Uniform, TW-UQ, and a Deep Ensemble.
Critical weaknesses
  1. Unmatched training budgets. Standard CP and CQR use a single (E_1) epoch stage, while TACR uses (E_1+E_2), with additional updates to both the surrogate and scale head. TACR uses a different lr from other baselines on AhmedML dataset . Any changes in the predictor impact the SIM% and g-MAE, this is backed by the correlation between NEEDS% and g-MAE. This makes me unable to attribute the improvements to the method rather than the training. Either all models need to be train with the full epochs or the surrogate predictor weights need to be fixed and only the scale and additional heads are finetuned.

  2. The routing mechanism is not isolated from surrogate quality. The model predicts (\hat y), from which (\hat g=g(\hat y)) is computed. Consequently, changes to the surrogate output affect g-MAE, NEEDS%, and routing behavior. A frozen-surrogate comparison is missing.

  3. Ensemble comparison does not fully support the claims. Each ensemble member is trained on the shorter (E_1) schedule, while TACR uses (E_1+E_2), resulting in five weaker-members. The results support an observation about the chosen ensemble configuration, not a conclusion about predictive uncertainty.

  4. No direct selective-routing baseline is included. CQR is an appropriate conformal baseline, but it is not specifically trained to identify when the surrogate-derived (g) estimate is likely to be wrong. A calibrated NEEDS/error gate would provide a more direct comparison. The authors discuss Uddin et al. (2026) approach (described as closest to the work) but never compare against it.

Questions for authors
  1. Can you report the results for CP, CQR, and Deep Ensemble trained with the same total training budget (E_1+E_2)?
  2. Can you reproduce the TACR main table results (for AhmedML) with the same base lr as other baselines.
  3. Can you freeze a common surrogate (f_\theta) and compare constant-scale CP, Uniform, and TACR under identical surrogate predictor?
  4. Can you compare TACR with a calibrated gate trained directly to predict the NEEDS event? Uddin et al. (2026) is discussed but never compared against.
  5. Can you reproduce Figure 3, under the settings of question 1&2 and while including CQR.
  6. Does the work mainly focus on downstream tasks, or also the full fields? My take from Section 3.1 is the former. However, in the case of the latter full field metrics need to be reported as well.

My concerns are directed at the experimental setup rather than the idea itself. If the questions are addressed and the method's gains persist, I would be happy to raise my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

During the discussion period, we responded to the concerns raised by all three reviewers through detailed clarifications and extensive new experiments. To summarize:

  • Reviewer Ziqh (2 → 3): We addressed their concerns about training-budget fairness, predictor attribution, and routing baselines through matched-budget experiments, frozen-predictor controls on all six datasets, and three routing-oriented baselines. The reviewer had stated they "would be happy to raise my score" if the gains persisted. After reviewing our results, they appreciated our "additional experiments and discussion," and increased their score to Weak Accept.
  • Reviewer jiJE (1 → 2): We addressed the objective-alignment concern through theoretical clarification and a direct-objective comparison. The reviewer acknowledged that "your clarification has helped me better understand the relationship between the two objectives" and increased their score.
  • Reviewer 1yiH (holding at 2): We addressed concerns about objective alignment, predictor attribution, and routing baselines through theoretical clarification and substantial new experiments. The reviewer thanked us for the "additional explanations and experiments" but maintained their score without identifying specific remaining concerns.

All new experiments and analyses are included in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.