- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Threshold-Aware Conformal Routing (TACR) to reduce expensive simulator calls when downstream decisions depend on a fixed scalar threshold. TACR learns input-dependent scales through a threshold-weighted objective that encourages tighter intervals near the decision boundary. Standard split-conformal calibration is then applied to preserve marginal coverage, and inputs whose intervals intersect the threshold are routed to simulation. Experiments on six datasets show reduced simulation rates compared with standard CP.
Critical strengths
-
The paper is generally well organized and easy to follow. The connection between CP and simulation routing is interesting and practically relevant.
-
The proposed method is well aligned with the problem formulation. The threshold-aware interval construction and routing rule directly correspond to the goal of reducing unnecessary simulator calls, while standard conformal calibration provides a principled coverage guarantee.
-
The experiments cover six datasets and several surrogate architectures. The threshold and backbone sensitivity analyses further strengthen the empirical evaluation.
Critical weaknesses
-
Eq. (3) aims to minimize the expected routing rate, whereas Eq. (6) minimizes a threshold-weighted approximation of average interval width. Minimizing the latter does not necessarily imply a lower routing probability. Moreover, the theoretical analysis in Section 3.3 mainly establishes the standard conformal coverage guarantee and its implied decision-error bound, rather than explaining why the proposed objective should improve routing efficiency. A more direct justification of this connection would strengthen the paper's central claim.
-
Since the proposed TACR fine-tunes the predictor, the improvements over Standard CP may arise from various reasons, e.g., changes in predictor accuracy and the learned interval allocation. Although the Uniform ablation partially addresses this concern, similar g-MAE does not necessarily imply comparable threshold-decision accuracy. Moreover, the improvements over Uniform are less convincing on AhmedML and ShapeNet-Car (Table 3).
-
Most of the baselines focus on conventional uncertainty estimation or interval construction, rather than directly optimizing the decision of whether to defer to a more expensive model. Given that the paper discusses selective prediction, learning-to-defer, and decision-aware conformal methods in Section 2, comparisons with stronger routing-oriented alternatives, where the settings can be made comparable, would provide a more convincing assessment of the proposed method.
Questions for authors
-
Can the authors evaluate TACR and Uniform using the same frozen predictor, training only their respective scale heads? This would more directly isolate the contribution of threshold-local interval allocation from changes in prediction quality.
-
Is there empirical or theoretical evidence that minimizing Eq. (6) consistently decreases the actual routing probability in Eq. (3)? In particular, how sensitive is the training objective to the mini-batch quantile approximation?
-
Can the authors try more relavant routing-oriented baselines?
Rebuttal Summary
During the discussion period, we responded to the concerns raised by all three reviewers through detailed clarifications and extensive new experiments. To summarize:
All new experiments and analyses are included in the revised manuscript.