- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes SUIT, which separates prediction interval widths into a scale function learned from simulator data and a multiplicative scalar correction calibrated on real data. The function captures how error magnitudes vary across inputs, while the correction is obtained from quantiles of normalized residuals, globally or within strata. The paper analyzes coverage under simulator mismatch and develops a coverage audit to guide recalibration.
Critical strengths
The problem setting is real and well-motivated: target labels are expensive, a stochastic simulator is available, and the simulator is imperfect.
Critical weaknesses
-
The writing and presentation are difficult to follow, particularly in the abstract and introduction. These sections introduce many technical concepts and claims without first making the central idea clear, making it unclear what problem the method solves, how the proposed method addresses it, and what the main contribution is?
-
The core score and interval construction follow locally weighted split conformal prediction [1], which calibrates absolute residuals divided by an estimated local error scale. Although SUIT learns this scale from simulations, the additional practical benefits from its shift analysis and coverage audit remain insufficiently established. In particular, the audit is not compared with directly estimating coverage from real data.
[1] Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., and Wasserman, L. (2018). Distribution-Free Predictive Inference for Regression. Journal of the American Statistical Association, 113(523), 1094-1111.
Questions for authors
-
The paper calls a "shape" function, while Assumption 2 uses "shape" to describe the distribution of . Could the authors distinguish these two meanings and clarify what is assumed to transfer from the simulator to the real environment?
-
Standard conformal prediction guarantees marginal coverage under exchangeability without Assumptions 1-2. Which additional guarantees rely on these assumptions, and what evidence supports their applicability to the NDV and species tasks?
-
The paper describes zero-shot deployment, but the deployment audit requires real labels. Could the authors clarify the total label budget, distinguishing labels used for auditing from those used for calibration? Can the same observations be reused, and what guarantees support this?
-
Section 5.1 states that CQRD is fitted on the calibration points themselves. Are separate samples used for fitting and conformal calibration? Please clarify the experimental setup and explain the very low NDV coverage in Table 2.
-
Table 2 targets coverage of but highlights the smallest widths among methods reaching only . Why is this lower threshold used?