acceptodds
Under review as a conference paper at ICLR 2027

Learn the Shape, Calibrate the Scale: Simulator-Trained Conformal Prediction under Sim-to-Real Shift

Abstract

Conformal prediction provides finite-sample coverage guarantees, but adaptive intervals are difficult to deploy when labeled target data are scarce, even when a simulator can provide abundant synthetic data. We propose SUIT, a two-stage framework that learns the input-dependent uncertainty shape from the simulator and uses a small number of real samples to calibrate a scalar scale correction. This decomposition separates transferable uncertainty structure from the sim-to-real gap: zero-shot prediction is possible when the simulator is faithful, scalar calibration removes arbitrary global scale shifts, and stratification controls the residual spatial heterogeneity of the scale ratio. A conservative, computable audit of zero-shot degradation provides a stratum-wise lower bound on coverage from a small number of real audit points and flags inconclusive strata for recalibration. Experiments on distinct-value estimation, species-richness estimation, simulation-based inference, and synthetic regression show near-nominal zero-shot coverage under faithful simulation and coverage recovery with only 100 real samples under a 10× scale mismatch, at substantially narrower intervals than global conformal baselines.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, sim-to-real, heteroscedastic UQ, amortized inference, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bqGFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SUIT, which separates prediction interval widths into a scale function learned from simulator data and a multiplicative scalar correction calibrated on real data. The function captures how error magnitudes vary across inputs, while the correction is obtained from quantiles of normalized residuals, globally or within strata. The paper analyzes coverage under simulator mismatch and develops a coverage audit to guide recalibration.

Critical strengths

The problem setting is real and well-motivated: target labels are expensive, a stochastic simulator is available, and the simulator is imperfect.

Critical weaknesses
  • The writing and presentation are difficult to follow, particularly in the abstract and introduction. These sections introduce many technical concepts and claims without first making the central idea clear, making it unclear what problem the method solves, how the proposed method addresses it, and what the main contribution is?

  • The core score and interval construction follow locally weighted split conformal prediction [1], which calibrates absolute residuals divided by an estimated local error scale. Although SUIT learns this scale from simulations, the additional practical benefits from its shift analysis and coverage audit remain insufficiently established. In particular, the audit is not compared with directly estimating coverage from real data.

[1] Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., and Wasserman, L. (2018). Distribution-Free Predictive Inference for Regression. Journal of the American Statistical Association, 113(523), 1094-1111.

Questions for authors
  • The paper calls q(x)q(x) a "shape" function, while Assumption 2 uses "shape" to describe the distribution of ZZ. Could the authors distinguish these two meanings and clarify what is assumed to transfer from the simulator to the real environment?

  • Standard conformal prediction guarantees marginal coverage under exchangeability without Assumptions 1-2. Which additional guarantees rely on these assumptions, and what evidence supports their applicability to the NDV and species tasks?

  • The paper describes zero-shot deployment, but the deployment audit requires real labels. Could the authors clarify the total label budget, distinguishing labels used for auditing from those used for calibration? Can the same observations be reused, and what guarantees support this?

  • Section 5.1 states that CQRD is fitted on the calibration points themselves. Are separate samples used for fitting and conformal calibration? Please clarify the experimental setup and explain the very low NDV coverage in Table 2.

  • Table 2 targets coverage of 0.950.95 but highlights the smallest widths among methods reaching only 0.900.90. Why is this lower threshold used?

Reviewer_oEV4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SUIT, a simulator-trained conformal prediction framework that separates input-dependent uncertainty shape from a domain-dependent scale. The shape is learned using abundant simulated data, while a small number of real observations are used for scalar calibration. The authors establish theoretical coverage bounds under multiplicative heteroscedasticity, introduce stratification to control residual scale heterogeneity, and develop a moment-based certificate for auditing zero-shot coverage. Experiments across four problem families demonstrate promising coverage–efficiency trade-offs, particularly under substantial global scale shifts.

Critical strengths
  1. The methodological framework can be practically useful. I think the shape–scale decomposition provides a novel strategy for leveraging abundant simulated data while minimizing real-label requirements (though the corresponding structural assumptions may be restrictive). The ability to reuse a single uncertainty model across coverage levels and deployment domains is particularly appealing.

  2. The theoretical analysis are well-motivated and supported by diverse experiments. The level–dispersion decomposition explains why scalar calibration can absorb arbitrarily large global scale shifts, while stratification controls remaining heterogeneity. The controlled shift experiments and real-world NDV results provide useful empirical support for this mechanism.

Critical weaknesses
  1. Violation of the official ICLR formatting requirements (major concern). The submitted PDF appears to use approximately 1-inch side margins and a 6.5-inch text width, whereas the official ICLR template specifies 1.5-inch side margins and a 5.5-inch text width. This provides approximately 18% additional horizontal space and substantially increases the amount of content accommodated per page. Since the main paper already occupies nine pages, compliance with the page limit under the required formatting is questionable. The authors should correct the formatting and ensure that the paper satisfies the official submission requirements.

  2. The structural assumptions in this paper are substantially restrictive. The theoretical guarantees rely on multiplicative heteroscedasticity and shape preservation, requiring standardized residuals to be independent of covariates and identically distributed across simulated and real environments. These assumptions exclude realistic shifts in residual distribution shapes, such as changes in tail behavior. Moreover, zero-shot validity under disjoint covariate supports requires accurate extrapolation of the learned scale function. These strong assumptions limit the generality and practical applicability of the proposed guarantees.

  3. The validation for zero-shot audit is insufficient. The proposed certificate can remain highly conservative even with thousands of labeled audit samples, raising questions about its practical usefulness in a label-scarce setting. A particularly important missing baseline is direct coverage certification using exact binomial confidence bounds on the same audit samples, which requires substantially weaker assumptions. Equal-budget comparisons against this simple alternative, together with validation under more realistic noise distributions, would strengthen the empirical contribution.

Questions for authors
  1. How robust is SUIT when the standardized residual distribution differs between the simulator and the real environment, particularly in terms of tail behavior or covariate-dependent shape changes?
  2. Given that the proposed zero-shot audit already requires labeled real-world observations, why not directly estimate coverage using binomial confidence bounds on the observed coverage indicators?
Reviewer_irzBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Simulator-trained Uncertainty Intervals via Two-stage calibration (SUIT), which decomposes conformal predictive uncertainty, in heteroskedastic regression settings, into a shape and scale factor. The premise of this approach is that the shape, the more complex term, can be learned from nearly unlimited synthetic training examples, while the scale, which ensures marginal coverage under exchangeability, can be learned from a few real calibration samples and absorbs the sim-to-real gap. The authors then introduce two coverage lower bounds, one which quantifies the marginal coverage degradation under zero shot transfer from the simulator to real data, and the other which is a high probability bound on the conditional coverage after calibrating against real data. Simulations show that their approach is able to attain at least the nominal α=0.95\alpha = 0.95 confidence level with tighter prediction sets than a wide range of baselines approaches.

Critical strengths
  1. This paper studies an important problem which is how to construct adaptive conformal prediction intervals using synthetic data. Synthetic data is often abundantly cheap to generate, whereas reserving real observations for calibration is ``expensive'' since decreases the number of samples for training the underlying model. Being able to offload most of the calibration work to cheap synthetic data can, therefore, be highly beneficial.

  2. The authors provide several new theoretical results on conformal prediction describing how coverage degrades as a function of simulator fidelity and shape learning. In particular, the high probability bound on the conditional coverage as a function of calibration is quite interesting.

Critical weaknesses
  1. This manuscript uses heavily modified version of the ICLR template to fit more material in 9 pages than is allowed.

  2. While the application to this setting is novel, the proposed method seems to be the classic studentized residual method introduced in Papadopoulos et al (2002) with the variance model trained on simulator data and rescaled on calibration data.

  3. Simulations include a wide variety of datasets but do not investigate coverage as a function of shape mismatch or shape error. Assumption 2 posits that the shape is the same from sim to real, but this is strong assumption that seems likely to be violated in practice. Theorem 3 in the appendix does provide a lower bound in terms of the KS distance, but its not clear how strongly it could deviate from the nominal level in practice. It would have been good to see simulations or real data examples showing how coverage degrades as the shape-gap increases.

Questions for authors
  1. Section 5.2 claims that when the simulator is faithful, then no calibration data are needed and no other method is defined. However, could it not be argued that in this case your simulator data IS your calibration data, so all methods would work?

  2. Are the target variables in all of these datasets univariate? Would it still be reasonable to learn a single shape model on simulation and expect it to generalize to the real data in higher dimensional settings?

  3. Why is SUIT's, and to some extent SUIT-G's, empirical coverage so high in Table 4 across all datasets? Relatedly, on the SSB dataset most methods have rel. width on the order of 10510^5, is there an estimation error here?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.