acceptodds
Under review as a conference paper at ICLR 2027

How Synthetic Labels Improve Conformal Prediction: A Perspective on Conditional Coverage

Abstract

Conformal prediction provides distribution-free, finite-sample marginal coverage, but post-hoc calibration data may be too limited to learn how uncertainty varies across inputs. At the same time, abundant covariates can often be labeled cheaply by existing domain models or general-purpose large language models. We study whether these synthetic labels can improve conditional coverage when only a small trusted sample is available. Building on the connection between conditional coverage and score-quantile regression, we introduce prediction-powered quantile learning: a large synthetic-labeled sample estimates pinball risk, a paired trusted sample corrects its bias, and an independent trusted split performs final conformalization. The final conformal correction changes which learning gains survive deployment. It exactly removes global threshold shifts, locally projects the remaining shape error through the boundary density, and leads to a three-resource expansion and a benefit–cost rule for synthetic power. Across eight classical regression benchmarks, synthetic-powered quantile learning substantially improves downstream conditional coverage while preserving marginal validity and producing more compact prediction sets. A realistic human-rating study further shows that labels from general-purpose language models improve final conditional coverage under a practical quality–quantity–cost tradeoff.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, conditional coverage, synthetic label, semi-supervision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QngoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether synthetic labels can improve the conditional coverage of conformal prediction when only a limited number of trusted labels are available. The authors propose prediction-powered quantile learning, which uses a bias-corrected pinball-loss objective to incorporate synthetic labels into conditional quantile estimation, followed by independent conformal calibration to preserve finite-sample marginal coverage. Theoretically, the paper analyzes the effect of final conformalization using a profiled pinball risk and a post-conformal influence map. It further derives a three-resource asymptotic error decomposition and an optimal synthetic-power rule. Experiments on eight regression datasets and HelpSteer2 demonstrate improvements in conditional coverage and prediction-set efficiency.

Critical strengths
  1. The paper goes beyond studying conditional quantile estimation accuracy and analyzes how learning errors propagate through the final conformal calibration step. In particular, the global-shift invariance and density-weighted post-conformal influence analysis provide useful insights into which components of estimation error affect downstream conditional coverage.

  2. The three-resource expansion separates the contributions of trusted learning data, synthetic-pool estimation, and final calibration. The resulting optimal-power rule provides a theoretical explanation of when synthetic labels can improve downstream efficiency and why synthetic-label quality alone does not determine their usefulness.

  3. The paper evaluates the proposed approach on eight regression datasets and an LLM-generated rating task. The experiments include comparisons with several existing conformal prediction methods and investigate synthetic-label quality, synthetic-pool size, and label scarcity.

Critical weaknesses
  1. Limited novelty of the core learning algorithm: The prediction-powered quantile-learning objective in Eq. (6) appears to be a relatively direct application of existing PPI control-variate techniques to the pinball loss, followed by established CQR or RCP calibration procedures. Related work on prediction-powered semi-supervised learning has already investigated bias-corrected learning objectives and power selection. Therefore, the main novelty of this paper seems to lie primarily in its post-conformal theoretical analysis rather than in the proposed learning algorithm itself.

  2. Insufficient empirical validation of the optimal synthetic-power analysis: Although Corollary 3.8 derives a theoretically optimal synthetic power, the main experiments use a fixed λ=1\lambda=1. The RCP-PPI-CV baseline selects λ\lambda based on trusted pinball validation, rather than the proposed post-conformal efficiency criterion. A systematic sensitivity analysis of λ\lambda, together with experiments using the theoretically motivated power estimator, would help demonstrate the practical value of the theoretical results.

  3. Insufficient justification of the augmentation baseline: RCP-Aug uses a fixed synthetic-loss coefficient of 0.5 without a clear justification or sensitivity analysis. This choice may influence its comparison with RCP-PPI, especially since RCP-Aug achieves better MSCE on some datasets. Furthermore, in the oracle-label experiments, the synthetic labels correspond to hidden true outcomes, making uniform pooling over the n+Nn+N correctly labeled samples a natural reference. Although RCP-Pool is discussed in the appendix, its results are not included.

  4. Limited analysis of the connection between theoretical efficiency and empirical conditional coverage: Theorem 3.7 characterizes stochastic error around the population-calibrated reference π0\pi_0, whereas total MSCE under model misspecification also includes approximation and interaction terms. Although the paper acknowledges this distinction, its practical implications are not thoroughly investigated. In particular, it remains unclear how well the theoretically optimal power predicts the best downstream conditional coverage performance on real datasets.

Questions for authors
  1. Could the authors clarify the methodological differences between Eq. (6) and existing prediction-powered semi-supervised learning objectives, particularly those employing bias-corrected losses and adaptive power selection? Beyond the post-conformal theoretical analysis, what are the main algorithmic innovations?

  2. Could the authors provide a λ\lambda-sensitivity analysis on the main regression benchmarks and explain why λ=1\lambda=1 is used throughout the main experiments instead of the theoretically optimal power derived in Corollary 3.8 or λ=0.5\lambda=0.5 used in RCP-Aug?

  3. What motivates the fixed coefficient of 0.5 in RCP-Aug? Would the conclusions change under different augmentation weights? In Table 11, since oracle labels correspond to hidden true outcomes, could the authors include uniform pooling over all n+Nn+N correctly labeled observations as a reference?

  4. In Table 1, RCP-Aug achieves lower MSCE than RCP-PPI on SGEMM and WEC and sometimes outperforms RCP-PPI-CV. Could the authors further investigate these cases, particularly whether the differences arise from synthetic-label bias, estimation variance, or the choice of λ\lambda?

  5. Since the optimal-power rule is derived from the leading stochastic error around π0\pi_0, how sensitive is it to misspecification of the conditional quantile model? Could the authors provide additional experiments examining whether minimizing the theoretical efficiency criterion also improves total MSCE?

Reviewer_AkVyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • This paper aims to improve the conditional coverage of conformal prediction by leveraging synthetic labels with limited trusted labels.
  • The method combines PPI-based bias-corrected quantile learning with independent conformal calibration.
  • The paper also theoretically analyzes estimation errors after conformalization and demonstrates empirical improvements across multiple benchmarks.
Critical strengths
  • The paper addresses a practical problem, which is improving conditional coverage when labeled data are scarce. This is a common setup in reality.

  • The theoretical analysis is systematic and well-developed. It characterizes the errors that remain after conformalization using the gradient and Hessian of the profiled risk. This analysis further quantifies the benefits of synthetic labels and derives their optimal contribution.

  • The experiments demonstrate clear improvements across diverse regression benchmarks and an LLM-generated labeling task.

Critical weaknesses
  • The methodological novelty appears somewhat incremental. The proposed bias-corrected quantile learning objective resembles existing PPI-based semi-supervised learning methods. It is unclear what algorithmic contributions go beyond applying this objective to the quantile learning component of RCP.

  • The theoretical contributions are not fully reflected in the proposed algorithm. Theorem 3.7 shows the effects of synthetic labels on post-conformal errors and leads to an optimal synthetic power. However, the main experiments use a fixed λ\lambda=1. It is unclear how the new theoretical analysis translates into practical advantages beyond existing PPI-based learning.

  • The scope of the theoretical improvement guarantees is limited. Theorem 3.7 shows asymptotic errors around a population-calibrated stationary target. Connecting this result to improvements in the actual conditional coverage error requires additional model specification assumptions.

Questions for authors
  • What are the main algorithmic contributions of the proposed method compared to directly applying PPI-based semi-supervised learning methods to conditional quantile regression?

  • Can the post-conformal theoretical analysis guide the practical selection of synthetic power? Would this yield additional improvements over the fixed λ\lambda=1 setting?

  • Is there a reason for restricting synthetic labels to quantile learning? Is it possible to utilize the synthetic informations to conformity score design or calibration while preserving marginal validity?

  • Does the proposed method remain more label-efficient under an equal trusted-label budget that includes the cost of training the auxiliary labeler?

3 comments

Thank you for the detailed and thoughtful rebuttal. I appreciate the additional experiments and the careful clarification of the theoretical contributions, which address many of my concerns.

I have one further question regarding scalability to high-dimensional covariate spaces, such as long-form LLM responses in LLM-as-a-judge evaluation or high-dimensional feature embeddings in scientific discovery.

Since the proposed method relies on learning an input-dependent conditional quantile through a PPI-corrected objective, I wonder whether limited trusted labels could lead to unstable optimization, high-variance bias correction, or overfitting as the covariate dimension increases. Estimating the curvature and covariance terms required for optimal synthetic-power selection may also become challenging. Although marginal coverage remains valid through independent calibration, would meaningful improvements in conditional coverage still be expected in these settings?

I understand that additional experiments may be difficult within the rebuttal period, so a conceptual discussion of these challenges, possible remedies, and expected performance would be sufficient. I would be happy to reconsider my assessment based on further clarification.

Thank you again for the thoughtful follow-up. We agree that the central question is not whether independent calibration preserves marginal validity, but whether learning the input-dependent threshold remains statistically useful when the embedding dimension exceeds the number of trusted training examples. We therefore performed a new held-out-prompt evaluation and complementary diagnostics. We report both improvements and their uncertainty below.

1. High-dimensional gains on held-out LLM-judging data

On HelpSteer2, the quantile learner receives 768-dimensional frozen prompt–response embeddings, with 100 trusted quantile-training examples, 400 auxiliary synthetic examples, a separate 100-example trusted development set for checkpoint selection, and 400 calibration observations. We evaluated 20 paired training seeds on 7,196 previously unused prompts (one fixed response per prompt). Predictions were frozen before the held-out human ratings were used for evaluation. All comparisons use matched supervised RCP, the same representation and calibration protocol.

Our empirical conditional-coverage proxy is mass-weighted, 10-group MSCE MSCE^10=∑g=110pg(c^g−0.9)2\widehat{\mathrm{MSCE}}_{10} = \sum_{g=1}^{10} p_g (\hat{c}_g - 0.9)^2. Lower is better. We additionally report 30-group and prompt-disjoint split-half estimates.

Method (768D)K10 MSCEK30 MSCESplit-half K10Empirical coverage
Supervised RCP0.0029350.0036850.0028200.8825
Flash-PPI, λ=1\lambda=10.0021230.0031400.0019980.8917
Pro-PPI, λ=0.5\lambda=0.50.0025180.0033170.0023980.8878
Pro-PPI, λ=1\lambda=10.0022490.0031360.0021210.8937

Thus, the K10 reductions are 27.7% (Flash, full power), 14.2% (Pro, half power) and 23.4% (Pro, full power) relative to matched RCP. The K30 and split-half metrics agree in direction. To assess uncertainty, we used 5,000 paired prompt-cluster bootstrap replicates; a second sensitivity also resamples the 20 training seeds:

Comparison (PPI − RCP)Δ\DeltaK10 ×103\times 10^3Prompt 95% CIPrompt + seed 95% CI
Flash, λ=1\lambda=1−0.812[−1.173, −0.512][−1.520, −0.166]
Pro, λ=0.5\lambda=0.5−0.417[−0.702, −0.162][−0.993, +0.097]
Pro, λ=1\lambda=1−0.685[−1.208, −0.248][−1.538, +0.140]

Both Pro variants improve the held-out-prompt point estimate and have negative prompt-bootstrap intervals; Flash remains negative even under joint seed-and-prompt resampling. We do not claim that Pro's gain is significant under this more demanding resampling. The intervals condition on the shared pilot and calibration sets, rather than representing repeated independent data collections. As a further sensitivity, with one response per calibration prompt (m=200m=200), Flash still has ΔK10=−0.000554\Delta\mathrm{K10}=-0.000554 and joint 95% CI [−0.001137,−0.000030][-0.001137,-0.000030].

2. The improvement is not merely better marginal coverage

For each seed, let U^=∑gpgc^g\hat U=\sum_g p_g\hat c_g. The empirical metric decomposes exactly as

MSCE^10=∑gpg(c^g−U^)2⏟between-group coverage heterogeneity+(U^−0.9)2.\widehat{\mathrm{MSCE}}_{10}=\underbrace{\sum_g p_g(\hat c_g-\hat U)^2}_{\text{between-group coverage heterogeneity}}+(\hat U-0.9)^2.

Averaging this decomposition within seeds gives:

Contrast vs. RCP (×103\times 10^3)Δ\DeltaMSCEΔ\Delta heterogeneityΔ\Delta squared marginal offset
Flash, λ=1\lambda=1−0.812−0.525−0.287
Pro, λ=0.5\lambda=0.5−0.417−0.384−0.033
Pro, λ=1\lambda=1−0.685−0.415−0.270

For the two full-power methods, approximately 65% (Flash) and 61% (Pro) of the observed grouped-MSCE reduction comes from reduced between-group coverage heterogeneity, rather than a shift in overall coverage. These are fixed-group empirical conditional-coverage diagnostics—not claims of pointwise conditional guarantees. The controlled experiment in our next comment evaluates pointwise conditional error using a known conditional CDF.

The response length and exchangeability limits are also relevant: the frozen encoder truncates each prompt/response to 256 wordpieces, so this establishes high-dimensional embedding-space, not full-length-context, scalability. Marginal rank-validity is a separate result under calibration/test score exchangeability; the observed coverage numbers and repeated-response dataset structure should not be mistaken for a validation of that assumption.

Following our held-out-prompt evidence, we address correction variance, overfitting, failure and remedies, and optimal-power estimation.

3. Does PPI correction become too noisy with few trusted labels?

Yes, it can. For fixed parameters, let gTg_T and gSg_S be trusted- and synthetic-label gradients, and let nn and NN be paired and auxiliary sample sizes. If the paired and auxiliary samples are independent and their synthetic marginals match,

Cov⁡(g^λ)=Cov⁡(gT−λgS)n+λ2Cov⁡(gS)N.\operatorname{Cov}(\hat g_\lambda)=\frac{\operatorname{Cov}(g_T-\lambda g_S)}{n}+\frac{\lambda^2\operatorname{Cov}(g_S)}{N}.

Increasing NN does not eliminate the scarce-label term. At a shared initialization in 768D, gradient-variance ratios relative to supervised learning were approximately 1.49 (Flash) and 1.59 (Pro) at λ=1\lambda=1, versus 0.92 and 0.96 at λ=0.5\lambda=0.5. Ratios at 64D were similar; the measured relative variance did not explode with dimension. It is not a trained-model variance guarantee.

In fact, with only n=25n=25 trusted training examples, full-power Flash worsened grouped MSCE by +0.000830+0.000830 (paired-seed 95% CI [+0.000144,+0.001516][+0.000144,+0.001516]). In our new n=100n=100 real-data results, PPI improves the mean group-coverage pattern while increasing between-training-seed variability. Thus we do not attribute every gain to variance reduction. Bounded threshold outputs, gradient clipping, and checkpoint selection on an independent trusted development set help with numerical stability; they do not eliminate statistical overfitting.

4. A 2,048D experiment demonstrates failure and recovery

We constructed a DGP with 2,048 ambient dimensions, four latent signal dimensions, n/N/m=100/400/400n/N/m=100/400/400, and 20 seeds. Teacher A has useful true–synthetic residual dependence (correlation 0.70.7); teacher C permutes synthetic scores. PCA32 uses independent unlabeled covariates. We fixed λ\lambda in advance and evaluated conditional error using the known conditional CDF, never used for training or model selection, on 20,000 test covariates. Each comparison uses RCP with the same representation.

Representation; teacher; λ\lambdaMatched RCPPPIΔ\Delta conditional MSCE [95% paired CI]
Native 2048D; A; 10.0208720.024395+0.003523 [+0.001207, +0.005840]
Native 2048D; A; 0.50.0208720.021561+0.000689 [−0.001486, +0.002864]
PCA32; A; 0.50.0187270.015165−0.003562 [−0.004898, −0.002226]
PCA32; A; 10.0187270.017710−0.001016 [−0.002742, +0.000709]
PCA32; permuted C; 0.50.0187270.027348+0.008621 [+0.006637, +0.010606]

Full-power PPI fails in the native space. PCA32 plus half power reduces known-CDF conditional MSCE by 19.0% even compared with PCA32-supervised RCP, improving in 18/20 seeds. Hence this is not merely PCA improving both methods. Permuting synthetic labels reverses the gain. This remedy works when low-dimensional signal is recoverable and synthetic labels are informative, not universally. Full power after PCA remains inconclusive.

5. Must the optimal power's curvature/covariance be estimated in 768D?

No. Theorem 3.7 characterizes a fixed-dimensional local post-conformal efficiency criterion. Our implementation fixes λ∈{0.5,1}\lambda\in\{0.5,1\} and needs no neural Hessian or covariance inverse. We do not claim reliable high-dimensional plug-in optimal-power estimation with 100 labels, or extend the theorem to arbitrary overparameterized networks. Appendix E.2 already documents limitations of even a small-parameter plug-in criterion. Thus optimal-power estimation is a possible refinement, not a prerequisite for implementing PPI.

We also checked stronger supervised baselines: full-rank PCA preprocessing (still 768D) reduces HelpSteer2 supervised K10 MSCE from 0.002935 to 0.000994. Matched Pro half power further reaches 0.000818, but its interval crosses zero; PCA64 Pro can instead worsen error. We therefore do not claim PPI beats every well-conditioned supervised model. A prompt-disjoint sensitivity reverses Pro's point estimate but also changes unique trusted prompts (~90 to 50); thus overlap is not causally isolated.

In summary: high-dimensional, label-scarce conditional gains are possible and observed on new prompts, but correction noise, effective representation complexity, and synthetic information quality determine whether they persist. Our experiments demonstrate both failure and a concrete reduced-power/low-rank recovery regime. The separate marginal-validity statement still requires calibration–test score exchangeability; it is not substituted for these conditional-coverage measurements.

Reviewer_VhjqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses synthetic labels from an auxiliary labeler (a domain model or an LLM) to improve the conditional coverage of conformal prediction when trusted labels are scarce. Building on RCP, it learns the score quantile with a bias-corrected prediction-powered pinball loss and keeps an untouched trusted split for the final correction, which preserves marginal validity. Theory links the profiled pinball risk to post-calibration coverage error and gives a rule for the synthetic power. Experiments on eight regression benchmarks and HelpSteer2 show better conditional coverage and smaller sets.

Critical strengths
  1. Prediction-powered quantile learning improves conditional coverage over its supervised counterpart on almost all datasets for both RCP and CQR. RCP-PPI improves WSC and L1-ERT on every dataset and MSCE on all but one, while also producing more compact sets. The comparisons are tightly controlled, with shared splits, architectures, and initializations, and a permutation placebo for the augmentation variant shows that synthetic labels carry input-specific information beyond generic regularization.

  2. Profiling the pinball risk over the final correction shows that only threshold-shape errors weighted by boundary density survive conformalization. It follows that a labeler’s usefulness is governed by boundary-aligned information rather than global accuracy, which the HelpSteer2 labeler comparison illustrates well. The analysis also yields an interpretable benefit–cost rule for the synthetic power.

Critical weaknesses
  1. While the insight in Strength 2 is valuable, the theory does not cover the regime of the experiments. The labelers are deterministic given XX, so by the paper’s own Cor. B.3, PPI can help to first order only through model misspecification. The quantitative checks of the theory (App. A.10), however, assume a different regime, and the fixed-dimensional analysis is hard to apply to networks with ~10410^4 parameters trained on a few hundred points. Since the method itself is a fairly direct combination of PPI and RCP, this matters: the theory is meant to be the main contribution, and it doesn’t yet explain the observed gains.

  2. Simple augmentation (RCP-Aug) performs as well as or better than RCP-PPI on several datasets, and on most datasets with the weak labeler, and no experiment shows the bias failure that PPI is supposed to prevent. Without such evidence, it’s hard to see why one should prefer the proposed method over the simpler baseline.

Questions for authors
  1. (suggestion) Could you estimate bb and cc on the real datasets using the cross-fitting procedure from App. C.6? It would be interesting to see whether the rule 2b^>(1+κ)c^2\hat b > (1+\kappa)\hat c actually predicts where unit PPI helps and where it doesn’t.

  2. (suggestion) Could you add an experiment where the labeler is deliberately biased in a covariate-dependent way, and report paired RCP-PPI vs. RCP-Aug comparisons?

  3. (clarification) What weight decay and output bounds do you use? Given that the unit-power objective may be ill-posed (Prop. C.5), how sensitive are the results to these choices and to the number of epochs?

Minor comments:

  • HelpSteer2 is not introduced in the experimental setup (Sec. 4.1) and is not cited. In the main text, “Flash” and “Pro” are never identified; the DeepSeek models are named only in App. A.8, without citation or version/access date. The eight regression benchmarks and the all-MiniLM-L6-v2 encoder are not cited anywhere in the paper.
  • Sec. 1.1: “Related Works” → “Related Work”.
  • Eq. (3): HET is never expanded, and MSCE is first spelled out only in Sec. 4.1. That expansion (“mean-squared conditional error”) also differs from the one in the Table 1 caption (“mean-squared conditional-coverage error”).
  • Terminology: “labeler” and “teacher” are used interchangeably (Sec. 4.2); please use one term.
  • Dataset names are inconsistent across tables (Diamond vs. Diamonds; Cond. vs. Supercond.).

Comments on the paper

2 comments

Follow-up on Author Responses

We sincerely thank the reviewers and AC for their time and consideration.

As the discussion period is approaching its end, we would greatly appreciate it if the reviewers could revisit our detailed responses and manuscript revisions, and share whether their main concerns have been adequately addressed.

If any substantive concerns remain, we would be happy to clarify them further. We would also appreciate the AC's help in facilitating the discussion before the deadline.

Dear Authors,

Thank you for your message and for your efforts in addressing the reviewers' comments. The reviewers have been reminded to examine your responses, and have been encouraged to engage in further discussion if they deem it necessary.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.