acceptodds
Under review as a conference paper at ICLR 2027

Contrast-First Calibration: Reusing Labels to Compare Synthetic Sources

Abstract

A cheap synthetic source is useful when its errors can be corrected with affordable real labels. Choosing that correction also uses labels, which should remain available for training. We study how to evaluate these choices when the labels are reused. Our central protocol conditions on a finite, paid source bank and allows its data to determine the correction subspaces. For hard corrections with a common real-data count, a pairwise risk error separates into two random quantities and a deterministic retention factor. One confidence event therefore covers every later real-data budget with the same radius as the corresponding fixed-budget construction. The assessment extends to unknown Gaussian noise variance through regression residuals. For a broader pooled-data selector, a risk-localized oracle bound replaces a discrepancy-dependent square-root penalty with an additional cost on the learning-risk scale. Label-level experiments use 29 recipes, a learned correction basis, estimated variance, and cost-matched holdout certificates. They identify both successful source correction and a shared-error failure mode. A matched diagnostic separates the probability-allocation cost of a general confidence event from its coefficient-envelope cost. The resulting framework provides computable risk comparisons for retaining calibration labels and choosing subsequent acquisition plans.

Primary area:
learning theory
Keywords:
synthetic data, data acquisition, label reuse, risk calibration, source correction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DUteOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Linear regression models can be fit using expensive real data or cheaper synthetic data from approximate simulators. Assuming multiple simulators are already available, the authors ask whether predictors fit to synthetic data can be improved with corrections, which replace coefficients estimated from synthetic data with coefficients estimated from real data.

Each combination of a simulator + choice of which coefficients to correct produces a different regression model. The authors want to identify which of these models has the lowest prediction error on real data. However, using the same data to fit the models and to select the model would introduce bias. To account for this, the authors compare differences in prediction error between pairs of models, leveraging the fact that some measurement noise cancels because both are evaluated with the same data (eq. 5). Theorem 1 gives a confidence bound on how much worse the selected model is than the best possible model, including a model trained entirely on real data.

Experiments use only simulated linear regression data with Gaussian noise. The authors simulate a ground truth data generating process, along with 2 simulators with errors in their synthetic data. They compare 29 candidate models---2 models trained on synthetic data, each with 14 choices for how many coefficients to replace with real data estimates, and 1 model trained entirely on the ground truth process (“real data”). The authors find that using real data generally improves prediction when the synthetic models have errors in directions where they disagree. However, when both synthetic models share the same error, comparing them cannot reveal with direction needs correction, so the proposed comparison correction strategy fails.

Critical strengths

Theorem 1 provides a statistical guarantee for evaluating a selected regression model. However, it applies to a rather restrictive setting and relies on some strong assumptions.

Critical weaknesses
  1. Novelty seems limited. The main idea of comparing prediction errors to cancel shared noise is well established, e.g. in Stein’s unbiased risk estimation (SURE). The authors say that their risk estimator reduces exactly to SURE in a special case, and that existing work already establishes related model selection guarantees. Thus, it is unclear how much Theorem 1 (the main result) is actually extending prior work.

  2. The experiments consider 29 candidate models, based on 2 synthetic sources and 14 possible corrections per source. The choice of 14 seems arbitrary; it is unclear whether this choice was tailored to the experimental setting, and how sensitive the results are to other correction choices and other error structures.

  3. The experiments are very limited. All experiments are in toy settings with linear regression data with Gaussian noise. No real data or nonlinear models are considered.

  4. I find the language of this paper to be so bad that it significantly obstructs my ability to understand the paper. Everything is written in a contorted, overly compressed language that smacks of unfiltered Claude output.

Questions for authors
  1. Could the authors clarify what the main novelty of Theorem 1 is relative to existing SURE-based risk estimation? To my reading, the main distinction is the guarantee for adaptively chosen data budgets. What is the practical significance of this guarantee? Could the authors provide justification showing that adaptive budget selection offers a meaningful advantage over fixed budget selection?

  2. What motivates the choice of 14 correction strengths per synthetic source? How sensitive are the results to other correction choices?

2 comments
AuthorsOpenReview ↗

Response to Reviewer DUte (1/2): novelty relative to SURE, and choosing the budget after the pilot

Thank you for the summary and the direct criticism. Terms are as in the general response.

W1 and Q1: novelty relative to SURE. The overlap with SURE is substantial, and §1 and Appendix E now say so. At α=1\alpha=1 our score (3) is exactly SURE for the recipe's affine predictor, and for α<1\alpha<1 it is SURE minus a known term. At α=1\alpha=1, our identity (5) also follows from Eq. (12) of Trippe et al. (JASA 2024) combined with our score (3), and the revision now cites it. The selection results we cite (Birgé and Massart; Cauchois et al.; Baraud et al.) are oracle inequalities, not bounds computed from the data.

Neither they nor SURE give a confidence bound on the selected recipe's regret, and the paper adds three results for this. Theorem 1 bounds the errors of all pairwise risk-difference estimates at once with finite-sample confidence, so the resulting certificate holds for whichever recipe is selected. Because the later label count enters (5) only through the known factor 1+α21+\alpha^2, the same confidence statement holds for every later count, provided later batches repeat the pilot's design. Corollary 2 handles an unknown noise level.

The new Theorem 1′ (§3, Appendix H) builds the same statement from exact quantiles. It needs isotropic noise and a nested menu. In the disagreement geometry, Theorem 1′ certificates are 2.7–3.3 times narrower than those of Theorem 1. In that geometry and at a fixed budget, they are about as narrow as certificates built from the c-value bounds (median per-dataset ratio of our width to theirs: 0.88–1.05). Theorem 1′ does not dominate these bounds. The c-value certificate is narrower at dimension d=2048d=2048. It is also narrower at d=128d=128 in the shared-bias geometry, where it relies on Trippe et al.'s approximate Bound 4.1 for every pair, while ours are finite-sample.

Q1: the value of choosing the budget after the pilot. Because one confidence statement covers every later count, the pilot alone shows which counts, if any, would give a certificate within a chosen tolerance. A fixed-budget bound used this way would need a union bound over counts. In §6.2 a trial is certified when its certificate is within the tolerance 0.1σ^2d/p0.1\widehat\sigma^2d/p, where σ^2\widehat\sigma^2 estimates the noise variance, and later labels come in 0 to 4 batches of dd. On 400 pilots at d=2048d=2048, Algorithm 1, which picks the later count after the pilot, certifies 74 trials with Theorem 1. A union bound over the five counts would certify only 5, and a union bound over the two end counts, with interpolation between them, 32. For Theorem 1′, covering every count makes the certificates only 5–8% wider.

  • With Theorem 1′ at the §6.2 tolerance (Table 16, Appendix H.1), almost every pilot is certified without later labels. At the pilot, its certificate valid for every later count certifies 376 of 400 trials, its fixed-budget version 379, and the c-value certificate 377. Algorithm 1 certifies 377 and buys 0.005d0.005d extra labels on average.
  • At half the tolerance, Algorithm 1 certifies 267 trials, against 219, 262 and 274 for the same three certificates at the pilot.
  • Mean risk differs by at most 0.2% between Algorithm 1 and rules that always buy a fixed number of extra batches.

Against fixed budgets, the gain is therefore modest: in every setting we ran, our fixed-budget certificate or the c-value one, whichever was better at the pilot, certified at least as many trials as Algorithm 1.

AuthorsOpenReview ↗

Response to Reviewer DUte (2/2): the correction grid, the experiments and the writing

W2 and Q2: the 14 correction strengths. A recipe with correction strength kk uses real-label estimates for the first kk directions of a basis learned from source data. Each source uses kk = 0, 1, 2, 3, 4, 6, 8, 12, 16, 24, 32, 48, 64 and 96, a roughly geometric grid that is the same at every dimension dd, and §6.1 now defines the strength and lists the grid. We chose the grid while developing the setting and cannot show that it was fixed before the first runs, so we tested sensitivity (Appendix F.6 and Table 13 for Theorem 1, and Table 19 in Appendix H.1 for Theorem 1′).

  • With 7 or 23 strengths per source, mean risk is essentially unchanged. Certificates grow with the number of recipes, partly because the error probability is split over more pairs. For example, at d=512d=512 in the disagreement geometry, the median fixed-budget Theorem 1′ certificate rises from 0.234R00.234R_0 with 7 strengths per source to 0.302R00.302R_0 with 23.
  • We added three error structures: a bias component shared by both sources, of size 0.2 or 0.4, and a bias of one source spread over four further directions. None of our certificates or gates failed. Under these structures the holdout certificate was narrower than that of Theorem 1. Its median was still larger than the fixed-budget Theorem 1′ median in every setting except those where both medians are zero (Table 19).
  • The holdout gate still deploys a correction more often than the Theorem 1′ gate in some settings (for example, in 41.75% against 2.25% of datasets at d=512d=512 with a shared bias component of size 0.2), with lower deployed risk in some and higher in others.

W3: only simulated linear-Gaussian data. §7 states this limitation. We have not run a real-data study, because exact coverage checks need the true target. Predictors must be linear in a fixed representation, so nonlinear models are not covered. Within the theory's assumptions, the simulations are not idealized: they fit the sources from finite data, learn the basis and estimate the noise level. We also added two checks outside the theory.

  • Under t5t_5, Laplace, exponential and heteroskedastic noise, none of our certificates or gates failed. Under t3t_3 noise, each theorem's certificate failed in 3 of 1,600 datasets (Tables 14 and 20).
  • On a semi-synthetic task with real covariates and residuals from the California housing data, no certificate or gate failed in 8,000 trials (Appendix H.1). However, the heavy-tailed residuals made the nominal 99% interval for the noise variance cover it in only 89–90% of trials. No recipe from the two sources, fitted on other regions, beat all-real, so the task tests validity rather than usefulness.

W4: the writing. We agree that the submission was too compressed. The revision rewrites the abstract, §1–3 and §5–7 in plainer, shorter sentences and defines the main terms, mostly in §1, §2 and §6.1. §1 now lists the four contributions and what is not new, and it ends with a roadmap. §2, §3, §4 and §6 open by saying what they show, and §2 points to a notation table in Appendix F.8. The secondary protocol (§4–5) is shorter, and §4 now says that §6 does not depend on it. The final version will move §4–5 to the appendix.

Please tell us which proof step concerns you, and we will check it.

Reviewer_mBNbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to compare synthetic data sources and their corrections while reusing expensive real labels for training. The authors propose a framework for comparing the risks of candidate estimators across future data budgets, with additional results for unknown noise variance and data acquisition costs. Experiments are conducted on simulated regression data.

My main concerns are: (i) the presentation is difficult to follow despite careful reading, particularly the different problem settings and theoretical guarantees, and the novelty (contributions) relative to prior work remains unclear (W1); and (ii) the experiments do not sufficiently demonstrate the practical benefits of the proposed certification and adaptive acquisition procedures (W2). I would consider increasing my rating if these issues are both addressed.

Critical strengths
  • A concrete theoretical result. Under the frozen-bank setting, Theorem 1 establishes confidence bounds that remain valid across future real-data budgets without an additional uniformity penalty. Although the assumptions are restrictive, this provides a specific guarantee for budget selection after calibration.
  • Useful experimental observations. The experiments include cases where Holdout produces tighter certificates (Table 2) and where source disagreement fails to identify shared bias (§6.1). These results help clarify when the proposed approach may not be beneficial.
Critical weaknesses

W1. Insufficient differentiation from prior work

  • Presentation. The paper is difficult to follow, especially the differences between the acquisition protocols and their guarantees. Although Table 1 and Appendix E provide some clarification, the main contributions and their novelty should be explained more clearly.

  • Moreover, the connection to R1 is quite direct.

    • In 📍§2, Eq. (1), when N=pN=p, the proposed estimator becomes f^a(Y)=(I−Pa)Y+PaGa,Aa=I−Pa,ba=PaGa.\widehat f_a(Y)=(I-P_a)Y+P_aG_a,\quad A_a=I-P_a,\quad b_a=P_aG_a. This is an affine estimator.
    • Similarly, R1, §4, Approximate Bound 4.1 considers general affine estimators of the form θ^(y)=Ay+k,θ∗(y)=Cy+ℓ.\widehat{\theta}(y)=Ay+k,\qquad \theta^*(y)=Cy+\ell. For two recipes aa and bb, we can set A=I−Pa,C=I−Pb,k=PaGa,ℓ=PbGb.A=I-P_a,\quad C=I-P_b,\quad k=P_aG_a,\quad \ell=P_bG_b.
    • Furthermore, 📍R1, §4, Eq. (12) derives the pairwise loss-difference identity W(θ,y)=2ε⊤G(y)+∣θ^(y)−y∣2−∣θ∗(y)−y∣2.W(\theta,y)=2\varepsilon^\top G(y)+|\widehat{\theta}(y)-y|^2-|\theta^*(y)-y|^2. Substituting the estimators above into this identity and using the risk score in 📍Eq. (3) recovers the paper's 📍Eq. (5) at α=1\alpha=1: ea(1)−eb(1)=2gab⊤ε+2(ε⊤Dabε−tr⁡(DabS)).e_a(1)-e_b(1)=2g_{ab}^{\top}\varepsilon+2\left(\varepsilon^\top D_{ab}\varepsilon-\operatorname{tr}(D_{ab}S)\right). Thus, the fixed-budget comparison has a direct connection to an existing approach, rather than merely using a similar estimator.

I understand that R1's general affine bounds are approximate, whereas this paper provides finite-sample guarantees under specific assumptions. The budget-uniform guarantee may also be an important contribution. However, the paper does not clearly establish how much is gained over existing methods, especially for fixed-budget comparisons.

I suggest a direct theoretical and experimental comparison with R1, including assumptions, coverage guarantees, confidence bound widths, and the additional benefits of budget-uniform certification.

W2. Insufficient evidence that certification improves acquisition decisions

The main concern is whether the proposed certification framework actually helps make better data acquisition decisions. The theoretical guarantee is interesting, but the experiments do not directly establish its practical benefits.

  • §6.1: The main comparison is between Reuse and Holdout. Reuse uses all 4d4d real labels for fitting, while Holdout uses only 3d3d for fitting and reserves dd for validation. Although the comparison is cost-matched, using more labels for fitting is an expected benefit of label reuse. However, this experiment does not isolate the contribution of the proposed certification method. In particular, at N=pN=p, the selection score is SURE, as acknowledged in Appendix E. Therefore, the observed prediction improvement does not directly establish the additional value of the new confidence bounds. Comparisons with existing confidence-based methods using the same labels and candidate predictors would better demonstrate this contribution.
  • §6.2: The authors demonstrate adaptive budget selection, but only 74 of 400 trials meet the certification tolerance at some budget. More importantly, the experiments do not show whether adaptive acquisition actually saves labels or improves prediction compared with fixed-budget strategies. Also, according to 📍Eq. (6), with a fixed pilot, the confidence radius generally does not approach zero as the future sample count increases. Thus, collecting more data does not necessarily resolve the uncertainty in the initial assessment. It remains unclear when additional acquisition is useful and when the initial pilot is the limiting factor.

I suggest comparing adaptive and fixed-budget strategies under the same settings, reporting average labeling costs, final prediction errors, and certification success rates. Comparisons of empirical coverage and certificate widths against stronger confidence-based baselines would also help demonstrate the value of the proposed method. Experiments with different pilot sizes and certification thresholds could further clarify when the approach is most useful.

References

[R1] Confidently Comparing Estimates with the c-value. Trippe et al., Journal of the American Statistical Association, 2024.

Questions for authors
  1. Benefit of certification. In §6.1, beyond SURE-based selection, what additional practical decisions are enabled by the proposed certificate? Could the authors directly compare its empirical coverage, confidence bound widths, and decision outcomes with existing confidence-based methods?
  2. Adaptive acquisition. In §6.2, only 74 of 400 trials achieve certification. How does the adaptive strategy compare with fixed-budget alternatives in terms of labeling costs and prediction errors, including uncertified trials?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mBNb (1/2): relation to R1, the new Theorem 1′, a direct comparison, and the presentation

Thank you for the careful review and for the derivation connecting our Eq. (5) to R1, which we checked and agree with. Terms are as in the general response.

Relation to R1. As you show, R1's Eq. (12) with our score (3) gives our Eq. (5) at α=1\alpha=1, so the fixed-budget identity is not new, and at a fixed budget our score is SURE. The revision cites R1 throughout, and the text after Corollary 2 (unknown variance) credits R1's Remark 3.3.

What changed: Theorem 1′ (§3, Appendix H). The comparison you asked for showed that Theorem 1, which uses generic tail inequalities, was 2.5–3.2 times wider than certificates built from R1's bounds. The new Theorem 1′ uses exact quantiles and R1's noncentral chi-square confidence bound (R1, Eq. 9). It needs isotropic noise and a nested menu, as in our main experiments.

Q1: direct comparison. Table 3 (p. 10) summarizes it; Appendix F.1 and Table 15 (Appendix H.1) give details. On the 3,200 datasets of §6.1, we applied R1 to the K−1K-1 pairs that contain the selected recipe (K=29K=29 recipes). We used R1's finite-sample Bound S4.1 where it applies and its approximate Bound 4.1 otherwise, at our certificates' error probability. Unless stated, results below are for the disagreement geometry.

R1Theorem 1Theorem 1′
SettingBound 4.1: any affine pair. Bound S4.1: MLE against shrinkage toward a subspace, isotropic noise.Hard projections built from source data, a common real count, Gaussian pilot noise.As Theorem 1, with isotropic noise and a nested menu.
CoverageS4.1 is finite-sample; 4.1 is approximate.Finite-sample, from generic tail bounds.Finite-sample, from exact quantiles.
Later countsNot covered.One event, with the radius of a fixed count.One event; certificates 5–8% wider than the fixed-budget version.
Median certificate over R0R_0 at dd = 128, 512, 1024, 20481.13, 0.311, 0.142, 0.04833.31, 0.768, 0.352, 0.1541.01, 0.271, 0.126, 0.0526 at a fixed budget; 1.05, 0.287, 0.136, 0.0566 for every count

No certificate failed, and no reference gate deployed a recipe worse than all-real. The fixed-budget version of Theorem 1′ is narrower than R1 in 274, 342, 318 and 182 of 400 datasets at dd = 128, 512, 1024 and 2048. Theorem 1′ splits the confidence interval for the noise variance into four subintervals (Appendix H). With the worst case over the whole interval instead, its median certificate would be wider than R1's at every dd. R1 is narrower in all 400 datasets at d=128d=128 in the shared-bias geometry and for every single pair in Table 18 but one. In the disagreement geometry, the Theorem 1′ gate deploys a synthetic recipe in 1.25%, 93.5%, 100% and 100% of datasets at the four values of dd, against 7%, 96.25%, 100% and 100% for the gate built from R1's Bound S4.1 (Table 3).

What remains ours. Unlike R1, Theorem 1′ gives a finite-sample guarantee for every pair, and because one event covers every later count, it can certify a count before those labels are bought.

W1: presentation. We agree and have rewritten the main text. §1 now lists the contributions and what is not new, and it names the sections of each protocol. In the main protocol (§2, §3 and §6), the synthetic bank is frozen before the real pilot, only real labels are added later, and Algorithm 1 picks the later count after the pilot. Theorems 1 and 1′ and Corollary 2 bound the selected recipe's regret at any later count. In the secondary protocol (§4–5), both real and synthetic data may be added later, all counts are fixed before the pilot, and Theorem 3 bounds expected risk. Table 1 now summarizes the results by protocol. §4 now opens by saying that §6 does not depend on §4–5 and that Theorem 3 is not a certificate computed from the pilot. §5 is rewritten in plain language. The main terms are now defined, and §2 points to a notation table in Appendix F.8.

AuthorsOpenReview ↗

Response to Reviewer mBNb (2/2): does certification improve decisions? (§6.1, §6.2 and Eq. 6)

W2 (§6.1) and Q1: what a certificate adds to SURE. We agree that §6.1 did not isolate the certificate's contribution. Reuse's lower risk comes from fitting with 4d4d rather than 3d3d labels and from SURE selection, and §6.1 now says so. The certificate supports three decisions that SURE alone does not.

  1. It decides whether to deploy a correction or fall back to all-real. Appendix F.3 adds a bias that source disagreement cannot reveal and varies its size. When the best synthetic recipe is about as good as all-real, selecting by SURE alone deploys a predictor worse than all-real in 50.25% of datasets at d=512d=512, while every gate falls back to all-real. Without the added bias, a gate costs accuracy: at d=512d=512 the mean risk is 0.0831 with the Theorem 1′ gate, 0.0724 with the gate built from R1's bounds and 0.0600 with SURE (Tables 8 and 17). R1's gate gives the same protection, so this decision does not need our theory.
  2. It certifies, at a stated confidence level, that the selected recipe is within a given tolerance of the best recipe in the menu, even though the same labels chose it. R1's bounds give about the same width but are finite-sample only for some pairs.
  3. It decides, before labels are bought, whether and at which later count the choice will be certified. This needs the dependence of (5) on the later label count, which R1 does not cover.

W2 (§6.2) and Q2: adaptive versus fixed budgets. Table 16 and Appendix H.1 compare the rules on 400 pilots at d=2048d=2048 and d=1024d=1024, averaging label costs and risks over all trials, certified or not. The submission's 74 of 400 came from Theorem 1. Below, the fixed rules and Algorithm 1 use Theorem 1′'s event for every later count, a batch is dd real labels, and numbers are for d=2048d=2048 unless stated.

  • At the §6.2 tolerance, fixed rules that always buy jj = 0 to 4 extra batches (jdjd labels) certify 368 to 376 trials, and Algorithm 1 certifies 377 with 0.005d0.005d extra labels on average. The fixed-budget certificates, valid only at the pilot count, certify 379 (Theorem 1′) and 377 (R1).
  • At half the tolerance, where later labels matter, the fixed rules certify 192 to 239 trials and Algorithm 1 certifies 267 with 0.165d0.165d extra labels on average. It beats our pilot-only certificate (262) but not R1's (274).
  • Mean risk differs by at most 0.2% at d=2048d=2048 and 0.4% at d=1024d=1024 between Algorithm 1 and the fixed rules.

The acquisition gain is therefore modest. In every setting we ran, our fixed-budget certificate or R1's, whichever was better at the pilot, certified at least as many trials as Algorithm 1. In the shared-bias geometry, Algorithm 1 stops at the pilot because it targets certified regret rather than risk, although four more batches would have halved the risk there.

Eq. (6): when is the pilot the limiting factor? We agree. With a fixed pilot the radius does not vanish, because its linear term does not depend on α\alpha and the other terms shrink by at most half. The paragraph "When later labels cannot help" at the end of §3 (p. 6) now opens with this point and adds its consequence: if the certificate exceeds the tolerance for every α∈[0,1]\alpha\in[0,1], no later count can certify, and the pilot alone shows this. With Theorem 1, 287 of the 326 uncertified pilots were limited in this way (Appendix F.5). A larger pilot helps. At d=2048d=2048 and a fixed absolute tolerance, pilots of 3d3d, 4d4d, 6d6d and 8d8d labels certify 298, 376, 399 and 400 trials at the pilot with Theorem 1′ (Appendix H.1). Table 12 (Appendix F.5) varies the tolerance τ=cσ^2d/p\tau=c\widehat\sigma^2d/p over cc = 0.05, 0.1, 0.2 and 0.5 with Theorem 1, where §6.2 uses c=0.1c=0.1. At d=2048d=2048 in the disagreement geometry, Algorithm 1 then certifies 0, 74, 391 and 400 of 400 trials.

We sincerely thank the authors for their detailed responses, substantial revisions, and additional theoretical and experimental comparisons with R1. We appreciate the effort made to address our concerns.

However, I still find the presentation somewhat difficult to follow, particularly the notation and the relationships between the different problem settings and theoretical guarantees. Although the authors have added explanations and a notation table, many symbols and quantities remain difficult to interpret without repeatedly referring to other sections or appendices. For example, the relationships among the pilot size, subsequent data budgets, confidence certificates, and different risk quantities could be explained more explicitly. A more self-contained presentation of the main theoretical results, with clearly defined notation and assumptions, would significantly improve readability.

Regarding the experimental evidence, I appreciate the new comparisons and the authors' candid acknowledgment that the practical gains from adaptive acquisition are modest. Nevertheless, these results leave me somewhat uncertain about the practical advantage of the proposed framework over existing fixed-budget approaches.

I thank the authors again for their thoughtful responses and the considerable effort invested in improving the manuscript.

Reviewer_mS97OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on how to compare synthetic predictors against real-data corrections when the same real labels are used both to choose a correction and to fit the final predictor. The authors propose a confidence bound for pairwise risk differences in a finite source-conditioned menu of hard-projection corrections. The proposed method performs on simulated Gaussian regression data, and the simulation results demonstrate the effectiveness of the theoretical idea, however they come from controlled Gaussian regression instead of real image or text tasks.

Critical strengths
  1. This paper is well motivated. Reusing calibration labels for fitting can invalidate post-selection comparisons.
  2. In Eq.5, the authors directly compare two candidate corrections. This can avoid some noise they share, making the comparison more informative than estimating each candidate’s risk separately.
Critical weaknesses
  1. The main concern is that this paper lacks practical results. The results/evidence are from simulations.
  2. The authors make very strong hypotheses, like frozen source bank, hard orthogonal projections, independent Gaussian target noise, etc. This limits the practical claim in this paper.
  3. The experiments provide little evidence that the more elaborate guarantee usually changes an acquisition decision for the better. In this paper, holdout certificates are tighter at smaller dimensions, and the uniform-event construction can inflate the radius.
  4. The border proposed in this paper seems to be oversold. The contribution of this result should be stated narrowly or under stronger hypotheses.
Questions for authors
  1. How to test it on a practical task with actual synthetic and real datasets, rather than only simulated Gaussian regression?
  2. How sensitive is it to non-Gaussian noise or an incorrectly estimated noise variance?
  3. How much do the confidence bounds affect decisions?
  4. Could the guarantee be extended beyond hard orthogonal projections?
2 comments
AuthorsOpenReview ↗

Response to Reviewer mS97 (1/2): practical evidence and assumptions

Thank you for the review and for valuing direct comparison of candidates. Terms are as in the general response.

W1 and Q1: practical results. We agree that this is the main gap. We have not run a real-data study, because exact coverage checks need the true target. A practical study would fix a representation, such as frozen pretrained features, and fit a linear head. Each cheap source would be a linear head fitted on the same frozen features of its synthetic examples, and the sources and the menu would be built without real labels. A large held-out real set would approximate the target, so coverage checks would be approximate. With a balanced or whitened pilot design and homoskedastic Gaussian noise, the certificates for buying no later labels would hold for the fixed-design risk, but not for a misspecified linear head.

As a partial step, Appendix H.1 adds a semi-synthetic check with real covariates and resampled residuals from the California housing data. No certificate failed in 8,000 trials, whether it came from Theorem 1, from the new exact-quantile Theorem 1′ or from the c-value bounds of Trippe et al. (2024). No reference gate accepted a recipe worse than all-real. However, the heavy-tailed residuals made the nominal 99% variance interval cover the noise variance in only 89–90% of trials. No recipe from the two sources, fitted on other regions, beat all-real, so this checks validity rather than usefulness.

W2: strong assumptions. We agree that they limit the practical claim, and the abstract and §7 now state them. The frozen bank limits only when data are bought; the sources may be biased, non-Gaussian and fitted from finite data. The protocol requires a menu built from source data only and no synthetic data after the pilot. Hard orthogonal projections with a common real-label count make the random error terms in Eq. (5) independent of the later real-label count (Appendix A.2). Theorem 1 needs Gaussian pilot noise and allows any orthogonal projectors, and its later-count statement needs later batches that repeat the pilot's design. Theorem 1′ also needs isotropic noise and a nested menu, as when each recipe corrects the first kk directions of a basis learned from source data.

AuthorsOpenReview ↗

Response to Reviewer mS97 (2/2): decisions, claims, robustness and extensions

W3 and Q3: does the guarantee change decisions? In some settings it does.

  • The large radius inflation you noted arises only in the secondary protocol (§5), where one confidence event must cover a predeclared range of later real and synthetic counts. Its figure is now Figure 2 in Appendix C.3, and §5 and the caption state that this inflation does not apply to the main, frozen-bank protocol. There, covering every later count adds no width for Theorem 1 and 5–8% for Theorem 1′.
  • In the disagreement geometry, Theorem 1′ is about as narrow at a fixed budget as the c-value bounds, with a median per-dataset ratio of our width to theirs of 0.88–1.05. The c-value certificate is narrower at dimension d=2048d=2048, and at d=128d=128 in the shared-bias geometry. As you noted, the holdout certificate is narrower than that of Theorem 1 at small dd (Table 2), but its median is larger than that of the fixed-budget Theorem 1′ certificate in every setting except those where both medians are zero (Tables 3, 5, 15 and 19). Even so, the holdout gate deploys a correction more often than the Theorem 1′ gate in several settings, with lower deployed risk in some and higher in others.
  • When a bias hidden from source disagreement makes the best synthetic recipe about as good as all-real, choosing the smallest score without a certificate (SURE selection) deploys a predictor worse than all-real in 50.25% of datasets at d=512d=512, while every gate, ours or the c-value one, falls back to all-real (Appendix F.3 and Table 17). This protection therefore does not need our theory. When synthetic recipes are clearly better, a gate costs accuracy: at d=512d=512 the mean risk is 0.0831 with the Theorem 1′ gate, 0.0724 with the c-value gate and 0.0600 without a gate.
  • At d=2048d=2048 and the §6.2 tolerance (regret at most 0.1σ^2d/p0.1\widehat\sigma^2d/p), the Theorem 1′ certificate valid for every later count certifies 376 of 400 trials at the pilot. At half that tolerance, choosing the later count after the pilot (Algorithm 1) certifies 267 trials, against 219 for that certificate at the pilot, but the fixed-budget Theorem 1′ and c-value certificates certify 262 and 274 at the pilot. Mean risk differs by at most 0.2% between Algorithm 1 and rules that always buy a fixed number of extra labels, so the acquisition gain is modest.

W4: the claims are too broad. We agree. The abstract and §1 now state each claim with its assumptions, and §1 lists what is not new: the SURE score, the pairwise identity at a fixed budget, and the c-value's noncentral chi-square confidence bound. We do not claim narrower intervals than the c-value, and the abstract presents the expected-risk bound (Theorem 3) as a secondary result.

Q2: non-Gaussian noise or a misestimated variance. Neither theorem covers non-Gaussian noise, so Tables 14 and 20 measure its effect on 1,600 datasets per distribution. Under t5t_5, Laplace, exponential and heteroskedastic noise, none of our certificates or gates failed. Under t3t_3 noise, each theorem's certificate failed in 3 datasets, and the 99% variance interval failed to cover the true variance in 403 of 800 datasets at d=512d=512 and in 509 of 800 at d=2048d=2048. With Theorem 1, using the variance point estimate instead of the interval never caused a failure. A variance off by 20–25% in either direction made some pairwise bound fail in up to 235 of 400 datasets, but the selected certificate and the gate held. One off by a factor of two broke the selected certificate or the gate in most datasets in five of eight settings (Theorem 1, Appendix F.7). Theorem 1′ is narrower and may be more sensitive.

Q4: beyond hard orthogonal projections? The guarantee extends only in part. At a fixed budget, Trippe et al.'s finite-sample Bound S4.1 already compares a uniformly shrunk correction with all-real. In our secondary protocol (§4–5), Theorem 3 (an expected-risk bound) and Appendix C (certificates over a predeclared range of counts, at extra width) cover soft diagonal weights. The single event for every later count without extra width does not extend, because with soft corrections the cross terms between bias and noise depend on the later count. Nonlinear predictors beyond a fixed representation are not covered.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: a sharper bound (Theorem 1′), a rewritten main text, and new experiments

Thank you for the detailed reviews. We have uploaded a revised PDF. This note lists the main changes, with the sections where they appear, and defines the terms used in all replies.

  1. A sharper bound. The new Theorem 1′ (§3, Appendix H) replaces the generic tail bounds of Theorem 1 with exact quantiles for isotropic noise and a nested menu, as in our main experiments, and one confidence statement still covers every later label count. At a fixed budget it is about as narrow as the c-value bounds of Trippe et al. (JASA 2024), which compare two estimates computed from the same data, but it does not dominate them.
  2. A rewritten main text. The abstract, §1–3 and §5–7 are rewritten in plainer sentences, and the secondary protocol (§4–5), where synthetic data can also be bought later, is shorter.
  3. New experiments, simulated or semi-synthetic, are in Appendices F and H.1.

Terms. The pilot is the first batch of pp real labels, a share α=p/N\alpha=p/N of the final NN. A recipe is a synthetic source whose coefficients in some directions are replaced by real-label estimates, the menu is the list of recipes, and all-real is the recipe that uses real labels only. A recipe's regret is how much worse it is than the best recipe in the menu; its certificate is our confidence bound on that regret. A menu is nested if all recipes correct directions of one basis and, of any two recipes, one corrects a superset of the other's directions. The reference gate deploys a synthetic recipe only if it is certified to be no worse than all-real. Holdout fits on 3d3d of the 4d4d real labels and validates on the rest (dd is the dimension). In the disagreement geometry of §6.1 the two sources err by different amounts mainly along one direction, and in the shared-bias geometry they also share an error that their disagreement cannot reveal. R0R_0 is the risk of spending the whole budget on real labels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.