- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Linear regression models can be fit using expensive real data or cheaper synthetic data from approximate simulators. Assuming multiple simulators are already available, the authors ask whether predictors fit to synthetic data can be improved with corrections, which replace coefficients estimated from synthetic data with coefficients estimated from real data.
Each combination of a simulator + choice of which coefficients to correct produces a different regression model. The authors want to identify which of these models has the lowest prediction error on real data. However, using the same data to fit the models and to select the model would introduce bias. To account for this, the authors compare differences in prediction error between pairs of models, leveraging the fact that some measurement noise cancels because both are evaluated with the same data (eq. 5). Theorem 1 gives a confidence bound on how much worse the selected model is than the best possible model, including a model trained entirely on real data.
Experiments use only simulated linear regression data with Gaussian noise. The authors simulate a ground truth data generating process, along with 2 simulators with errors in their synthetic data. They compare 29 candidate models---2 models trained on synthetic data, each with 14 choices for how many coefficients to replace with real data estimates, and 1 model trained entirely on the ground truth process (“real data”). The authors find that using real data generally improves prediction when the synthetic models have errors in directions where they disagree. However, when both synthetic models share the same error, comparing them cannot reveal with direction needs correction, so the proposed comparison correction strategy fails.
Critical strengths
Theorem 1 provides a statistical guarantee for evaluating a selected regression model. However, it applies to a rather restrictive setting and relies on some strong assumptions.
Critical weaknesses
-
Novelty seems limited. The main idea of comparing prediction errors to cancel shared noise is well established, e.g. in Stein’s unbiased risk estimation (SURE). The authors say that their risk estimator reduces exactly to SURE in a special case, and that existing work already establishes related model selection guarantees. Thus, it is unclear how much Theorem 1 (the main result) is actually extending prior work.
-
The experiments consider 29 candidate models, based on 2 synthetic sources and 14 possible corrections per source. The choice of 14 seems arbitrary; it is unclear whether this choice was tailored to the experimental setting, and how sensitive the results are to other correction choices and other error structures.
-
The experiments are very limited. All experiments are in toy settings with linear regression data with Gaussian noise. No real data or nonlinear models are considered.
-
I find the language of this paper to be so bad that it significantly obstructs my ability to understand the paper. Everything is written in a contorted, overly compressed language that smacks of unfiltered Claude output.
Questions for authors
-
Could the authors clarify what the main novelty of Theorem 1 is relative to existing SURE-based risk estimation? To my reading, the main distinction is the guarantee for adaptively chosen data budgets. What is the practical significance of this guarantee? Could the authors provide justification showing that adaptive budget selection offers a meaningful advantage over fixed budget selection?
-
What motivates the choice of 14 correction strengths per synthetic source? How sensitive are the results to other correction choices?
2 comments
Response to Reviewer DUte (2/2): the correction grid, the experiments and the writing
W2 and Q2: the 14 correction strengths. A recipe with correction strength uses real-label estimates for the first directions of a basis learned from source data. Each source uses = 0, 1, 2, 3, 4, 6, 8, 12, 16, 24, 32, 48, 64 and 96, a roughly geometric grid that is the same at every dimension , and §6.1 now defines the strength and lists the grid. We chose the grid while developing the setting and cannot show that it was fixed before the first runs, so we tested sensitivity (Appendix F.6 and Table 13 for Theorem 1, and Table 19 in Appendix H.1 for Theorem 1′).
- With 7 or 23 strengths per source, mean risk is essentially unchanged. Certificates grow with the number of recipes, partly because the error probability is split over more pairs. For example, at in the disagreement geometry, the median fixed-budget Theorem 1′ certificate rises from with 7 strengths per source to with 23.
- We added three error structures: a bias component shared by both sources, of size 0.2 or 0.4, and a bias of one source spread over four further directions. None of our certificates or gates failed. Under these structures the holdout certificate was narrower than that of Theorem 1. Its median was still larger than the fixed-budget Theorem 1′ median in every setting except those where both medians are zero (Table 19).
- The holdout gate still deploys a correction more often than the Theorem 1′ gate in some settings (for example, in 41.75% against 2.25% of datasets at with a shared bias component of size 0.2), with lower deployed risk in some and higher in others.
W3: only simulated linear-Gaussian data. §7 states this limitation. We have not run a real-data study, because exact coverage checks need the true target. Predictors must be linear in a fixed representation, so nonlinear models are not covered. Within the theory's assumptions, the simulations are not idealized: they fit the sources from finite data, learn the basis and estimate the noise level. We also added two checks outside the theory.
- Under , Laplace, exponential and heteroskedastic noise, none of our certificates or gates failed. Under noise, each theorem's certificate failed in 3 of 1,600 datasets (Tables 14 and 20).
- On a semi-synthetic task with real covariates and residuals from the California housing data, no certificate or gate failed in 8,000 trials (Appendix H.1). However, the heavy-tailed residuals made the nominal 99% interval for the noise variance cover it in only 89–90% of trials. No recipe from the two sources, fitted on other regions, beat all-real, so the task tests validity rather than usefulness.
W4: the writing. We agree that the submission was too compressed. The revision rewrites the abstract, §1–3 and §5–7 in plainer, shorter sentences and defines the main terms, mostly in §1, §2 and §6.1. §1 now lists the four contributions and what is not new, and it ends with a roadmap. §2, §3, §4 and §6 open by saying what they show, and §2 points to a notation table in Appendix F.8. The secondary protocol (§4–5) is shorter, and §4 now says that §6 does not depend on it. The final version will move §4–5 to the appendix.
Please tell us which proof step concerns you, and we will check it.
Response to Reviewer DUte (1/2): novelty relative to SURE, and choosing the budget after the pilot
Thank you for the summary and the direct criticism. Terms are as in the general response.
W1 and Q1: novelty relative to SURE. The overlap with SURE is substantial, and §1 and Appendix E now say so. At α=1 our score (3) is exactly SURE for the recipe's affine predictor, and for α<1 it is SURE minus a known term. At α=1, our identity (5) also follows from Eq. (12) of Trippe et al. (JASA 2024) combined with our score (3), and the revision now cites it. The selection results we cite (Birgé and Massart; Cauchois et al.; Baraud et al.) are oracle inequalities, not bounds computed from the data.
Neither they nor SURE give a confidence bound on the selected recipe's regret, and the paper adds three results for this. Theorem 1 bounds the errors of all pairwise risk-difference estimates at once with finite-sample confidence, so the resulting certificate holds for whichever recipe is selected. Because the later label count enters (5) only through the known factor 1+α2, the same confidence statement holds for every later count, provided later batches repeat the pilot's design. Corollary 2 handles an unknown noise level.
The new Theorem 1′ (§3, Appendix H) builds the same statement from exact quantiles. It needs isotropic noise and a nested menu. In the disagreement geometry, Theorem 1′ certificates are 2.7–3.3 times narrower than those of Theorem 1. In that geometry and at a fixed budget, they are about as narrow as certificates built from the c-value bounds (median per-dataset ratio of our width to theirs: 0.88–1.05). Theorem 1′ does not dominate these bounds. The c-value certificate is narrower at dimension d=2048. It is also narrower at d=128 in the shared-bias geometry, where it relies on Trippe et al.'s approximate Bound 4.1 for every pair, while ours are finite-sample.
Q1: the value of choosing the budget after the pilot. Because one confidence statement covers every later count, the pilot alone shows which counts, if any, would give a certificate within a chosen tolerance. A fixed-budget bound used this way would need a union bound over counts. In §6.2 a trial is certified when its certificate is within the tolerance 0.1σ2d/p, where σ2 estimates the noise variance, and later labels come in 0 to 4 batches of d. On 400 pilots at d=2048, Algorithm 1, which picks the later count after the pilot, certifies 74 trials with Theorem 1. A union bound over the five counts would certify only 5, and a union bound over the two end counts, with interpolation between them, 32. For Theorem 1′, covering every count makes the certificates only 5–8% wider.
Against fixed budgets, the gain is therefore modest: in every setting we ran, our fixed-budget certificate or the c-value one, whichever was better at the pilot, certified at least as many trials as Algorithm 1.