acceptodds
Under review as a conference paper at ICLR 2027

The Resolution of Noisy Validation : Model Selection From Repeated Measurements

Abstract

More proxy labels can estimate an observed distribution ever more precisely while leaving the best predictor unidentified. We characterize this obstacle for repeated measurements of a positive target. In a fixed finite panel allowing dependent reader errors, two admissible worlds share the complete observation law yet reverse the risks of the same nonconstant candidate pair. Matching bounds give a cubic resolution floor from second-order information, a quartic floor plus interface error from marginal third-order or matched signed information, and a sampling-limited frontier under exact transport. Candidate-disagreement directions map channel uncertainty to risk comparisons; one simultaneous matrix certifies exact selection, tolerance and an entire curve of minimum-improvement promises. Boundary-uniform moment geometry makes the information operational at finite noise. A pre-specified paid comparison tests the same-cost value of acquiring and exploiting signed information; same-information controls isolate processing gains. A stochastic-target control preserves every candidate risk difference and calibration interval while increasing evaluation variance. Together, these results identify what measurement information resolves a comparison, how strongly the available data can support an improvement, and which remaining uncertainty blocks the decision

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
model selection, measurement error, partial identification, uncertainty quantification, repeated measurements, weak supervision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AaHEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to compare prediction models when the true values that they are trying to predict cannot be observed directly. Instead, these values are measured by different people or instruments, and the measurements contain errors. The authors consider the setting where the measurements may be biased, where their error distributions are not fully known, and where errors from different measurements may be related.

The authors show that, under these assumptions, two settings can produce the same observed data but give opposite rankings for the same prediction models (Theorem 1). They then characterize how much the models’ true averaged squared errors can differ while it remains impossible to determine which model is better. They show that, with only information about the measurement errors’ means and variances, this ambiguity scales with the cube of the maximum error magnitude. Sufficiently precise information about the asymmetry of the errors reduces this to a fourth power, and knowing the correction for the bias removes the ambiguity entirely under unlimited data. Based on these results, the authors develop a method that estimates a range of possible corrections and uses confidence bounds to determine whether one model is better than the other.

The authors mostly test their method on simulated data. In their main experiment, the proposed method identifies 95 out of 600 cases where replacing the current model would improve prediction accuracy by a specified amount, compared with 70 for a simpler method using the same measurement budget. The authors also compare against several variants of their approach, but not directly against other established methods from related work. Experiments on real data with artificially added measurement errors show similar benefits. However, the proposed method cannot recommend any model replacements on data with actual human measurements. Thus, while the proposed method has potentially promising results in controlled settings, there is not much evidence that it is effective on real world measurement errors.

Critical strengths

The paper addresses a practically relevant problem. Their main result (Theorem 1) appears to be sound.

Critical weaknesses
  1. Model evaluation under measurement error is an established research area. In particular, there is already work on comparing models using noisy measurements and bounding model performance when true labels are unavailable. It is unclear how these results advance related work, or why these particular bounds are important in practice.

  2. The experimental results seem sensitive to the setting. We get the positive main result at a particular improvement threshold and measurement budget, but at smaller budgets, the proposed method rarely recommends any model replacements.

  3. The experiments do not show much evidence that the proposed method is effective. Most of the positive evidence comes from simulations where the measurement errors follow specified distributions. The comparisons are also mostly against variants of the proposed method, rather than methods from related work.

  4. The writing is very difficult to follow. The language is overly compressed, and there does not seem to be much attempt to have made this draft human-readable.

Questions for authors
  1. Could the authors clarify what Theorem 1 establishes beyond existing work on model comparison under measurement error and partial identification? In particular, what practical insights do the cubic/quartic bounds provide that were not available in prior work?

  2. Could the authors better justify the choice of improvement threshold and measurement budget in the main experiment? The results seem to be sensitive to these choices. Under what conditions should we expect the proposed method to outperform other alternatives?

1 comment
AuthorsOpenReview ↗

Advance over measurement-error and partial-identification work; pre-specified threshold and budget; readability

We thank the reviewer for an accurate summary and for confirming that Theorem 1 appears sound.

  1. What Theorem 1 adds. We agree that evaluation under measurement error and partial identification are established areas. Our results differ in four ways. (a) Strength of non-identification. Partial-identification evaluation (e.g., Polo et al., 2024) bounds a model's performance under weak supervision; robust-loss approaches (Patton, 2011; Hoga, 2023) preserve rankings when a conditionally unbiased proxy exists. Theorem 1 gives a stronger obstruction: for the same fixed pair of nonconstant models, two admissible worlds share the complete joint law of inputs and all readers, yet reverse the winner. This defeats every estimator, even one using all readers jointly, and allows dependent reader errors, where independence-based repeated-measurement deconvolution (e.g., Capitao-Miniconi et al., 2025) does not apply. (b) Rates, not only existence. We give matching upper and lower bounds on the unresolvable risk gap as a function of noise magnitude b and the available information: b^3 with mean/variance information, b^4 plus a cap with third-order information, and zero with exact correction (leaving n^(-1/2)). To our knowledge these orders are new. (c) The target is a comparison. Separate risk bounds for each model are usually too loose to rank models. What matters is channel uncertainty projected onto the direction in which candidates disagree (Section 4.1): the width of a risk-difference interval is exactly |beta_fg| times the multiplier width. (d) A certificate: one simultaneous matrix yields exact-selection, tolerance and minimum-improvement guarantees, plus the largest supported improvement.

Practical insights of the cubic/quartic bounds. (i) Stopping rule: if candidates differ by less than about b^3 and only variances are known, more readings of the same kind cannot decide the comparison; the budget should buy different information. (ii) Value of information: halving the noise radius shrinks the floor 8-fold with variance information but 16-fold with third-order information (small cap), and the third-order route needs a declared closure condition that data cannot verify. (iii) Abstention: by Corollary 2, an error-controlled rule must abstain on such pairs; abstention is forced by the information, not by conservative analysis. We will add a paragraph making these consequences explicit.

  1. Threshold and budget. Both were fixed before outcomes: 600 seeds, five budgets, tau = 1e-4 and one primary endpoint were locked with code/configuration hashes before the confirmation ran. Table 4 and Figure 3 re-score the same matrices at other thresholds; dual is never below second order (394 vs 381 at tau = 0; 280 vs 258 at 0.5e-4; 95 vs 70 at 1e-4; 12 vs 0 at 2e-4; 0 vs 0 at 5e-4), with zero shortfalls. Also, the improvement curve (Section 4.3) removes the need to pick tau: one matrix certifies every tau below the reported maximal supported improvement, with no new union bound.

Small budgets. At the three smallest budgets all certified methods abstain. Theory predicts this: the evaluation radius scales as n^(-1/2) and exceeds candidate gaps of order 1e-4. The uncertified point rule does update there, but with 421, 240 and 74 shortfalls per 600 pools. Keeping the incumbent is then the correct output.

When to expect gains. The margin identity (Section 4.3) gives the condition: signed acquisition helps when the reduction in calibration loss (|beta| times the interval-width reduction) exceeds the increase in evaluation radius caused by moving readings to calibration. This holds for skewed channels with valid closure at sufficient budgets (95 vs 70 confirmatory; 147 vs 60 at the moment boundary) and fails under nonzero closure (39 vs 65), which we report.

  1. Baselines and real data. Polo et al. (weak-supervision label models) and Dorner and Hardt (binary labels) target different estimands and do not apply directly to continuous positive targets with multiplicative reader bias. Our comparators instantiate the standard alternatives: the second-order certificate is the partial-identification approach using only identified first and second moments (sharp Seneta-Weber bounds), and the point rules are plug-in moment corrections that score with a single estimated multiplier. We agree that evidence under natural noise is limited: diabetes uses real features with simulated noise, and on Chaksu no action is certified. That analysis is diagnostic: neither a narrower calibration interval nor more evaluation alone would suffice, consistent with the frontier. We will scope the claims accordingly.

  2. Readability. We accept this criticism. The revision will add a plain-language example and pipeline figure before the formalism, a glossary of terms, an interpretation paragraph after each theorem, and less compressed prose, with protocol detail moved to the appendix.

Reviewer_9hgAOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper derives matching information-dependent limits on model comparison, including cubic noise resolution from second-order information, quartic resolution plus third-order interface uncertainty, and sampling-limited resolution with exact transport. This converts calibrated intervals into simultaneous selection and improvement certificates, then separates acquisition, processing, and evaluation effects experimentally.

Critical strengths

The impossibility result goes beyond examples of estimator bias: two settings with identical complete observation laws can favor opposite models within the same fixed pair of nonconstant candidates. The matching upper and lower bounds, including the sampling term under exact transport, clarify what additional information is needed to resolve a comparison. The experiments are also carefully controlled, separating acquisition strategies at the same cost from processing methods using the same information. Across 600 confirmation pools, the proposed method produces 95 useful updates versus 70 for the comparison method at the primary endpoint. The stochastic-target control further isolates evaluation uncertainty while preserving all risk differences.

Critical weaknesses

This paper's validity depends on strong channel assumptions and externally justified support and closure bounds that repeated measurements alone cannot establish; the cost of validating these inputs is also excluded from the budget comparison. The certificate applies to a fixed candidate pool, reader, and request family, leaving adaptive selection and repeated evaluation outside its current guarantees. Although the allocation controls are informative, the fixed 35/35/30 split does not establish an advantage over equally optimized second-order alternatives. Empirical support under natural measurement noise is limited: the diabetes experiments use simulated noise, and the natural-data study certifies no requested action. Finally, the benefits depend on the channel and reliability objective: signed dual produces fewer useful updates under nonzero closure (39 versus 65), while point rules achieve lower mean regret at high budgets, leaving practitioners without a validated rule for choosing the appropriate acquisition–reliability tradeoff.

Questions for authors

How are support and closure caps justified independently in the natural-data example, and what acquisition cost is omitted from the reading ledger? Can observable quantities predict when signed acquisition will lose to more evaluation, as in the nonzero-closure cohort, without using latent targets after selection? What additional assumptions or statistical accounting would extend the certificate to adaptive candidate generation or repeated evaluation, and which parts of the present guarantee would remain unchanged?

1 comment
AuthorsOpenReview ↗

Unverifiable inputs and their cost, an observable diagnostic for acquisition, and extension to adaptive evaluation

We thank the reviewer for a careful and accurate reading.

Q1. Caps and omitted costs in the natural-data example. The Chaksu analysis uses only the second-order interval, so no closure cap enters it. The support radius is not justified independently: we deliberately treat it as a scenario variable (log-error radii 0.35, 0.5, 1, 2 crossed with 5 full and 5 reader-removal panels, 100 scenarios), because the released tables contain no external evidence that could certify it. The study is therefore a diagnosis, not a validated deployment: no scenario certifies an action, and at fixed multiplier centers removing either the multiplier width or the evaluation radius alone unlocks nothing, while removing both admits 80 of 100. The ledger omits the cost of the external evidence that would justify the radius (an instrument or rating-protocol specification, or an anchored gold-standard subsample) and, for signed calibration, evidence for closure such as protocol-level separation of readers. In the synthetic studies these inputs hold by construction. We will state this omission explicitly in the ledger paragraph.

These inputs are not an artifact of our method. In the cubic construction (Appendix A), two worlds share the observed law and the observed signed mean, yet differ in their closure residuals; a small observed signed mean thus cannot certify closure, just as sample extrema cannot certify support. Any procedure claiming these resolutions must import this information; the frontier quantifies what it buys.

Q2. Predicting when signed acquisition loses, without latent targets. The exact margin identity of Section 4.3, U = -Gamma + B + V, separates the two terms that differ between methods. Their observable bounds, |beta| * |Lambda| for calibration loss and 2r for evaluation loss, are computable from readings alone; Gamma is latent but common to both methods for a given challenger. The nonzero-closure reversal is visible in observable quantities: the declared closure cap leaves the signed interval only modestly narrower (mean |Lambda| 0.0302 vs 0.0420), while moving 70% of validation readings to calibration inflates the evaluation radius by roughly sqrt(65/30) ~ 1.5. A pilot-based rule comparing predicted width reduction with predicted radius inflation is therefore observable. It is not yet validated as a pre-registered rule, and a pilot must charge its readings and failure probability. We will present it as a diagnostic, not a guarantee.

Q3. Adaptive candidates and repeated evaluation. Theorem 3 conditions on a pool, reader set and request family fixed before evaluation. Three parts are unchanged under adaptivity: the frontier (Theorem 1, Corollary 2) concerns population information and binds every rule; the calibration interval stays valid if calibration data are not used to generate candidates; and the deterministic implication "if every Delta <= U, the exact, tolerance and improvement outputs are correct" holds for any pool. Only the evaluation event changes. Candidates generated from training data or separate folds keep the guarantee by conditioning. Candidates generated using evaluation data require fresh evaluation objects, or a union bound over a predeclared reachable set (e.g., a finite search tree with alpha spending). Repeated evaluation can replace the fixed-n empirical Bernstein radius with a time-uniform empirical-Bernstein confidence sequence, keeping the endpoint-affinity argument and the M(M-1) allocation. We will add this as a remark with the required accounting.

Fixed 35/35/30 split. Agreed: the primary test compares one pre-specified signed allocation with two native second-order allocations (35/0/65 and 70/0/30), not with equally optimized ones. We do not claim allocation optimality and will say so explicitly. The 95/70 result is a same-cost test of one locked design; the 28-update dual-over-Taylor contrast on identical data does not depend on the split.

Natural noise. Correct: the diabetes study uses real features with simulated channels, and the natural-reader study certifies nothing. We will scope the claims: the evidence supports the mechanism under specified channels, and Chaksu shows diagnostic use (locating a joint bottleneck), not a deployment gain.

Reliability objective. Certificates and point rules serve different objectives. Point rules have lower mean regret at high budgets, but 421, 240, 74, 32 and 12 shortfalls per 600 pools across budgets, whereas certificates have zero. The choice is set by the objective: when a replacement must exceed tau with probability 1 - alpha, use a certificate; when only average regret matters and the budget is large, a point rule may be preferable. Between signed and second-order acquisition, the observable margin diagnostic above is our proposed guide, and the nonzero-closure cohort (39 vs 65) stays reported as a counterexample.

Reviewer_uVKWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper seems to theoretically characterize the problem of noisy measurements. Specifically, when we fail to observe the “true” label but rather only noisy measurements, the nature of the (unobservable) noise can vastly change downstream conclusions. This paper tries to provide an information theoretic characterization of what information is available to us, what sort of repeated measurements can help and how to make downstream decisions.

Critical strengths

see below

Critical weaknesses

I was unable to understand this paper. Despite two passes, I cannot say with any certainty what this paper is trying to prove or even address. There is heavy technical language used throughout, but terms are rarely properly defined or the context of these terms made clear. No examples of the problem or setup under consideration are given. None of the theoretical development is motivated. There is no discussion around the results explaining their signifi- cance or meaning. There may be interesting things in this paper, but the current version needs serious redrafting to make its message clear.

Questions for authors

n/a

1 comment
AuthorsOpenReview ↗

Plain-language restatement of the problem, the three results, and concrete presentation revisions

We thank the reviewer for the frank assessment. If a careful reader cannot extract the message after two passes, the presentation has failed, whatever the technical content. Below we restate the paper in plain terms, then list the concrete revisions.

  1. The problem, by example. We want to know which of two prediction models, f or g, has lower mean squared error against a positive target Y (e.g., a lab concentration or a clinical ratio). Y is never observed. Instead, several readers (annotators or instruments) each report a noisy log measurement Z_j = log Y + bias_j + noise_j. A common practice is to score both models against the readings X_j = exp(Z_j). But exponentiated noise inflates the reading: E[lambda_j X_j | Y] = Y only for an unknown correction factor lambda_j that depends on the bias and on the whole noise distribution. Running example (Section 2): with f = 0.45, g = 0.55 and E X = 0.515, raw scoring (lambda = 1) prefers g; if lambda lies in [0.94, 0.96], f is actually better; if we only know lambda in [0.94, 0.98], the data cannot tell.
  2. Question 1: do more readings fix this? No (Theorem 1). We construct two noise models that produce exactly the same joint distribution of inputs and all reader measurements, yet reverse which of the same two fixed models is better. Unlimited objects and readers therefore cannot decide such a comparison, and no better estimator of the observed distribution can either. Corollary 2 makes this practical: any rule whose error probability is at most alpha must abstain with probability at least 1 - 2*alpha on such a pair.
  3. Question 2: how large can the undecidable gap be, and what information shrinks it? Theorem 1 gives matching upper and lower bounds in terms of the noise magnitude b:
  • knowing only the noise mean and variance: gap of order b^3;
  • additionally knowing the noise asymmetry (third moment), or a cap on a measurable reader-disagreement statistic: order b^4 plus the cap;
  • knowing the exact correction factor: zero, leaving only the usual sampling error of order n^(-1/2). In words: the resolution of a noisy validation set is set by the kind of information available about the noise, not only by the number of readings. If two models differ by less than about b^3 and only variances are known, collecting more readings of the same kind is wasted; different information is needed.
  1. Question 3: what should a practitioner do? Section 4 turns this into an algorithm. Differences between readers on the same object cancel the unknown Y, so they yield confidence intervals for the biases, variances and optionally third moments. These give an interval for lambda. One matrix of simultaneous upper bounds on all pairwise risk differences then certifies, with probability at least 1 - alpha (Theorem 3), (a) the best model, (b) a model within tolerance epsilon, or (c) a replacement improving the incumbent by more than tau; otherwise the incumbent is kept. The same matrix reports the largest improvement the data support.
  2. Evidence. Paired worlds confirm the reversal numerically (Figure 1). In a pre-specified study of 600 candidate pools at equal measurement cost, buying third-order information yields 95 certified useful replacements versus 70 when the same budget buys more evaluation readings, with zero shortfalls for both (Table 1). We also report where it does not help: nonzero-closure channels, small budgets, and a natural-reader dataset where no action can be certified. Concrete revisions: (i) An opening paragraph and a pipeline figure built on the example above, before any formalism. (ii) A short glossary defining reader, panel, channel, bias, support radius, closure, correction (transport) multiplier, certificate, useful update and shortfall, each in one sentence at first use. (iii) A "what this means" paragraph after Theorem 1, Corollary 2 and Theorem 3, stating the practical consequence as in points 2-4. (iv) One motivating sentence at the start of each section linking it to the three questions above. (v) Moving dense protocol details from Section 5 to the appendix, and replacing compressed phrases (e.g., "interface", "transport", "promise curve") with plain descriptions. We hope this summary makes the contribution assessable. The complete proofs are in Appendices A-B, and the other reviews also assess the main theorem as sound; we would be grateful if the reviewer would reconsider the paper in light of this clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.