- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how to compare prediction models when the true values that they are trying to predict cannot be observed directly. Instead, these values are measured by different people or instruments, and the measurements contain errors. The authors consider the setting where the measurements may be biased, where their error distributions are not fully known, and where errors from different measurements may be related.
The authors show that, under these assumptions, two settings can produce the same observed data but give opposite rankings for the same prediction models (Theorem 1). They then characterize how much the models’ true averaged squared errors can differ while it remains impossible to determine which model is better. They show that, with only information about the measurement errors’ means and variances, this ambiguity scales with the cube of the maximum error magnitude. Sufficiently precise information about the asymmetry of the errors reduces this to a fourth power, and knowing the correction for the bias removes the ambiguity entirely under unlimited data. Based on these results, the authors develop a method that estimates a range of possible corrections and uses confidence bounds to determine whether one model is better than the other.
The authors mostly test their method on simulated data. In their main experiment, the proposed method identifies 95 out of 600 cases where replacing the current model would improve prediction accuracy by a specified amount, compared with 70 for a simpler method using the same measurement budget. The authors also compare against several variants of their approach, but not directly against other established methods from related work. Experiments on real data with artificially added measurement errors show similar benefits. However, the proposed method cannot recommend any model replacements on data with actual human measurements. Thus, while the proposed method has potentially promising results in controlled settings, there is not much evidence that it is effective on real world measurement errors.
Critical strengths
The paper addresses a practically relevant problem. Their main result (Theorem 1) appears to be sound.
Critical weaknesses
-
Model evaluation under measurement error is an established research area. In particular, there is already work on comparing models using noisy measurements and bounding model performance when true labels are unavailable. It is unclear how these results advance related work, or why these particular bounds are important in practice.
-
The experimental results seem sensitive to the setting. We get the positive main result at a particular improvement threshold and measurement budget, but at smaller budgets, the proposed method rarely recommends any model replacements.
-
The experiments do not show much evidence that the proposed method is effective. Most of the positive evidence comes from simulations where the measurement errors follow specified distributions. The comparisons are also mostly against variants of the proposed method, rather than methods from related work.
-
The writing is very difficult to follow. The language is overly compressed, and there does not seem to be much attempt to have made this draft human-readable.
Questions for authors
-
Could the authors clarify what Theorem 1 establishes beyond existing work on model comparison under measurement error and partial identification? In particular, what practical insights do the cubic/quartic bounds provide that were not available in prior work?
-
Could the authors better justify the choice of improvement threshold and measurement budget in the main experiment? The results seem to be sensitive to these choices. Under what conditions should we expect the proposed method to outperform other alternatives?
Advance over measurement-error and partial-identification work; pre-specified threshold and budget; readability
We thank the reviewer for an accurate summary and for confirming that Theorem 1 appears sound.
Practical insights of the cubic/quartic bounds. (i) Stopping rule: if candidates differ by less than about b^3 and only variances are known, more readings of the same kind cannot decide the comparison; the budget should buy different information. (ii) Value of information: halving the noise radius shrinks the floor 8-fold with variance information but 16-fold with third-order information (small cap), and the third-order route needs a declared closure condition that data cannot verify. (iii) Abstention: by Corollary 2, an error-controlled rule must abstain on such pairs; abstention is forced by the information, not by conservative analysis. We will add a paragraph making these consequences explicit.
Small budgets. At the three smallest budgets all certified methods abstain. Theory predicts this: the evaluation radius scales as n^(-1/2) and exceeds candidate gaps of order 1e-4. The uncertified point rule does update there, but with 421, 240 and 74 shortfalls per 600 pools. Keeping the incumbent is then the correct output.
When to expect gains. The margin identity (Section 4.3) gives the condition: signed acquisition helps when the reduction in calibration loss (|beta| times the interval-width reduction) exceeds the increase in evaluation radius caused by moving readings to calibration. This holds for skewed channels with valid closure at sufficient budgets (95 vs 70 confirmatory; 147 vs 60 at the moment boundary) and fails under nonzero closure (39 vs 65), which we report.
Baselines and real data. Polo et al. (weak-supervision label models) and Dorner and Hardt (binary labels) target different estimands and do not apply directly to continuous positive targets with multiplicative reader bias. Our comparators instantiate the standard alternatives: the second-order certificate is the partial-identification approach using only identified first and second moments (sharp Seneta-Weber bounds), and the point rules are plug-in moment corrections that score with a single estimated multiplier. We agree that evidence under natural noise is limited: diabetes uses real features with simulated noise, and on Chaksu no action is certified. That analysis is diagnostic: neither a narrower calibration interval nor more evaluation alone would suffice, consistent with the frontier. We will scope the claims accordingly.
Readability. We accept this criticism. The revision will add a plain-language example and pipeline figure before the formalism, a glossary of terms, an interpretation paragraph after each theorem, and less compressed prose, with protocol detail moved to the appendix.