- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The CARD treats active dataset repair as a game over candidate conditional corruption laws. It defines worst-case expected residual error, relates the exact finite game to a least-favourable Bayesian mixture, and studies minimax, posterior, and tempered policies. Five image datasets, 23 arms, menu ablations, and small exact-planning episodes evaluate repair quality and model-relative certificates.
Critical strengths
The strength of the paper is its explicit distinction between model-set coverage and policy weighting. The finite-game formulation, Hedge-based bounds, and comparison between planning and greedy acquisition provide a coherent treatment of robust expected repair risk, while the authors acknowledge the approximation limits of the dataset-scale implementation. The evaluation is also broad, with informative controls covering published cleaners under both their native repair rules and a shared repair engine, alternative weightings, different candidate menus, and exact two-step planning episodes. I appreciate that the authors report increases in realized error and cases where it exceeds the certificate, rather than presenting only favorable certificate values. This makes the practical tradeoffs easier to assess.
Critical weaknesses
My main concern is the interpretation of the certificate. A bound on expected risk is not a confidence bound on realized error: even a correctly specified Bernoulli model with expected loss 0.1 can realize loss 1. Consequently, the reported 18/25 and 24/25 counts do not establish calibrated coverage for the released dataset. The guarantee also depends on the true conditional law belonging to the candidate hull, a condition that neither model weighting nor agreement with an observed error rate can verify. There is also a gap between the exact-game guarantees and the dataset-scale implementation. One-step replanning is a heuristic rather than an exact full-horizon best response, so the minimax bracket does not establish optimality for the deployed policy. The menu-size ablation needs clarification as well: Section 4.2 describes certification over all 24 laws, whereas Figure 3 and Appendix D.3 report menu-24 certificates over the original 16-law subset. Those values cannot establish a guarantee over the expanded uncertainty set. Finally, the experimental comparisons do not fully isolate the benefit of robust audit placement. Using ALC’s ranking with CARD’s repair engine recovers most of the headline CIFAR-10H improvement, suggesting that much of the gain comes from repair. Moreover, Appendix E uses a static ALC variant with frozen posteriors because the original update loop relies on a retired training stack. The shared-repair controls are informative, but they do not replace a comparison against a fully functioning adaptive baseline.
Questions for authors
What are the menu-24 certificates when the maximum is computed over all 24 laws, rather than the 16-law subset? What measurable optimization guarantee, if any, applies to the dataset-scale receding-horizon policy? How much advantage remains after matching the repair engine and using a functioning adaptive ALC baseline? Can existing results isolate robust acquisition gains from repair gains and the static-baseline adaptation?
4 comments
Dear Reviewer,
we would be grateful to know whether our reply and the revised PDF (changes in blue) address your concerns. We are happy to clarify anything in the remaining days of the discussion.
Thank you for your time.
Response to author
Thank you for the detailed response and the revised experiments. Recomputing the certificates over all 24 noise models resolves my concern about the menu-size comparison. The adaptive ALC baseline with a shared repair engine also addresses my attribution question. It shows that the repair engine accounts for much of the reduction in realized error and that the additional benefit of CARD’s audit placement varies by dataset and budget. The revised text also clarifies the scope of the guarantee for the deployed one-step policy.
My main remaining concern is how to interpret the new 95% certificate on real datasets. Its guarantee depends on the true corruption process being represented in the candidate menu. The 25 reported cases in which realized error fell below the certificate leave that assumption unverified. For the proposed random-audit check, can you quantify how likely it is to detect menu misspecification large enough to compromise the certificate, and how many gold-label audits that would require? This would help readers assess when they can rely on the certificate in practice and how much of the audit budget that assessment would consume.
thank you for your comment. I believe the remaining question can be answered quantitatively:
The check rejects the menu when the errors found among n random records exceed what the most pessimistic noise model in the menu considers plausible at the 95% level. Its power depends on one quantity only: the gap between the true error rate of the released dataset and the rate the menu expects. This is exactly the misspecification that matters for the certificate. A menu that is wrong in a way that does not raise the error above the certified level leaves the certificate valid.
For a given gap, the number of random audits follows from a one-sided binomial test at the 5% level:
| Error rate the menu expects | Certificate too low by | Audits for 80% power | Audits for 90% power |
|---|---|---|---|
| 10% | 10 points | 78 | 110 |
| 10% | 5 points | 278 | 393 |
| 10% | 2 points | 1,528 | 2,172 |
| 25% | 10 points | 132 | 186 |
| 25% | 5 points | 501 | 720 |
| 25% | 2 points | 3,031 | 4,234 |
On the Clothing1M release (certificate 24.7%, realised error 32.6%) this calculation predicts rejection in 45%, 67% and 97% of draws with 80, 160 and 400 records. We measured 43%, 71% and 94% (App. D.10). The 8-model menu of the ablation, whose certificate is too low by 10 points, was rejected in every draw.
What this means for a practitioner.
- The cost is fixed and small. The numbers above do not grow with the dataset: about 300 to 500 random audits detect a certificate that is too low by 5 points with 80% power, which is under 1% of a 50,000-record dataset.
- A passed check is informative. With 400 random audits, a release that passes rules out, with about 90% power, a certificate that is too low by more than about 5 points at a 10% error level. A reader can therefore attach a concrete tolerance to the certificate: "valid up to a gap of X points", with X read from the table for the audits spent.
- The check is optional and separate from the repair budget. The certificate itself is available from the first audit; the check is the additional step for releases where an independent safeguard on the menu is wanted, and the table tells the user in advance what that step costs.
We will add the table and the reading in point 2 to App. D.10, so that readers can choose the number of check audits for the tolerance they need.
We thank the reviewer for the detailed comments.
1. Expected error and realised error.
We agree that a bound on expected risk is not a bound on realised error. The certificate is defined in the paper as a bound on expected error (Section 2), not as a confidence bound, and we did not intend the 18/25 and 24/25 counts as a coverage guarantee. But we did write that the certificate "covers" the realised error, and we agree that this needs a statement about realised error. We add it (Proposition B.4): a 95% certificate, defined as the largest error that any noise model in the menu considers plausible at the 95% level. If the true noise process is in the menu, the realised error exceeds it at most 5% of the time. A textbook concentration bound might not apply here, since in our noise models the errors of one annotator can be correlated, so we compute the 95% certificate by simulation from each noise model. Table 2 now shows both certificates. The realised error is below the original certificate in 18 of 25 minimax cases and below the 95% certificate in 25 of 25. We no longer call these counts coverage.
2. Verifying the menu.
We agree that the data cannot verify whether the true process is in the menu. This is true of every guarantee that rests on a model. Every label-cleaning method assumes some structure in the noise, and most assume one (Goswami et al., 2023, Section 3). Ours assumes a menu of noise models instead of a single one, which is the point of the paper; the certificate is stated relative to the menu throughout. What we add is a check that can expose a wrong menu without gold labels, from a few hundred random audits (Section 2, App. D.10); how it works and how often it catches a wrong menu are in our reply to Reviewer kJtn. The paper also now reports Clothing1M as a dataset on which the menu is wrong.
3. Question 2.
We agree that one-step replanning is a heuristic, and the paper does not claim optimality at dataset scale. Section 3 presents the dataset-scale policy as one-step replanning, and Theorem B.4 of the submission (Theorem B.5 in the revised PDF) shows that one-step rules can be arbitrarily worse than exact planning. Section 3 now says this in one explicit sentence. However, the certificate is computed after the audits, so it is valid for any policy, including this one. Where the exact game can be solved (small episodes), exact planning lowers the certified risk by about 6% of its value on CIFAR-10H (0.0006 on 0.0100; App. C.6). We do not claim that this carries over to dataset scale.
4. Question 1.
We agree that the menu-24 certificates must be computed over all 24 noise models, not 16, and we have corrected this everywhere. Over all 24 noise models:
The certificate holds at every budget, so the claim of Section 4.3(c) stands. One number changes: the larger menu costs 1.7 to 6.1 points of certificate on Dopanim (we had written 1.7 to 3.4).
5. Questions 3 and 4.
CARD has three parts: a rule that places the audits, a repair that uses every noise model in the menu, and a certificate. Any placement rule can be combined with our repair and certificate, which is why the paper reports each baseline "with our repair". We built the adaptive ALC you asked for (ALC's selection rule unchanged, its model refit after every round of audits, within the paper's protocol of fixed image features). It matches the static version within 0.3 points at every budget.
Error (%) with 1% of records audited (all budgets: Table 13):
Repair. Repairing with the whole menu, the core idea of the paper, lowers ALC's error by 2 to 11 points in the table above, on its own.
Placement. With the same repair, our placement lowers the error further on the development set at every budget (1.0 to 2.4 points) and on CIFAR-10H; on Dopanim ALC's placement is ahead at 1% and ours at 5%. The larger difference is in the certificate. ALC audits the records most likely to be wrong, which corrects errors quickly but reveals little about how each source behaves. Consistent with this, its certificate stays large: 15.6 to 19.0% on the development set against 11.4 to 15.2% for CARD-minimax, and 6.0 to 6.4% on CIFAR-10H against 0.9 to 5.4%.