- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper focuses on the label noise detector design issue and formulates the detector design as common search space search. This paper also provides a characterization of exactly what information determines a detector's response to a proposed intervention, then a mechanism for turning that characterization into cheaper algorithm search. The three contributions 'Unified characterisation', 'Mechanism prediction' and 'Economical algorithm search' are tightly cooperated.
Critical strengths
-
The source–operator–query decomposition is clean, genuinely unifying, and maps credibly onto the major detector families.
-
Four Theorems are proposed to ensure the therotical proof of the solution.
-
The experiment discussion are sufficient.
Critical weaknesses
-
The evaluation datasets are small. More large-scale dataset is highly suggested. Also, the advantage of Economical algorithm search can be tested.
-
In the related work section, the cited works are old. More recent works should be discussed.
-
Overclaims issue. Some exepriment results can not support the claims in the abstract, such as "zero development regret under 0.1 pp tolerance" will not reassure a reader who cares about whether the selected detector actually generalizes.
Questions for authors
Some visualization results of the label denoising are suggested via the proposed method.
3 comments
Response to Reviewer 35Jy (Part 2/2)
W3. Development selection and generalisation. We have revised the abstract to state explicitly that the method matches “development-set selection accuracy”, identifying the objective on which the reported comparison is established. The revised search-cost table also distinguishes zero observed development regret from the prescribed 0.1-pp certification tolerance.
Matching exhaustive selection is useful for configuring a detector for the dataset under investigation. To address the additional question of population performance, we now make the statistical consequence explicit in Section 4 and Appendix J.1. For a fixed candidate family and an independent, representative development sample, simultaneous empirical–population AUROC error at most gives
\leq \tau + 2\epsilon,$$ where $\tau$ is the computational selection regret and $2\epsilon$ is the statistical uncertainty. The uniform bound accounts for selection across the candidate family, including adaptive computational refinement. We state the sampling and independence assumptions explicitly; constructing additional candidates using development labels requires further statistical control. This connects our computational guarantee to population performance through standard model-selection reasoning. Please also see our response to Reviewer guvR, W3 and Q4, for further discussion of the statistical assumptions, control of development-set selection overfitting, and held-out evaluations. **We have also added substantial held-out evidence for this rebuttal.** Relative to the predefined interior reference configurations, the new 60-family study yields **59/60 positive held-out changes**, with median **$+2.334$ pp**. The new large-scale CIFAR-100N study yields **6/6 positive held-out changes**, with median **$+4.653$ pp**. These are dependent family-level comparisons that reuse datasets and partitions. The earlier mixed held-out outcomes remain reported. The revised manuscript therefore presents three complementary pieces of evidence: **computational accuracy relative to exhaustive selection, a population-regret consequence under stated assumptions, and measured held-out performance**. The abstract now describes the first precisely, while the main text provides the statistical connection and empirical transfer results. ### Response to Questions **Q1. Qualitative visualisation.** We have added a 16-image Food-101N illustration in **Appendix T.3, Figure 4**, showing how the selected configuration changes individual review decisions. **Review rank** orders images from most to least suspicious according to the detector’s score: rank 1 receives the highest review priority. With a review budget of 80 images, ranks 1–80 enter the review queue. Each arrow shows the rank under the reference configuration followed by the rank under the selected configuration upon applying our method. Recorded labels and their correctness statuses are supplied by Food-101N. The illustration contains four categories: - **Incorrect labels brought into review:** The first three columns of the top two rows show six annotation errors previously outside the review queue. For example, an image incorrectly labelled *crème brûlée* moves from rank **436 to 35**, while another incorrectly labelled *peking duck* moves from **286 to 56**. These errors become accessible within the same review budget. - **Incorrect labels narrowly missed:** The final column of the top two rows shows errors promoted to ranks **98 and 94**. Their priority improves, often quite significantly (from 168 and 225 respectively), but they remain just outside the queue. - **Correct labels removed from review:** The first three columns of the bottom two rows show six correctly labelled images no longer flagged. For example, a correctly labelled pancake image moves from **36 to 608**, and a correctly labelled chicken-curry image moves from **62 to 503**, avoiding unnecessary review. - **Correct labels still flagged:** The final column shows remaining false alarms near the cutoff, with modest rank changes of **78 to 70** and **68 to 63**. These examples make the practical consequence of configuration selection concrete: changing the configuration can surface previously missed annotation errors and release correctly labelled examples from unnecessary review. Together with the quantitative selection and timing studies, the panel illustrates what economical configuration search enables at the sample level, while also showing the mistakes that remain.Large-scale Evaluation and Further Evidence on Held-out Generalisation
We have now completed the promised large-scale evaluation on the full Food-101N dataset, which comprises 310,009 images of which the train set has 294,851 images across all 101 classes and the remainder is split into development and test. This setting is this over seven times larger than the initial 40,000-example subset evaluated in the original submission. The results are incorporated into Section 5 and Appendix T.5, and the anonymous code package has been updated.
a. Selection accuracy at substantially larger scale
The full-population core study uses three fresh seeds and two candidate layouts, giving six search families, each containing 486 detector–training configurations. Screening, certification and optimised successive halving recover every exhaustive development optimum, with zero observed selection regret. The prescribed certification tolerance remains 0.1 pp.
These experiments use fixed image features and trainable multiclass prediction models. We additionally completed twelve full-population searches at 32 and 64 feature dimensions; point prediction, four-trajectory screening and halving again recover every optimum. Training uses recorded labels, with verification statuses reserved for evaluation.
b. Further held-out evidence at scale
Our previous responses introduced the population-selection analysis and expanded cross-dataset held-out evaluation. The new Food-101N study adds evidence at a substantially larger training scale.
Every configuration selected in the six full-population core families improves held-out detection AUROC over its predefined interior reference, with a median improvement of +5.301 pp. Configuration choices are fixed before held-out scores are evaluated.
These results extend the held-out evidence already reported for CIFAR-100N and the cross-dataset robustness study. The population bound and assumptions described in our previous responses continue to provide the statistical interpretation of selection.
c. Computational efficiency against stronger baselines
We compare our forecasting, screening and certified-selection procedures against optimised successive halving and exhaustive evaluation. The three variants of our method serve different purposes:
- Point prediction chooses a promising training trajectory from inexpensive forecasts and then executes it.
- Four-trajectory screening forecasts candidate performance, trains four promising trajectories and selects using their scores.
- Certified selection additionally maintains performance bounds, refining comparisons or evaluating further trajectories until no alternative can improve upon the selected configuration by more than 0.1 pp, under the stated assumptions.
On a predeclared full-population case, three paired repetitions give:
| Search policy | Role | Median search time |
|---|---|---|
| Point prediction | Our simplest forecasting control | 8.022 s |
| Four-trajectory screening | Our economical search procedure | 9.104 s |
| Query-refined certification | Our certified search procedure | 15.349 s |
| Optimised successive halving | Comparison baseline | 18.028 s |
| Optimised exhaustive evaluation | Exact reference baseline | 33.201 s |
Our screening procedure is 1.99× faster than halving and 3.66× faster than exhaustive evaluation. Our certified procedure achieves corresponding speed-ups of 1.17× and 2.16×. Every repetition has zero observed development regret.
Point prediction remains cheapest; screening provides additional verification, while certification supplies explicit selection-error control. Appendix T.5 reports the complete accuracy–cost comparison.
Response to Reviewer 35Jy (Part 1/2)
We thank the reviewer for their constructive assessment and suggestions. We have revised the manuscript and updated the accompanying anonymous code package with the additional experiments and reproduction instructions. In particular, we have added larger-scale and broader search evaluations, strengthened the discussion of recent related work, clarified the connection between development-set selection and population performance, and included an image-level visualisation. We address each concern below.
Response to Weaknesses
W1. Scale, diversity and economical algorithm search. In direct response to this concern, we have conducted new large-scale experiments for this rebuttal on CIFAR-100N, using 40,000 training examples, all 100 classes and human-corrupted labels. Three fresh seeds and two candidate layouts give six search families, each containing 486 detector–training configurations. Screening, certified search and successive halving each recover all six exhaustive development optima with zero observed selection regret. This complements our existing 40,000-example Food-101N evaluation across all 101 classes.
The new rebuttal experiments directly measure the advantage of economical search against both optimised exhaustive evaluation and successive halving. Three clean paired repetitions on a predeclared CIFAR-100N case give median screening speed-ups of 13.57× over exhaustive evaluation and 6.67× over halving. Certified search achieves 7.71× and 3.79×, respectively. The existing large Food-101N experiment independently demonstrates a 6.83× certified speed-up over exhaustive evaluation.
For this rebuttal, we also completed a new 60-family robustness study across Dry Bean, PenDigits, Food-101N and human-noise CIFAR-10N, varying seeds, Cartesian and irregular candidate layouts, and 8–64 feature components. The unchanged four-trajectory screen recovers every development optimum. Searches vary detector readouts—loss and area under the margin—example weights, regularisation, learning rates and stopping times. Together with the propagation-selection study and the new controlled feature-learning extension, this consistency supports the generality of the information–prediction–selection approach across datasets and algorithmic mechanisms.
To extend the scale further, we are additionally in the process of evaluating the entire Food-101N dataset, which has more then 290,000 training examples covering all 101 classes -- over seven times our current largest training population. We will provide an update to our rebuttal once we have obtained the respective results. Together, these evaluations broaden the evidence across training populations, class counts, representations and candidate families. If the reviewer has further suggestions with regards to this point, we would welcome them.
W2. Recent related work. We have expanded and reorganised Section 7 to distinguish recent detector developments from broader frameworks that organise the detector algorithm space. The revised discussion includes model probing (George et al., TMLR 2024), which organises detector construction through models, probes, ensembles and aggregation, and ADGym (Jiang et al., NeurIPS 2023), which constructs and selects component-based anomaly-detection pipelines. We also added discussion of DynaCor (CVPR 2024) and LEMoN (ICML 2025) to connect this conceptual positioning to contemporary detector developments.
The distinction matters because our principal contribution concerns a characterisation of the algorithm space and a principled way to navigate it. Entirely new organising paradigms and unified characterisations are comparatively uncommon; newer individual detectors often develop particular mechanisms within an existing paradigm. Within this specific conceptual scope, our literature review has not identified a more recent direct precedent for the characterisation-to-search programme developed here.
To our knowledge, ours is the first unified operator-theoretic characterisation of label-noise detection that connects intervention-specific information requirements to reusable performance predictions and economical configuration selection with explicit selection-error control. The revised related-work discussion makes these connections and their relationship to existing approaches explicit.
We would welcome specific references the reviewer considers relevant, particularly newer organising frameworks, and would be pleased to discuss their relationship to our contribution.