- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CleanCore is a coreset selection method for training deep models on large tabular datasets that contain several kinds of errors at once. It works in stages, using training signals each time. It first sorts samples by error type (label, feature or mixed), then applies a light fix to each type and adjusts each sample’s reliability weight. Finally it selects a small, representative coreset weighted by reliability. Tested against 11 baselines on 11 public datasets with MLP and tabular ResNet models, it gets the best accuracy and F1 in almost every setting. It also runs about 2–24× faster than full data-cleaning methods.
Critical strengths
- Combining data cleaning with coreset selection is an interesting idea.
- The method’s design makes sense at a high level.
- End-to-end results are good, with consistently higher accuracy/F1 and lower runtime.
Critical weaknesses
- 7 of the 11 datasets have fewer than 100K instances, which makes coreset selection unnecessary for them. More realistic, large-scale datasets (more instances and more features) would be better.
- Standard deviations are not reported, which matters because the performance gaps are not large.
- Several key assumptions are not verified, either theoretically or empirically. This includes the accuracy of the high-risk screening filter and of the label-consistency and feature-sensitivity checks.
- The method has many hyperparameters. It's unclear how to set the values in practice.
Questions for authors
- What are the precision and recall of the simple filter used for high-risk screening?
- What are the precision and recall of the label-consistency and feature-sensitivity checks?
- Does the coreset selection part follow or extend an existing framework? If so, it needs to be cited.
- How should the hyperparameters be set in practice?
8 comments
To W2
Thank you for pointing out the importance of reporting variability in experimental results. We selected MAGIC, Sensorless, MiniBooNE, and Seismic Bumps to cover different sample sizes, binary and multiclass classification tasks, and balanced and imbalanced class distributions. We used MLP as the downstream classifier and conducted paired repetitions of CleanCore and five baselines with five new random seeds (72–76). The table below reports ACC (%), F1 (%), and TIME (s) as means ± sample standard deviations.
Table 2. Predictive performance and runtime across random seeds.
| Dataset | Metric | CTRL | Cleanlab | CoreTab | GradMatch | Deem | CleanCore |
|---|---|---|---|---|---|---|---|
| MAGIC | ACC | 84.62 ± 0.27 | 81.46 ± 0.36 | 80.22 ± 0.79 | 80.58 ± 1.27 | 83.84 ± 0.60 | 84.82 ± 0.19 |
| MAGIC | F1 | 82.67 ± 0.28 | 76.95 ± 0.42 | 78.37 ± 1.05 | 78.03 ± 0.37 | 81.08 ± 0.85 | 82.68 ± 0.47 |
| MAGIC | TIME | 43.46 ± 6.05 | 23.67 ± 6.81 | 5.04 ± 0.41 | 2.94 ± 0.27 | 3.48 ± 0.23 | 3.16 ± 0.18 |
| Sensorless | ACC | 93.35 ± 0.16 | 92.78 ± 0.13 | 87.04 ± 0.68 | 91.28 ± 0.73 | 72.07 ± 2.33 | 94.58 ± 0.55 |
| Sensorless | F1 | 93.21 ± 0.19 | 92.74 ± 0.14 | 86.96 ± 0.73 | 91.13 ± 0.81 | 70.91 ± 2.58 | 94.54 ± 0.56 |
| Sensorless | TIME | 93.98 ± 4.48 | 101.75 ± 4.38 | 16.78 ± 6.51 | 13.28 ± 2.36 | 12.95 ± 7.29 | 9.24 ± 3.10 |
| MiniBooNE | ACC | 89.17 ± 0.06 | 88.91 ± 0.18 | 87.99 ± 0.61 | 79.18 ± 5.59 | 85.61 ± 2.18 | 89.43 ± 0.20 |
| MiniBooNE | F1 | 86.57 ± 0.05 | 85.21 ± 0.14 | 84.39 ± 1.02 | 68.40 ± 8.50 | 79.60 ± 4.53 | 86.93 ± 0.21 |
| MiniBooNE | TIME | 182.91 ± 12.10 | 208.54 ± 7.33 | 27.07 ± 4.50 | 28.58 ± 12.33 | 27.13 ± 3.87 | 21.66 ± 3.89 |
| Seismic Bumps | ACC | 91.45 ± 0.75 | 91.88 ± 0.31 | 86.81 ± 2.00 | 91.49 ± 0.39 | 91.95 ± 0.62 | 92.92 ± 0.40 |
| Seismic Bumps | F1 | 52.83 ± 1.94 | 53.89 ± 3.96 | 52.59 ± 1.99 | 49.33 ± 2.36 | 52.81 ± 4.47 | 57.80 ± 2.25 |
| Seismic Bumps | TIME | 17.71 ± 1.90 | 15.31 ± 4.22 | 2.16 ± 0.39 | 2.83 ± 0.28 | 2.47 ± 0.15 | 1.77 ± 0.50 |
In the experiments repeated over five random seeds, CleanCore achieved the highest mean ACC and F1 among the six methods on all four datasets. On each dataset, CleanCore's sample standard deviations for ACC and F1 were lower than the average sample standard deviations of the five baselines for the corresponding metrics. These repetitions show that CleanCore maintains high classification performance on datasets of different sizes and class distributions and performs relatively consistently across random seeds. CleanCore achieved better predictive performance than the data cleaning baselines while delivering approximately 7.5–13.8× speedups. With runtime efficiency comparable to or better than that of the coreset selection baselines, CleanCore achieved better predictive performance. It combines the advantages of data cleaning and coreset selection and improves both predictive performance and runtime efficiency.
We have added the setup and analysis of the paired repetitions to Appendix E.6 of the revised manuscript and report the mean and sample standard deviation of each metric in Table 14.
To W3, Q1, and Q2
Thank you for pointing out that error identification needs further validation. Following your suggestion, we use loss and gradient signals from training to explain the key assumptions underlying high-risk screening and error typing and the rationale for them. We also use the detection records from the repeated experiments in W2 to evaluate how high-risk screening and error typing perform in practice.
Approximate consistency of training updates. Assumption 1 first requires that actual training and ideal clean training start from the same model and have sufficiently close parameter updates within the current sliding window. MisDetect (Deng et al., 2024) observes that models tend to fit clean samples first during early training. This supports the premise that early updates in actual training can still follow the classification patterns in the clean data. In later stages, CleanCore progressively corrects identified errors, down-weights suspicious samples, and prioritizes reliable samples as coreset representatives. This reduces the interference of corruption with subsequent updates, making the actual updates more likely to be close to those in ideal clean training.
Correctness of ideal error typing. Assumption 1 also requires that, for samples identified as erroneous at the current stage, the signals from ideal clean training yield the correct type under the same decision rules and thresholds. The rationale is that the relationships between features and labels learned by the ideal model from clean data can help identify the source of a high loss. If the features still carry information about the true class, replacing the label with one consistent with those features may substantially reduce the loss and bring it close to normal levels. If the abnormal loss mainly stems from a few corrupted features, it may be more sensitive to those features. Keeping the label fixed and making a small adjustment to those features may also reduce the loss.
The above analysis explains the rationale for Assumption 1. CleanCore aims to reduce the interference of data corruption with model training and improve predictive performance. Error typing guides subsequent handling without requiring identification of the true error type of every corrupted sample. As described in the paper, samples retained in the training pool can be checked again in later stages. CleanCore can therefore focus on accurately typing detected samples without having to detect every error in a single stage, reducing the risk of incorrectly modifying correct labels or features. To evaluate actual screening and typing performance, we compare the full-run detection records for the four datasets and five random seeds in W2 against the known injected error types, sample by sample. Each sample is counted once per seed, and the counts are pooled across the five seeds to calculate precision and recall.
Thank you for carefully reviewing our paper and for your specific suggestions on the coverage of large-scale datasets, variability in experimental results, validation of error identification, the relationship between our coreset selection and existing methods, and practical hyperparameter settings. In response, we have added experimental results and clarifications.
In the revised manuscript, substantive changes made specifically in response to your comments are marked in orange, and changes addressing comments from multiple reviewers are marked in purple. The corresponding W/Q labels appear in orange in the right margin.
To W1
Thank you for suggesting that we evaluate CleanCore on datasets with more samples and features. Our existing experiments cover a range of sample sizes, task types, and class distributions. For example, Pendigits is a relatively balanced small-scale multiclass classification dataset, Seismic Bumps is a markedly imbalanced small-scale binary classification dataset, and SUSY is a large-scale binary classification dataset with five million samples. Following your suggestion, we added a comparison on Santander, which contains 200,000 samples and 200 features, to further evaluate CleanCore in a setting with more than 100,000 samples and many features.
We selected five baselines with good predictive performance or efficiency in the main experiments, covering data cleaning, coreset selection, and hybrid methods. The paired repetitions in W2 use the same five baselines. The table below reports ACC (%), F1 (%), and TIME (s) as means ± sample standard deviations for each method over five random seeds (72–76).
Table 1. Predictive performance and runtime on Santander.
On Santander, CleanCore achieves the highest mean ACC and F1 and the shortest mean runtime. Its mean F1 is close to that of Cleanlab, and its overall runtime is about 7% of Cleanlab's. With a runtime similar to GradMatch's, CleanCore achieves a 5.00 percentage point higher mean F1.
We have added the comparative experiment on Santander to Appendix E.8 of the revised manuscript, including the experimental setup, complete results, and analysis.