- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets the problem of coreset selection, where the goal is to select a subset of the training dataset while retaining as much performance as possible. The paper targets this via a frozen pretrained foundation model such as CLIP ViT-L/14 for vision, BERT's [CLS] text classification and Qwen3-Embedding-4B for LLM fine-tuning and reasoning. The approach involves extracting the embedding for the target dataset using the pretrained model. Then the proposed algorithm C-GeoVal is applied which is a 3 phase algorithm. In the first phase, the local density weight is calculated using the cosine similarity to k nearest neighbors which the paper dubs "density-weight-coverage" . Then the bottom 12% of the density quantile is removed. In next phase the goal is to obtain the topology component of the score for the subset . The second phase effectively builds the coreset using the density part of the coreset valuation score. Phase 3 looks for rare topological structure and swaps samples/points to retain structure. The paper then presents experimental results on vision (CIFAR10N), GoEmotions for text classification and AIME-24 and MATH-500 for supervised fine tuning.
Critical strengths
The idea of preserving structure in the coreset is something that can add value and is missing broadly in existing work (a few recent contemporary works attack the same problem [1, 2]).
The paper also perform the evaluation on two modes, vision and language to show the applicabilityof the method
References:
[1] Roy, Arjun, et al. "TopoPrune: Robust Data Pruning via Unified Latent Space Topology." arXiv preprint arXiv:2602.02739 (2026).
[2] Yehuda, Ofer, et al. "Active learning through a covering lens." Advances in Neural Information Processing Systems 35 (2022)
Critical weaknesses
-
One of the major concerns is that the gains are likely coming from the base frozen embedding model. If this model has already seen the downstream eval dataset, the performance gains are hardly surprising. There is evidence in the paper that points to this, in Table 17, only the largest model CLIP ViT-L 14 outperforms random. Then there is the added confounder that the downsteam model for reasoning, Qwen3-8B is the same family as Qwen3-Embedding. Experimental setup needs to be carefully evaluated so that such confounders can be removed.
-
The topology component, a big part of the claim is not well supported. The algorithm does not really optimize for equation 4. The landmarks are restricted to the dense region so structure in sparse regions are not captured, and rare subpoluations are likley sparse. The claims of structure/loops being rare is not supported.
-
Some of the experimental setup needs work:
a. In GoEmotions ablation, w/o topology is 9 points blow facility location, I am not sure I understand why, should that be similar?
b. The math setup disables topology (lines 53, 75-76)
c. The mislabelled experiment is missing a bunch of baselines such as SimiFeat [1], Curvature [2], CSL [3], SSFT [4]. It is likley the noisy label works because the noise matches outlier images, a better setup woould be to use various noisy setups such as symmetric noise or otherwise. Clean lab seems to be run with ResNet18 and not CLIP features.
d. Real-noise datasets at larger scale are missing such as Red Mini-ImageNet/CNWL, WebVision, Clothing1M. And missing some older coreset baselines [5, 6, 7]
e. "Beats full data" is measured against plain cross-entropy training on noisy labels. On clean labels the coreset loses by 2.8 points to full data.
References:
[1] Zhu, Zhaowei, Zihao Dong, and Yang Liu. "Detecting corrupted labels without training a model to predict." ICML 2022.
[2] Garg et al. Memorization through the lens of curvature of loss function around samples. ICML 2024
[3] Ravikumar, et al. "Towards memorization estimation: Fast, formal and free." ICML 2025
[4] Maini, Pratyush, et al. "Characterizing datapoints via second-split forgetting." NeurIPS 2022
[5] Zheng, H., Liu, R., Lai, F., & Prakash, A. Coverage-centric coreset selection for high pruning rates. ICLR 2023
[6] I. Garg and K. Roy, "Samples with Low Loss Curvature Improve Data Efficiency," CVPR 2023
[7] Zheng, H., Liu, R., Lai, F., & Prakash, A. (2022). Coverage-centric coreset selection for high pruning rates. ICLR 2023
[8] Nagaraj, Manish, et al. "TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuning." ICML 2026\
Questions for authors
- Do the results transfer to a larger real-noise dataset such as Red Mini-ImageNet/CNWL, WebVision, Clothing1M.?
- Does the base model have to be supervised can DinoV3 or unsupervised model be used?
- See weakness a, c
3 comments
Thank you again for your review. We would like to clarify the scope of our results.
On CIFAR-10N, C-GeoVal and ZCore share the same CLIP embeddings and training protocol, reaching 91.36% and 83.25% at the 80% budget. This comparison isolates their difference under that representation, but the gain depends strongly on the encoder: C-GeoVal reaches 82.39% with DINOv2, below Random at 83.17%.
Algorithm 1 optimizes coverage and then applies topology-guided swaps; the approximation guarantee applies only to coverage. Density-restricted landmarks do not establish preservation of sparse rare subpopulations, and topology swaps are disabled in the math experiments.
Finally, the “beats full data” claim compares against ordinary training on noisy labels. With clean labels, C-GeoVal reaches 92.56% versus 95.36% for full-data training. Larger real-noise datasets and the additional requested baselines remain untested. We hope these clarifications help you reconsider the paper within the scope supported by its evidence.
Thank you for your response, however it does not address the fact that the gains are might be attributed to the base encoder. No new evidence was provided to show that the proposed method was the reason for the delta.
Thank you for the reviewer RzA3 comments. We sincerely appreciate the valuable suggestions. We address each point below.
Response: In the main CIFAR-10N comparison, C-GeoVal and ZCore use the same CLIP embeddings and the same downstream training protocol. At the 80% budget, they reach 91.36% and 83.25%, respectively. This comparison measures the difference in selection under a given representation. Table 17 of the manuscript already tests the self-supervised DINOv2 ViT-B/14. Under this representation, C-GeoVal reaches 82.39% at the 80% budget, which is below the 83.17% of Random at the same budget in the manuscript. We will therefore state explicitly that the selector can use self-supervised representations, while the measured gain depends on the encoder. In the math experiment, C-GeoVal and DDCF share the same Qwen3-Embedding features.
Response: Eq. (4) defines the set utility used for valuation, and Eq. (6) defines the expected marginal value of a sample. Algorithm 1 first greedily optimizes the density-weighted coverage and then performs local swaps guided by the reference persistence diagram. The approximation guarantee in Section 3.4 applies to the coverage stage, over the retained candidate set and under the fixed sparse-graph support. The landmarks are drawn from the higher-density region, so our conclusion about topological repair is limited to correcting the H1 discrepancies reflected by these landmarks. For whether low-density variants are retained, please see the binned results in our response to R1. For the plain facility-location control on GoEmotions, and for why the math experiment can only evaluate the density and coverage stages, please see our response to R2.
Response: The "exceeds full data" conclusion in the manuscript refers to standard cross-entropy training on noisy labels. On clean labels, the 80% C-GeoVal subset in the manuscript reaches 92.56%, compared with 95.36% for full-data training. The manuscript also reports results of noise-robust training on the full data, such as ELR+. The new factorial experiment tests the density preference using both the noisy labels and the clean labels of the same image pool, and the scope of the study remains limited to the existing data and noise conditions. Large-scale real-world noisy datasets, independent random label noise, and other detection baselines such as SimiFeat are controls that could further test the range of applicability. When these methods are compared, it should be stated whether each of them relies on frozen representations, observed labels, a clean validation set, or upstream training. We will also clarify in the related work the relationship to topological pruning methods such as TopoPrune, and describe our design specifically as local swaps, guided by a reference persistence diagram, applied to a coverage subset at a given budget.
We sincerely hope that this detailed response addresses the reviewer's remaining concerns, and we respectfully ask the reviewer to consider raising the score in light of the strengthened evaluation.