- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a training-free, label-free dataset pruning method that frames unbiased subset selection as optimizing an orientation-averaged variance surrogate over the polytope of unbiased -selections, using closed-form pairwise kernels on the sphere and torus and a swap-based vertex walk.
Critical strengths
The paper provides a theoretical framework connecting unbiased selection, convex polytope vertices, and orientation-averaged variance on the sphere/torus, replacing geometric heuristics with a derived objective.
Critical weaknesses
The inverse-distance surrogate is motivated by kernel envelopes, but the paper does not prove that decreasing it decreases the true orientation-averaged variance.
Algorithm 1 precomputes all pairwise scores at cost, which is prohibitive for large datasets (e.g., ImageNet).
The paper benchmarks against herding, -center greedy, and uniform sampling, but does not compare against other recent training-free, label-free methods.
The paper argues that biased selections can drive accuracy below uniform sampling under class imbalance, but does not test this claim empirically. All experiments use relatively balanced datasets (CIFAR-10, MNIST, CelebA, IMDb), leaving the class-imbalance motivation unvalidated.
Questions for authors
The models tested are relatively small and old (ResNet, VAE, and MLP). It is not clear whether the finding holds for large-scale modern LLMs/VLMs, for which the scale of the training data is crucial. Can you apply the method to GPT-style models?
We thank the reviewer for the comments and for recognizing that the paper replaces geometric heuristics with a derived objective. We address each weakness and the question below.
W1. The inverse-distance surrogate is not proven to decrease the true variance. The surrogate has an upper-bound interpretation rather than being the exact orientation-averaged variance objective. Over the bounded distance range, inverse distance can be positively rescaled and shifted to bound a decreasing envelope without changing the optimization. Choosing the smallest such shift gives a nontrivial bound, whose minimum lies below the envelope's maximum. This does not guarantee that decreasing the surrogate decreases the true orientation-averaged variance. We have clarified this interpretation and its limitation in Appendix A.2, referenced from Section 4.
To examine the practical relationship with gradient variance, we have added an experiment comparing 18 partitions along three optimization paths on 2,048 CelebA images, with ResNet-18 fixed at initialization and batch size 128. Lower inverse-distance scores generally correspond to lower gradient variance (Pearson r=0.93), with final variances 14.2–22.5% below their random-start values, although the reductions are not monotone. Section 6.2 and Figure 5 report this diagnostic, which provides empirical support rather than a monotonicity guarantee.
W2. Quadratic cost of Algorithm 1. For large datasets, we provide the fast variant in Section 5, which avoids storing pairwise scores by using binary embeddings and per-bucket bit counts. Excluding encoding, its assignment cost is O(mNb∣T∣), linear in dataset size for a fixed selection fraction and number of bits. The runtime benchmark records 153.1 seconds for 222≈4.2 million 4096-dimensional embeddings on one GPU (Figure 4). By streaming inputs and outputs in fixed-size chunks, the fast variant can also keep GPU working memory independent of dataset size at a fixed selection fraction (Appendix B.4). We also use this variant in the new experiment on 3.84 million reviews, obtaining significantly lower Qwen3-0.6B gradient variance than uniformly random mini-batches of the same size (Section 6.2, Table 5 of the revised manuscript).
W3. Other training-free, label-free baselines. We selected herding and k-center greedy because they are the two methods covered in the training-free, geometry-based category of Moser et al. (2026, Section III-B), alongside uniform sampling as a reference baseline. Across the four pruning benchmarks, our evaluation comprises 13,000 selection-and-training runs. Comparisons with additional methods would exceed our computational budget. We have added this baseline-selection rationale to the introduction.
W4. The class-imbalance motivation is not tested. We have reworded the motivation in Section 2 to state that geometry-based methods can perform worse than uniform sampling even on balanced datasets. The revised text points to our CIFAR-10 and MNIST results (Table 1), which directly illustrate this behavior, rather than invoking class imbalance.
Q. Large modern models. We have added an experiment with Qwen3-0.6B, a modern autoregressive language model, on 3.84 million reviews. At fixed model weights, our fast variant reduces gradient variance relative to uniformly random mini-batches of the same size by 7.2% with pretrained weights and 1.9% at random initialization. Both reductions are statistically significant (Table 5 of the revised manuscript). The experiment is described in Section 6.2 of the revised manuscript.
We thank the reviewer for encouraging us to clarify the theoretical connection and extend the evaluation to larger datasets and modern language models.