- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies coreset selection from the perspective of generalization under distribution shift. Motivated by a multi-source domain-generalization bound, it proposes GenDS, which scores samples according to image-text alignment, subset invariance, and classification-structure consistency, and then performs coverage-aware selection with class and subset balancing and redundancy reduction. Experiments on three standard image-classification benchmarks and three multi-domain benchmarks show consistently competitive results, particularly at low selection ratios and under label noise and image corruptions.
Critical strengths
-
The paper studies data selection under distribution shift, where preserving the full-data ERM solution may not ensure good generalization. This is an important and practical problem.
-
The experiments cover six standard and multi-domain benchmarks, multiple selection ratios, robustness settings, ablations, runtime, and cross-architecture transfer. The results are consistently strong, and the paper is clear and well organized.
Critical weaknesses
-
The paper does not show that GenDS produces a coreset that better matches an unseen test distribution. It does not compare the selected coresets using MMD, linear separability, or other distribution-discrepancy measures with respect to the test set.
-
The paper does not directly validate its motivating claim that ERM-oriented selection underperforms because it overlooks distribution shift. The component ablations show that each component contributes to GenDS, and the image-only variants partly assess the effect of text information. However, these experiments do not isolate distribution shift as the reason for the gap over existing methods. A controlled comparison using matched representations under different levels of train-test shift would better support this claim.
Questions for authors
-
Can the authors compare the discrepancy between the held-out distribution and equal-sized, class-matched coresets selected by GenDS and the baselines, using the measures, such as MMD or domain-classifier accuracy? Without using target data during selection, do these post-hoc measurements correlate with test accuracy?
-
Can the authors provide a controlled comparison between GenDS and ERM-oriented selection using the same representations and training protocol across i.i.d. and shifted settings? Does the relative gain of GenDS increase with the degree of distribution shift?
3 comments
Thank you for the detailed response. The additional experiments substantially address my two main concerns. The remaining limitations concern the scope of the new evaluations and the heuristic connection between the domain-generalization bound and the practical scoring criteria. These limitations do not materially affect my assessment of the core contribution. I therefore maintain my rating.
Response to Reviewer BWMD’s Reply
Thank you for your follow-up and for acknowledging our additional experiments. We would like to further address the remaining points you raised.
Regarding the connection between the generalization bound and the scoring criteria, the criteria are theoretically motivated by an upper-bound decomposition, as detailed below.
(1) Source risk and . Let denote the adapted image feature, the category predictor constructed from the adapted textual class prototypes, and the classifier induced by for sample scoring. For the InfoNCE loss computed over all class prototypes, the source 0-1 classification error of satisfies
\leq \frac{1}{\log 2} \sum_i\pi_i \mathbb E_{(x,y)\sim P_i} \left[ \mathcal L_{\mathrm{InfoNCE}} \bigl(G_y(\tilde f_I(x)),y\bigr) \right],$$ where $P_i$ denotes the distribution of training subset $i$, and $\pi_i$ its corresponding mixture weight. The correct-class logit is $S_{\mathrm{ITA}}(x)/\tau'$, where $\tau'$ is the softmax temperature. With competing logits fixed, increasing this alignment score decreases the InfoNCE loss. This provides a loss-based theoretical rationale for using $S_{\mathrm{ITA}}$ to prioritize samples with reliable class supervision. **(2) Cross-subset discrepancy and $S_{\mathrm{SI}}$.** Let $\widetilde P_i$ denote the distribution of the adapted feature $\tilde f_I(x)$ for $x\sim P_i$. Using the standard normalization of $\mathcal H$-divergence and natural logarithms, $$d_{\mathcal{H}}(\widetilde{P}_i,\widetilde{P}_j)\leq 2\sqrt{2\mathrm{JS}(\widetilde{P}_i,\widetilde{P}_j)}.$$ Moreover, under the uniform subset mixture $P_S=\frac1M\sum_iP_i$, if the independent probe $G_{d2}$ recovers the true subset posterior given the adapted feature, the uniformly weighted generalized JS divergence satisfies $$\mathrm{JS}(\widetilde{P}_1,\ldots,\widetilde{P}_M)=\log M-\mathbb{E}_{x\sim P_S}[S_{\mathrm{SI}}(x)].$$ This analysis motivates prediction entropy as a feature-space proxy for subset invariance. We use the independently trained probe $G_{d2}$ to approximate these posterior probabilities and assess residual subset predictability. **(3) Ideal joint error and $S_{\mathrm{CSC}}$.** Under the assumptions in Appendix C, combining Eqs. (15) and (24) gives $$\lambda_{\mathcal H,(P_\pi,Q)} \leq R_{\pi,\alpha} + \frac{M(M-1)}{2} \mathbb E_{x\sim P_\pi}[\overline D_g(x)],$$ where $\overline D_g(x)$ is the average pairwise classification disagreement defined in Eq. (19), and we use $R_{\pi,\alpha}$ to denote the sum of the two residual mismatch terms in Eq. (24), which are generally not directly observable during coreset selection. The pairwise-disagreement term identifies a training-side component of the upper bound, providing analytical support for the pairwise-consistency structure used by $S_{\mathrm{CSC}}$. These relationships provide theoretical motivation for the scoring design, while the practical scores approximate the corresponding training-side quantities or reflect their underlying structure. As complementary evidence, we independently select coresets using each score on OfficeHome (R) and compare the corresponding evaluation metrics with those obtained through random selection: | Scoring criterion / Evaluation metric | 20% Random | 20% Score-based | 30% Random | 30% Score-based | |---|---:|---:|---:|---:| | $S_{\mathrm{ITA}}$ / Source risk (%) | 3.932 | **0.481** | 3.820 | **0.379** | | $S_{\mathrm{SI}}$ / Average A-distance | 1.8752 | **1.7842** | 1.8767 | **1.8099** | | $S_{\mathrm{SI}}$ / Maximum A-distance | 1.9011 | **1.8498** | 1.9146 | **1.8647** | | $S_{\mathrm{CSC}}$ / Minimum prediction disagreement (%) | 23.795 | **17.050** | 21.602 | **19.421** | Source risk is measured by the classification error of a ResNet-50 trained on the selected training subset. We report both average and maximum pairwise A-distances as proxies for cross-subset discrepancy. For $S_{\mathrm{CSC}}$, we report the minimum prediction disagreement across source and target distributions between two ResNet-50 classifiers trained on the selected training subset and a target-domain diagnostic set, respectively, as a proxy for ideal joint error. At both selection ratios, each score reduces its corresponding evaluation metric relative to random selection. These results provide empirical evidence that the proposed scores capture their intended generalization-related properties. Regarding the **scope of the new evaluations**, we may not have fully understood your point and would appreciate any further details you could share. If you have any other concerns, please share them with us. We look forward to continuing the discussion and further improving our work. Thank you again for your feedback!
Response to Reviewer BWMD’s Official Review
Thanks for the helpful feedback!
W1 & Q1: Do GenDS Coresets Better Match the Test Distribution?
To investigate the distribution discrepancy between the selected coresets and the held-out distribution, as well as its correlation with test accuracy on the held-out distribution, we conducted additional experiments on OfficeHome (R), comparing GenDS with Random, CPDS, and Moderate-DS at different selection ratios. At each ratio, all baselines were matched to GenDS in both total subset size and per-class sample counts. Importantly, the held-out domain was used only for post-hoc evaluation and was not involved in coreset selection.
The results are presented below. Lower MMD² and domain-classifier accuracy indicate closer alignment with the held-out distribution. GenDS achieves the lowest discrepancy under both measures and the highest test accuracy at all selection ratios.
We further examined the association between distribution discrepancy and test accuracy. The mean Spearman correlations across selection ratios are reported below. Both discrepancy measures exhibit negative correlations with test accuracy. This indicates that, at a fixed selection ratio, coresets closer to the held-out distribution tend to achieve higher test accuracy. These findings suggest that the reduced distribution discrepancy of the coresets selected by GenDS is associated with improved generalization.
W2 & Q2: Does GenDS’s Advantage Increase with Distribution Shift?
To examine the relationship between GenDS’s performance and distribution shift, we conduct controlled comparisons in both i.i.d. and multi-domain settings, comparing GenDS with Random, CPDS and ZCore under the same representations and downstream training protocol. Evaluation sets are constructed by mixing target samples with held-out source samples:
Dtestnew=αtDtest+(1−αt)Dtrainingheld-out,αt∈0,1/3,2/3,1.
Here, a larger αt indicates a greater proportion of target samples. The held-out source samples are excluded from selection and training. The candidate pool, selection budget, and evaluation set size remain fixed across mixing ratios. We use αt=0 as the reference distribution.
To measure how GenDS's relative advantage changes with distribution shift, we calculate the gain increase over the best-performing baseline relative to the reference distribution:
ΔGαt=(AccGenDSαt−AccBestBaselineαt)−(AccGenDS0−AccBestBaseline0).
A positive value indicates an increased relative advantage over the reference distribution.
i.i.d. setting - Tiny-ImageNet
Multi-domain setting – Office-Home (R)
We also report the Spearman correlation between the target proportion αt and the original GenDS gain over the best baseline.
These results show a positive correlation between the target proportion and GenDS’s gain, with a stronger correlation in the multi-domain setting, indicating that its relative advantage over the baselines tends to increase with distribution shift.