acceptodds
Under review as a conference paper at ICLR 2027

Towards Theory-Guided and Generalizable Data Selection

Abstract

Data selection aims to construct compact coresets that reduce training cost while preserving model performance. Most existing methods select data by preserving statistical, geometric, or optimization properties of the full training set, reflecting an empirical risk minimization (ERM)-oriented perspective. However, discrepancies between observed training data and unseen test data can arise from dataset collection, construction, and sampling effects, suggesting that preserving full-data ERM behavior alone may be insufficient for generalization. We therefore revisit data selection from a generalization perspective. Drawing on a domain generalization error bound, we identify four key factors associated with generalization and formulate corresponding principles for data selection: preserving predictive utility, encouraging cross-subset invariance, maintaining semantic consistency, and promoting distributional coverage. Guided by these principles, we propose GenDS, a theory-guided Generalizable Data Selection framework that operationalizes them through three complementary sample-level scores and a coverage-aware coreset selection strategy. Specifically, the scoring stage evaluates image-text alignment, subset invariance, and classification-structure consistency to capture predictive utility, cross-subset invariance, and semantic consistency, respectively, while the subsequent selection stage promotes distributional coverage and reduces redundancy. Extensive experiments on six standard and multi-domain image classification benchmarks demonstrate that GenDS consistently outperforms representative data selection methods across varying selection ratios, with particularly strong gains under limited data budgets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Data Selection, Generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BWMDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies coreset selection from the perspective of generalization under distribution shift. Motivated by a multi-source domain-generalization bound, it proposes GenDS, which scores samples according to image-text alignment, subset invariance, and classification-structure consistency, and then performs coverage-aware selection with class and subset balancing and redundancy reduction. Experiments on three standard image-classification benchmarks and three multi-domain benchmarks show consistently competitive results, particularly at low selection ratios and under label noise and image corruptions.

Critical strengths
  • The paper studies data selection under distribution shift, where preserving the full-data ERM solution may not ensure good generalization. This is an important and practical problem.

  • The experiments cover six standard and multi-domain benchmarks, multiple selection ratios, robustness settings, ablations, runtime, and cross-architecture transfer. The results are consistently strong, and the paper is clear and well organized.

Critical weaknesses
  • The paper does not show that GenDS produces a coreset that better matches an unseen test distribution. It does not compare the selected coresets using MMD, linear separability, or other distribution-discrepancy measures with respect to the test set.

  • The paper does not directly validate its motivating claim that ERM-oriented selection underperforms because it overlooks distribution shift. The component ablations show that each component contributes to GenDS, and the image-only variants partly assess the effect of text information. However, these experiments do not isolate distribution shift as the reason for the gap over existing methods. A controlled comparison using matched representations under different levels of train-test shift would better support this claim.

Questions for authors
  • Can the authors compare the discrepancy between the held-out distribution and equal-sized, class-matched coresets selected by GenDS and the baselines, using the measures, such as MMD or domain-classifier accuracy? Without using target data during selection, do these post-hoc measurements correlate with test accuracy?

  • Can the authors provide a controlled comparison between GenDS and ERM-oriented selection using the same representations and training protocol across i.i.d. and shifted settings? Does the relative gain of GenDS increase with the degree of distribution shift?

3 comments
AuthorsOpenReview ↗

Response to Reviewer BWMD’s Official Review

Thanks for the helpful feedback!

W1 & Q1: Do GenDS Coresets Better Match the Test Distribution?

To investigate the distribution discrepancy between the selected coresets and the held-out distribution, as well as its correlation with test accuracy on the held-out distribution, we conducted additional experiments on OfficeHome (R), comparing GenDS with Random, CPDS, and Moderate-DS at different selection ratios. At each ratio, all baselines were matched to GenDS in both total subset size and per-class sample counts. Importantly, the held-out domain was used only for post-hoc evaluation and was not involved in coreset selection.

The results are presented below. Lower MMD² and domain-classifier accuracy indicate closer alignment with the held-out distribution. GenDS achieves the lowest discrepancy under both measures and the highest test accuracy at all selection ratios.

Selection ratioMetricRandomCPDSModerate-DSGenDS
20%MMD² ↓0.016660.026920.013760.01023
Domain accuracy (%) ↓83.1585.8182.8879.37
Test accuracy (%) ↑75.4773.6575.3778.52
30%MMD² ↓0.016160.021000.013920.01173
Domain accuracy (%) ↓83.9084.5983.9681.48
Test accuracy (%) ↑76.8577.1277.5079.14
40%MMD² ↓0.016620.016180.014310.01289
Domain accuracy (%) ↓84.5384.1784.6382.93
Test accuracy (%) ↑77.9178.8978.5580.22

We further examined the association between distribution discrepancy and test accuracy. The mean Spearman correlations across selection ratios are reported below. Both discrepancy measures exhibit negative correlations with test accuracy. This indicates that, at a fixed selection ratio, coresets closer to the held-out distribution tend to achieve higher test accuracy. These findings suggest that the reduced distribution discrepancy of the coresets selected by GenDS is associated with improved generalization.

Discrepancy measureMean Spearman correlation with test accuracy
MMD²−0.858
Domain-classifier accuracy−0.725

W2 & Q2: Does GenDS’s Advantage Increase with Distribution Shift?

To examine the relationship between GenDS’s performance and distribution shift, we conduct controlled comparisons in both i.i.d. and multi-domain settings, comparing GenDS with Random, CPDS and ZCore under the same representations and downstream training protocol. Evaluation sets are constructed by mixing target samples with held-out source samples:

Dtestnew=αtDtest+(1−αt)Dtrainingheld-out,αt∈0,1/3,2/3,1.\mathcal{D}_{\text{test}}^{\text{new}} = \alpha_t \mathcal{D}_{\text{test}} + (1-\alpha_t)\mathcal{D}_{\text{training}}^{\text{held-out}}, \quad \alpha_t \in \\{0, 1/3, 2/3, 1\\}.

Here, a larger αt\alpha_t indicates a greater proportion of target samples. The held-out source samples are excluded from selection and training. The candidate pool, selection budget, and evaluation set size remain fixed across mixing ratios. We use αt=0\alpha_t=0 as the reference distribution.

To measure how GenDS's relative advantage changes with distribution shift, we calculate the gain increase over the best-performing baseline relative to the reference distribution:

ΔGαt=(AccGenDSαt−AccBestBaselineαt)−(AccGenDS0−AccBestBaseline0).\Delta G_{\alpha_t} = \left(\mathrm{Acc}_{\mathrm{GenDS}}^{\alpha_t} - \mathrm{Acc}_{\mathrm{BestBaseline}}^{\alpha_t}\right) - \left(\mathrm{Acc}_{\mathrm{GenDS}}^{0} - \mathrm{Acc}_{\mathrm{BestBaseline}}^{0}\right).

A positive value indicates an increased relative advantage over the reference distribution.

i.i.d. setting - Tiny-ImageNet

Selection rateαt\alpha_tGain increase (pp)
20%00
1/30.13
2/30.14
10.18
30%00
1/30.10
2/30.12
10.22
40%00
1/30.17
2/30.32
10.35

Multi-domain setting – Office-Home (R)

Selection rateαt\alpha_tGain increase (pp)
20%00
1/30.25
2/31.13
11.47
30%00
1/30.02
2/30.65
10.87
40%00
1/30.05
2/30.69
11.17

We also report the Spearman correlation between the target proportion αt\alpha_t and the original GenDS gain over the best baseline.

SettingSpearman ρ\rho
i.i.d. – Tiny-ImageNet0.5182
Multi-domain – Office-Home (R)0.8760

These results show a positive correlation between the target proportion and GenDS’s gain, with a stronger correlation in the multi-domain setting, indicating that its relative advantage over the baselines tends to increase with distribution shift.

Thank you for the detailed response. The additional experiments substantially address my two main concerns. The remaining limitations concern the scope of the new evaluations and the heuristic connection between the domain-generalization bound and the practical scoring criteria. These limitations do not materially affect my assessment of the core contribution. I therefore maintain my rating.

AuthorsOpenReview ↗

Response to Reviewer BWMD’s Reply

Thank you for your follow-up and for acknowledging our additional experiments. We would like to further address the remaining points you raised.

Regarding the connection between the generalization bound and the scoring criteria, the criteria are theoretically motivated by an upper-bound decomposition, as detailed below.

(1) Source risk and SITAS_{\mathrm{ITA}}. Let f~I(x)\tilde f_I(x) denote the adapted image feature, GyG_y the category predictor constructed from the adapted textual class prototypes, and hh the classifier induced by Gy(f~I(x))G_y(\tilde f_I(x)) for sample scoring. For the InfoNCE loss computed over all class prototypes, the source 0-1 classification error of hh satisfies

\leq \frac{1}{\log 2} \sum_i\pi_i \mathbb E_{(x,y)\sim P_i} \left[ \mathcal L_{\mathrm{InfoNCE}} \bigl(G_y(\tilde f_I(x)),y\bigr) \right],$$ where $P_i$ denotes the distribution of training subset $i$, and $\pi_i$ its corresponding mixture weight. The correct-class logit is $S_{\mathrm{ITA}}(x)/\tau'$, where $\tau'$ is the softmax temperature. With competing logits fixed, increasing this alignment score decreases the InfoNCE loss. This provides a loss-based theoretical rationale for using $S_{\mathrm{ITA}}$ to prioritize samples with reliable class supervision. **(2) Cross-subset discrepancy and $S_{\mathrm{SI}}$.** Let $\widetilde P_i$ denote the distribution of the adapted feature $\tilde f_I(x)$ for $x\sim P_i$. Using the standard normalization of $\mathcal H$-divergence and natural logarithms, $$d_{\mathcal{H}}(\widetilde{P}_i,\widetilde{P}_j)\leq 2\sqrt{2\mathrm{JS}(\widetilde{P}_i,\widetilde{P}_j)}.$$ Moreover, under the uniform subset mixture $P_S=\frac1M\sum_iP_i$, if the independent probe $G_{d2}$ recovers the true subset posterior given the adapted feature, the uniformly weighted generalized JS divergence satisfies $$\mathrm{JS}(\widetilde{P}_1,\ldots,\widetilde{P}_M)=\log M-\mathbb{E}_{x\sim P_S}[S_{\mathrm{SI}}(x)].$$ This analysis motivates prediction entropy as a feature-space proxy for subset invariance. We use the independently trained probe $G_{d2}$ to approximate these posterior probabilities and assess residual subset predictability. **(3) Ideal joint error and $S_{\mathrm{CSC}}$.** Under the assumptions in Appendix C, combining Eqs. (15) and (24) gives $$\lambda_{\mathcal H,(P_\pi,Q)} \leq R_{\pi,\alpha} + \frac{M(M-1)}{2} \mathbb E_{x\sim P_\pi}[\overline D_g(x)],$$ where $\overline D_g(x)$ is the average pairwise classification disagreement defined in Eq. (19), and we use $R_{\pi,\alpha}$ to denote the sum of the two residual mismatch terms in Eq. (24), which are generally not directly observable during coreset selection. The pairwise-disagreement term identifies a training-side component of the upper bound, providing analytical support for the pairwise-consistency structure used by $S_{\mathrm{CSC}}$. These relationships provide theoretical motivation for the scoring design, while the practical scores approximate the corresponding training-side quantities or reflect their underlying structure. As complementary evidence, we independently select coresets using each score on OfficeHome (R) and compare the corresponding evaluation metrics with those obtained through random selection: | Scoring criterion / Evaluation metric | 20% Random | 20% Score-based | 30% Random | 30% Score-based | |---|---:|---:|---:|---:| | $S_{\mathrm{ITA}}$ / Source risk (%) | 3.932 | **0.481** | 3.820 | **0.379** | | $S_{\mathrm{SI}}$ / Average A-distance | 1.8752 | **1.7842** | 1.8767 | **1.8099** | | $S_{\mathrm{SI}}$ / Maximum A-distance | 1.9011 | **1.8498** | 1.9146 | **1.8647** | | $S_{\mathrm{CSC}}$ / Minimum prediction disagreement (%) | 23.795 | **17.050** | 21.602 | **19.421** | Source risk is measured by the classification error of a ResNet-50 trained on the selected training subset. We report both average and maximum pairwise A-distances as proxies for cross-subset discrepancy. For $S_{\mathrm{CSC}}$, we report the minimum prediction disagreement across source and target distributions between two ResNet-50 classifiers trained on the selected training subset and a target-domain diagnostic set, respectively, as a proxy for ideal joint error. At both selection ratios, each score reduces its corresponding evaluation metric relative to random selection. These results provide empirical evidence that the proposed scores capture their intended generalization-related properties. Regarding the **scope of the new evaluations**, we may not have fully understood your point and would appreciate any further details you could share. If you have any other concerns, please share them with us. We look forward to continuing the discussion and further improving our work. Thank you again for your feedback!
Reviewer_EV8UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GenDS, a data selection framework motivated by a domain-generalization error bound. It interprets the bound as suggesting four principles for generalizable coreset construction: preserving predictive utility, encouraging cross-subset invariance, maintaining classification-structure consistency, and promoting distributional coverage. GenDS implements these principles using an Image-Text Alignment Score (SITA), a Subset-Invariance Score (SSI), and a Classification-Structure Consistency Score (SCSC), followed by class- and subset-aware budget allocation with redundancy reduction.

Experiments are conducted on three standard image-classification benchmarks and three multi-domain benchmarks, with comparisons against 13 representative data selection baselines. The evaluation further includes noisy-label and image-corruption settings, component and hyperparameter ablations, runtime analysis, and cross-architecture transfer. The reported results show consistent gains over the tested baselines across selection ratios, with particularly strong improvements at smaller data budgets.

Critical strengths

1. A coherent generalization-oriented formulation of data selection.

GenDS organizes data selection around several complementary properties that may matter beyond reproducing the full-data ERM solution: semantic reliability, cross-subset invariance, classification-structure consistency, and distributional coverage. The resulting design is well structured, with SITA, SSI, and SCSC targeting different aspects of sample quality and the subsequent selection stage explicitly encouraging coverage while reducing redundancy.

2. Broad empirical evaluation.

The paper evaluates GenDS on six standard and multi-domain benchmarks against 13 baselines, including recent methods such as CPDS, ZCore, and FAST. It further studies robustness to label noise and image corruption, component contributions, hyperparameter sensitivity, runtime, and transfer to alternative downstream architectures.

Critical weaknesses

1. The theoretical motivation is not sufficiently connected to the concrete scoring criteria.

The domain-generalization bound motivates four design principles, but GenDS implements them through heuristic sample-level proxies rather than directly estimating or optimizing the corresponding bound terms. In particular, SITA measures image-text alignment, SSI uses the entropy of an independently trained subset probe, and SCSC measures agreement among subset-specific classifiers. Appendix C provides a conditional theoretical motivation for using pairwise classification consistency, but it retains unobservable residual terms and does not establish that the complete scoring-and-selection procedure reduces target-domain risk. This limits the strength of the theoretical contribution and leaves the connection between the proposed scores and improved generalization insufficiently validated.

2. The strongest domain-generalization evidence relies on explicit domain annotations, while the annotation-free latent partitioning mechanism is not directly validated under domain shift.

GenDS uses available subset/domain annotations on the multi-domain benchmarks, whereas it relies on CLIP-based latent subset discovery when such annotations are unavailable. The leave-one-domain-out results on VLCS, Office-Home, and miniDomainNet therefore establish the effectiveness of GenDS when source-domain identities are known, while the latent-partitioning variant is mainly evaluated on standard image-classification benchmarks. The paper does not directly test whether latent subset discovery can recover the same generalization benefit when explicit source-domain labels are withheld in a genuine domain-generalization setting.

Questions for authors

1. Theoretical justification of the proposed scoring criteria.

The domain-generalization bound motivates four design principles, but the practical scores are proxies for the corresponding theoretical factors. In particular, SITA measures image-text alignment, SSI measures subset-prediction entropy, and SCSC measures agreement among subset-specific classifiers. Could the authors provide empirical evidence that these scores capture their intended generalization-related properties, for example through analyses relating individual scores to held-out-domain performance? This would help clarify which aspects of GenDS are theoretically motivated and which are supported primarily by empirical evidence.

2. Validation of latent subset discovery under domain shift.

GenDS uses explicit domain annotations when they are available, while relying on CLIP-based latent subset discovery otherwise. The leave-one-domain-out experiments on VLCS, Office-Home, and miniDomainNet therefore evaluate the annotation-aware setting, whereas latent partitioning is mainly tested on standard classification benchmarks. Could the authors report a leave-one-domain-out experiment in which source-domain annotations are withheld during subset discovery and GenDS relies only on its latent partitioning mechanism? This would directly test whether the proposed annotation-free partitioning strategy preserves the reported generalization benefit under genuine domain shift.

3. Scoring protocol for SSI and SCSC.

Sections 4.2 and 4.4 describe training a subset probe and subset-specific classifiers to compute SSI and SCSC, respectively, but do not explicitly state whether the samples being scored are also used to train the corresponding models. Could the authors clarify this protocol? If in-sample predictions are used, could they explain why these scores reliably reflect subset invariance and cross-subset classification consistency, rather than partly reflecting model fit to the scored samples? A cross-fitted sensitivity analysis would be useful if this effect is potentially substantial.

4. Relationship to CPDS.

A CLIP-Powered Framework for Robust and Generalizable Data Selection (CPDS) (Yang et al., ICLR 2025) already uses CLIP adaptation, image-text semantic alignment, diversity-aware scoring, and selection optimization for robust and generalizable data selection. GenDS appears to differ primarily through its explicit cross-subset modeling—particularly SSI and SCSC—and its coverage-aware allocation motivated by a domain-generalization bound. Could the authors clarify this distinction more explicitly and, if possible, indicate which of these new components accounts for the improvement over CPDS?

5 comments
AuthorsOpenReview ↗

Response to Reviewer EV8U’s Official Review (1/2)

Thanks for the helpful feedback!

W1 & Q1: Theoretical justification of the proposed scoring criteria.

The paper does not verify whether the selected coresets actually improve any of the terms in the generalization bound.

To validate whether each scoring criterion captures its intended generalization-related property, we independently select coresets using each score and evaluate the corresponding properties. We then evaluate proxies for the corresponding theoretical terms and compare the results with those obtained through random selection:

(1) SITAS_{\mathrm{ITA}} → Source risk (∑iπiϵPi(h)\sum_i\pi_i\epsilon_{P_i}(h))

We train a ResNet-50 classifier on each selected coreset and estimate the source risk by evaluating its classification error.

(2) SSIS_{\mathrm{SI}} → Cross-subset discrepancy (ρ/2\rho/2)

We use pairwise A-distances as a proxy for the cross-subset discrepancy term. We report both the maximum and average pairwise A-distances across all subsets.

(3) SCSCS_{\mathrm{CSC}} → Ideal joint error (λH,(P,Q)\lambda_{\mathcal H,(P,Q)})

Since the ideal joint error involves an optimization over the hypothesis space, we adopt the following metric as a proxy for the ideal joint error [1]:

λ′=min⁡{ϵS(hs,ht),ϵT(hs,ht)}.\lambda' = \min\left\lbrace \epsilon_S(h_s,h_t), \epsilon_T(h_s,h_t) \right\rbrace.

where hsh_s is a ResNet-50 trained on the selected training subset, hth_t is a ResNet-50 trained on a target-domain diagnostic set, and ϵ(hs,ht)\epsilon(h_s,h_t) denotes the prediction disagreement between the two models.

We conduct the above experiments on OfficeHome (R), with the results reported below:

Scoring Criterion / Evaluation Metric20% Random20% Score-based30% Random30% Score-based
SITAS_{\mathrm{ITA}} / Source Risk (%)3.9320.4813.8200.379
SSIS_{\mathrm{SI}} / Average A-distance1.87521.78421.87671.8099
SSIS_{\mathrm{SI}} / Maximum A-distance1.90111.84981.91461.8647
SCSCS_{\mathrm{CSC}} / Minimum Prediction Disagreement (%)23.79517.05021.60219.421

At all selection ratios, selecting samples with high scores reduces all reported metrics compared with random selection. These results provide empirical evidence that the proposed scores capture their intended generalization-related properties.

[1] Shai Ben-David et al. A theory of learning from different domains. Machine Learning, 2010.

W2 & Q2: Validation of latent subset discovery under domain shift.

... Could the authors report a leave-one-domain-out experiment in which source-domain annotations are withheld during subset discovery and GenDS relies only on its latent partitioning mechanism? ...

To evaluate the effectiveness of our annotation-free latent partitioning mechanism under genuine domain shift, we conducted an additional leave-one-domain-out experiment on OfficeHome, where the Real World domain serves as the held-out target domain. In this experiment, source-domain annotations were withheld during subset discovery, and GenDS relied exclusively on the proposed CLIP-based latent partitioning mechanism to construct subsets. All other experimental settings remained identical to those in the original manuscript. The experimental results are presented below:

Method20%30%40%
GenDS (with domain annotations)78.52 ± 0.2979.14 ± 0.2180.22 ± 0.33
GenDS (with latent partitioning)78.32 ± 0.3178.84 ± 0.4480.09 ± 0.52

As shown in the table, GenDS with latent partitioning achieves performance comparable to that obtained using explicit domain annotations, with only minor performance differences across different selection ratios. These results suggest that the proposed latent partitioning mechanism can largely preserve the generalization performance of GenDS, even without access to source-domain annotations during subset discovery.

We will incorporate these additional experimental results into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer EV8U’s Official Review (2/2)

Q3: Scoring protocol for SSI and SCSC

... but do not explicitly state whether the samples being scored are also used to train the corresponding models. ... could they explain why these scores reliably reflect subset invariance and cross-subset classification consistency, rather than partly reflecting model fit to the scored samples? ...

In our original implementation, we uses in-sample predictions to compute the selection scores. To assess the potential influence of in-sample model fitting on the computed scores, we conducted an additional sensitivity analysis using four-fold cross-fitting. Specifically, we divided the source data into four folds. For each fold, we independently trained the image adapter, shared text adapter, domain-specific text adapter, adversarial domain discriminator (Gd1G_{d1}), and independent scoring discriminator (Gd2G_{d2}) using the other three folds. The held-out fold was then used to compute the SCSCS_{CSC}, SSIS_{SI}, and SITAS_{ITA} scores. The resulting scores were aggregated according to the original sample indices.

We compared the cross-fitted scores with those from the original implementation using Spearman's rank correlation coefficient and mean absolute error (MAE). The results are presented below:

MetricSCSCS_{CSC}SSIS_{SI}SITAS_{ITA}
Spearman ρ0.83230.75530.9507
MAE0.022310.003580.03875

Furthermore, we evaluated the downstream performance of GenDS using cross-fitted scores on Office-Home (R), keeping the original scoring formulas and all other experimental settings unchanged.

Method20%30%40%
GenDS (Original)78.52 ± 0.2979.14 ± 0.2180.22 ± 0.33
GenDS (4-fold cross-fitting)77.89 ± 0.4579.02 ± 0.5180.01 ± 0.58

The results show that cross-fitting produces comparable downstream performance to the original implementation, with only minor differences across selection ratios. This suggests that the effectiveness of the proposed scoring mechanism is not primarily driven by in-sample model fitting.

Q4: Relationship to CPDS

... CPDS already uses ... for robust and generalizable data selection ... indicate which of these new components accounts for the improvement over CPDS?

We would like to clarify that the notions of "robust" and "generalizable" in CPDS differ from the generalization objective of our GenDS framework, and our method is not simply an extension of CPDS with additional scoring functions or selection strategies.

In CPDS, "robust" and "generalizable" refer to two properties of the coreset selected by its heuristic algorithm: robustness against noise perturbations and adaptability across architectures, respectively. Our method also exhibits these properties, as demonstrated by Table 1 and Figure 3 for robustness, and Tables 7 and 8 for cross-architecture adaptability. In contrast, GenDS explicitly aims to select coresets that facilitate model generalization to unseen distributions through the Generalizable Data Selection (GenDS) framework. Thus, generalization in GenDS is an explicit design motivation rather than merely an observed property.

We reiterate that GenDS is a generalization-oriented data selection framework guided by the Domain Generalization Error Bound theory. Specifically, GenDS aims to identify a coreset with low overall generalization error through the joint contribution of four theoretically guided principles: Preserve Predictive Utility, Encourage Cross-Subset Invariance, Maintain Semantic Consistency, and Promote Distributional Coverage, rather than a simple combination of two heuristic methods as in CPDS.

AuthorsOpenReview ↗

Look forward to your feedback

Dear Reviewer EV8U,

Thank you very much for your insightful and valuable comments! In our previous response, we have addressed your queries regarding the theoretical justification of the proposed scoring criteria (W1, Q1), the validation of latent subset discovery under domain shift (W2, Q2), the scoring protocol for SSIS_{\text{SI}} and SCSCS_{\text{CSC}} (Q3), and the relationship to CPDS (Q4).

We would be happy to provide further responses. Look forward to your feedback.

Thanks for your comment again!

Best Regards,

All Authors

Reviewer_ciamOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper points out that most existing data selection methods aim to approximate empirical risk minimization (ERM) on the full training set; however, when the training distribution differs from the test distribution, this approach may fail to ensure good generalization. Drawing on an error bound for domain generalization (Theorem 1), the authors derive four principles: preserving predictive utility, promoting invariance across subsets, maintaining semantic consistency, and enhancing distribution coverage. These principles are operationalized through three sample-level scoring metrics—SITA (image-text alignment), SSI (subset invariance), and SCSC (class-structure consistency)—combined with a coverage-based selection strategy that accounts for redundancy. The evaluation covers six benchmark datasets (including both multi-domain and standard benchmarks) and examines model robustness in scenarios involving noisy labels and image corruptions.

Critical strengths
  1. The authors reinterpret the problem of data selection—moving away from an empirical risk minimization (ERM) approximation perspective—by framing it in terms of what must be retained to satisfy a generalization bound. By linking each selection criterion to a corresponding term within the generalization bound, they provide a framework that is both logically rigorous and intuitive.
  2. The failure of an adversarial discriminator does not necessarily imply that all relevant information has been eliminated from the subset; therefore, the authors introduce a separate probe, Gd2G_{d2}, to quantify the remaining recoverability (Equation 3). This approach effectively sidesteps a well-known pitfall in domain adversarial learning.
Critical weaknesses
  1. The paper does not verify whether the selected coresets actually improve any of the terms in the generalization bound. Theorem 1 is cited from existing literature (Albuquerque et al., 2020); the authors themselves acknowledge that translating the four terms in the theorem into four principles serves merely as "design guidance." Furthermore, the implementation of these principles via SITA, SSI, and SCSC represents yet another heuristic leap—for instance, using the pairwise Jensen-Shannon (JS) divergence (or "JS agreement") between classifiers trained on specific subsets to characterize the ideal joint error λH,(P∗,Q)\lambda_{H,(P*,Q)} (Section 4.3), with the justification for this approach relegated to Appendix C. A critical omission: ρ\rho (the maximum pairwise subset divergence—Principle 2) could easily be estimated on the training set (using A-distance or the accuracy of a domain discriminator as a proxy). However, the paper fails to measure whether the coresets selected by GenDS actually reduce this metric. If the ρ\rho value for GenDS coresets is no lower than that of Random sampling, Moderate-DS, or CPDS, then the paper’s core selling point—that it is "theoretically guided"—lacks empirical support, and the so-called "four principles" are merely "post-hoc packaging" for four heuristic scoring criteria. Please provide the following: (a) Calculate proxy metrics for subset divergence on the full dataset as well as on the coresets selected by each method; (b) Evaluate the error distribution of hypotheses trained on these coresets using held-out subsets (i.e., are the errors more uniform or lower?), thereby directly testing the "cross-subset invariance" principle; (c) Conduct a control experiment that optimizes only one of these metrics to verify whether it produces the effect predicted by the corresponding principle.
  2. The scope of the runtime analysis is not clearly defined, which may lead to a significant underestimation of the actual computational cost incurred by GenDS during the selection process. Section 5.5 mentions a runtime of 4.97 minutes on Tiny-ImageNet (26.3% faster than the 6.74 minutes required by CPDS) and compares this with MoSo/FAST (>120 minutes) in Figure 4. However, according to the description of the method itself, the scoring phase involves the following steps: performing CLIP encoding for all samples; conducting domain-adversarial fine-tuning of the CLIP-Adapter (incorporating Gd1G_{d1}) based on Equation (2); training a separate detector Gd2G_{d2} with frozen parameters; training a classifier hDih_{Di} for each of the MM subsets; and performing Variational Fair Clustering.
  3. Equation (6) employs a multiplicative aggregation approach—specifically, U=Norm(SITA)⋅Norm(SSI)⋅Norm(SCSC)U = \text{Norm}(\text{SITA}) \cdot \text{Norm}(\text{SSI}) \cdot \text{Norm}(\text{SCSC})—yet the text does not explicitly define the meaning of "Norm\text{Norm}" (e.g., whether it refers to min-max normalization, rank normalization, or quantile normalization).
Questions for authors

See Critical Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer ciam’s Official Review

Thanks for the helpful feedback!

W1: Empirical Validation of the Theory-Guided Selection Principles

We have conducted the following three additional experiments to provide more direct empirical evidence validating the effectiveness of the selected coresets in reducing the corresponding terms in generalization bound:

(a) Proxy Measurement of Maximum Pairwise Subset Divergence

We evaluate on OfficeHome (R), comparing the full dataset with the coresets selected by Random, Moderate-DS, CPDS, and GenDS. We calculate a proxy for the maximum pairwise subset divergence, namely the maximum pairwise A-distance across the three source domains. Lower values indicate lower separability between different source domains.

Method20%30%40%
Full Dataset1.92301.92301.9230
Random1.87801.88231.8891
Moderate-DS1.90161.87941.8833
CPDS1.91771.87511.8802
GenDS1.85061.85601.8568

At all selection ratios, GenDS achieves the lowest values, indicating its effectiveness in reducing subset divergence.

(b) Evaluation of Cross-Subset Error Distribution

On OfficeHome (R), we evaluate hypotheses trained on each coreset using a common evaluation set consisting of the held-out 40% of the original source-domain data. We report the mean source-domain error (Err.) and the standard deviation across source-domain errors (SD). Lower values indicate lower overall error and more uniform errors, respectively.

Method30% Err. (%)30% SD (pp)40% Err. (%)40% SD (pp)
Random30.22648.249528.76758.1883
Moderate-DS31.708410.183629.40119.8843
CPDS35.20138.676931.65038.3720
GenDS30.06767.792828.33906.9946

The results show that GenDS achieves lower mean error and more uniform error distributions across held-out subsets than all baselines.

(c) Single-Criterion Control: Selection Based Solely on SSIS_{\mathrm{SI}}

To examine whether optimizing SSIS_{\mathrm{SI}} alone produces the predicted effect, we construct a coreset by selecting source samples with high SSIS_{\mathrm{SI}} scores.

Method20%30%40%
CPDS1.91771.87511.8802
GenDS1.85061.85601.8568
SSIS_{\mathrm{SI}}-only1.65811.74571.7520

At all selection ratios, SSIS_{\mathrm{SI}}-only achieves lower maximum pairwise A-distance than other baselines and GenDS. This is because SSIS_{\mathrm{SI}}-only selects samples solely based on SSIS_{\mathrm{SI}}, whereas GenDS is designed to jointly consider all four design principles derived from the generalization bound rather than minimizing subset divergence alone. These results demonstrate that optimizing SSIS_{\mathrm{SI}} alone can reduce subset separability among the selected subsets.

W2: Clarification on the Scope of Runtime Analysis

The scope of the runtime analysis is not clearly defined ... (A) CLIP encoding; (B) domain-adversarial fine-tuning Gd1G_{d1}; (C) separate detector Gd2G_{d2}; (D) classifier hDih_{D_i} for MM subsets; (E) Variational Fair Clustering

The reported runtime of GenDS is 4.97 minutes (297.91 seconds), representing the total computational cost of the entire data selection pipeline, starting from the original training dataset and generating subsets across all selection ratios from 20% to 90%, rather than only the final selection phase.

The detailed runtime breakdown of GenDS across different phases is provided in the following table. The experiments were conducted on an Intel Xeon Silver 4314 CPU and an NVIDIA RTX 4090 GPU.

PhaseRuntime (s)Percentage
Phase 1: Subset partition (IID datasets only)69.4123.30%
└─ Multi-layer CLIP feature extraction and data loading (A)36.4112.22%
└─ Feature reduction and fair clustering (E)33.0011.08%
Phase 2: Multi-dimensional scoring222.9474.83%
└─ CLIP-Adapter/DANN and subset-specific classifier training (B, D)186.4162.57%
└─ Independent subset-probe training (C)6.522.19%
└─ Multi-score inference and storage30.0110.07%
Phase 3: Theory-guided selection5.561.87%
└─ Data preparation and score normalization0.340.11%
└─ Coverage-Aware Coreset Selection5.221.75%
Total297.91100%

W3: Clarification of the Rank Normalization Strategy in Equation (6)

... the text does not explicitly define the meaning of Norm(⋅)\mathrm{Norm}(\cdot) in Equation (6) ...

The normalization strategy is already specified in Appendix E (Implementation Details), under the Coverage-aware Coreset Selection paragraph: "we independently apply rank normalization to each of the three sample-level scores over all samples for consistent score scaling, mapping them into [0, 1]."

We will explicitly state this normalization strategy in the main text near Equation (6) in the revised manuscript.

AuthorsOpenReview ↗

Look forward to your feedback

Dear Reviewer ciam,

Thank you very much for your insightful and valuable comments! In our previous response, we have addressed your queries regarding the empirical validation of the theory-guided selection principles (W1), the scope of the runtime analysis (W2), and the clarification of the normalization strategy in Equation (6) (W3).

We would be happy to provide further responses. Look forward to your feedback.

Thanks for your comment again!

Best Regards,

All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.