acceptodds
Under review as a conference paper at ICLR 2027

Rethinking representativeness and diversity in dynamic data selection

Abstract

Dynamic data selection accelerates training by sampling a changing subset of the dataset while preserving accuracy. We re-examine two core notions underlying sample selection: representativeness and diversity. We define representativeness as weighted coverage of dataset-level high-frequency feature factors, moving beyond local geometric centrality, and treat diversity as a process-level constraint that requires the selection trajectory to gradually include complementary rare factors over training, rather than dispersion within a single subset. Based on this view, we propose a dynamic selection framework with three components. First, we score representativeness in a plug-in feature space to prioritize samples covering high-frequency feature factors. We instantiate this with a sparse autoencoder trained on the target dataset, using sparse unit activations to summarize both individual samples and dataset-wide factor statistics. Second, we realize process-level diversity by combining rare-factor sampling with a usage-frequency penalty that promotes sample rotation, provably discourages monopoly, and reduces gradient bias. Third, we couple the two-dimensional scoring with a smooth scheduler that transitions selection from core-pattern consolidation to rare-factor exploration, without extra gradients, influence estimates, or second-order computations on the training model. Extensive experiments across vision, text, medical, language modeling, and generative multimodal tasks and different models show that our method yields a better accuracy-efficiency frontier than prior selection methods: it retains most of the full-data accuracy at – lower training cost, and under a matched-propagation protocol it even surpasses full-data training on ImageNet-1K. Code is provided in the supplementary material.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
dynamic data selection, representativeness, diversity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Mb8wOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies dynamic data selection. The authors train a sparse autoencoder (SAE) on CLIP features and use the activation frequencies of SAE units across the dataset to compute representativeness and diversity scores for each sample. During training, an epoch-wise scheduler balances the two scores, together with a penalty based on historical usage. The top-p samples are then selected for training at each epoch. The authors evaluate this method on CIFAR-10/100, Tiny-ImageNet, ImageNet-1K, and several text, medical, and multimodal tasks.

Critical strengths
  • The method is straightforward to implement. The scores are computed offline, so the additional computational cost is low.
  • The experiments cover a broad range of domains, including vision, text, medical, and multimodal tasks.
Critical weaknesses

1. The method is weakly aligned with the narrative of representativeness and diversity. The main claim needs stronger evidence.

The paper argues that SAE activation statistics provide a different view of representativeness and diversity than conventional dense feature-space methods. However, to establish the practical value of this approach, the authors need to demonstrate at least two things:

First, the proposed method should outperform selection directly in CLIP's dense feature space. The baseline should also be dynamic. For example, at each epoch, graph cut could be used for selection, with a similar schedule for the coefficient balancing representativeness and diversity.

Second, the SAE should provide useful structure beyond what can be obtained from a simple transformation of dense CLIP features. One straightforward experiment would be to randomly project CLIP features to 16,384 dimensions, retain the top 64 dimensions as the activation set, and keep the rest of the pipeline unchanged. Comparing the results with those obtained using the SAE would help isolate the contribution of the learned SAE features.

Only if both are demonstrated would the claim that the method captures representativeness and diversity at a different level be convincing.

Moreover, the selection method itself weakens the narrative. The objective in Eq. 1 is a sum of per-sample scores, making it a modular function. Optimization simply amounts to selecting the top-p samples. However, with the usage-frequency penalty, the selection counts of different samples may become increasingly balanced over time. Appendix Figure 8 also shows that most samples are selected a similar number of times. If this pattern holds, the method may eventually behave similarly to RS2 w/o in terms of sample exposure. The main difference may be the order in which samples are selected. The authors need to clarify how much of the improvement comes from this ordering itself.

2. The experimental evaluation is limited in scale, and some key results lack directly comparable baselines.

For vision tasks, the most comprehensive baseline comparisons are conducted on relatively small datasets such as CIFAR-10 and CIFAR-100. On ImageNet-1K, Table 4a does not provide a directly comparable evaluation of the proposed method against the listed selection baselines.

Table 4a uses a matched-propagation protocol for the proposed method. This means that, despite the low per-epoch selection fraction, the method processes the same total number of samples as full-data training. However, the static baselines in the table use a fixed-epoch protocol, with results taken from their original papers. These results are therefore not directly comparable. Although the abstract explicitly identifies the matched-propagation protocol, highlighting this result may overstate its practical significance for training efficiency.

Moreover, on ResNet-50 with CIFAR-100, at the 70% selection fraction, the proposed method (79.1) outperforms the strongest random resampling baseline, RS2 w/r (78.4), by only 0.7 percentage points. At the 30% selection fraction, the improvement is only 0.6 points (75.7 vs. 75.1). No variance is reported for either comparison.

In Table 4b, the "only Pen" variant, which does not use any SAE scores and behaves similarly to deterministic sample rotation, already achieves 78.1. This suggests that the performance gains may come primarily from balanced sample rotation rather than SAE factor-based scoring.

The reduced-data ImageNet results (Tables 2 and 3), which do reflect actual training savings, are compared only against full-data training, without any dynamic selection baselines such as RS2 or InfoBatch under the same protocol.

Questions for authors

Please clarify the selection fraction used in the ablation in Table 4b. In addition, on ResNet-50, RS2 w/o is substantially lower than RS2 w/r (75.3 vs. 78.4 at 70%; 72.7 vs. 75.1 at 30%), whereas the two are nearly identical on ResNet-18 (76.9 vs. 77.1). What explains this discrepancy?

7 comments
AuthorsOpenReview ↗

Rebuttal to ReviewerMb8w

[ImageNet results]

The ImageNet comparison adopts a matched-propagation protocol while baselines use a fixed-epoch protocol, so the results are not directly comparable; the reduced-data results are compared only against full-data training.

Thank you for the reminder. We have revised the caption of Table 4(a) in the revised manuscript: it now states that baselines' results are quoted from the original papers under their own fixed-epoch protocols and serve only as contextual references, and that comparisons with dynamic methods follow the SeTa protocol and are reported as relative performance. For experiments on ImageNet, considering reproducibility difficulty and computational cost, we mainly follow the experimental protocol from [1] and adopt its reported results for comparison under identical settings, with the primary goal of verifying the performance benefits of our method. For comparisons with dynamic methods, we refer to the experimental data from [2] and Table 2 in our paper. Due to differences in training protocols, we only report relative performance. The SeTa method [2] suffers a 0.5% performance drop at a pruning ratio of 45.5%, while our method achieves only a 0.3% performance drop at a 50% pruning ratio.

[Experiment settings]

No variance is reported for the narrow improvements over RS2 w/r on CIFAR-100 / ResNet-50 (0.7 points at 70%, 0.6 points at 30%); are the gains robust?

We appreciate the reviewer's question regarding the robustness of our experimental results. Except for ImageNet-1k, which is excluded due to prohibitive computational cost, all other experiments are run with three random seeds, and the standard deviations are within 0.4. Research on training acceleration for datasets such as CIFAR is relatively mature. As a result, consistent small gains on CIFAR are considered meaningful. Recent state-of-the-art data selection work such as SeTa [1] achieves an average accuracy improvement of 0.4 over prior methods across multiple CIFAR settings, closely approaching full-data performance. Our method outperforms SeTa consistently under most experimental settings (please refer to our response to reviewer fKm3 W3). Beyond the final accuracy gains, our main contribution lies in introducing a plug-and-play, extremely lightweight penalty term together with its scheduling strategy. This mechanism enables static scoring to become dynamic and better leverages information from the full dataset.

[1] Zhou, Qing et al. "Scale Efficient Training for Large Datasets." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025): 20458-20467.

[SAE effectiveness]

The method must (1) outperform dynamic selection directly in CLIP's dense feature space, and (2) show that the SAE provides structure beyond a simple transformation of dense features (e.g., random projection to 16,384 dims, keep top-64) — otherwise the claim of capturing representativeness/diversity "at a different level" is not convincing.

The comment's premise — that the paper claims SAE features buy accuracy — is not what §3.2 says: the selector is defined with a swappable encoder and the SAE is presented as a sparse-unit probe. But swappable is not the same as behaviourally interchangeable, and that is directly testable.

With selector, schedule, penalty and per-class budget fixed, and the dense code matched to the SAE's 0.39% activation rate (3/768 positive CLIP dims vs 64/16,384 SAE units), we compared them at a tight budget where the selector rather than data volume is binding (CIFAR-10, 5%, no final refinement; ResNet-18, 200 ep):

Encoder (matched 0.39% activation rate)Test accuracy (%)
SAE dictionary82.54 ± 0.91
Dynamic dense CLIP (top-3/768 dims)80.09 ± 0.92
Paired difference (SAE − dense)+2.45 (positive on 5/5 seeds)

The reason is visible in the code itself: at matched sparsity the dense CLIP code has 193 active units and 12.8 distinct Rep values per class, with 87% of a class sitting on the modal value, so Rep-only and Div-only select almost the same set (overlap 0.648; SAE: 0.039) and the α schedule stops being a curriculum. This is a property of the construction, not of one draw: across five dense constructions (CLIP top-3, k=6, k=12, DINOv2 top-3, random projection) the best reaches 81.67, still 0.9 below both SAE dictionaries (82.54, 82.58); the five dense constructions span 1.58 points while the two SAE dictionaries differ by 0.04.

Encoder familyVariantsAccuracy range (%)
Dense constructionsCLIP top-3, k=6, k=12, DINOv2 top-3, random projectionbest 81.67; spread 1.58
SAE dictionariesCLIP dict, DINOv2 dict82.54 / 82.58; spread 0.04

What the SAE provides is then the measurement interface itself — a discrete, learned, interpretable factor vocabulary with well-defined frequency/coverage/rarity, a transferable probe (Table 9b), and stable behavior across 12 hyperparameter settings (Table 5). We have revised Appendix A.4's "prerequisite" wording to "interpretable default".

[Q1]

Please clarify the selection fraction used in the ablation in Table 4b.

Thank you for pointing out the inadequate description of our experimental setup. Our ablation experiments are conducted on CIFAR-100 with ResNet-50, under a fixed selection ratio of 70%. We have added the detailed configurations (CIFAR-100, ResNet-50, 200 epochs, fixed 70% per-epoch selection ratio) to the notes of Table 4(b) in the revised manuscript.

[Q2]

Why is RS2 w/o substantially lower than RS2 w/r on ResNet-50, whereas the two are nearly identical on ResNet-18?

Regarding RS2, we only reproduce it in this work to compare its strengths and weaknesses against our method. We do not include further analysis of the underlying mechanism of RS2 in the paper. Thank you for your understanding.

Dear Reviewer Mb8w,

Thanks for your time and effort in reviewing this paper. Your comments and suggestions are valuable to improve the quality of this paper. As the author-reviewer discussion period ends in about 3 days, we kindly request that you read the rebuttal and confirm whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Reviewer_LSoCOpenReview ↗
Rating
4(+2)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper talks of dynamic dataset selection. Dataset selection often talks of measures which are submodular in nature. The paper examines two specfic metrics representative and diversity. Thr authors use SAE to gather item/data level statistics. The autors measure represetative by frequency level i.e common objects/items and rate item for diversity. Experiments across vision, text, medical, language modeling, and generative multimodal tasks and different models show that our method yields a better accuracy-efficiency frontier than prior selection methods

Critical strengths

The dynamic aspect of DS devoid of gradient and influence functions makes this plug and play part really useful. I also find the part of ablations and multiple model study very through.

Critical weaknesses

(i)Eq. 1 is a sum of per-sample scores under a cardinality constraint, so the argmax is top-k ranking. This implies within-subset redundancy ( i.e marginal gain of a sample ) is independent of S. This implies the score is modular not sub-modular.

  • "Coverage" and "process-level diversity" are used as set/trajectory properties, but Rep, Div and Pen are per-sample quantities. Clarify this part
  • H(di,t)H(d_i,t) in Eq. 1 vs H(i,t)H(i,t) in Eq. 7 hides dependence on selection history (u_i(t)) and dataset-level statistics (cj,Rar)(c_j, Rar).

(ii)The papers claim is that selection rotates through rare factors over time. But Pen only counts how often a given sample was picked. Two different samples can share the same rare unit can both be selected every epoch without penalty. I don't see how factor-level rotation follows from the mechanism, so the claim needs a form of evidence.

(iii)Fig. 2 compares MMD against K-Center only, and MMD is measured in the same feature space the score is built from. MMD also suffers from variance related issues. Would weighted version of MMD help ?

Questions for authors

(i)Is Fig. 1's min-max normalization applied in the implementation? If so, over what set (full dataset, per class, current candidates)? (ii)How does random selection with the same usage penalty compare to the full method? (iii)Does the selection actually rotate across factors, or only across samples? Could you show the fraction of rare units covered per epoch? (iv)What happens on a dataset with label noise, where rare factors may correlate with corrupted samples?

Minor comments In L60 authors mention 'A frequent factor like color ir material is spread across...' Any references for this D^2 pruning looks as if its needs latex formatting

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer LSoC

[W1]

Eq. (1) is modular rather than submodular; "coverage" and "process-level diversity" are set-/trajectory-level properties while Rep, Div and Pen are per-sample quantities; and Eq. (1) vs Eq. (7) hides the dependence on selection history and dataset-level statistics.

We thank the reviewer for this careful observation, which we agree with. For a fixed epoch, Eq. (1) sums per-sample scores under a cardinality constraint; its argmax is simply top-p selection. Conditioned on the selection history at epoch start, the objective is modular rather than submodular, and each sample's marginal gain is independent of the selected set S. We do not claim Eq. (1) maximizes a submodular coverage function, and the revised manuscript contains no such wording. In the revision, we have clarified the formulation and notation with three changes:

  1. Section 3.1 now explicitly states that Rep(i), Div(i), and Pen(i, t) are per-sample scores, and that per-epoch selection follows a modular top-k rule.
  2. Section 3.1 clarifies that "coverage" and "process-level diversity" are dataset- and trajectory-level measurement properties, not the set function optimized by Eq. (1). The history-aware penalty couples these per-epoch modular subproblems into a long-term trajectory.
  3. Section 3.5 now writes the score as H(i, t; u(t), D), where u(t) encodes the selection history — u(i, t) counts the selections of sample i before epoch t — and D denotes the dataset-level statistics used by Rep and Div.

Section 3.5 also specifies the offline precomputed quantities (Rep, Div), the online-updated u(i, t), and the set over which the scores are normalized (the full training set; see also the new appendix subsection "Normalization of the Selection Scores").

[W2]

Pen only counts how often a given sample was picked — two different samples sharing the same rare unit can both be selected every epoch without penalty. How does factor-level rotation follow from the mechanism? Evidence is needed.

We appreciate the reviewer's careful observation on the gap between sample-level rotation and factor-level rotation. Although the penalty is defined on sample selection counts, factor-level coverage is guaranteed via sample rotation: whenever two samples share the same rare unit, covering that factor requires selecting either of them, and the usage penalty prevents any single sample — and hence the factors concentrated in it — from being repeatedly over-selected. To directly verify this claim at the factor level, we conduct a simple experiment that measures how much of its fair exposure each rare factor actually receives:

  • c(i): times sample i was selected (selection phase only); c̄ = mean of c(i) over samples
  • A(i): the 64 SAE units activated by sample i
  • freq(u): number of samples i with u ∈ A(i); exp(u): sum of c(i) over samples i with u ∈ A(i)
  • r(u) = exp(u) / (freq(u) · c̄) — 1.0 = uniform sampler's fair share
  • T: active units with freq(u) ≤ max(2, 0.001·N) — CIFAR-10: freq ≤ 50
  • RFED = 1 − mean of min(r(u), 1) over units u ∈ T — 0 = no rare factor under-served
  • S(a): fraction of units u ∈ T with r(u) ≥ a
  • p10: 10th percentile of r(u) over T

min(r(u), 1) prevents over-exposure of one factor from compensating for starvation of another.

C = CLIP scoring dictionary (11,132 units), D = independent DINOv2 dictionary (16,350 units). RFED/S are percentages; p10 is a rate.

MethodRFED ↓ C / DS75 ↑ C / Dp10 ↑ C / DTail cov. ↑ C / D
Ours0.6 / 2.4100.0 / 99.90.991 / 0.915100 / 100
Ours w/o penalty12.5 / 14.979.2 / 77.20.489 / 0.45999.9 / 96.4
InfoBatch22.5 / 22.254.6 / 53.40.496 / 0.534100 / 100
Random + same penalty21.7 / 14.168.6 / 78.60.233 / 0.554100 / 100

Experimental results demonstrate that our method ensures uniform coverage of rare factors across both the CLIP feature space and the independent DINOv2 feature space, i.e., the selection procedure indeed rotates across factors rather than merely across samples.

[W3]

Fig. 2 compares MMD against K-center only, and MMD is measured in the same feature space the score is built from; would a weighted version of MMD help?

We agree with the reviewer's suggestion to report results using weighted MMD. To avoid prior bias introduced by the CLIP feature space, we conduct the tests in the DINO feature space. The results are presented below.

CIFAR-10 (z, lower = closer to the full-data distribution)

rule0.5% β=00.5% β=11% β=01% β=13% β=03% β=1
ours — Rep+Div α=0.212.310.220.117.822.919.6
class-center30.739.348.358.676.790.0
k-means medoid54.780.482.1110.7125.8144.9
K-center (FPS)22.418.037.230.073.260.8
EL2N-hard12.011.815.915.053.349.0
EL2N-easy75.164.9101.785.3171.0141.6

As summarized in the table, our method obtains the best or runner-up results under all experimental configurations, which verifies that our definition of representativeness captures information overlooked by geometric and difficulty-aware sampling strategies.

[Q1]

Is Fig. 1's min-max normalization applied in the implementation? If so, over what set (full dataset, per class, current candidates)?

Yes, min-max normalization is adopted in our implementation. This is because Rep and Div operate on vastly different scales: Rep is computed with inverse class-coverage weights for frequent factors, whereas Div uses inverse activation-frequency weights over the active factors of each sample. As a result, their raw value ranges differ by roughly two orders of magnitude. Without normalization, the coefficient α in Equation 7 would be dominated by the term with the larger scale, failing to achieve the intended trade-off between the two objectives. We therefore normalize each score individually to [0, 1] prior to combination. This normalization is performed only once on the full training set before training starts. It is not computed per class, not restricted to the current candidate pool, and not re-evaluated at each epoch. We have clarified this detail explicitly in Section 3.5 of the revised manuscript and in a new appendix subsection ("Normalization of the Selection Scores").

[Q2]

How does random selection with the same usage penalty compare to the full method?

We ran this control and now report it; it therefore isolates what the penalty does on its own. The penalty flattens usage counts but gives the rotation no direction: with a frozen order it cannot choose which samples or factors to revisit. In short: the penalty equalises counts, the scores supply the direction — which is also the evidence that factor-level rotation does not follow from count flattening alone (W(ii)).

All comparisons are on CIFAR-10 (30%, 200 epochs) unless otherwise noted:

MetricRandom + penaltyOurs
Usage-count Gini0.5830.132
Epochs to reach 90% rare-unit coverage93
Rarity-weighted coverage AUC0.9550.993
Share of selections to top-10% Rep samples (first 20 epochs)10.0% (uniform throughout)21.7%
Selection-phase gradient bias — CIFAR-1053.2 ± 4.4%12.7 ± 0.7%
Selection-phase gradient bias — CIFAR-10056.1 ± 1.8%7.5 ± 0.6%
Weighted MMD at β=14.80σ2.38σ
Final accuracy — CIFAR-100 (3 seeds)73.62 ± 0.4576.14 ± 0.27

[Q3]

Does the selection actually rotate across factors, or only across samples? Could you show the fraction of rare units covered per epoch?

We understand the reviewer's concern. Factor-level coverage is guaranteed via sample rotation. We validate this conclusion with a simple experiment. We define the rare-tail SAE units as those activated by at most 50 out of the 50,000 training images (5,736 units for CIFAR-10). The experiment is conducted on CIFAR-10 with a selection budget of 30%, reproduced using the empirically measured sample selection counts.

MethodEpoch 1Epoch 2Epoch 3Epoch 5Epoch 10Epoch 20Epoch 50
Ours72.485.392.197.699.8100100
Random + penalty77.681.684.487.591.494.498.1
Ours (penalty removed)72.472.472.472.472.672.673.4

The results demonstrate that the selection procedure indeed rotates across different factors, and this rotation effect arises jointly from the penalty term and sample scores. Scores alone lead to repeated selection of high-Rep samples at the early stage of training, yielding inferior factor coverage compared with the random baseline equipped with the penalty.

[Q4]

What happens on a dataset with label noise, where rare factors may correlate with corrupted samples?

Regarding the reviewer's concern about the noise robustness of our method, relevant results have been presented in Table 7 of the appendix. We reproduce the table here for easy reference. This experiment evaluates noise robustness on CIFAR-100 with ResNet-50 under 20% label noise. We report top-1 accuracy (%) for clean and noisy training, as well as the accuracy drop Δ = Clean − Noisy.

Setting20% Clean20% Noisy20% Δ30% Clean30% Noisy30% Δ
CLIP54.1046.058.0561.9058.343.56
MoSo-31.01-58.2743.7314.54
MoDS51.8940.2511.6456.0048.537.47
InfoBatch56.9054.492.4163.8155.678.14
Ours64.2960.333.9675.7060.8614.84

Under label noise, our method achieves the highest final top-1 accuracy among all compared data selection baselines. We provide a tentative explanation for this observation: our scheme prioritizes high-Rep samples in the early training phase, combined with the learning rate decay schedule. Samples containing abundant rare factors are mostly introduced at later training stages. At this point, the learning rate is relatively small, and the model has already learned the dominant patterns of each class. This enables our method to maintain strong final performance even with label noise present.

[M]

Minor: please add references for the L60 claim about frequent factors, and fix the LaTeX formatting of D^2 pruning.

We appreciate the reviewer's suggestions on formal writing, and we have revised the relevant content in the revised manuscript. Regarding the reviewer's concern about the supporting evidence for the statement that high-frequency factors such as color and texture are widely distributed across different categories, the Introduction now cites the corresponding appendix figure directly at this claim (the universal-unit visualization in Appendix Fig. 7). For instance, the high-frequency factor "yellow" appears across multiple classes, including chairs, oranges and sofas. The D² Pruning citation formatting has also been checked and corrected in the reference list.

Thank you for your response. I appreciate the clarification.

Reviewer_fKm3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies dynamic data selection for reducing training cost while preserving model performance. It formulates representativeness as the coverage of dataset-level high-frequency feature factors and diversity as the exposure to complementary rare factors over the training trajectory. The proposed framework trains a sparse autoencoder on features from a plug-in encoder to obtain sparse units, scores each sample according to its high-frequency and rare-unit activations, penalizes samples that have been selected repeatedly, and uses a curriculum scheduler to shift from representativeness to diversity during training. Experiments across image classification and several additional tasks report competitive accuracy under reduced data usage, including results close to full-data training on ImageNet-1K.

Critical strengths
  • The motivation and overall structure of the proposed framework are clear.

  • The usage-frequency penalty is simple and computationally inexpensive. It offers a practical way to discourage the repeated selection of a small group of samples without requiring gradients or influence estimates from the training model.

  • The empirical evaluation is broad, covering multiple datasets, model families, selection ratios, component ablations, and tasks beyond standard image classification.

Critical weaknesses
  • The proposed scoring rule does not directly optimize or track the subset-level and trajectory-level factor coverage emphasized by the paper. Representativeness and diversity are computed independently for each sample, without considering whether its factors have already been covered by other selected samples. The usage-frequency penalty tracks sample selection counts rather than factor coverage. It may encourage sample rotation, but it does not ensure that the selected samples cover complementary factors over time.

  • The paper provides insufficient evidence that the proposed representativeness captures information missed by geometric or difficulty-based selection. The motivating claim that cluster-center or difficult-example selection may miss distributed frequent factors is not empirically demonstrated. Figure 2 compares the proposed method with K-center, which differs from selecting samples near cluster or class centers. Moreover, it evaluates MMD rather than high-frequency factor coverage and does so in the same CLIP feature space used to construct the proposed scores.

  • The experimental evaluation omits recent and closely related dynamic selection methods, such as RePB [1]. In addition, the paper does not clearly report the number of random seeds for the proposed method.

[1] Qing Zhou and Tao Yang and Bingxuan Zhao and Hongyuan Zhang and Junyu Gao and Qi Wang. Inconsistency Biases in Dynamic Data Pruning. In ICLR 2026.

  • The theoretical analysis does not support the claim that the usage-frequency penalty reduces gradient bias. The anti-monopoly result only shows that overused samples eventually lose priority. It neither bounds gradient bias nor proves that it decreases.

  • The claims of broad generalization and practical efficiency are weakly supported. RSD-15K uses a static combination of the two scores, while WikiText-2 uses only representativeness, so neither experiment validates the full dynamic framework. The ImageNet speedup is also estimated without a clear end-to-end accounting of feature extraction, storage, scoring, and selection overhead.

Questions for authors
  • Since the method tracks sample usage rather than factor usage, what mechanism ensures that the selected samples add complementary factors instead of repeatedly covering the same high-scoring factors? Do the authors have measurements of cumulative unique-factor coverage and redundancy over the training trajectory?

  • Can the authors provide a controlled experiment that directly tests the motivating failure case? In particular, please compare the proposed representativeness score with class-center, cluster-medoid, K-center, and difficulty-based selection under the same budget, and report direct high-frequency factor coverage using independently defined factors or an independent representation space.

  • Can the authors provide comparisons with recent dynamic data selection methods such as SeTa and RePB under the same total data exposure, number of optimizer updates, refinement policy, and end-to-end wall-clock accounting?

  • Do the authors have either a theoretical bound or direct gradient measurements showing that the usage-frequency penalty reduces gradient bias, rather than only flattening the distribution of sample-selection counts?

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer fKm3

W1: The scoring rule does not directly track subset-/trajectory-level factor coverage, since Rep, Div and the penalty are all computed at the sample level. Q1: What mechanism ensures that selected samples add complementary factors, and do you have measurements of cumulative unique-factor coverage and redundancy over the trajectory?

In our method, selection operates at the sample level: the score assigned to each individual datum essentially determines its ordering, i.e., which data should appear with higher priority and which should be introduced at later stages of training. The penalty term serves as a concise and effective mechanism to suppress the overrepresentation of samples containing high-frequency factors, thereby ensuring coverage of rare feature factors in practice. We conduct a simple experiment to verify the above claim:

These metrics measure how much of its fair exposure each rare factor actually receives.

  • c(i): times sample i was selected (selection phase only); c̄ = mean of c(i) over samples
  • A(i): the 64 SAE units activated by sample i
  • freq(u): number of samples i with u ∈ A(i); exp(u): sum of c(i) over samples i with u ∈ A(i)
  • r(u) = exp(u) / (freq(u) · c̄) — 1.0 = uniform sampler's fair share
  • T: active units with freq(u) ≤ max(2, 0.001·N) — CIFAR-10: freq ≤ 50
  • RFED = 1 − mean of min(r(u), 1) over units u ∈ T — 0 = no rare factor under-served
  • S(a): fraction of units u ∈ T with r(u) ≥ a
  • p10: 10th percentile of r(u) over T

min(r(u), 1) prevents over-exposure of one factor from compensating for starvation of another.

C = CLIP scoring dictionary (11,132 units), D = independent DINOv2 dictionary (16,350 units). RFED/S are percentages; p10 is a rate.

MethodRFED ↓ C / DS75 ↑ C / Dp10 ↑ C / DTail cov. ↑ C / D
Ours0.6 / 2.4100.0 / 99.90.991 / 0.915100 / 100
Ours w/o penalty12.5 / 14.979.2 / 77.20.489 / 0.45999.9 / 96.4
InfoBatch22.5 / 22.254.6 / 53.40.496 / 0.534100 / 100
Random + same penalty21.7 / 14.168.6 / 78.60.233 / 0.554100 / 100

Experimental results demonstrate that our method ensures uniform coverage of rare factors across both the CLIP feature space and the new DINO feature space. Please refer to our response to W2 for the analysis of high-frequency factors.

[W2/Q2]

W2: The motivating claim that geometric/difficulty-based selection misses distributed frequent factors is not empirically demonstrated — Fig. 2 compares against K-center only and evaluates MMD in the same CLIP space used to build the scores. Q2: Please provide a controlled comparison against class-center / cluster-medoid / K-center / difficulty-based selection, and report direct factor coverage using independent factors or an independent representation space.

We appreciate the reviewer's suggestion. Following your suggestion and that of reviewer LSoC, we compare our representative scoring against class-center, k-means medoid, K-center, and difficulty-based sampling under identical data budgets. We further report results using the weighted MMD metric with independently defined factors and an independent representation space:

CIFAR-10 (z, lower = closer to the full-data distribution)

rule0.5% β=00.5% β=11% β=01% β=13% β=03% β=1
ours — Rep+Div α=0.212.310.220.117.822.919.6
class-center30.739.348.358.676.790.0
k-means medoid54.780.482.1110.7125.8144.9
K-center (FPS)22.418.037.230.073.260.8
EL2N-hard12.011.815.915.053.349.0
EL2N-easy75.164.9101.785.3171.0141.6

As summarized in the table, our method obtains the best or runner-up results under all experimental configurations, which verifies that our definition of representativeness captures information overlooked by geometric and difficulty-aware sampling strategies.

We note two caveats when interpreting these comparisons: (i) our representative method inherently selects samples that activate the most high-frequency feature factors, which makes direct comparisons with other methods unfair in this aspect; (ii) feature factors can achieve over 80% coverage at a very small fraction of the budget, leading to subtle numerical differences.

[W3/Q3]

W3: The evaluation omits recent and closely related dynamic selection baselines such as RePB, and the number of random seeds is not clearly reported.

We appreciate the reviewer's suggestion of these extra baselines. These works contribute distinct viewpoints to data selection research, and we have included the relevant citations and discussions in the revised manuscript (Section 2, Related Work: SeTa and RePB are now cited and discussed alongside the existing dynamic-selection baselines). We have carried out the suggested experiments, as shown below.

Data / backboneMethod70%30%
CIFAR-10 / R18Ours96.195.3
CIFAR-10 / R18RePB95.694.9
CIFAR-10 / R18SeTa95.795.0
CIFAR-100 / R18Ours79.776.7
CIFAR-100 / R18RePB78.477.2
CIFAR-100 / R18SeTa79.477.4

We follow the training protocol of SeTa. Consequently, our results differ slightly from those in Table 1 of the paper. Under this setting, our method attains the best performance in most scenarios and yields a steady yet modest improvement over current SOTA approaches.

[W4/Q4]

W4: The anti-monopoly result neither bounds nor decreases gradient bias. Q4: Please provide either a theoretical bound or direct gradient measurements showing that the usage-frequency penalty reduces gradient bias.

We agree the original lemma only shows over-used samples lose priority and does not bound gradient bias. We have removed this implication — the abstract and the contribution list now claim only provable anti-monopoly behavior and empirically measured gradient-bias reduction — and introduced Proposition 1 in Appendix B.1, stated alongside (not replacing) the anti-monopoly theorem.

Let u(i) denote the selection count of sample i over the T selection epochs and K the per-epoch selection budget, so the total number of selections is TK. Let ḡ(T) denote the selection-weighted average gradient at a fixed θ — the average of the per-sample gradients ∇ℓ(i; θ) weighted by u(i) — and let g(θ) denote the full-data average gradient. Defining B as the maximum deviation of any sample's selection count from its uniform share TK/N, we have:

‖ḡ(T) − g(θ)‖ ≤ (B / TK) · Σ ‖∇ℓ(i; θ)‖, where the sum runs over all samples.

This bounds sampling-composition bias at fixed θ; it does not cover bias from evolving θ, and we state this scope explicitly in Appendix B.1 of the revised manuscript.

We directly measure bias at the final θ via one full-set forward pass and three backward passes for uniform, full-trajectory and selection-phase weights. In the key ablation (scores and curriculum fixed, penalty removed):

  • CIFAR-10: bias drops from 73.74±6.54% to 12.74±0.65% (5.8× reduction; cosine similarity: 0.60–0.77 → 0.995–0.996)
  • CIFAR-100: bias drops from 70.21±1.24% to 7.48±0.63% (9.4× reduction; cosine similarity: 0.86 → 0.997)

Without the penalty, many samples are never selected over 170 selection epochs; all samples get selected with the penalty. The measured value satisfies the bound, though the worst-case bound is loose by construction. We therefore rely on these empirical measurements as our primary evidence.

As raised by the reviewer, one natural question is whether this effect merely stems from count flattening. Across 13 variants and 47 runs, usage Gini strongly predicts bias (Pearson r = 0.990; bias ≈ 102 · Gini − 1.5, with R² = 0.98). The penalty reduces bias by lowering the Gini coefficient, which suggests count flattening serves as its underlying mechanism. Proposition 1 formalizes this quantitative connection instead of leaving it as an informal assumption; the proposition, its proof, and these measurements are now included in Appendix B.1 of the revised manuscript.

Thank you for the detailed response. The additional experiments and clarifications improve the evaluation. I have three remaining questions:

  • The controlled comparison for W2 reports only 0.5%, 1%, and 3% sampling ratios, which are substantially lower than most ratios used in the main experiments. Does the proposed representativeness score retain its advantage over class-center, k-means medoid, K-center, and difficulty-based selection at the original sampling ratios? What's the meaning of \beta?
  • Since the rebuttal includes comparisons with RePB and SeTa, could these results be added to the revised manuscript or appendix, together with the full training protocol, total data exposure, and results across multiple random seeds?
  • The revised manuscript reports additional efficiency measurements, but RSD-15K uses a static combination of the scores and WikiText-2 uses only representativeness. How should these experiments be interpreted with respect to the full dynamic framework? Also, does the reported ImageNet speedup include feature extraction, feature storage, scoring, selection, and other preprocessing costs? If not, could the claims about cross-task generalization and end-to-end efficiency be qualified accordingly?

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers and the Area Chair for the constructive and detailed discussions throughout the rebuttal period. The questions raised have substantially improved the paper: they prompted new controlled experiments, a new theoretical result, direct gradient-bias measurements, additional budget-matched comparisons, and a correction to our own reported baselines. Below we summarize (1) the aspects of our work that reviewers valued, (2) the concerns raised by each reviewer and how we addressed them, and (3) the corresponding revisions, all marked in green in the updated manuscript.

1. Strengths recognized by the reviewers

We are encouraged that reviewers highlighted the following aspects of our work:

  • Lightweight, plug-and-play design. The usage-frequency penalty is simple, computationally inexpensive, and discourages repeated selection of a small group of samples "without requiring gradients or influence estimates from the training model" (Reviewer fKm3). Reviewer LSoC similarly noted that "the dynamic aspect of DS devoid of gradient and influence functions makes this plug-and-play part really useful," and Reviewer Mb8w acknowledged that the method is straightforward to implement with scores computed offline at low additional cost.
  • Clear motivation and framework structure (Reviewer fKm3), with a re-examination of representativeness (dataset-level high-frequency factor coverage) and diversity (process-level rotation over the training trajectory).
  • Broad empirical evaluation, covering multiple datasets, model families (ResNet-18/34/50, VGG, ViT), selection ratios, component ablations, and tasks beyond image classification — vision, text, medical, language modeling, and generative multimodal tasks (all three reviewers). Reviewer LSoC also found the ablations and multi-model study thorough.

2. Concerns raised and how we addressed them

Reviewer fKm3

(a) Sample-level scoring vs. factor-level coverage (W1/Q1). The concern was that per-sample scores and a sample-count penalty do not directly track subset-/trajectory-level factor coverage. We clarified the mechanism (the penalty suppresses over-selection of any single sample, so samples carrying rare factors are rotated in) and provided direct factor-level evidence: new RFED / S75 / p10 / tail-coverage metrics measuring each rare factor's actual exposure relative to its fair share, evaluated in both the CLIP scoring dictionary and an independent DINOv2 dictionary. Our method achieves near-zero rare-factor exposure deficit (RFED 0.6 / 2.4 vs. 12.5 / 14.9 without the penalty and 21.7–22.5 for InfoBatch and random+penalty controls), with essentially full tail coverage.

(b) Motivating comparison against geometric/difficulty-based selection (W2/Q2). We ran the requested controlled comparison against class-center, k-means medoid, K-center, and EL2N-based difficulty sampling under identical budgets, reporting weighted MMD in an independent DINO feature space with independently defined factors. Our scoring obtains the best or runner-up results under all configurations. For the follow-up question on the original (30%–70%) ratios — where high-frequency factor coverage saturates for all methods — we additionally compared early-epoch convergence speed at a 30% ratio, where our method converges fastest (e.g., 59.08 ± 6.87 at epoch 7 vs. 50.97–56.86 for the alternatives), indirectly verifying that more representative samples are selected early.

(c) Missing recent baselines and seed reporting (W3/Q3). We cited and discussed SeTa and RePB in Section 2, ran the requested comparisons under the SeTa training protocol, and added both methods to the main table with an explicit protocol footnote. Under this protocol our method attains the best result in three of the four dataset–ratio cells and runner-up in the fourth, a steady yet modest improvement over these recent SOTA methods. All experiments except ImageNet-1K use three random seeds with standard deviations within 0.4.

(d) Gradient-bias claim (W4/Q4). We agreed the original anti-monopoly theorem does not bound gradient bias and corrected the claim: the abstract and contributions now state provable anti-monopoly behavior plus empirically measured bias reduction. We introduced Proposition 1 (Appendix B.1, alongside the original theorem), which bounds the sampling-composition gradient bias by the maximum deviation of selection counts from the uniform share, with an explicit scope statement (fixed parameters; evolving-parameter bias is not covered). Direct measurements show the penalty reduces selection-phase gradient bias from 73.74 ± 6.54% to 12.74 ± 0.65% on CIFAR-10 (5.8×) and from 70.21 ± 1.24% to 7.48 ± 0.63% on CIFAR-100 (9.4×); across 13 variants and 47 runs, usage Gini strongly predicts bias (Pearson r = 0.990, R² = 0.98).

(e) Cross-task experiments and efficiency accounting (W5 and follow-up). We clarified the roles of RSD-15K (a few-epoch regime where most dynamic selectors are unsuitable) and WikiText-2 (a distribution-shifted test setting where representativeness alone is the appropriate objective, analogous to noisy-data learning), and confirmed that the ImageNet efficiency analysis accounts for feature extraction, feature storage, scoring, and selection overheads.

Reviewer LSoC

(a) Modularity and notation (W1). We agreed Eq. (1) defines a modular (not submodular) per-epoch objective whose argmax is top-k selection, and clarified that coverage and process-level diversity are dataset-/trajectory-level measurement properties rather than the optimized set function. Section 3.1 and Section 3.5 now state this explicitly, and the score is rewritten as H(i, t; u(t), D) to expose the dependence on selection history u(t) and dataset-level statistics D, including which quantities are precomputed offline and which are updated online.

(b) Factor-level rotation evidence (W2, Q3). Beyond the RFED exposure analysis above, we reported per-epoch rare-unit coverage: our method reaches 90%+ coverage of the rare tail (5,736 units) by epoch 3 and 100% by epoch 20, whereas removing the penalty freezes coverage at ~72–73% and random+penalty reaches only 98.1% by epoch 50 — confirming that rotation happens across factors, not only across samples, and that it arises jointly from the scores and the penalty.

(c) Weighted MMD in an independent space (W3). Provided as described above (DINO space, independently defined factors).

(d) Clarification questions (Q1, Q2). We specified min-max normalization precisely (each score normalized individually to [0,1], computed once on the full training set before training, not per class, not per candidate pool, not re-evaluated per epoch; Section 3.5 and a new appendix subsection). The random+penalty control isolates the penalty's role: it flattens usage counts (Gini 0.583 → ours 0.132) but supplies no direction — slower rare-unit coverage, higher gradient bias, and ~2.5 points lower final accuracy on CIFAR-100 — showing that the scores supply the direction of rotation.

(e) Label noise (Q4) and minor comments. We reproduced the noise-robustness table (CIFAR-100, ResNet-50, 20% label noise), where our method achieves the highest noisy-training accuracy, with a tentative explanation based on late introduction of rare-factor samples under a decayed learning rate. We also added a direct citation to the appendix universal-unit visualization for the L60 claim and fixed the D² Pruning citation formatting. We thank the reviewer for the follow-up acknowledgment.

Reviewer Mb8w

(a) SAE effectiveness vs. dense-feature selection (W1). We ran both requested controls with the selector, schedule, penalty, and per-class budget fixed, matching the dense code to the SAE's 0.39% activation rate, at a tight budget where the selector is binding (CIFAR-10, 5%, no refinement). The SAE dictionary outperforms dynamic dense CLIP by +2.45 points (82.54 ± 0.91 vs. 80.09 ± 0.92; positive on 5/5 seeds). The mechanism is visible in the code statistics: at matched sparsity the dense code yields only 12.8 distinct Rep values per class with 87% of a class on the modal value, so Rep-only and Div-only select nearly the same set (overlap 0.648 vs. 0.039 for SAE) and the scheduler degenerates. Across five dense constructions (CLIP top-3, k=6, k=12, DINOv2 top-3, random projection), the best reaches 81.67 — still below both SAE dictionaries (82.54, 82.58), which differ by only 0.04. We revised Appendix A.4 to present the SAE as an interpretable default, not a necessary component.

(b) ImageNet protocol and efficiency claims (W2). We revised the Table 4(a) caption to state that baseline results are quoted from original papers under their own fixed-epoch protocols and serve as contextual references, and that dynamic-method comparisons follow the SeTa protocol with relative performance (SeTa: 0.5% drop at 45.5% pruning; ours: 0.3% drop at 50% pruning). The overhead accounting covers feature extraction, storage, scoring, and selection.

(c) Refinement and fair comparison with RS2 (follow-up questions). We clarified that RS2 baselines follow the original paper without the refinement stage, and added the requested matched-exposure comparisons (three seeds, identical total processed samples including refinement) for the full method, Only-Pen, and RS2 (w/o) at 30% and 70% on CIFAR-100 / ResNet-50: our method remains best at both ratios (75.35 ± 0.53 and 79.10 ± 0.42). An epoch-170 snapshot (before refinement takes effect) shows our method already ahead, and multi-seed analysis shows refinement itself contributes only +0.25 ± 0.35 on average; the ablation discussion and a budget-matched appendix subsection were updated accordingly.

(d) Data inconsistency identified during discussion. The reviewer noted a 3-point gap between Only-Pen and RS2 (w/o) at epoch 170 despite their conceptual similarity. Re-examining all logs, we found the main-table RS2 (w/o) runs used batch size 256 while Only-Pen and our method used 128, making them not directly comparable. We withdrew those RS2 (w/o) numbers and updated the main table with results under unified hyperparameters (77.50 at 70%, 74.63 at 30%, still 1–2 points below our method), cross-checked the settings of all other comparative experiments, and updated the appendix tables accordingly. We further identified the source of the residual Only-Pen vs. RS2 (w/o) gap: stratified per-class sampling in our framework versus hypergeometric fluctuations in RS2's realized per-class counts (about 150 ± 10 per class at 30% and 350 ± 19 at 70%). We thank the reviewer for the careful examination that helped us correct this issue.

3. Summary of manuscript revisions (marked in green)

  1. Abstract and contributions: gradient-bias claim reworded to empirical measurement, with a new bound (Proposition 1, Appendix B.1) stated alongside the anti-monopoly theorem, including proof, scope, and measurements.
  2. Sections 3.1 and 3.5: modular top-k formulation, history- and dataset-dependent notation, and score-normalization details; new appendix subsection "Normalization of the Selection Scores."
  3. Section 2: SeTa and RePB cited and discussed; both methods added to the main table under the SeTa protocol with an explicit footnote.
  4. Table 4(a) caption: protocol differences for ImageNet baselines stated explicitly; Table 4(b) notes now include the full ablation configuration (CIFAR-100, ResNet-50, 200 epochs, fixed 70% per-epoch selection ratio).
  5. Appendix: budget-matched multi-seed comparisons and refinement analysis added; RS2 (w/o) results updated under unified hyperparameters; Appendix A.4 wording on the SAE revised to "interpretable default."
  6. Minor: L60 claim now cites the appendix universal-unit visualization; D² Pruning citation formatting corrected.

We hope these additions and clarifications address the reviewers' concerns, and we remain happy to answer any further questions. Thank you again for the time and care devoted to reviewing our work.

— The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.