- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies dynamic data selection. The authors train a sparse autoencoder (SAE) on CLIP features and use the activation frequencies of SAE units across the dataset to compute representativeness and diversity scores for each sample. During training, an epoch-wise scheduler balances the two scores, together with a penalty based on historical usage. The top-p samples are then selected for training at each epoch. The authors evaluate this method on CIFAR-10/100, Tiny-ImageNet, ImageNet-1K, and several text, medical, and multimodal tasks.
Critical strengths
- The method is straightforward to implement. The scores are computed offline, so the additional computational cost is low.
- The experiments cover a broad range of domains, including vision, text, medical, and multimodal tasks.
Critical weaknesses
1. The method is weakly aligned with the narrative of representativeness and diversity. The main claim needs stronger evidence.
The paper argues that SAE activation statistics provide a different view of representativeness and diversity than conventional dense feature-space methods. However, to establish the practical value of this approach, the authors need to demonstrate at least two things:
First, the proposed method should outperform selection directly in CLIP's dense feature space. The baseline should also be dynamic. For example, at each epoch, graph cut could be used for selection, with a similar schedule for the coefficient balancing representativeness and diversity.
Second, the SAE should provide useful structure beyond what can be obtained from a simple transformation of dense CLIP features. One straightforward experiment would be to randomly project CLIP features to 16,384 dimensions, retain the top 64 dimensions as the activation set, and keep the rest of the pipeline unchanged. Comparing the results with those obtained using the SAE would help isolate the contribution of the learned SAE features.
Only if both are demonstrated would the claim that the method captures representativeness and diversity at a different level be convincing.
Moreover, the selection method itself weakens the narrative. The objective in Eq. 1 is a sum of per-sample scores, making it a modular function. Optimization simply amounts to selecting the top-p samples. However, with the usage-frequency penalty, the selection counts of different samples may become increasingly balanced over time. Appendix Figure 8 also shows that most samples are selected a similar number of times. If this pattern holds, the method may eventually behave similarly to RS2 w/o in terms of sample exposure. The main difference may be the order in which samples are selected. The authors need to clarify how much of the improvement comes from this ordering itself.
2. The experimental evaluation is limited in scale, and some key results lack directly comparable baselines.
For vision tasks, the most comprehensive baseline comparisons are conducted on relatively small datasets such as CIFAR-10 and CIFAR-100. On ImageNet-1K, Table 4a does not provide a directly comparable evaluation of the proposed method against the listed selection baselines.
Table 4a uses a matched-propagation protocol for the proposed method. This means that, despite the low per-epoch selection fraction, the method processes the same total number of samples as full-data training. However, the static baselines in the table use a fixed-epoch protocol, with results taken from their original papers. These results are therefore not directly comparable. Although the abstract explicitly identifies the matched-propagation protocol, highlighting this result may overstate its practical significance for training efficiency.
Moreover, on ResNet-50 with CIFAR-100, at the 70% selection fraction, the proposed method (79.1) outperforms the strongest random resampling baseline, RS2 w/r (78.4), by only 0.7 percentage points. At the 30% selection fraction, the improvement is only 0.6 points (75.7 vs. 75.1). No variance is reported for either comparison.
In Table 4b, the "only Pen" variant, which does not use any SAE scores and behaves similarly to deterministic sample rotation, already achieves 78.1. This suggests that the performance gains may come primarily from balanced sample rotation rather than SAE factor-based scoring.
The reduced-data ImageNet results (Tables 2 and 3), which do reflect actual training savings, are compared only against full-data training, without any dynamic selection baselines such as RS2 or InfoBatch under the same protocol.
Questions for authors
Please clarify the selection fraction used in the ablation in Table 4b. In addition, on ResNet-50, RS2 w/o is substantially lower than RS2 w/r (75.3 vs. 78.4 at 70%; 72.7 vs. 75.1 at 30%), whereas the two are nearly identical on ResNet-18 (76.9 vs. 77.1). What explains this discrepancy?
7 comments
[SAE effectiveness]
The method must (1) outperform dynamic selection directly in CLIP's dense feature space, and (2) show that the SAE provides structure beyond a simple transformation of dense features (e.g., random projection to 16,384 dims, keep top-64) — otherwise the claim of capturing representativeness/diversity "at a different level" is not convincing.
The comment's premise — that the paper claims SAE features buy accuracy — is not what §3.2 says: the selector is defined with a swappable encoder and the SAE is presented as a sparse-unit probe. But swappable is not the same as behaviourally interchangeable, and that is directly testable.
With selector, schedule, penalty and per-class budget fixed, and the dense code matched to the SAE's 0.39% activation rate (3/768 positive CLIP dims vs 64/16,384 SAE units), we compared them at a tight budget where the selector rather than data volume is binding (CIFAR-10, 5%, no final refinement; ResNet-18, 200 ep):
| Encoder (matched 0.39% activation rate) | Test accuracy (%) |
|---|---|
| SAE dictionary | 82.54 ± 0.91 |
| Dynamic dense CLIP (top-3/768 dims) | 80.09 ± 0.92 |
| Paired difference (SAE − dense) | +2.45 (positive on 5/5 seeds) |
The reason is visible in the code itself: at matched sparsity the dense CLIP code has 193 active units and 12.8 distinct Rep values per class, with 87% of a class sitting on the modal value, so Rep-only and Div-only select almost the same set (overlap 0.648; SAE: 0.039) and the α schedule stops being a curriculum. This is a property of the construction, not of one draw: across five dense constructions (CLIP top-3, k=6, k=12, DINOv2 top-3, random projection) the best reaches 81.67, still 0.9 below both SAE dictionaries (82.54, 82.58); the five dense constructions span 1.58 points while the two SAE dictionaries differ by 0.04.
| Encoder family | Variants | Accuracy range (%) |
|---|---|---|
| Dense constructions | CLIP top-3, k=6, k=12, DINOv2 top-3, random projection | best 81.67; spread 1.58 |
| SAE dictionaries | CLIP dict, DINOv2 dict | 82.54 / 82.58; spread 0.04 |
What the SAE provides is then the measurement interface itself — a discrete, learned, interpretable factor vocabulary with well-defined frequency/coverage/rarity, a transferable probe (Table 9b), and stable behavior across 12 hyperparameter settings (Table 5). We have revised Appendix A.4's "prerequisite" wording to "interpretable default".
[Q1]
Please clarify the selection fraction used in the ablation in Table 4b.
Thank you for pointing out the inadequate description of our experimental setup. Our ablation experiments are conducted on CIFAR-100 with ResNet-50, under a fixed selection ratio of 70%. We have added the detailed configurations (CIFAR-100, ResNet-50, 200 epochs, fixed 70% per-epoch selection ratio) to the notes of Table 4(b) in the revised manuscript.
[Q2]
Why is RS2 w/o substantially lower than RS2 w/r on ResNet-50, whereas the two are nearly identical on ResNet-18?
Regarding RS2, we only reproduce it in this work to compare its strengths and weaknesses against our method. We do not include further analysis of the underlying mechanism of RS2 in the paper. Thank you for your understanding.
Dear Reviewer Mb8w,
Thanks for your time and effort in reviewing this paper. Your comments and suggestions are valuable to improve the quality of this paper. As the author-reviewer discussion period ends in about 3 days, we kindly request that you read the rebuttal and confirm whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.
Regards from the authors.
Rebuttal to ReviewerMb8w
[ImageNet results]
Thank you for the reminder. We have revised the caption of Table 4(a) in the revised manuscript: it now states that baselines' results are quoted from the original papers under their own fixed-epoch protocols and serve only as contextual references, and that comparisons with dynamic methods follow the SeTa protocol and are reported as relative performance. For experiments on ImageNet, considering reproducibility difficulty and computational cost, we mainly follow the experimental protocol from [1] and adopt its reported results for comparison under identical settings, with the primary goal of verifying the performance benefits of our method. For comparisons with dynamic methods, we refer to the experimental data from [2] and Table 2 in our paper. Due to differences in training protocols, we only report relative performance. The SeTa method [2] suffers a 0.5% performance drop at a pruning ratio of 45.5%, while our method achieves only a 0.3% performance drop at a 50% pruning ratio.
[Experiment settings]
We appreciate the reviewer's question regarding the robustness of our experimental results. Except for ImageNet-1k, which is excluded due to prohibitive computational cost, all other experiments are run with three random seeds, and the standard deviations are within 0.4. Research on training acceleration for datasets such as CIFAR is relatively mature. As a result, consistent small gains on CIFAR are considered meaningful. Recent state-of-the-art data selection work such as SeTa [1] achieves an average accuracy improvement of 0.4 over prior methods across multiple CIFAR settings, closely approaching full-data performance. Our method outperforms SeTa consistently under most experimental settings (please refer to our response to reviewer fKm3 W3). Beyond the final accuracy gains, our main contribution lies in introducing a plug-and-play, extremely lightweight penalty term together with its scheduling strategy. This mechanism enables static scoring to become dynamic and better leverages information from the full dataset.