- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a data selection method that uses sparse autoencoders (SAEs) to decompose pretrained DINOv3 features into sparse, more interpretable concept activations. It then applies submodular optimization with diminishing returns on accumulated concept support, which reallocates the training budget toward under-represented concepts rather than preserving the source distribution, aiming to improve downstream generalization and robustness. A local-support term additionally credits samples whose concepts recur among their neighbors, to avoid overemphasizing isolated rare samples.
Critical strengths
-
Strong performance on fine-grained and spurious-correlation benchmarks at low selection rates (e.g., +3.2 accuracy on NABirds at 10%, and +7.6/+7.5 WGA on Waterbirds at 10%/30% over the strongest baseline).
-
Comprehensive experiments and ablations (representation choice, local-support weight, neighborhood size, utility function, SAE pretraining data, downstream architecture). These make the contribution of individual components reasonably traceable, which increases my confidence that the gains do not hinge on a single design choice.
Critical weaknesses
I do not see any major weaknesses that would independently justify rejection.
However, the method implicitly assumes that SAE sparsification makes individual dimensions more semantically meaningful than the original DINOv3 features. The paper offers some support (Fig. 4 shows multi-modal per-dimension activation distributions; Table 16 shows that K-Center also benefits from SAE features), but this evidence is descriptive. A multi-modal activation distribution does not by itself establish that the dimensions correspond to more meaningful concepts, nor does it explain why the concept-coverage objective benefits from them. A more mechanistic or theoretical explanation would strengthen the paper.
Questions for authors
1. Fairness of feature extractors.
The paper does not state which feature extractor or scoring model each baseline uses, only that the original protocols are followed. Could the authors list this explicitly for every baseline?
I note that K-Center (whose Table 1 numbers match K-Center (DINO) in Table 16), RAM-APL, and SDM already rely on foundation-model features. However, training-dynamics-based baselines (Forgetting, EL2N, Moderate, CCS, D2-Pruning, InfoMax) derive their scores from early training of the downstream model, which is much weaker than DINOv3. For these comparisons, it is difficult to rule out that part of the improvement comes from the stronger representation rather than the selection objective. Could the authors report, for at least a few of these baselines (e.g., CCS and D2-Pruning), a variant whose scores are computed from DINOv3 features (e.g., via a linear probe on DINOv3)?
2. Within-class vs. global submodular optimization.
After inspecting the released code, I noticed that all labeled classification datasets use independent within-class submodular optimization, while CC12M uses global selection. This is not described in Section 3, which presents a single global objective.
My concern is that concept support is then not accumulated across classes. For example, if bird species A and B both frequently appear on branches, global selection would account for the accumulated support of this concept across both classes, whereas within-class selection treats them independently, so the support from species A does not affect selection within species B. This makes the optimized quantity class-conditional concept coverage, while the paper's framing and analyses (Fig. 1, Table 4) concern dataset-level concept support.
Could the authors: (a) explain the motivation for this design and state it explicitly in the paper; (b) clarify how per-class quotas are set, and whether the baselines use the same per-class allocation, since per-class allocation can itself act as class balancing on imbalanced datasets such as NABirds; (c) report results with global selection on at least one classification dataset (e.g., NABirds or Waterbirds)? This seems computationally feasible given that global selection on CC12M takes about 1.3 hours (Appendix A.3).
In particular, on Waterbirds, within-class selection may itself drive the increase in bias-conflicting examples (Table 5): once the water-background concept saturates within the waterbird class, the objective naturally turns to land-background waterbirds. It would be informative to know whether the robustness gains persist under global selection.
5 comments
Rebuttal part 2 of 3
Q1. Please specify the feature extractor or scoring model used by each baseline. Could part of the gain come from stronger representations rather than the selection objective? Please report DINOv3-based scoring variants for a few baselines, such as CCS and D², for example using linear probes.
Thank you for suggesting this comparison. We trained linear classifiers on frozen DINOv3 features and used their prediction errors (EL2N scores) to run CCS and D², with matched class quotas and downstream training settings. The original and stronger-feature results are listed together below; each cell gives the 10% / 30% / 50% results, averaged over three runs.
| Method and feature setting | NABirds accuracy (%) | Waterbirds WGA (%) |
|---|---|---|
| CCS, ResNet-50 scores | 39.7 / 65.5 / 73.6 | 52.8 / 67.1 / 69.9 |
| CCS, DINOv3 linear-probe EL2N scores | 35.4 / 64.2 / 73.7 | 48.8 / 75.1 / 79.1 |
| D², ResNet-50 scores and features | 25.8 / 66.6 / 74.7 | 47.0 / 56.4 / 60.8 |
| D², DINOv3 linear-probe EL2N scores and features | 40.6 / 65.8 / 73.3 | 62.0 / 69.9 / 72.6 |
| K-Center, DINOv3 patch average | 31.6 / 64.9 / 74.1 | 57.5 / 64.2 / 79.4 |
| K-Center, DINOv3 CLS | 26.3 / 63.2 / 74.6 | 56.1 / 73.7 / 75.4 |
| Ours | 42.9 / 67.4 / 75.2 | 68.6 / 77.9 / 79.3 |
The DINOv3-based variants show mixed gains: at 10% on NABirds, D² improves from 25.8% to 40.6%, while CCS changes from 39.7% to 35.4%. Ours reaches 42.9% and outperforms both variants at every budget on both datasets, even with their stronger features and additional classifier-training cost.
The table below lists the main-result configurations and additional baseline variants.
| Method | Scoring model / criterion | Features for distance or graph |
|---|---|---|
| Random | Uniform sampling | — |
| Forgetting | ResNet-50 forgetting events | — |
| EL2N | ResNet-50 probability-error norms | — |
| K-Center | Geometric coverage | DINOv3 patch average (main); DINOv3 CLS or SAE activations (additional) |
| Moderate | Class-median distance | ResNet-50 |
| CCS | ResNet-50 forgetting scores (main); DINOv3 linear-classifier EL2N scores (additional) | ResNet-50 (main) |
| D² | ResNet-50 forgetting scores (main); DINOv3 linear-classifier EL2N scores (additional) | ResNet-50 (main); DINOv3 (additional) |
| Supervised InfoMax | Class-normalized forgetting scores | L2-normalized ResNet-50 |
| RAM-APL | CLIP ViT-L/14 and DINOv2-base | CLIP / DINOv2 |
| SDM | SAE concept-distribution matching | Image-level SAE activations |
| Ours | Accumulated SAE concept support | DINOv3-B → SAE |
For multimodal pretraining, CLIPScore and its score-based baseline variants use CLIP ViT-B/16 image–text similarity.
In the revised manuscript, we have added the baseline configurations in Appendix A.2, Table 7, and the stronger-feature comparisons in Section 4.2, Table 3(b), and Appendix D.7, Table 23. Due to time constraints, these additional comparisons currently cover NABirds and Waterbirds at 10%, 30%, and 50% selection; we will add the complete results to the main tables in the future.
Rebuttal part 3 of 3
Q2. The code uses within-class selection for labeled datasets and global selection for CC12M, while Section 3 presents a global objective. Please explain and document this design and reconcile class-conditional coverage with dataset-level support analyses. How are per-class sample counts set, and do baselines match them to control for class balancing? Please report global-selection results on a classification dataset and assess whether the Waterbirds robustness gains persist under global selection. Could within-class selection itself increase bias-conflicting examples and contribute to those gains?
Thank you for asking about this design. We use within-class support accumulation because shared contexts need training support in different classes. For example, seeing a hand with species A does not give species B training examples in the same context. With global support accumulation, hand examples from A lower the reward for adding hand support in B, even if B has no such examples yet. Within-class accumulation preserves the reward for support across classes.
For labeled classification, neighbors and support accumulation are class-specific, with proportional quotas , where is the class size and is the selection rate. Classification baselines use the same class quotas; global controls match the total budget . Figure 1 and Table 5 report raw SAE support aggregated across all classes after selection, summarizing the dataset-level allocation produced by the class-conditional objective.
We compare within-class and global selection on NABirds and Waterbirds. Within-class selection selects separately within each class. Global selection with fixed class counts selects across classes while keeping the same per-class sample counts and neighbor graphs. Global selection uses a global graph without fixing class counts. Random (global) samples uniformly from the full pool. All settings use the same total budget and downstream training. Each entry below reports the 10% / 30% / 50% results.
| Selection mode | NABirds accuracy (%) | Waterbirds WGA (%) |
|---|---|---|
| Within-class selection | 42.9 / 67.4 / 75.2 | 68.6 / 77.9 / 79.3 |
| Global selection with fixed class counts | 32.7 / 63.4 / 72.5 | 48.6 / 53.3 / 65.7 |
| Global selection | 30.1 / 59.7 / 71.4 | 59.8 / 62.8 / 72.0 |
| Random (global) | 35.2 / 63.9 / 73.5 | 51.6 / 63.4 / 74.9 |
Within-class selection performs better at every budget when the class counts and graphs are fixed, isolating the contribution of separate support accumulation.
The hand concept occurs in 108 classes in the full NABirds training pool. The table below compares hand-context class coverage under within-class selection and global selection with fixed class counts.
| Selection budget | Within-class selection | Global selection with fixed class counts |
|---|---|---|
| 10% | 50 / 108 classes | 37 / 108 classes |
| 30% | 74 / 108 classes | 64 / 108 classes |
| 50% | 88 / 108 classes | 80 / 108 classes |
Within-class selection retains hand-containing images in more classes at every budget, even with the same number of samples for each class. On Waterbirds, Table 6 further shows that matching all four label–background group counts raises Random’s WGA from 52.3% to 63.7%, while Ours reaches 68.6%. Thus, a more balanced background composition explains part of the gain, but which images provide support within each group also matters.
For large-scale image–text pretraining on CC12M, we use global support accumulation to promote broad concept coverage across diverse image–text pairs. This matches the pretraining goal of learning visual–language associations across the full data pool. Global accumulation reduces the marginal reward for repeatedly supported patterns and directs the selection budget toward under-supported concepts across the pool.
In the revised manuscript, we have clarified the implementation in Section 3.2 and baseline allocations in Appendix A.2, and added the controlled comparisons and hand-context analysis in Appendix D.3, Tables 18–19.
Rebuttal part 1 of 3
Dear reviewer,
We sincerely thank the reviewer for the constructive feedback and for recognizing our strong low-budget results and comprehensive experiments and ablations. Our responses are listed below.
W. Multimodal activation histograms and K-Center gains do not establish that SAE latents are more semantically meaningful than DINOv3 features. Please clarify this assumption and provide a mechanistic or theoretical explanation of why the concept-coverage objective benefits from SAE activations.
Thank you for raising this point. We first clarify that our method does not assume SAE latents are more semantically meaningful than DINOv3 features. Monosemanticity is a goal of SAE feature decomposition; our method uses the resulting activation patterns to measure support for recurring visual properties, without requiring each latent to correspond to a single human-interpretable concept.
The main advantage of sparse representations lies in how they interact with our concept-coverage objective. In dense representations, individual dimensions can receive activations from many images with different visual characteristics, making their accumulated support less discriminative. In contrast, SAE sparsity allows each image to contribute to a limited subset of dimensions, providing more selective measurements of recurring and under-supported visual patterns. This enables our objective to distinguish concepts that have already received substantial training support from those that remain underrepresented.
More specifically, for a candidate image i of class y, the marginal gain is
Δy(i∣Sy)=∑c=1Clog!(1+1+My,c(Sy)x~i[c]),My,c(Sy)=∑j∈Syx~j[c].
Here, x~i includes local support. The same activation receives a larger reward when its latent has less accumulated support. Sparse activations make this allocation more selective across visual patterns, while diminishing returns dynamically adjust their importance throughout selection.
Figure 3 and Table 14 test this connection, with SAE local support fixed; using SAE latents for concept coverage raises NABirds accuracy at 10% selection budget from 35.4% to 42.9% compared with DINO features. Table 21 also shows that SAE features improve K-Center in five of six settings, while our objective gives further gains. These experiments establish the representation's usefulness for this selection objective. We use the multi-peaked activation histograms to describe the distribution of activations.
We have added a discussion of how SAE activations support concept allocation in Appendix D.5 of the revised manuscript.