- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes the CORE score for label-free data selection. A corruption–reconstruction network is trained on the unlabeled pool with masking and noise, and the CORE score of an image is its expected reconstruction loss over random corruptions. The authors argue this is a conditional code length (Proposition 1) that splits into feature bits and residual bits (Eq 7), and they keep the top-scored half of the pool. MAE trained on the top-scored half at half the gradient steps reaches 62.1% linear probe, against 62.3% for the full set and 56.7% for a random half on ImageNet-1k.. At equal steps it exceeds the full set by 5.9 points. The order top > random > bottom holds under fine-tuning for five pixel-prediction backbones and for DINO. A score-selected 1.28M subset of ImageNet-21k beats ImageNet-1k by 0.32 points, and a scorer trained on half of the pool ranks the other half with ρs = 0.89.
Critical strengths
-
I find the empirical evidence broad and mostly well controlled. The order top > random > bottom holds for five pixel-prediction learners and also for DINO, which does not reconstruct pixels, so the result is not only about matching the scorer's objective. The held-out scoring in Table 7 deals with most of my worry about memorisation. I also appreciate that the authors report seeds, a paired t-test, and the cost of the scorer with a break-even point.
-
The paper shows tests about the texture concern. The correlation with compressed size, the low-scored textures in Figure 2, and the mask-ratio ablation, where a 25% mask falls below random selection, support the claim that heavy masking discounts high-frequency texture.
Critical weaknesses
-
I find the novelty limited, and the closest prior work is missing. Proposition 1 is the standard link between likelihood and code length, Proposition 2 adds and subtracts log p, and Equation 7 is the chain rule, as the authors partly say themselves. The selection rule, training a model on the pool and ranking samples by its loss, is the same as perplexity-based data pruning for language models (Marion et al., 2023; Ankner et al., 2025), and perplexity filtering was already used in CCNet (Wenzek et al., 2020). Appendix I says that language-model selectors use other signals, which is not correct. Hard Patches Mining (Wang et al., 2023) also uses reconstruction loss as a measure of difficulty in masked image modeling.
-
I also find the equal-steps result in Table 2 hard to understand. A random half beats the full set by 2.6 linear-probe points with the same number of steps and the full-set MAE run reaches only 62.3%, which is well below the linear-probe numbers usually reported for MAE ViT-B. The full baseline may therefore be under-trained.
-
I am not sure the code-length observation holds, because the supplementary code does not compute the score defined in the paper. In saliency_guided_masking, the noise timestep is set from the image itself using the number of high-affinity tokens. The mask is also chosen by saliency at a fixed 60% instead of uniformly at 75%, and the noise is added to all token embeddings, including the mask tokens, while Eq 2 noises only the visible tokens. The CORE score is then the L1 loss against a noised copy of the image (noise_x). This means part of S(x) depends on how much noise each image was given, which does not fit Proposition 1, where \pie is the same for all images. I also found that the denoising term reuses loss_recon, so the visible tokens are not trained to denoise.
Minor: the code cannot run because models/vision_transformer.py and models/swin_transformer.py are missing.
Questions for authors
- How is CORE different from perplexity-based data pruning for language models (Marion et al., 2023; Ankner et al., 2025)? There, a small model is trained on the pool and samples are ranked by its loss. Apart from the image domain and the corruption, I do not see a difference in method. The paper should discuss this work, and also Hard Patches Mining (Wang et al., 2023), which uses reconstruction loss as difficulty in masked image modeling.
- Section 2.2 says that Section 4.3 tests whether the rankings from S and S_lat agree, but I could not find any S_lat results in the paper. Can you report the rank correlation between S and S_lat, and the DINO results when selecting with S_lat? Without this, the split into feature bits and residual bits in Equation 7 is not tested.
- Why does a random half beat the full set by 2.6 linear probe points at equal steps? Does your 800-epoch full-data MAE run match the published MAE numbers, and what do you get with full data at 1600 epochs?
- The fine-tuning gain over a random half is only 0.24 points on average. The linear-probe, ADE20K and equal-steps results are just for MAE. Can you report the linear probe and ADE20K for at least one other learner, and results for other keep ratios? Did you try keeping the middle of the ranking, as in perplexity-based pruning?
- The label-free baselines in Table 4 are weak. Laplacian variance keeps textures, and the SSL prototype metric was designed for supervised training. How does CORE compare with SemDeDup (Abbas et al., 2023)? Table 4 also says the pretrained MAE loss uses a model trained outside the pool, but the public MAE checkpoint was trained on ImageNet-1k. Which checkpoint did you use, and what do you get with a plain MAE trained on the pool and scored in the same way?
- For CORE-1k, how much do the selected classes and images overlap with ImageNet-1k? Was the ImageNet-1k validation set removed from the ImageNet-21k pool before scoring? The scorer for this setting also costs about 10 times more than on ImageNet-1k, so when does it pay off?
- The code does not match the paper in several places. In saliency_guided_masking, the noise timestep seems to depend on the image, and the CORE score is computed against a noised copy of the image instead of x. Please report the correlation between the CORE score and the timestep, and the selection results with a fixed π and the clean image as the target, as defined in Eqs 2 and 5. Also, how was the mask-ratio ablation in Table 10 run if the ratio is fixed in the code? The denoising term reuses loss_recon, so the visible tokens are not really trained to denoise. Minor: ρ is used for the Pearson correlation, the Spearman correlation and the keep ratio.
5 comments
Follow-up questions
I thank the authors for the response and the new experiments. The new baselines in Table 4 (SemDeDup, compressed size and an MAE trained on the pool), the full-data run at 1600 epochs, the keep ratios, the middle of the ranking, the SimMIM linear probe and the S_lat result answer most of my questions. The related work on perplexity pruning is now discussed. I have a few remaining points.
-
The response says a scorer trained with the new code “agrees with the paper’s ranking at ρs ≈ 0.9”, and that its top half reaches 61.3 LP against 62.1 in the paper. I understand this as follows: the rankings behind Tables 1 to 10 came from the original archive, with the saliency-guided mask, the image-dependent timestep and the noised target. Is that right? My Q7 asked for the correlation between the score and the timestep for that scorer, and the author response only covered the new code. If the results in the paper come from the original scorer, please clarify and state in the paper. Then either describe that scorer as the method, or report the main results (at least Tables 2 and 3) with the scorer of Eq. 2-5. Please also say which code was used for the mask-ratio ablation in Table 10. The ratio was fixed in the original code, and “the corruption decides the sign” is now one of your three main findings.
-
With the scorer of Eq. 2-5, the top half reaches 61.3 LP. An MAE trained on the pool and scored in the same way reaches 59.7. So with the score as defined, CORE is 1.6 LP above the simplest version of the idea, and 3.0 of the 5.4 LP over random already come from the plain MAE loss. I think the paper should state this clearly.
-
At 1.42 Cp, a random half (65.8 LP) is better than both full data (63.5) and CORE (62.1). At about 2 Cp, CORE (68.2) is compared with full data (65.8), but not with a random half trained to the same total budget, which is the stronger baseline. Do you have this value, or can you estimate it from the learning curves in App. D? If not, this is a limitation of the paper.
-
A random half beats full data by 2.6 LP at equal steps, and you say there is no verified explanation for this. Then 2.6 of the 5.9 points over the full set do not come from the selection. I would ask that the abstract leads with the gain over a random half at equal steps (3.3 LP), and not with 5.9.
-
S_lat is meant to estimate the feature bits that a latent learner pays for. For DINO, however, it is not better than S (73.3 against 74.11 k-NN, and 75.4 against 75.3 linear). So the experiments do not support contribution (ii), that the paper shows which part of the code length matters. Please revise this contribution and the text around Eq. 7.
-
For CORE-1k, class selection gives +0.42 and image selection +0.09 to +0.19, which is close to the seed-level spread in Table 6 (±0.18). Since the class selection uses labels, please make sure the abstract does not present the 0.32 gain as a label-free result.
I am inclined to raise my score if the above points are clarified.
Response to the follow-up questions
Thank you very much for reading our response and the revision so carefully, and for the constructive follow-up. We agree with points 2–6 and have revised the paper accordingly (in red); point 1 is a clarification.
1. Which scorer produced the rankings in the paper
No. The rankings behind Tables 1–10 were not produced with the saliency-guided mask, the image-dependent timestep, or the noised target. They come from a scorer that follows Eq. 2–5: a uniform mask, drawn independently of the image, and the clean image as the target. The components you found in the original archive belong to other methods in that codebase. App. B now states this.
- Why and 61.3 rather than 62.1. The check in App. C retrains a ViT-S scorer from scratch, whereas the default scorer of the paper is ViT-B. An agreement of 0.9 across architecture and seed is at the level of the 0.89 between two scorers trained on disjoint halves (Table 7), and the ViT-S scorer is also slightly below ViT-B in Table 10 (82.73 vs. 82.90 FT).
- Score and timestep. In this scorer does not depend on the image, so its correlation with the score is zero in expectation. We did not use the saliency-guided scorer for the paper and have no ranking from it to report.
- Table 10. The mask ratio is a configuration value of the same scorer (
corruption.mask_ratio); the 25 / 50 / 75% rows were obtained by changing it.
2. Share of the plain MAE loss
We agree, and Sec. 4.4 now states it: 3.0 of the 5.4 LP over a random half are already obtained by a plain MAE trained on the pool; the mask-plus-noise corruption adds 2.4 (62.1 vs. 59.7 with the default scorer). With the smaller ViT-S scorer of App. C the margin is 1.6, as you compute.
3. Random half at about
We do not have this run yet. Sec. 6 now lists it as a limitation, together with the fact that at a random half is above both full data and CORE. The run (random half, 3072 epochs, ) is now in progress. We will report the measured value in Table 12 of the final version, whichever way it comes out, and will post it here if it finishes during the discussion period.
An estimate, as you suggest: the random half gains 0.9 LP from 1600 to 2272 epochs (64.9 → 65.8). Extending this slope linearly to 3072 epochs () gives about 66.9, against 68.2 for CORE. This is an extrapolation and not a measurement; the fine-tuning curves in App. D flatten over this range.
4. Abstract
Revised as you ask. The abstract, Sec. 1, and Sec. 7 now lead with the gain over a random half at equal steps (+3.3 LP, +0.8 mIoU) and give the gain over the full set (5.9, 1.1) in parentheses, together with the part that a random half already obtains (2.6, 0.3).
5. Contribution (ii) and Eq. 7
We agree that the experiments do not show which part of the code length matters. Contribution (ii) now reads "we show when this rule works on images: the corruption decides it". Sec. 2.2 presents Eq. 7 as an interpretation and not as a tested claim, and Sec. 4.3 states that does not improve on for DINO.
6. CORE-1k
The abstract now says "selected from ImageNet-21k with class membership (class-aware curation, not label-free)" and that the gain comes mostly from the choice of classes. App. J adds that the image-selection effect (+0.09 to +0.19) is close to the seed-level spread of Table 6 (±0.18).
Thank you again for your time. Your comments have made the claims of the paper more precise, and we are glad to clarify anything further.
To Reviewer s7rS
Thank you for the expert review, the pointers to perplexity pruning, and the code check. Your weaknesses recur in your questions, so we answer them there: W1 → Q1; W2 → Q3; W3 and minor → Q7.
A (W1). These works are now in Sec. 5, and App. I is corrected. No prior work shows whether, or under which corruption, a pool-trained reconstruction loss selects images for self-supervised pretraining. The image setting behaves differently from text:
A. Thank you for catching this: ρs=0.78; DINO on the Slat half reaches 73.3 k-NN / 75.4 linear (S: 74.11 / 75.3; random: 72.29 / 73.3). Slat does not improve on S here, and the paper now presents it as an alternative (Sec. 4.3).
A (W2).
A. All added (App. J). SimMIM: +4.5 LP and +0.50 mIoU for top over random.
Keep 25 / 50 / 75% (MAE LP): top 55.5 / 62.1 / 62.6; random 49.7 / 56.7 / 60.2; middle 52.5 / 58.9 / –.
A. Both are added (GR-3). SemDeDup: 82.65 FT / 57.7 LP. Plain MAE on the pool: 82.78 / 59.7. CORE: 82.90 / 62.1. We used the public MAE ViT-B checkpoint; "Ext." meant pretrained outside our pipeline, and the column is clarified.
A. 337 of 1,000 classes and 38.9% of images overlap; validation images were excluded before scoring (Sec. 4.1). Class selection adds +0.42, image selection +0.09 to +0.19 (App. J). By Eq. 10 the break-even on that pool is n⋆=2 (App. D).
A (W3, minor). The original archive was packaged from a broader codebase that supports several methods and carries components beyond the method of this paper. We have replaced it. (GR-1).
corruption.mask_ratio). Denoising term: one loss against the clean image.