acceptodds
Under review as a conference paper at ICLR 2027

Select What Resists Compression: Reconstruction Code Length for Label-Free Data Selection

Abstract

Selecting training data typically requires labels, captions, or an externally pretrained encoder to assess data quality. A pool of unlabeled images offers none of these, and what makes an image useful for self-supervised pretraining is itself unsettled. We show that the per-image loss of a corruption–reconstruction network trained on the pool is a conditional code length: the number of nats needed to encode the image given a masked and noised view of it. We rank the pool by this quantity, which we call the CORE score. In other words, we keep the images that a model of the pool compresses worst. On ImageNet-1k, MAE pretrained on the top-scored 50% at half the gradient steps reaches 62.1% under linear probing against 62.3% for the full set, whereas a random 50% reaches 56.7%; at equal steps the top-scored half exceeds the full set by 5.9 linear-probe points and 1.1 ADE20K mIoU, and a random half by 3.3 and 0.8. The ordering top >random > bottom holds under fine-tuning for five pixel-prediction backbones and for DINO, which never reconstructs pixels, and a scorer trained on half of the pool ranks the other half almost identically (ρs = 0.89). A 1.28M-image subset selected from ImageNet-21k exceeds the human-curated ImageNet-1k by 0.32 points on average. The score correlates with lossless compressed size (ρ = 0.61), yet the lowest-scored images include high-frequency repetitive textures: heavy masking discounts the bits that a generic compressor counts. We report the one-time cost of the scorer together with the number of pretraining runs after which it pays off. Code is provided in the supplementary material.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, data selection, data pruning, masked image modeling, minimum description length, code length, denoising, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_s7rSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the CORE score for label-free data selection. A corruption–reconstruction network is trained on the unlabeled pool with masking and noise, and the CORE score of an image is its expected reconstruction loss over random corruptions. The authors argue this is a conditional code length (Proposition 1) that splits into feature bits and residual bits (Eq 7), and they keep the top-scored half of the pool. MAE trained on the top-scored half at half the gradient steps reaches 62.1% linear probe, against 62.3% for the full set and 56.7% for a random half on ImageNet-1k.. At equal steps it exceeds the full set by 5.9 points. The order top > random > bottom holds under fine-tuning for five pixel-prediction backbones and for DINO. A score-selected 1.28M subset of ImageNet-21k beats ImageNet-1k by 0.32 points, and a scorer trained on half of the pool ranks the other half with ρs = 0.89.

Critical strengths
  1. I find the empirical evidence broad and mostly well controlled. The order top > random > bottom holds for five pixel-prediction learners and also for DINO, which does not reconstruct pixels, so the result is not only about matching the scorer's objective. The held-out scoring in Table 7 deals with most of my worry about memorisation. I also appreciate that the authors report seeds, a paired t-test, and the cost of the scorer with a break-even point.

  2. The paper shows tests about the texture concern. The correlation with compressed size, the low-scored textures in Figure 2, and the mask-ratio ablation, where a 25% mask falls below random selection, support the claim that heavy masking discounts high-frequency texture.

Critical weaknesses
  1. I find the novelty limited, and the closest prior work is missing. Proposition 1 is the standard link between likelihood and code length, Proposition 2 adds and subtracts log p, and Equation 7 is the chain rule, as the authors partly say themselves. The selection rule, training a model on the pool and ranking samples by its loss, is the same as perplexity-based data pruning for language models (Marion et al., 2023; Ankner et al., 2025), and perplexity filtering was already used in CCNet (Wenzek et al., 2020). Appendix I says that language-model selectors use other signals, which is not correct. Hard Patches Mining (Wang et al., 2023) also uses reconstruction loss as a measure of difficulty in masked image modeling.

  2. I also find the equal-steps result in Table 2 hard to understand. A random half beats the full set by 2.6 linear-probe points with the same number of steps and the full-set MAE run reaches only 62.3%, which is well below the linear-probe numbers usually reported for MAE ViT-B. The full baseline may therefore be under-trained.

  3. I am not sure the code-length observation holds, because the supplementary code does not compute the score defined in the paper. In saliency_guided_masking, the noise timestep is set from the image itself using the number of high-affinity tokens. The mask is also chosen by saliency at a fixed 60% instead of uniformly at 75%, and the noise is added to all token embeddings, including the mask tokens, while Eq 2 noises only the visible tokens. The CORE score is then the L1 loss against a noised copy of the image (noise_x). This means part of S(x) depends on how much noise each image was given, which does not fit Proposition 1, where \pie is the same for all images. I also found that the denoising term reuses loss_recon, so the visible tokens are not trained to denoise.

Minor: the code cannot run because models/vision_transformer.py and models/swin_transformer.py are missing.

Questions for authors
  1. How is CORE different from perplexity-based data pruning for language models (Marion et al., 2023; Ankner et al., 2025)? There, a small model is trained on the pool and samples are ranked by its loss. Apart from the image domain and the corruption, I do not see a difference in method. The paper should discuss this work, and also Hard Patches Mining (Wang et al., 2023), which uses reconstruction loss as difficulty in masked image modeling.
  2. Section 2.2 says that Section 4.3 tests whether the rankings from S and S_lat agree, but I could not find any S_lat results in the paper. Can you report the rank correlation between S and S_lat, and the DINO results when selecting with S_lat? Without this, the split into feature bits and residual bits in Equation 7 is not tested.
  3. Why does a random half beat the full set by 2.6 linear probe points at equal steps? Does your 800-epoch full-data MAE run match the published MAE numbers, and what do you get with full data at 1600 epochs?
  4. The fine-tuning gain over a random half is only 0.24 points on average. The linear-probe, ADE20K and equal-steps results are just for MAE. Can you report the linear probe and ADE20K for at least one other learner, and results for other keep ratios? Did you try keeping the middle of the ranking, as in perplexity-based pruning?
  5. The label-free baselines in Table 4 are weak. Laplacian variance keeps textures, and the SSL prototype metric was designed for supervised training. How does CORE compare with SemDeDup (Abbas et al., 2023)? Table 4 also says the pretrained MAE loss uses a model trained outside the pool, but the public MAE checkpoint was trained on ImageNet-1k. Which checkpoint did you use, and what do you get with a plain MAE trained on the pool and scored in the same way?
  6. For CORE-1k, how much do the selected classes and images overlap with ImageNet-1k? Was the ImageNet-1k validation set removed from the ImageNet-21k pool before scoring? The scorer for this setting also costs about 10 times more than on ImageNet-1k, so when does it pay off?
  7. The code does not match the paper in several places. In saliency_guided_masking, the noise timestep seems to depend on the image, and the CORE score is computed against a noised copy of the image instead of x. Please report the correlation between the CORE score and the timestep, and the selection results with a fixed π and the clean image as the target, as defined in Eqs 2 and 5. Also, how was the mask-ratio ablation in Table 10 run if the ratio is fixed in the code? The denoising term reuses loss_recon, so the visible tokens are not really trained to denoise. Minor: ρ is used for the Pearson correlation, the Spearman correlation and the keep ratio.
5 comments
AuthorsOpenReview ↗

To Reviewer s7rS

Thank you for the expert review, the pointers to perplexity pruning, and the code check. Your weaknesses recur in your questions, so we answer them there: W1 → Q1; W2 → Q3; W3 and minor → Q7.

Q1. How is CORE different from perplexity-based data pruning for language models (Marion et al., 2023; Ankner et al., 2025)?

A (W1). These works are now in Sec. 5, and App. I is corrected. No prior work shows whether, or under which corruption, a pool-trained reconstruction loss selects images for self-supervised pretraining. The image setting behaves differently from text:

  • Which part to keep. Marion et al. find the middle best; Ankner et al. that it depends on the domain. On images the top beats the middle: 62.1 vs. 58.9 (50%); 55.5 vs. 52.5 (25%).
  • The corruption decides the sign. A 25% mask is below random (Table 10); perplexity has no such control.
  • Transfer. The ranking also orders the subsets for DINO (Table 3). HPM uses the loss to choose patches to mask within an image; we rank images.

Q2. Can you report the rank correlation between S and S_lat, and the DINO results when selecting with S_lat?

A. Thank you for catching this: ρs=0.78\rho_s = 0.78; DINO on the SlatS_{\text{lat}} half reaches 73.3 k-NN / 75.4 linear (SS: 74.11 / 75.3; random: 72.29 / 73.3). SlatS_{\text{lat}} does not improve on SS here, and the paper now presents it as an alternative (Sec. 4.3).

Q3. Why does a random half beat the full set by 2.6 linear probe points at equal steps? Does your 800-epoch full-data MAE run match the published MAE numbers, and what do you get with full data at 1600 epochs?

A (W2).

  • Full data, 1600 epochs: 65.8 LP / 83.36 FT / 47.90 mIoU, below CORE top 50% (68.2 / 83.57 / 48.39); now in Table 2 (total budget: GR-2).
  • Published numbers: the official checkpoint gives 67.6 under our probe (published 68.0); our runs are about two points lower at both schedules (62.3 vs. 64.4; 65.8 vs. 67.8) because we use batch size 1024 and unnormalized pixel targets in all rows (App. B).
  • Why: prior work predicts parity at equal iterations; FT and ADE20K are near parity. We have no verified explanation for the larger LP gap (Sec. 4.2).

Q4. linear probe and ADE20K for at least one other learner, and results for other keep ratios? … keeping the middle of the ranking

A. All added (App. J). SimMIM: +4.5 LP and +0.50 mIoU for top over random.

SimMIMFullCORE topRandomCORE bottom
LP / ADE20K56.8 / 47.2056.7 / 47.0952.2 / 46.5947.7 / 45.94

Keep 25 / 50 / 75% (MAE LP): top 55.5 / 62.1 / 62.6; random 49.7 / 56.7 / 60.2; middle 52.5 / 58.9 / –.

Q5. How does CORE compare with SemDeDup … Which checkpoint did you use, and what do you get with a plain MAE trained on the pool and scored in the same way?

A. Both are added (GR-3). SemDeDup: 82.65 FT / 57.7 LP. Plain MAE on the pool: 82.78 / 59.7. CORE: 82.90 / 62.1. We used the public MAE ViT-B checkpoint; "Ext." meant pretrained outside our pipeline, and the column is clarified.

Q6. how much do the selected classes and images overlap with ImageNet-1k? Was the ImageNet-1k validation set removed … when does it pay off?

A. 337 of 1,000 classes and 38.9% of images overlap; validation images were excluded before scoring (Sec. 4.1). Class selection adds +0.42, image selection +0.09 to +0.19 (App. J). By Eq. 10 the break-even on that pool is n⋆=2n^\star = 2 (App. D).

Q7. The code does not match the paper in several places.

A (W3, minor). The original archive was packaged from a broader codebase that supports several methods and carries components beyond the method of this paper. We have replaced it. (GR-1).

  • Fixed π\pi, clean target: in our runs a scorer trained with it agrees with the paper's ranking at ρs≈0.9\rho_s \approx 0.9; its top half reaches 61.3 LP (random: 56.7; App. C).
  • Timestep: tt is drawn independently of the image, so no per-image noise level exists to correlate with.
  • Mask ratio: a config option (corruption.mask_ratio). Denoising term: one loss against the clean image.
  • Notation: ρ\rho Pearson, ρs\rho_s Spearman, keep ratio κ\kappa.
Reviewer_s7rSOpenReview ↗

Follow-up questions

I thank the authors for the response and the new experiments. The new baselines in Table 4 (SemDeDup, compressed size and an MAE trained on the pool), the full-data run at 1600 epochs, the keep ratios, the middle of the ranking, the SimMIM linear probe and the S_lat result answer most of my questions. The related work on perplexity pruning is now discussed. I have a few remaining points.

  1. The response says a scorer trained with the new code “agrees with the paper’s ranking at ρs ≈ 0.9”, and that its top half reaches 61.3 LP against 62.1 in the paper. I understand this as follows: the rankings behind Tables 1 to 10 came from the original archive, with the saliency-guided mask, the image-dependent timestep and the noised target. Is that right? My Q7 asked for the correlation between the score and the timestep for that scorer, and the author response only covered the new code. If the results in the paper come from the original scorer, please clarify and state in the paper. Then either describe that scorer as the method, or report the main results (at least Tables 2 and 3) with the scorer of Eq. 2-5. Please also say which code was used for the mask-ratio ablation in Table 10. The ratio was fixed in the original code, and “the corruption decides the sign” is now one of your three main findings.

  2. With the scorer of Eq. 2-5, the top half reaches 61.3 LP. An MAE trained on the pool and scored in the same way reaches 59.7. So with the score as defined, CORE is 1.6 LP above the simplest version of the idea, and 3.0 of the 5.4 LP over random already come from the plain MAE loss. I think the paper should state this clearly.

  3. At 1.42 Cp, a random half (65.8 LP) is better than both full data (63.5) and CORE (62.1). At about 2 Cp, CORE (68.2) is compared with full data (65.8), but not with a random half trained to the same total budget, which is the stronger baseline. Do you have this value, or can you estimate it from the learning curves in App. D? If not, this is a limitation of the paper.

  4. A random half beats full data by 2.6 LP at equal steps, and you say there is no verified explanation for this. Then 2.6 of the 5.9 points over the full set do not come from the selection. I would ask that the abstract leads with the gain over a random half at equal steps (3.3 LP), and not with 5.9.

  5. S_lat is meant to estimate the feature bits that a latent learner pays for. For DINO, however, it is not better than S (73.3 against 74.11 k-NN, and 75.4 against 75.3 linear). So the experiments do not support contribution (ii), that the paper shows which part of the code length matters. Please revise this contribution and the text around Eq. 7.

  6. For CORE-1k, class selection gives +0.42 and image selection +0.09 to +0.19, which is close to the seed-level spread in Table 6 (±0.18). Since the class selection uses labels, please make sure the abstract does not present the 0.32 gain as a label-free result.

I am inclined to raise my score if the above points are clarified.

AuthorsOpenReview ↗

Response to the follow-up questions

Thank you very much for reading our response and the revision so carefully, and for the constructive follow-up. We agree with points 2–6 and have revised the paper accordingly (in red); point 1 is a clarification.

1. Which scorer produced the rankings in the paper

No. The rankings behind Tables 1–10 were not produced with the saliency-guided mask, the image-dependent timestep, or the noised target. They come from a scorer that follows Eq. 2–5: a uniform mask, tt drawn independently of the image, and the clean image as the target. The components you found in the original archive belong to other methods in that codebase. App. B now states this.

  • Why ρs≈0.9\rho_s \approx 0.9 and 61.3 rather than 62.1. The check in App. C retrains a ViT-S scorer from scratch, whereas the default scorer of the paper is ViT-B. An agreement of 0.9 across architecture and seed is at the level of the 0.89 between two scorers trained on disjoint halves (Table 7), and the ViT-S scorer is also slightly below ViT-B in Table 10 (82.73 vs. 82.90 FT).
  • Score and timestep. In this scorer tt does not depend on the image, so its correlation with the score is zero in expectation. We did not use the saliency-guided scorer for the paper and have no ranking from it to report.
  • Table 10. The mask ratio is a configuration value of the same scorer (corruption.mask_ratio); the 25 / 50 / 75% rows were obtained by changing it.

2. Share of the plain MAE loss

We agree, and Sec. 4.4 now states it: 3.0 of the 5.4 LP over a random half are already obtained by a plain MAE trained on the pool; the mask-plus-noise corruption adds 2.4 (62.1 vs. 59.7 with the default scorer). With the smaller ViT-S scorer of App. C the margin is 1.6, as you compute.

3. Random half at about 2 Cp2\,C_p

We do not have this run yet. Sec. 6 now lists it as a limitation, together with the fact that at 1.42 Cp1.42\,C_p a random half is above both full data and CORE. The run (random half, 3072 epochs, 1.92 Cp1.92\,C_p) is now in progress. We will report the measured value in Table 12 of the final version, whichever way it comes out, and will post it here if it finishes during the discussion period.

An estimate, as you suggest: the random half gains 0.9 LP from 1600 to 2272 epochs (64.9 → 65.8). Extending this slope linearly to 3072 epochs (1.92 Cp1.92\,C_p) gives about 66.9, against 68.2 for CORE. This is an extrapolation and not a measurement; the fine-tuning curves in App. D flatten over this range.

4. Abstract

Revised as you ask. The abstract, Sec. 1, and Sec. 7 now lead with the gain over a random half at equal steps (+3.3 LP, +0.8 mIoU) and give the gain over the full set (5.9, 1.1) in parentheses, together with the part that a random half already obtains (2.6, 0.3).

5. Contribution (ii) and Eq. 7

We agree that the experiments do not show which part of the code length matters. Contribution (ii) now reads "we show when this rule works on images: the corruption decides it". Sec. 2.2 presents Eq. 7 as an interpretation and not as a tested claim, and Sec. 4.3 states that SlatS_{\text{lat}} does not improve on SS for DINO.

6. CORE-1k

The abstract now says "selected from ImageNet-21k with class membership (class-aware curation, not label-free)" and that the gain comes mostly from the choice of classes. App. J adds that the image-selection effect (+0.09 to +0.19) is close to the seed-level spread of Table 6 (±0.18).

Thank you again for your time. Your comments have made the claims of the paper more precise, and we are glad to clarify anything further.

Reviewer_Y3H2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies image data selection for self-supervised pretraining without labels, captions, or externally pretrained encoders. The proposed method, CORE, trains a corruption–reconstruction network on the candidate pool and ranks images by their expected reconstruction loss under masking and noise. The authors interpret this score as a conditional code length and retain images that the pool’s own model compresses least effectively. Experiments show that the selected 50% of ImageNet-1k approaches full-data performance with half the pretraining steps and outperforms random selection under matched training schedules. The paper evaluates several pixel-prediction learners and DINO, alongside transfer experiments, scorer ablations, held-out scoring, and computational cost analysis.

Critical strengths

The problem is relevant, and the proposed approach is simple and practical.

Selecting useful pretraining data without external semantic supervision is a meaningful problem. CORE provides a straightforward pipeline, and the resulting ranking can be reused across different learners.

Frozen-feature and transfer evaluations provide convincing evidence of selection quality.

The strongest results go beyond the relatively small differences after full supervised fine-tuning. Under equal pretraining steps, CORE outperforms a random 50% subset by 3.3 percentage points in linear probing and approximately 0.8 mIoU on ADE20K. With half the steps, it also approaches the full-data linear-probe performance. These evaluations more directly demonstrate the value of the selected data for representation learning.

The paper includes useful controls for alternative explanations.

The DINO results suggest that the benefit is not restricted to learners sharing the scorer’s pixel-reconstruction objective. The held-out scoring experiment directly addresses potential memorization effects, showing strong agreement with in-sample rankings and similar downstream performance.

Critical weaknesses

The code-length interpretation does not sufficiently establish either methodological novelty or a connection to downstream utility.

Proposition 1 primarily applies the standard correspondence between reconstruction loss, negative log-likelihood, and code length. This clarifies what the score measures, but does not explain why a larger conditional code length should imply greater downstream training value. Operationally, the method remains high-loss selection using a reconstruction proxy.

The paper should more clearly distinguish its conceptual interpretation from its algorithmic contribution and empirical findings. As presented, the theoretical framing is stronger than the justification it provides for the selection rule.

The baselines do not fully isolate the contribution of the proposed scorer.

Several baselines in Table 4 underperform random selection, and the paper acknowledges that some were designed for different settings. The mask-only and noise-only ablations are useful, but comparisons should explicitly match architecture, training budget, reconstruction target, and scoring procedure.

A particularly relevant missing baseline is selection by lossless compressed size. Given its reported correlation with CORE, evaluating this inexpensive alternative would help establish how much the learned scorer contributes beyond basic image complexity.

The efficiency claims require comparisons under a complete computational budget.

The default scorer costs approximately 0.92Cp0.92C_p, where CpC_p is the cost of full-data MAE pretraining. Consequently, the first half-step CORE run costs approximately 1.42Cp1.42C_p, while the equal-step CORE experiment costs approximately 1.92Cp1.92C_p.

Although these costs are disclosed, phrases such as “half the compute” and “equal compute” can be interpreted as describing the entire pipeline. A critical missing comparison is against full-data or random-subset training given the same total budget, including scorer training. Reuse can amortize the overhead, but the break-even point also depends on the target learner’s cost.

The operating range and robustness of high-loss selection remain insufficiently characterized. The main experiments use only a 50% retention ratio. It is therefore unclear whether the ranking remains effective under more aggressive pruning, where coverage and diversity may become more important.

High reconstruction loss may also identify corrupted images, sensor noise, or irrelevant outliers. The qualitative texture examples do not rule out these failure modes. Experiments across retention ratios and controlled contamination levels would clarify when selecting the highest-scoring images is beneficial.

The ImageNet-21k experiment contains confounding factors. CORE-1k changes both the selected classes and the images within those classes. Its improvement over ImageNet-1k therefore cannot be attributed specifically to image-level selection; class composition and downstream-task relevance may also contribute.

Moreover, this experiment uses class information and should be clearly distinguished from the paper’s strictly label-free setting. Controls with fixed class sets and randomly selected classes would make the result easier to interpret.

Questions for authors

Please refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

To Reviewer Y3H2

Thank you very much for the thoughtful, balanced review and the positive assessment. Every suggestion led to a concrete addition, and the paper is clearer and better scoped as a result.

W1. The paper should more clearly distinguish its conceptual interpretation from its algorithmic contribution and empirical findings.

A. We followed this in Sec. 1 and Sec. 2.2.

  • Interpretation: the score is a conditional code length (Prop. 1, standard).
  • Algorithm: selection by the loss of a mask-plus-noise reconstruction network trained on the pool.
  • Findings: no prior work shows whether, or under which corruption, such a loss selects images for self-supervised pretraining. The corruption decides whether the rule helps or hurts (a 25% mask falls below random), the top of the ranking beats the middle, and the ranking carries over to DINO.

W2. A particularly relevant missing baseline is selection by lossless compressed size.

A. Added to Table 4 with matched baselines (LP in App. J).

50% budget, MAERandomSemDeDupCompressed sizeMAE on the poolCORE
FT / LP82.61 / 56.782.65 / 57.782.68 / 58.282.78 / 59.782.90 / 62.1

Compressed size explains 1.5 of the 5.4 LP over random. The MAE trained on the pool is matched in pool, scoring procedure, and target; CORE is 2.4 LP above it.

W3. A critical missing comparison is against full-data or random-subset training given the same total budget, including scorer training.

A. Your accounting is correct. The abstract, Sec. 1, and Sec. 7 now say "pretraining steps" and state end-to-end costs. Same-total-budget comparison (LP, Table 12):

  • 1.42 Cp1.42\,C_p: CORE top (800 ep) 62.1; full (1136 ep) 63.5; random half (2272 ep) 65.8.
  • About 2 Cp2\,C_p: CORE top (1600 ep) 68.2 at 1.92; full (1600 ep) 65.8 at 2.00.

A single short CORE run does not pay off at 1.42 Cp1.42\,C_p; at about 2 Cp2\,C_p it exceeds full data by 2.4 LP. The break-even depends on the learner, as you note: the scorer costs about half of one SimMIM run (App. D).

W4. Experiments across retention ratios and controlled contamination levels

A. Both are added (App. J). The gap over random is +5.8 / +5.4 / +2.4 LP at 25 / 50 / 75%.

KeepCORE topMiddleRandom
25%55.552.549.7
50%62.158.956.7
75%62.6–60.2

Contamination (5% of the pool; kept in top 50% / CORE LP / random LP): Gaussian noise 96% / 60.4 / 56.1; JPEG or sensor noise 63% / 62.1 / 57.4; out-of-distribution 58% / 62.2 / 55.8. The top half stays above random; pure noise has a long code and is selected, as you anticipated, which Sec. 6 now states. We tested one contamination level.

W5. Controls with fixed class sets and randomly selected classes would make the result easier to interpret.

A. We ran the controls you propose (App. J).

Classes / imagesFT (mean of five learners)
Random / random82.69
ImageNet-1k / ImageNet-1k82.88
ImageNet-1k / CORE83.07
CORE / random83.11
CORE / CORE83.20

Class selection contributes +0.42 and image selection +0.09 to +0.19. The experiment is now labelled "class-aware curation" and kept apart from the label-free results.

Thank you again for the constructive suggestions and for your support of the paper.

I thank the authors for their detailed response. I believe their clarifications adequately address the concerns raised in my review. I will therefore retain my original score and continue to recommend acceptance.

Reviewer_6oQiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes CORE, a label-free score for selecting images for self-supervised pretraining. A single corruption–reconstruction network (masking plus diffusion-style noise) is trained on the unlabeled pool, each image is scored by its expected reconstruction loss over random corruptions, and the highest-scoring images are kept. The loss is interpreted as a conditional code length and decomposed into surprisal and excess terms. On ImageNet-1k at a 50% keep ratio, the top half outperforms a random half for five pixel-prediction learners under fine-tuning, and the same ordering holds for MAE linear probing, ADE20K transfer and DINO. Further experiments cover an ImageNet-21k subset, label-free baselines, held-out scoring, cost, cross-dataset transfer and a preliminary DataComp result.

Critical strengths
  1. A consistent selection signal with sensible controls.
    • Top > random > bottom holds for all five pixel-prediction backbones, and a five-seed MAE run gives direct evidence on variability (82.85 ± 0.02 vs 82.53 ± 0.18, Table 6).
    • The ordering also holds for DINO, which does not reconstruct pixels (+1.82 k-NN over random), and the ImageNet-trained scorer transfers to ADE20K, COCO, CUB and NABirds.
    • Class-balance and held-out-scoring controls (ρs=0.89\rho_s = 0.89) argue against imbalance and memorization as explanations.
    • Cheaper scorers pay off from the first run, with smaller gains. If the result holds, a pool-only score is practically valuable for unlabeled domains.
Critical weaknesses
  1. The large equal-step gains lack a total-budget baseline and are not well explained.

    • Under standard fine-tuning the effect is small: top minus random is +0.24 on average, and at fixed epochs the top half stays slightly below full data (82.83 vs 82.88).
    • The abstract's "+5.9 linear-probe / +1.1 mIoU over the full set" comes from the equal-step rows. In those rows a random half trained for 1600 epochs also beats the full pool at 800 epochs by 2.6 linear-probe points (64.9 vs 62.3). The cited data-size insensitivity of MIM does not predict this. The result may be genuine, but it needs variance, learning curves and a check of the full baseline against published MAE results.
    • The abstract's equal-step claim counts downstream steps only. A single end-to-end run of "CORE top, 1600 ep" costs about 1.92 CpC_p against 1 CpC_p for full data, and full ImageNet-1k at 1600 epochs is not reported; with the supplied 800-epoch scorer config this becomes about 2.84 CpC_p. The amortized comparison stands as reported.
    • Table 10 has no variance, and the default wins some rows by 0.04–0.05 points on ImageNet-1k validation accuracy; how the defaults were chosen should be disclosed.
  2. The theory does not explain why long codes are useful, and the supplied code does not compute the defined score.

    • Prop. 1 holds for any LpL_p loss and β>0\beta > 0, and Eq. 7 is the chain rule; neither explains why a high code length indicates utility. The two "testable predictions" do not follow from Prop. 2, which concerns average conditional entropy rather than per-image utility.
    • The excess-code-length term of Eq. 6 is never separated out, and Eq. 4 should also condition on tt unless tt can be recovered from x~\tilde x.
    • Unexplained variance beyond compressed size need not be useful, and Table 4 lacks a compressed-size baseline. High loss can also mark noise (Mindermann et al., 2022), but no experiment injects corrupted or out-of-distribution images.
    • The supplied implementation differs from the paper in verifiable ways:
      • (a) Mask: saliency_guided_masking in models/simmim.py (line 123) uses mask_ratio=0.6, versus 75% in App. B.
      • (b) Corruption: timesteps depend on a saliency count (lines 133–137), which the paper does not describe, and noise covers the whole token sequence rather than only visible tokens.
      • (c) Target: training (line 301) and score_pool.py (line 51) reconstruct noise_x, a freshly noised copy of the image, rather than the clean xx. The resulting noise floor means the score may partly reflect the saliency-to-timestep mapping.
      • (d) Loss region: the loss covers masked pixels only, whereas Eq. 3 defines it on the whole image.
      • (e) Loss bug: in models/simmim.py lines 301–306, the clean-target loss_denoise = F.l1_loss(x, x_rec, ...) is immediately overwritten by (loss_recon * visible).sum() / ..., and the final loss is loss_recon + loss_denoise * 0.05 (line 314). The only clean-image term never enters training.
      • (f) The config trains for 800 epochs rather than 400, and the imported ViT/Swin backbone modules are missing.
    • These differences do not by themselves refute the framing, but the supplied code does not establish the clean-image score of Eq. 3–5.
Questions for authors

see the weakness part.

2 comments
AuthorsOpenReview ↗

To Reviewer 6oQi

Thank you for the exceptionally careful review, including the code reading and the cost accounting.

W1. Equal-step gains and total budget

Under standard fine-tuning the effect is small

A. We agree. Fine-tuning narrows the gap between encoders; evaluations that keep the encoder frozen separate them: +5.4 LP / +1.35 mIoU for MAE (Table 2) and +4.5 LP / +0.50 mIoU for SimMIM (new, App. J).

it needs variance, learning curves and a check of the full baseline against published MAE results.

  • A1. Explanation. You are right: insensitivity predicts parity, not a gain, and we removed that explanation. FT and ADE20K are near parity; we have no verified explanation for the larger LP gap, and Sec. 4.2 now says so.
  • A2. Variance and learning curves. Over three runs LP is 62.30 ± 0.02 (full, 800 ep), 64.90 ± 0.12 (random, 1600 ep), and 68.20 ± 0.06 (top, 1600 ep) (App. D). Learning curves are in App. D.
  • A3. Published baseline. The official checkpoint gives 67.6 under our probe (published 68.0). Our runs are about two points below reported MAE numbers at both schedules (62.3 vs. 64.4; 65.8 vs. 67.8) because we use an effective batch size of 1024 (official 4096) and unnormalized pixel targets, shared by all rows (App. B).

full ImageNet-1k at 1600 epochs is not reported

A. Your cost of 1.92 for CORE at 1600 epochs is correct. Full data at 1600 epochs reaches 65.8 LP / 83.36 FT / 47.90 mIoU at 2.00 Cp2.00\,C_p; CORE reaches 68.2 / 83.57 / 48.39 at 1.92 Cp1.92\,C_p (Table 2; GR-2 adds the 1.42 Cp1.42\,C_p comparison). The abstract now says "pretraining steps" and states end-to-end costs.

Table 10 has no variance … how the defaults were chosen should be disclosed.

  • A1. Variance. We did not run multiple seeds for Table 10 (each cell is a full pipeline run). We rely only on effects beyond the seed-level spread (±0.18, Table 6): the mask ratio (1.5 points) and under-trained scorers (0.5–0.9). Differences of 0.04–0.05 are not read as an ordering.
  • A2. Defaults. Fixed in advance from prior work: the mask ratio follows MAE; the noise range, architecture, and schedule follow ECC (App. G).

W2. Theory and code

The two "testable predictions" do not follow from Prop. 2

A. We agree. Prop. 1 fixes what the score measures; utility rests on the experiments. L203–204 are now presented as expectations that Table 10 tests.

The excess-code-length term of Eq. 6 is never separated out, and Eq. 4 should also condition on tt

A. Eq. 4 now reads qθ(x∣x~,t)q_\theta(x \mid \tilde x, t). The excess term is not measurable per image; Sec. 2.2 now states that Prop. 2 is used for its pool-average statement only.

Table 4 lacks a compressed-size baseline … no experiment injects corrupted or out-of-distribution images.

  • A1. Table 4. Added: compressed size reaches 82.68 FT / 58.2 LP, against 82.61 / 56.7 for random and 82.90 / 62.1 for CORE (GR-3).
  • A2. Injection (5% of the pool, App. J). Kept in the top 50% / CORE LP / random LP: Gaussian noise 96% / 60.4 / 56.1; JPEG or sensor noise 63% / 62.1 / 57.4; out-of-distribution 58% / 62.2 / 55.8. The top half stays above random; pure noise has a long code and is selected, as you anticipated, and Sec. 6 states this limitation.

The supplied implementation differs from the paper in verifiable ways

A. The original archive was packaged from a broader codebase that supports several methods and carries components beyond the method of this paper. We have replaced it with an anonymized implementation of Eq. 2–5 (GR-1): (a) uniform 75% mask; (b) tt independent of the image, noise on visible tokens only; (c) clean target in training and scoring; (d) loss over the whole image; (e) a single loss term; (f) 400-epoch configs, self-contained. In our runs a scorer trained with it agrees with the paper's ranking at ρs≈0.9\rho_s \approx 0.9 (top half: 61.3 LP vs. 56.7 for random; App. C).

I thank the authors for their detailed response. I believe their clarifications have adequately addressed the concerns raised in my review. I will therefore increase my score accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their careful and constructive reviews. We are encouraged that all three credit the empirical signal: "a consistent selection signal with sensible controls" (6oQi), "convincing evidence of selection quality" (Y3H2), "broad and mostly well controlled" (s7rS). Revision (changes in red): the method, claims, and every original result are unchanged; we add the requested experiments (App. D, J; original table numbers are kept), clarify wording, add related work, and replace the supplementary code.

1. Code (6oQi, s7rS). We have replaced the supplementary archive with an anonymized implementation that follows Eq. 2–5. The original archive was packaged from a broader codebase that supports several methods and carries components beyond the method of this paper. In the new archive the mask is uniform at 75%, tt is drawn independently of the image, noise is applied to visible tokens only, the target is the clean image, and one whole-image loss is shared by training and scoring.

2. Baseline and total budget (all). Counting the full pipeline, CORE reaches 68.2 LP at 1.92 Cp1.92\,C_p; full data at 1600 epochs reaches 65.8 at 2.00 Cp2.00\,C_p.

MAE ViT-BTotal cost (CpC_p)LPFTADE20K
Full, 800 ep1.0062.382.9247.28
Random 50%, 1600 ep1.0064.983.2547.58
CORE top 50%, 800 ep1.4262.182.9047.22
CORE top 50%, 1600 ep1.9268.283.5748.39
Full, 1600 ep2.0065.883.3647.90
  • Baseline. The official 1600-epoch MAE checkpoint gives 67.6 under our probe (published 68.0). Our own runs are about two points below reported MAE numbers at both schedules (62.3 vs. 64.4; 65.8 vs. 67.8) because we use an effective batch size of 1024 (official 4096) and unnormalized pixel targets; every row shares this recipe (App. B).
  • At 1.42 Cp1.42\,C_p (LP, Table 12). CORE top (800 ep) 62.1; full (1136 ep) 63.5; random half (2272 ep) 65.8. A single short CORE run does not pay off at this budget.
  • Random half vs. full at equal steps. Prior work predicts parity at equal iterations (El-Nouby et al., 2021); FT and ADE20K are near parity, and we have no verified explanation for the larger LP gap (Sec. 4.2).
  • Wording. The abstract, Sec. 1, and Sec. 7 now say "pretraining steps" and state end-to-end costs.

3. Baselines (all). On the linear probe CORE is 3.9 points above lossless compressed size, 2.4 above an MAE trained on the same pool and scored identically, and 4.4 above SemDeDup (Table 4, App. J).

50% budget, MAERandomSemDeDupCompressed sizeMAE on the poolCORE
FT / LP82.61 / 56.782.65 / 57.782.68 / 58.282.78 / 59.782.90 / 62.1

4. What is new (s7rS, Y3H2). No prior work shows whether, or under which corruption, a pool-trained reconstruction loss selects images for self-supervised pretraining. Three findings are absent from, and in part contradict, the text literature on perplexity pruning: (i) the corruption decides the sign of the effect (a 25% mask is below random); (ii) the top of the ranking beats the middle (62.1 vs. 58.9), whereas Marion et al. (2023) find the middle best for text; (iii) a pixel-space ranking carries over to DINO. Prop. 1 is the standard likelihood–code-length correspondence and fixes what the score measures. The related work is now discussed in Sec. 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.