- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces a self-supervised learning method designed to operate as the first stage of the latent-generative models. They provide some insights from prior works and connect the pointwise mutual-information (PMI) to the generative denoising through variation diffusion model (VDM) formulation. The proposed method learns representations by matching their inner products to estimated conditional denoising improvements. Experiments show improved generation over direct pixel diffusion at comparable total training compute, while extrapolated scaling curves suggest potential encoder-training savings relative to DINOv2 and WebSSL.
Critical strengths
(S1) Obtaining a compute optimal method to reach generation friendly representations is an important open problem and this work makes some progress in that direction.
(S2) The insights from prior work connecting the PMI to denoising, and turning that into a framework of self-supervised learning is interesting.
Critical weaknesses
Major concerns:
(w1) A primary concern for this work is the lack of sufficient empirical results given the claim. First line of the manuscript opens up with introduction of an approach for an SSL method designed for generation. If the evaluation is restricted to generative modeling rather than broader SSL tasks, it should at minimum establish the method’s advantage within that setting through comparisons against representative alternatives such as RAE, EQ-VAE, and RAEv2 under matched total training budgets, accounting for both tokenizer and generator training.
(w2) Following from w1, the comparison that yields the 7-8x encoder-training compute savings is only computed from extrapolation and no matched training. If authors have already trained LeJEPA and DINOv3 on standard imagenet (stated in lines 369-370) it is unclear why the generations were not provided at matched budgets. Same expectation is around training the method presented here until plateaued and report gFID values comparable or at least in the same range as the baselines.
(w3) A separate concern here is that ablations seem to miss the isolation of the RePR loss. Without this it is not possible to understand whether the proposed RePR loss is doing real work, or the MAR framework itself introduces the gains for generation when trained with the bottom branch of the model showing in Figure1. To do this is it possible to only train the bottom branch and after training feed the unmasked image through the encoder and train the latent forcing generator on this?
(w4) Masking, timestep weighting, and loss-weight choices are primarily assessed through ImageNet classification accuracy, despite the paper’s motivation that discriminative representation quality need not predict generative usefulness (Table 1).
Minor weaknesses (must be fixed):
(w5) DINOv2 explicitly supervises patch features through its iBOT objective, making the characterization of its patch features as unsupervised misleading (§5.2).
Questions for authors
(Q1) Can authors experimentally validate the projected savings (7-8x)? This is especially important given the some of the baselines are already trained under authors own procedure
(Q2) Following from w3 can authors provide some ablation of the RePR loss? Ideally isolating whether the bottom branch of the framework learns these representations on its own or the RePR loss is needed? If it's not possible can authors explain this?
(Q3) Is it possible to provide matched comparisons of generation quality at convergence with other baselines?
3 comments
I thank the authors for their rebuttals.
The rebuttal does not establish that pretraining on more data would improve results on ImageNet itself. VA-VAE, for example, trains its autoencoder only on ImageNet and reports better ImageNet results than SD-VAE. Larger pretraining datasets may help with generalization beyond ImageNet hence this is not on its own an issue in investigating the scaling properties of other methods.
This is why I asked for a matched comparison with RAE. Table 2 includes MoCo v3 and I-JEPA-H, which prior RAE studies already found to perform worse than DINOv2. DINOv2-B is a stronger control and has approximately 86M parameters. To substantiate RePR’s claimed advantage under matched data and compute, it would be appropriate to compare the presented scaling laws with the scaling law of the baseline that is already used for this task.
Additional questions:
- Are the provided results in rebuttal regarding the pretrained DINO-v2 B or have authors trained this under same compute and data?
Main issue: I am currently struggling in establish the claim that such scaling law is only dedicated to the proposed method. There is no evidence that other SSL methods are not following the same law with respect to generation especially DINO-v2 that is know to yield best REPA targets and latents for RAE.
We thank the reviewer for engaging with our rebuttal. We believe there is a misunderstanding about DINOv2-B that is central to our paper's motivation.
DINOv2-B is not an 86M-parameter, ImageNet-scale control. It is distilled from a 1.1B ViT-g teacher trained on LVD-142M ( ImageNet-1K, curated to resemble it) with FLOPs, so no DINOv2 checkpoint is a matched-data or matched-compute control. This is precisely the gap our paper addresses (Sec. 2):
"RAE and REPA do not control for representation learner pretraining, where the chosen latent space, DINOv2-b (Oquab et al., 2024) is distilled from a 1.1 billion parameter model trained on over 100 times more data than ImageNet. As a result, the FLOP-efficiency of representation encoding is not currently understood."
VA-VAE has the same issue: although trained on ImageNet images, it is aligned to a frozen DINOv2 via its VF loss, and replacing DINOv2 with MAE worsens gFID from to (VA-VAE, Table 4).
The rebuttal does not establish that pretraining on more data would improve results on ImageNet itself.
At these compute scales, ImageNet-1K is data-limited: our largest encoder begins to overfit near FLOPs, while on GPIC ( more images) it continues to scale past ImageNet FLOP counts (Fig. 3, Table 8). DINOv2 uses that compute.
Are the DINOv2-B results pretrained, or trained under the same compute and data?
We use the official pretrained DINOv2-B checkpoint; retraining it under our data and compute would no longer be DINOv2. We therefore report it as an explicitly labeled topline. Even so, with less encoder compute and less data, RePR reaches FID-50K vs. DINOv2-B's in the 80-epoch RAE setting.
This is why we compare via scaling laws: placing the official DINOv2 at its true cost (Fig. 3; FID-10K at FLOPs), RePR's fitted scaling law predicts the same FID with fewer FLOPs.
There is no evidence that other SSL methods are not following the same law.
We do not claim that other methods lack scaling laws at small scale. Our claims (Sec. 1) are that (i) RePR exhibits tokenizer encoder scaling laws (Fig. 3), (ii) learning a representation is compute-optimal relative to direct pixel diffusion without external models (Table 3, Fig. 4), and (iii) RePR's representations have a theoretical characterization.
There is direct evidence that DINOv2 does not follow such a law: scaling the DINOv2 recipe, DINOv3 found that its patch-feature quality degrades with continued training, requiring re-alignment to an earlier checkpoint (Siméoni et al., 2025; Sec. 5.2). This is precisely what our theory addresses: DINO's tokens are intermediate patch features that no objective directly supervises, so nothing ties their quality to compute. RePR directly supervises its exported patch features by factorizing PMI, which is why its encoder scaling law is meaningful.
Together, (i) and (ii) are why scaling laws are necessary: training a tokenizer only pays off if additional encoder compute predictably improves generation (i) and outperforms spending the same FLOPs on pixel diffusion directly (ii), which no single pretrained checkpoint can establish.
In summary, RePR outperforms all FLOP- and data-matched ImageNet-only RAE encoders, and its scaling law extrapolation is more FLOP-efficient than DINOv2.
Rebuttal for 9VYN, all points addressed.
We thank the reviewer for recognizing the importance of the problem and our PMI–denoising framework.
We compare directly versus RAE baselines in our paper in Table 2 at FLOP-matched budgets, discussed in Sec. 6.5. We demonstrate that our method outperforms all prior FLOP- and data-matched RAE methods (RePR: 2.19×1020 FLOPs; I-JEPA-H: 2.45×1020). We also add new baselines and 80-epoch results:
We note that EQ-VAE uses an externally pretrained SD-VAE and RAE defaults to an externally pretrained DINOv2, making the raw results incomparable with FLOP-equal baselines. To control for this, we compare solely against ImageNet-only encoders for RAE, following the established baseline setup in RAEv2.
7–8×. These are scaling-law predictions and are presented as such in every instance (abstract, contributions, Sec. 6.2). Extrapolation is the only clean comparison available: DINOv2 trains on a private dataset ~111× the size of ImageNet-1K, so matching FLOPs on ImageNet-1K would measure dataset size instead of the method.
We scale up further and demonstrate that the trend holds where we can test it, however we cannot afford to scale up our largest model over 10x to match DINOv2:
Baselines. Most baselines are taken directly from prior work rather than trained under our own procedure: Latent Forcing + DINOv2 from Latent Forcing, JiT (Table 3) from Li & He (2026), and all RAE baselines except MAE at 20 epochs from RAEv1/RAEv2.
LeJEPA / DINOv3. These were 24-epoch runs designed to expose feature suppression on watermarked data, not tokenizers trained to convergence, so we did not report generation with them. LeJEPA is a regularization method and does not have a single canonical model checkpoint or architecture. The closest established baseline, however, is I-JEPA, from overlapping authors and the same model family, which has a published checkpoint that we outperform.
Removing the RePR loss and instead training solely with the MAR objective dramatically harms performance, obtaining 58.2 FID (vs. 35.7 with both losses). A related result requested by Reviewer 1 demonstrates, however, that the MAR features are necessary: training with only the RePR loss and a separate MAR model obtains 57.3 FID.
We similarly outperform the MAE baseline in RAE (table above). MAE shares the encoder/decoder architecture with MAR but is meant for learning features rather than direct generation and lets us compare against a published baseline. We also note that MAR representations suffer theoretically: similar to DINOv2 and DINOv3, the MAR objective has no direct causal theoretical grounding on its patch features.
We used probing for the hyperparameter sweeps to avoid over a dozen extra diffusion runs. The core claims were evaluated on generation: iterations (Table 1e), scaling (Fig. 3), and Tables 2–3. We agree that generation is a better evaluation, so we re-ran the most important ablation, timestep weighting, on generation (RePR-78M, 200 ep):
We distinguish between sending gradients into patch tokens (which both iBOT and CLS losses do) and directly supervising representation geometry, which DINOv2 does not. For both iBOT and CLS, a universal function approximator (a multilayer non-linear network) sits between the loss and the ViT patch features used as a tokenizer. The only difference is that iBOT features are connected by a residual connection, however this does not modify theoretical properties. Neither loss directly constrains patch geometry, and as a result, despite training with the iBOT loss, DINOv2/v3 tokenizers demonstrate significant instability at scale (outlier tokens requiring registers [Darcet et al., 2024] and late-training collapse requiring self-distillation in DINOv3 [Siméoni et al., 2025], Sec. 5.2). We will clarify Section 5.2 to state "unregularized representation geometry" rather than "unsupervised."