acceptodds
Withdrawn from ICLR 2027

Generative Representation Learning for Scalable Latent Tokenization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce an approach for self-supervised representation learning designed to be used as a tokenizer for generative modeling. Our objective supervises a representation loss by factorizing the pointwise mutual information of the data, which we estimate by taking information differences between conditional predictions from a learned generative model. We justify our algorithm theoretically and empirically, demonstrating that our generative approach to representation learning avoids dimensional collapse and feature suppression in cases where commonly used methods like DINO and LeJEPA fail. Moreover, we show our representation learning technique is scalable; expending more compute leads to predictably better latent tokenization. For the first time, we perform a careful analysis of the combined flop efficiency of tokenizer and generative model and demonstrate that our approach is a compute-optimal improvement over direct diffusion for pixel-space generation, obtaining an FID-50K of 2.18 with JiT-L. Applied to ImageNet and the text-to-image dataset GPIC, we show tokenizer encoder scaling laws for diffusability, where extrapolating our approach achieves representations in fewer FLOPs than DINOv2 on ImageNet and fewer FLOPs than WebSSL on GPIC.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, image tokenization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9VYNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces a self-supervised learning method designed to operate as the first stage of the latent-generative models. They provide some insights from prior works and connect the pointwise mutual-information (PMI) to the generative denoising through variation diffusion model (VDM) formulation. The proposed method learns representations by matching their inner products to estimated conditional denoising improvements. Experiments show improved generation over direct pixel diffusion at comparable total training compute, while extrapolated scaling curves suggest potential encoder-training savings relative to DINOv2 and WebSSL.

Critical strengths

(S1) Obtaining a compute optimal method to reach generation friendly representations is an important open problem and this work makes some progress in that direction.

(S2) The insights from prior work connecting the PMI to denoising, and turning that into a framework of self-supervised learning is interesting.

Critical weaknesses

Major concerns:

(w1) A primary concern for this work is the lack of sufficient empirical results given the claim. First line of the manuscript opens up with introduction of an approach for an SSL method designed for generation. If the evaluation is restricted to generative modeling rather than broader SSL tasks, it should at minimum establish the method’s advantage within that setting through comparisons against representative alternatives such as RAE, EQ-VAE, and RAEv2 under matched total training budgets, accounting for both tokenizer and generator training.

(w2) Following from w1, the comparison that yields the 7-8x encoder-training compute savings is only computed from extrapolation and no matched training. If authors have already trained LeJEPA and DINOv3 on standard imagenet (stated in lines 369-370) it is unclear why the generations were not provided at matched budgets. Same expectation is around training the method presented here until plateaued and report gFID values comparable or at least in the same range as the baselines.

(w3) A separate concern here is that ablations seem to miss the isolation of the RePR loss. Without this it is not possible to understand whether the proposed RePR loss is doing real work, or the MAR framework itself introduces the gains for generation when trained with the bottom branch of the model showing in Figure1. To do this is it possible to only train the bottom branch and after training feed the unmasked image through the encoder and train the latent forcing generator on this?

(w4) Masking, timestep weighting, and loss-weight choices are primarily assessed through ImageNet classification accuracy, despite the paper’s motivation that discriminative representation quality need not predict generative usefulness (Table 1).

Minor weaknesses (must be fixed):

(w5) DINOv2 explicitly supervises patch features through its iBOT objective, making the characterization of its patch features as unsupervised misleading (§5.2).

Questions for authors

(Q1) Can authors experimentally validate the projected savings (7-8x)? This is especially important given the some of the baselines are already trained under authors own procedure

(Q2) Following from w3 can authors provide some ablation of the RePR loss? Ideally isolating whether the bottom branch of the framework learns these representations on its own or the RePR loss is needed? If it's not possible can authors explain this?

(Q3) Is it possible to provide matched comparisons of generation quality at convergence with other baselines?

3 comments
AuthorsOpenReview ↗

Rebuttal for 9VYN, all points addressed.

We thank the reviewer for recognizing the importance of the problem and our PMI–denoising framework.

(w1 / Q3) Diffusion / RAE / EQ-VAE / RAEv2 comparisons

We compare directly versus RAE baselines in our paper in Table 2 at FLOP-matched budgets, discussed in Sec. 6.5. We demonstrate that our method outperforms all prior FLOP- and data-matched RAE methods (RePR: 2.19×10202.19×10^{20} FLOPs; I-JEPA-H: 2.45×10202.45×10^{20}). We also add new baselines and 80-epoch results:

RAE encoder (FID-50K)20 ep80 ep
MAE-B/1614.738.38 (RAEv1)
MoCoV3-B13.84–
I-JEPA-H12.46–
RePR-173M Iter 3 (ours)11.135.81
DINOv2-B (topline)–4.28

We note that EQ-VAE uses an externally pretrained SD-VAE and RAE defaults to an externally pretrained DINOv2, making the raw results incomparable with FLOP-equal baselines. To control for this, we compare solely against ImageNet-only encoders for RAE, following the established baseline setup in RAEv2.

(w2 / Q1) 7–8× relies on extrapolation; LeJEPA/DINOv3 at matched budgets.

  • 7–8×. These are scaling-law predictions and are presented as such in every instance (abstract, contributions, Sec. 6.2). Extrapolation is the only clean comparison available: DINOv2 trains on a private dataset ~111× the size of ImageNet-1K, so matching FLOPs on ImageNet-1K would measure dataset size instead of the method.

    We scale up further and demonstrate that the trend holds where we can test it, however we cannot afford to scale up our largest model over 10x to match DINOv2:

    • A second iteration of our 306M encoder improves FID-10K from 31.23 to 27.22 / 25.79 / 24.39 after 111 / 211 / 411 additional epochs.
    • On GPIC (~80× ImageNet), scaling continues past ImageNet FLOP counts (Fig 3, Table 8).
  • Baselines. Most baselines are taken directly from prior work rather than trained under our own procedure: Latent Forcing + DINOv2 from Latent Forcing, JiT (Table 3) from Li & He (2026), and all RAE baselines except MAE at 20 epochs from RAEv1/RAEv2.

  • LeJEPA / DINOv3. These were 24-epoch runs designed to expose feature suppression on watermarked data, not tokenizers trained to convergence, so we did not report generation with them. LeJEPA is a regularization method and does not have a single canonical model checkpoint or architecture. The closest established baseline, however, is I-JEPA, from overlapping authors and the same model family, which has a published checkpoint that we outperform.

(w3 / Q2) RePR loss isolation

Removing the RePR loss and instead training solely with the MAR objective dramatically harms performance, obtaining 58.2 FID (vs. 35.7 with both losses). A related result requested by Reviewer 1 demonstrates, however, that the MAR features are necessary: training with only the RePR loss and a separate MAR model obtains 57.3 FID.

We similarly outperform the MAE baseline in RAE (table above). MAE shares the encoder/decoder architecture with MAR but is meant for learning features rather than direct generation and lets us compare against a published baseline. We also note that MAR representations suffer theoretically: similar to DINOv2 and DINOv3, the MAR objective has no direct causal theoretical grounding on its patch features.

(w4) Probing ablations

We used probing for the hyperparameter sweeps to avoid over a dozen extra diffusion runs. The core claims were evaluated on generation: iterations (Table 1e), scaling (Fig. 3), and Tables 2–3. We agree that generation is a better evaluation, so we re-ran the most important ablation, timestep weighting, on generation (RePR-78M, 200 ep):

Timestep WeightingFID-10K ↓
(1−t)/t3(1-t)/t^3NaN
(1−t)/(0.05,t)3(1-t)/(0.05,t)^342.15
μ=1.2,σ=1.2\mu=1.2,\sigma=1.235.72

(w5) DINOv2 explicitly supervises patch features through iBOT.

We distinguish between sending gradients into patch tokens (which both iBOT and CLS losses do) and directly supervising representation geometry, which DINOv2 does not. For both iBOT and CLS, a universal function approximator (a multilayer non-linear network) sits between the loss and the ViT patch features used as a tokenizer. The only difference is that iBOT features are connected by a residual connection, however this does not modify theoretical properties. Neither loss directly constrains patch geometry, and as a result, despite training with the iBOT loss, DINOv2/v3 tokenizers demonstrate significant instability at scale (outlier tokens requiring registers [Darcet et al., 2024] and late-training collapse requiring self-distillation in DINOv3 [Siméoni et al., 2025], Sec. 5.2). We will clarify Section 5.2 to state "unregularized representation geometry" rather than "unsupervised."

I thank the authors for their rebuttals.

The rebuttal does not establish that pretraining on more data would improve results on ImageNet itself. VA-VAE, for example, trains its autoencoder only on ImageNet and reports better ImageNet results than SD-VAE. Larger pretraining datasets may help with generalization beyond ImageNet hence this is not on its own an issue in investigating the scaling properties of other methods.

This is why I asked for a matched comparison with RAE. Table 2 includes MoCo v3 and I-JEPA-H, which prior RAE studies already found to perform worse than DINOv2. DINOv2-B is a stronger control and has approximately 86M parameters. To substantiate RePR’s claimed advantage under matched data and compute, it would be appropriate to compare the presented scaling laws with the scaling law of the baseline that is already used for this task.

Additional questions:

  1. Are the provided results in rebuttal regarding the pretrained DINO-v2 B or have authors trained this under same compute and data?

Main issue: I am currently struggling in establish the claim that such scaling law is only dedicated to the proposed method. There is no evidence that other SSL methods are not following the same law with respect to generation especially DINO-v2 that is know to yield best REPA targets and latents for RAE.

We thank the reviewer for engaging with our rebuttal. We believe there is a misunderstanding about DINOv2-B that is central to our paper's motivation.

DINOv2-B is not an 86M-parameter, ImageNet-scale control. It is distilled from a 1.1B ViT-g teacher trained on LVD-142M (∼111×\sim 111\times ImageNet-1K, curated to resemble it) with 1.17×10221.17 \times 10^{22} FLOPs, so no DINOv2 checkpoint is a matched-data or matched-compute control. This is precisely the gap our paper addresses (Sec. 2):

"RAE and REPA do not control for representation learner pretraining, where the chosen latent space, DINOv2-b (Oquab et al., 2024) is distilled from a 1.1 billion parameter model trained on over 100 times more data than ImageNet. As a result, the FLOP-efficiency of representation encoding is not currently understood."

VA-VAE has the same issue: although trained on ImageNet images, it is aligned to a frozen DINOv2 via its VF loss, and replacing DINOv2 with MAE worsens gFID from 15.8215.82 to 19.8919.89 (VA-VAE, Table 4).

The rebuttal does not establish that pretraining on more data would improve results on ImageNet itself.

At these compute scales, ImageNet-1K is data-limited: our largest encoder begins to overfit near 1.2×10201.2 \times 10^{20} FLOPs, while on GPIC (∼80×\sim 80\times more images) it continues to scale past ImageNet FLOP counts (Fig. 3, Table 8). DINOv2 uses ∼97×\sim 97\times that compute.

Are the DINOv2-B results pretrained, or trained under the same compute and data?

We use the official pretrained DINOv2-B checkpoint; retraining it under our data and compute would no longer be DINOv2. We therefore report it as an explicitly labeled topline. Even so, with ∼53×\sim 53\times less encoder compute and ∼111×\sim 111\times less data, RePR reaches 5.815.81 FID-50K vs. DINOv2-B's 4.284.28 in the 80-epoch RAE setting.

This is why we compare via scaling laws: placing the official DINOv2 at its true cost (Fig. 3; 17.9517.95 FID-10K at 1.17×10221.17 \times 10^{22} FLOPs), RePR's fitted scaling law predicts the same FID with 7.8×7.8\times fewer FLOPs.

There is no evidence that other SSL methods are not following the same law.

We do not claim that other methods lack scaling laws at small scale. Our claims (Sec. 1) are that (i) RePR exhibits tokenizer encoder scaling laws (Fig. 3), (ii) learning a representation is compute-optimal relative to direct pixel diffusion without external models (Table 3, Fig. 4), and (iii) RePR's representations have a theoretical characterization.

There is direct evidence that DINOv2 does not follow such a law: scaling the DINOv2 recipe, DINOv3 found that its patch-feature quality degrades with continued training, requiring re-alignment to an earlier checkpoint (Siméoni et al., 2025; Sec. 5.2). This is precisely what our theory addresses: DINO's tokens are intermediate patch features that no objective directly supervises, so nothing ties their quality to compute. RePR directly supervises its exported patch features by factorizing PMI, which is why its encoder scaling law is meaningful.

Together, (i) and (ii) are why scaling laws are necessary: training a tokenizer only pays off if additional encoder compute predictably improves generation (i) and outperforms spending the same FLOPs on pixel diffusion directly (ii), which no single pretrained checkpoint can establish.

In summary, RePR outperforms all FLOP- and data-matched ImageNet-only RAE encoders, and its scaling law extrapolation is more FLOP-efficient than DINOv2.

Reviewer_m9LaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether learning image representations specifically for generation is more effective than reusing representations originally designed for recognition and classification. The authors argue that such objectives can lead to feature suppression, where the model learns some image features while ignoring others useful for generation. Therefore, the paper argues that representations learned through generative prediction preserve more useful information. The authors also expect generation quality to improve as they scale up the training of these representations.

This motivation is actually already present in published works (e.g., Yao et al., Towards Scalable Pre-training of Visual Tokenizers for Generation, ECCV 2026). This paper differs in its learning objective. The proposed method, RePR, measures how much better a model predicts a hidden part of an image when it can see more of that image. It uses this improvement to train an encoder.

The authors first test whether RePR preserves useful image information even when an easy shortcut is available. The baselines are trained to produce similar representations for different views of the same image. Repeating a unique watermark on each image’s patches lets them satisfy this objective by recognizing the watermark. ImageNet classification accuracy then drops for LeJEPA and Dinov3. The authors also report better image generation as they scale RePR on ImageNet and GPIC with a fixed generator. At similar total training budgets RePR achieves an FID-50K of 2.18, compared with 2.36 for direct pixel generation.

Critical strengths

The paper addresses an understudied aspect in image generation: they rely on off-the-shelf self-supervised encoders like Dino (e.g. RAE, REPA). This not just an objective mismatch as discriminative and contrastive methods encourage feature suppression, but also, it wasn't clear whether their 2-stage latent diffusion is better than pixel diffusion because of better representation or pretrained encoder enabling "more compute" and "more data". The authors run a FLOP-matched comparison on ImageNet for both encoder and generator training. With their experiments on the 78M encoder training, they show that representation learning can improve generation quality.

Another strength is the proposed learning objective. It uses pointwise mutual information (PMI) to measure how much seeing one masked view improves prediction of the other. This enables supervision for patch representations without negatives. The authors make it tractable by:

  1. introducing a timestep-weighted PMI that biases the objective toward semantic noise scales instead of high-frequency pixel noise,
  2. formulating a bootstrapped estimator that reduces full-image likelihood estimation to low-variance single-patch increments,
  3. using a masked autoregressive prediction head.

The authors explain what goes wrong with the naive PMI estimator and how their design choices address it. They also back up several of these choices with ablations.

Critical weaknesses

W1:. It is hard to isolate the gain of the PMI objective from MAR pretraining, as there is no MAR-only baseline evaluated on FID.

W2: The claim of matching Dinov2 (7x) and WebSSL (8x) fewer FLOPs relies on extrapolation across a fairly large gap (24.73 vs. 17.95 FID on ImageNet, and 325 vs. 277.1 FD on GPIC). Since the 306M encoder already starts overfitting on ImageNet after 200 epochs, it is unclear whether the scaling trend is sustainable; whether later iterations can reach Dino using ImageNet alone.

Questions for authors

Q1: How do representations from the MAR encoder trained without the RePR loss perform on generation FID under comparable data and training budgets?

Q2: Have you evaluated a second round of encoder training using feature prediction for the 306M model on ImageNet? Does this alleviate the overfitting observed with pixel prediction?

1 comment
AuthorsOpenReview ↗

Rebuttal for m9La, all points addressed.

We thank the reviewer for the detailed review.

W1/Q1: Isolating gains of PMI vs. MAR.

Removing the RePR loss catastrophically harms performance versus only training for the MAR objective.

  • RePR-78M [200 ep] → 35.7 FID
  • MAR loss, no RePR [100 ep MAR, 100 ep RePR] → 58.2 FID

We also believe the following experiment, provided for Reviewer 1, is informative. For RAE experiments, we compare against the MAE baseline from RAEv1, where MAE shares an encoder/decoder with MAR but is meant for learning features rather than direct generation (MAE is informationally equivalent to MAR when the noisy patch is full noise). In this case, we similarly outperform:

RAE encoder (FID-50K)20 ep80 ep
MAE-B/1614.738.38
RePR-173M Iter 3 (ours)11.135.81

We also note that MAR representations suffer theoretically: similar to DINOv2 and DINOv3, the MAR objective has no direct causal theoretical grounding on its patch features. Although this generally works at small scales, the DINO family, which performs an iBOT objective similar to MAR training (masked patch prediction), does not sufficiently supervise its representations at scale, leading to significant instability (Sec. 5.2).

W2/Q2: 7×/8× relies on extrapolation across a large gap; the 306M encoder overfits on ImageNet. Does a second round of feature prediction for the 306M model help?

Yes. Feature prediction keeps our largest model improving (FID-10K, JiT-B 80 ep):

306M encoderEncoder epochsFID-10K ↓
Iter 1 (pixel targets)21131.23
Iter 2 (feature targets)211 + 11127.22
Iter 2211 + 21125.79
Iter 2211 + 41124.39

Sustainability of the trend to reach DINOv2. Our algorithm shows consistent trend sustainability to reach DINOv2. First, DINOv2 trains on a private, internal dataset with 111× the data of ImageNet and roughly 97× the compute of our largest model. Therefore, under a linear relationship between compute and data scaling, our model is entirely consistent with DINOv2 performance at scale. We empirically validate that the early overfitting is a result of this dataset disparity rather than our proposed RePR algorithm by training on GPIC, where our 400-epoch large first-iteration model continues scaling past the point where it overfits on ImageNet (Fig. 3, Table 8).

Relation to Yao et al. (ECCV 2026). We admit that we are not the first work to address tokenizer scaling. However, Yao et al.'s scaling results couple the tokenizer encoder, our work’s focus and a component that has repeatedly demonstrated instability during scaling, with the tokenizer decoder, which is a part of the generative process. We differ in three ways:

  • We isolate gains to being from the encoder through a lossless pixel-space generator.
  • We account for tokenizer training compute against direct pixel diffusion (Table 3).
  • Our objective is theoretically grounded via PMI factorization and is robust to feature suppression (Fig. 2).
Reviewer_AUhdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to propose a representation learning method suitable for generative modeling, inspired by the relationship between pointwise mutual information and contrastive learning. Specifically, it uses differences in conditional denoising errors to estimate weighted PMI increments and supervises patch representations through the PMI-based objective. Experiments on ImageNet and GPIC show improved generation with Latent Forcing, including gains over pixel-space JiT after accounting for the representation training compute.

Critical strengths
  1. Some designs are clever and interesting, such as the bootstrapped PMI estimation, and the Gaussian corruption that can be composed into a single timestep weighting.
  2. Studying the combined training cost of the representation learner and downstream generator addresses an important question that comparisons using pretrained encoders often overlook.
Critical weaknesses
  1. The method presentation lacks clarity, and several important design choices remain insufficiently explained.
  • The notation changes across Section 4 and Section 5. The transition from two-view PMI (x/y, Eq. 1-14) to the one involving A/B/p (Eq. 15, Figure 1) is somewhat difficult to follow. Although Section 4.3 motivates the decomposition, its connection to the actual view construction, prediction targets, and supervision of the full spatial representation needs clearer explanation.
  • Section 4.2 introduces an elegant idea: using an additional ss to control information corruption, then combining it with the second time tt used for estimation. However, the paper does not spell out how the actual f(t)f(t) and g(t)g(t) follow from this construction. The relationship between g(t)g(t) and w(t)w(t) is also unclear, as well as Table 1(f). Why does the original weighting produce NaN, and how were μ=1.2,σ=1.2\mu=1.2,\sigma=1.2 selected?
  • More fundamentally, what information and tasks does this "inductive bias" favor? I assume the PMI-based method (Eq. 1-8) provides a general representation learning objective, it is unclear why these choices particularly suit diffusion generation, and broader perceptual tasks such as image segmentation are missing.
  • The prediction model and the representation model share the encoder, which is shaped by the MAR generative objective. This already biases the representation toward generative prediction, beyond the timestep weighting. How much of the downstream JiT gain comes from this generative pretraining, vs the PMI-based method? Controls such as separate encoders are missing.
  1. The experiments do not sufficiently isolate RePR's contribution and support the claims.
  • Table 3 compares RePR + Latent Forcing against standard pixel-space JiT, combining improvements from the representation and the co-denoising framework.
  • I understand that the paper emphasizes controlled compute budgets, so it is difficult to compare against foundation models such as DINOv2 and DINOv3 directly. However, the claimed 7x/8x efficiency over DINOv2/WebSSL relies on extrapolation, and the current comparison in Table 2 is not sufficient enough.
  • The paper evaluates "diffusability" mainly through Latent Forcing, whereas conventional VAE/tokenizer evaluation trains diffusion in the latent space. It therefore remains unclear how well the reported gains transfer to standard latent diffusion pipelines, and this can be somewhat misleading. Although the authors include a conventional RAE evaluation, it is limited to 20 epochs, one generator scale, and two baselines.
Questions for authors

Please refer to the weaknesses.

1 comment
AuthorsOpenReview ↗

Rebuttal for AUhd, all points addressed.

We thank the reviewer for taking a careful read of our paper.

W1. Notation changes across Sec. 4 and 5.

In Section 4, we use x,y,z to represent abstract views of data, while in Section 5 A,B,p represent instantiated image masks and patches. They correspond one-to-one: x,y,z=A,B,p, respectively. We will make this correspondence explicit in our updated paper. We use a random mask ratio for A,B to supervise all context sizes necessary for telescoping (Eq. 14, Sec. 5.3), while p is a random non-overlapping patch.

How f,g follow from q; why NaN; f, g selection.

In standard diffusion models (e.g., EDM; Karras et al., 2022), weighting is decomposed into timestep sampling f(t) and a loss weighting g(t), inducing an overall weighting of f(t)g(t). We never instantiate q(s), rather we use q(s) to demonstrate that timestep weighting is consistent with a definition of the PMI on noised data. The precise relationship is f(t)g(t)∝1−tt3Fq(t)f(t)g(t) \propto \tfrac{1-t}{t^3}F_q(t), where FqF_q is the CDF of q; we will add a derivation to the appendix. w(t) is the loss weight used to train the MAR denoiser (Eq. 16), while g(t) weights the Monte Carlo PMI integration (Eq. 15). To avoid introducing new hyperparameters, we choose g(t)=w(t) in practice.

Training with the timestep weighting of ordinary PMI, (1−t)/t3,(1-t)/t^3, leaves the raw ELBO weight, which diverges as t0, giving NaN. We swept a handful of timestep weighting strategies: clipping (JiT; Table 1f, row 2) and logit-normal sampling (Esser et al., 2024). With logit-normal μ=0.6,σ=1.2\mu=0.6,\sigma=1.2, we get an IN-1K top-1 of 55.5.

What does this inductive bias favor?

A key motivation for our work is performing PMI factorization without feature suppression (Sec. 1). Contrastive models also factorize PMI (Sec 3) and demonstrate a strong probing and segmentation performance [MoCov3]. However, generative modeling is a unique task in that all latent variables must be sampled, making it ideal to focus on for a non feature-suppressed objective.

Shared encoder; separate-encoder controls are missing.

We run your separate encoder decoder experiment, as well as an experiment that only trains for MAR.

  • RePR-78M [200 ep] → 35.7 FID
  • Separate encoder/decoder [100 ep MAR+100 ep RePR] → 57.3
  • No RePR (MAR loss only) [200 ep] → 58.2

Both losses are required. Although the RePR loss is sufficient in theory, we only obtain a gradient signal on a small number of patches per forward pass (p_repeat=16). Meanwhile, generative training has much richer gradients, predicting every masked patch (~220 on average), although the features aren't diffusable. Therefore, in practice it is useful to supervise shared parameters.

Table 3 combines gains from the representation and the co-denoising framework.

We choose Latent Forcing precisely to prevent this confounding. Latent Forcing is not an architecture, but rather solely modifies the time schedule, so there are zero improvements from the co-denoising framework: JiT training is recovered exactly by replacing the latent with the zero vector and using the JiT time-schedule hyperparameters. Our evaluation controls for identical training FLOPs (tokenizer + JiT), inference FLOPs, parameters, data, and lossless pixel-space generation, with no external models (DINOv2, DINO-GAN, LPIPS) (Sec. 6.2).

The 7×/8× efficiency relies on extrapolation.

Yes, these are scaling-law predictions, and we present them explicitly as extrapolation in every instance we quote them (abstract, contributions, Sec. 6.2). DINOv2 trains on LVD-142M, a private dataset 111× the size of ImageNet-1K, using 1.17×10221.17×10^{22} FLOPs. Matching DINOv2 FLOPs on ImageNet-1K would be heavily confounded with dataset size. Extrapolation is a useful and necessary way to interpret our results on academic compute and data.

RAE is limited to 20 epochs, one generator scale, and two baselines.

We train our RAE model to 80 epochs and compare against a new MAE baseline. With this, our baselines include all ImageNet-only RAE results reported in prior work (RAEv1, RAEv2). RAE experiments are expensive because the default generator is DiT-XL with DH heads: each 20-epoch RAE run takes over double the time of an 80-epoch Latent Forcing run on ImageNet. 20 epochs was not arbitrarily chosen by us, but rather is the official ablation setup from RAEv2.

RAE encoder (FID-50K)20 ep80 ep
MAE-B/1614.738.38 (RAEv1)
MoCoV3-B13.84–
I-JEPA-H12.46–
RePR-173M Iter 3 (ours)11.135.81
DINOv2-B (topline)–4.28

Our goal for RAE experiments is solely to demonstrate competitiveness with prior work, not a FLOP-wise state-of-the-art recipe. The RAE decoder relies on a DINOv1/8 model trained for 4.7×10204.7×10^{20} FLOPs, making it unclear whether gains result from a better encoder or instead an encoder 'compatible' with DINOv1, allowing a model to borrow FLOPs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Period Ending Soon

We have uploaded our rebuttal. We kindly ask the reviewers to respond, as the discussion period is ending soon. Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.