- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RepFlow, a framework that jointly trains a generator and an encoder through reciprocal supervision. A timestep-independent encoder extracts visual representations from masked clean images, while a timestep-conditioned predictor uses these representations to predict the generator’s internal states. In turn, EMA of the encoder extracts representations from full clean images to guide the generator. Experiments in both SiT and JiT demonstrate the effectiveness of the approach.
Critical strengths
-
The method jointly trains a generator and an encoder without relying on a pretrained teacher encoder. Assigning timestep conditioning to the predictor is a sensible design choice that allows the encoder to remain general.
-
Training costs are clearly reported, and the method retains its advantage over the native baseline under matched training budgets.
-
The paper is well written and easy to understand.
Critical weaknesses
-
A straightforward alternative is to first train an MAE and then use it to guide generator training. The paper does not establish whether the proposed joint training approach offers advantages over this alternative in generation quality or overall training efficiency.
-
Table 2 shows that the method achieves worse FID than REPA, which uses a pretrained visual encoder, while remaining competitive with SRA, which does not require one. Meanwhile, the learned encoder’s linear-probe accuracy is substantially lower than that of MAE. The paper should clarify the advantages of the proposed approach and the scenarios in which it would be preferable to these alternatives.
-
The one-step post-training experiments in Table 3 start from a RepFlow-JiT-XL checkpoint that has already received supervision from the RepFlow encoder during joint training, whereas MAE was not involved in this process. This prior co-adaptation may favor the RepFlow encoder. An additional comparison starting from a natively trained JiT checkpoint would help isolate the effectiveness of the two encoders as distribution-matching metrics.
Questions for authors
-
Please provides comparison between joint training and separate training. Using pretrained MAE is also accepetd.
-
Provides fair comparison in one-step post-training experiments.
5 comments
## Response to Reviewer QfXv (W2)
W2: Advantages and intended use cases
RepFlow unifies generation and representation learning through reciprocal supervision. Generator states train a timestep-free image encoder, whose features then guide generator updates. The same training process produces a generator and a separately usable encoder without an externally pretrained representation teacher.
Its generation benefits hold across unconditional and class-conditional generation in both latent and pixel spaces, with 19.2–40.8% lower FID than Native (submitted paper, Sec. 4.2 and Table 2). Table Q1 adds comparisons at matched training time. The differences from the alternatives are:
- REPA: DINOv2-based REPA achieves lower conditional FID in the submitted paper, but requires a pretrained representation teacher. RepFlow learns its teacher within generator training; the MAE-based comparison below evaluates the effectiveness of this learned supervision.
- SRA: It remains competitive on conditional SiT, but gives weaker results than RepFlow on conditional JiT and both unconditional backbones. On unconditional JiT, SRA gives 45.92 FID at matched time versus Native's 44.20, and only a small gain at 200 epochs (43.71 versus 44.20).
- Self-Flow: Its lower throughput limits progress at fixed training time. In our runs, throughput is 1.20/0.93 steps/s on SiT/JiT, compared with RepFlow-B's 1.82/2.70. It improves over Native at 200 epochs, while RepFlow gives the best results under both comparisons (Table Q1).
RepFlow is therefore useful when both a generator and a transferable encoder are needed, without separately pretraining a representation teacher.
Under the same pretraining budget of 800 epochs, RepFlow exceeds MAE by 24.21 and 20.46 percentage points on zero-shot SPair-71k correspondence (Table Q3). MAE has higher reported ImageNet linear-probe accuracy, while RepFlow has a clear advantage in spatial correspondence. Full RepFlow also improves over G2R-only on SPair-71k and Flowers102. Both evaluations use frozen encoders; SPair-71k requires no task-specific training, and Flowers102 uses a linear classifier.
Table Q3. Transfer of frozen representations. All SPair-71k encoders use the same 800-epoch pretraining budget; RepFlow uses SiT ViT-L student EMA checkpoints. Scores are percentages; pp denotes percentage points.
| Encoder | SPair-71k [email protected] ↑ | SPair-71k [email protected] ↑ | Flowers102 linear probe ↑ |
|---|---|---|---|
| Public MAE ViT-L | 18.14 | 11.14 | — |
| RepFlow-L, G2R-only | 41.89 | 30.17 | 76.17 |
| RepFlow-L, full reciprocal supervision | 42.35 | 31.60 | 77.05 |
| Full − G2R-only, percentage points | +0.46 | +1.43 | +0.88 |
| Full − MAE, percentage points | +24.21 | +20.46 | — |
The REPA-MAE comparison tests how effectively these representations supervise generator training. With ViT-L encoders in both methods, RepFlow-L reduces FID from 35.26 to 31.56 on SiT and from 35.86 to 34.52 on JiT at matched training time (Table Q1). At 200 epochs, FID falls from 29.91 to 27.86 on SiT and from 34.94 to 31.19 on JiT. These results favor jointly learned, evolving supervision over a fixed pretrained MAE teacher. The matched-time budget includes encoder learning.
The original one-step experiment in Table 3 of the submitted paper also shows that the encoder remains useful after it is frozen: from the same RepFlow-JiT initialization, it gives better FID, FDr6, and IS than MAE features. RepFlow's representation advantages therefore span spatial correspondence, supervision during generator training, and frozen-encoder reuse for post-training. We see this reuse across tasks and objectives as a promising direction for unified generation and representation learning.
## Response to Reviewer QfXv (W3 and Q2)
W3 and Q2: One-step post-training and prior co-adaptation
We added Native JiT and REPA-MAE JiT initializations, both post-trained with the same MAE-based FD loss. In the REPA-MAE control, MAE guides both generator pretraining and post-training, so the comparison includes prior interaction with the encoder in both complete training pipelines. Table Q4 combines these new controls with the two RepFlow-JiT results from Table 3 of the submitted paper.
Table Q4. New controls and original results for class-conditional one-step JiT post-training. All initializations use 400 pretraining epochs followed by 30 post-training epochs; evaluation uses NFE = 1 and CFG = 1. The final row is a calculated difference.
| Generator initialization | Frozen FD encoder | Result source | FID ↓ | FDr6 ↓ | IS ↑ |
|---|---|---|---|---|---|
| Native JiT | MAE ViT-L | New | 7.64 | 15.06 | 267.9 |
| REPA-MAE JiT | MAE ViT-L | New | 6.49 | 13.82 | 276.7 |
| RepFlow-JiT | MAE ViT-L | Submitted paper, Table 3 | 6.34 | 9.33 | 286.1 |
| RepFlow-JiT | RepFlow ViT-L | Submitted paper, Table 3 | 3.14 | 8.06 | 312.8 |
| Full pipeline vs. REPA-MAE pipeline, change | — | Calculated difference | −3.35 (−51.6%) | −5.76 | +36.1 |
With MAE fixed as the FD encoder, RepFlow initialization gives the best results among the three initializations. Replacing MAE with the RepFlow encoder then improves all three metrics. The complete RepFlow pipeline reduces FID from 6.49 to 3.14 relative to REPA-MAE, with improvements in FDr6 and IS as well.
This compares a pipeline guided by MAE in both stages with reciprocal training followed by reuse of the learned encoder. FDr6 is evaluated in six feature spaces, none of which is RepFlow's FD encoder.
## Response to Reviewer QfXv (W1 and Q1)
Thank you for suggesting the separately pretrained MAE comparison and the additional post-training controls. We report these below and clarify when joint generation and representation learning is useful.
W1 and Q1: Joint training versus a separately pretrained MAE
Following your suggestion, we use MAE ViT-L as the frozen teacher for REPA-style generator training. This directly compares separately pretrained MAE guidance with the representations learned jointly by RepFlow, using ViT-L encoders in the RepFlow-L comparison. All methods use unconditional ImageNet at 256 × 256 on eight H200 GPUs. Each budget equals Native's 200-epoch training time: approximately 33.3 hours for SiT and 22.1 hours for JiT. MAE's pretraining cost is excluded; RepFlow learns its encoder within the joint-training budget.
Table Q1. Unconditional FID-50K at matched wall-clock budgets and 200 epochs, including separately pretrained MAE guidance.
Both RepFlow variants outperform pretrained MAE guidance on both backbones at matched time and at 200 epochs. The joint-training budget includes learning the encoder.
Table Q2. Continued interaction after a shared warm-up. Unconditional SiT RepFlow-B starts from the same 50-epoch checkpoint and continues for another 50 epochs.
Continuing to update both models improves FID by 0.78 and student linear-probe accuracy by 1.1 percentage points over the respective frozen-teacher controls.