- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new objective-addition to train flow-matching models. The problem it solves is that during training, flow matching models are trained on the idealized target trajectories but, especially in early phases of training, these models can easily move off the ideal paths they saw during training, which puts them into a out-of-distribution situation leading to a gap between training-time and inference-time. The authors of this paper propose a simple solution: do not train the model on these idealized trajectories only, but instead ask the model to predict a current trajectory direction, and then also use a point on that trajectory and learn a correction towards to actually ground-truth velocity as well. This reminds me of popular predictor-corrector methods for numerical integration like Heun's method. Moreover, they propose a shallow-deep feature alignment loss term that forces earlier shallow model representations to be predictive of later and deeper model representations. The authors show that their additions improve sample efficiency and thus leads to faster convergence.
Critical strengths
- Clear problem formulation that is well motivated and the proposed solution is simple and intuitive and could therefore actually have an impact on this important area of research.
- This paper has a clear focus and is a good read, because it is not overloaded with unnecessary background or confusing formalizations.
Critical weaknesses
- Very little baselines for class-conditioned generation. I understand that comparison might be expensive, but I would have expected more baselines in the main tables beyond REPA. For example: SOAR, DEFAR, SelfFlow, and so on. If they are not a fair comparison, this should be more prominently stated and clarified in the rebuttal. Table 5 is more extensive, but lacks highlighting making it difficult to interpret.
- The shallow-deep feature alignment is not studied in detail. Which blocks are used? Why are they used? There are details missing, because it is presented as one monolithic addition without any ablations are detailed discussion. Especially for the text-conditioned generation, this feature alignment contributions appears to be more important than the model-induced rollout correction term. This alignment term is never shown in isolation or compared to the conceptually related approaches of SelfFlow and such. More details and careful experiments on this would significantly strengthen this work.
Questions for authors
Clarifications
- Can you elaborate on the connections of MiTraFlow to well-studied numerical integration methods like Heun's method and beyond? I think these connections need to be clarified and probably added to this work. If these predictor-corrector methods are not related then it should also be clarified.
- How expensive is a from-scratch training run? If I missed that part in the paper, please simple point me to where I can find it.
- Can you please provide more details on the shallow-deep feature alignment design and its importance for MiTraFlow? Can one use this feature alignment without the rollout loss component? Which one is more important?
Future work
- The experiments should be extended by ablations that clearly show the effects of the two proposed loss components (the model-induced rollout correct and the shallow-deep feature alignment).
- More baselines and/or a more extensive comparison to related methods would significantly strengthen this paper. It is currently presented as an extension to REPA. What about other underlying objectives?
4 comments
Response to Reviewer FJap
4. Applicability beyond REPA
The corrective objective depends solely on flow-matching states and velocity predictions, rather than external REPA supervision. In our representation-space experiments (Table 3 of the main paper), RAE () applies MiTraFlow directly on top of a non-REPA flow-matching objective () and improves unguided FID from to (and IS from to ), while PixNerd further demonstrates effectiveness in pixel space (FID ). We will clarify this generality in the revised text and distinguish the core corrective objective from optional external or self-alignment losses.
5. Related works and baseline comparisons
Table R18. Controlled comparison with SOAR, DEFAR, and MixFlow under identical DiT-L/2 + REPA-100K initialization settings at cumulative 400K iterations.
| Method | FID | sFID | IS | Precision | Recall | |
|---|---|---|---|---|---|---|
| REPA | 8.97 | 4.89 | 17.20 | 118.1 | 0.68 | 0.70 |
| SOAR | 8.48 | 5.22 | 17.22 | 125.1 | 0.69 | 0.66 |
| DEFAR† | -- | -- | -- | -- | -- | |
| MixFlow | 7.78 | 4.51 | 17.39 | 130.0 | 0.70 | 0.66 |
| MiTraFlow | 6.91 | 4.77 | 15.54 | 133.0 | 0.70 | 0.79 |
Note. † DEFAR diverged with FID exceeding 200 at cumulative 200K iterations.
MiTraFlow improves FID, , IS, and Recall over both SOAR and MixFlow (while MixFlow attains a lower sFID of 4.51 vs. 4.77). SOAR uses detached rollout followed by re-noising, and MixFlow uses slowed-interpolation mixtures for post-training, whereas MiTraFlow controls error exposure and jointly optimizes the earlier prediction and later correction. DEFAR combines directional rectification with frequency compensation, whereas MiTraFlow supervises the full corrective velocity. Under our DiT-L/2 + REPA-100K setup with DEFAR's default weights (, , ), training became unstable (FID by cumulative 200K iterations), suggesting sensitivity when transferring unmixed rollout () and heavily weighted directional losses across backbones and initializations without re-tuning.
In addition to these controlled implementations under our settings, we have updated Table 5 of the revised paper to include the full officially reported results of DEFAR and Self-Flow, and improved its readability by separating model-scale and representation-space blocks, pairing each + MiTraFlow row directly below its baseline, and highlighting both per-block best values (bold) and improvements over the direct baseline (underline).
Thank you for the additional effort in addressing my concerns. I appreciate the added baselines, the explanation of the connections to the predictor-corrector methods, and the additional ablations. Even though I am not fully confident in my assessment, I am still leaning positive on this work and will thus keep my current score.
If this paper is accepted, I would advise adding qualitative comparisons of samples between MiTraFlow, the baselines, and different ablated variants of MiTraFlow to the main paper.
Official Comment by Authors
Thank you for your thoughtful follow-up and continued positive assessment. We appreciate your suggestion and will add qualitative sample comparisons between MiTraFlow, the baselines, and its ablated variants to the main paper.
Response to Reviewer FJap
We thank Reviewer FJap for the positive assessment and for highlighting the clarity, simplicity, and practical motivation of our formulation. Below we address the connection to numerical integration, full training cost, shallow-deep alignment design and component ablations, applicability beyond REPA, and related works.
1. Connections to predictor-corrector integration
We agree that both approaches aim to limit error accumulation during sampling, but they address different sources of error. Heun is an ODE solver: given a trained velocity field, it improves the numerical accuracy with which the sampler follows that field. MiTraFlow instead addresses errors in the learned field through training. Its cross-step optimization teaches the model to correct deviations caused by its own predictions.
The distinction is therefore between more accurately integrating a model and training a model with better corrective ability. MiTraFlow retains the original sampling procedure without additional denoiser evaluations. We have added this discussion to Section 3.4 of the revised paper.
2. Full training cost
Table R14. Training speed and costs for the Large model on NVIDIA H20 GPUs with BF16 and global batch 256. GPU memory denotes peak allocated memory per GPU. Estimated hours are reported as x+y, where x is the shared 100K REPA initialization time on 8 GPUs and y is the subsequent training time.
Under the same x+y breakdown, the final Large model's 200-epoch recipe (1M iterations: 100K REPA initialization on 8 GPUs plus 900K corrective iterations on 16 GPUs) takes an estimated 10.25+95.15 hours, while the XL model's 900K iterations take approximately 112.61 hours in total on 16 H20 GPUs at 2.22 iterations/s.
3. Shallow-deep alignment design and component contributions
Alignment design and layer selection
For XL models, our alignment follows Self-Flow's design, including the 8/20 shallow-deep layer pairing, with two adaptations: we use the same MLP architecture as REPA and obtain the deep target from the online model rather than an EMA teacher. This target choice fits our training pipeline: unlike Self-Flow, which already evaluates an EMA teacher, MiTraFlow reuses the forward passes required by corrective learning and avoids an additional teacher evaluation. We also tested EMA-derived deep targets and observed comparable performance.
For the Large model, we compare two layer pairings in a 200K-iteration ablation (Table R15).
Table R15. Alignment-layer ablation on the Large model at 200K iterations.
We use 8/16 for the Large model. It gives slightly lower FID and sFID, while 7/17 has higher IS; overall, the two pairings yield comparable performance.
Separate component contributions
Table R16. Component comparison on ImageNet 256×256 at cumulative 400K iterations.
Corrective learning provides the larger FID improvement on ImageNet, and alignment further improves the combined model. The loss-removal control also shows that alignment can be trained without the corrective loss.
We add complementary T2I evaluations following the finetuned model settings in Table 6 of the main paper.
Table R17. T2I quality without classifier-free guidance. Lower FID and higher CLIP are better.
Both components improve FID and CLIP, and their combination gives the best results on both datasets. Alignment supplies representation supervision that complements corrective velocity learning.