acceptodds
Withdrawn from ICLR 2027

MiTraFlow: Learning Corrective Vector Fields on Model-Induced Trajectories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Diffusion and flow-matching models generate images by progressively transforming noise into data samples. However, standard training supervises ideal noising or interpolation states, whereas sampling follows states induced by the model's own predictions. This mismatch creates exposure bias: prediction errors can shift subsequent states away from the training distribution, where recovery is not directly supervised. We introduce MiTraFlow, a flow-matching objective that learns corrective velocity fields on model-induced trajectories. A controlled one-step rollout mixes the predicted and paired target velocities to construct a later state, at which the model learns the velocity needed to reach the paired data endpoint in the remaining time. Crucially, gradients propagate through both the rollout state and its corrective target, allowing the later loss to jointly optimize the correcting prediction and the earlier prediction that produced the state. We further align shallow features at the earlier state with detached deep features at the later state to regularize representations along the same transition. Our analysis shows that the corrective target compensates for the preceding velocity error, and ablations confirm that both controlled error exposure and cross-time gradients are essential. MiTraFlow adds no inference cost and yields larger gains at fewer sampling steps. On ImageNet , it converges approximately faster than REPA and achieves an FID of 1.29 with guidance; the improvements also generalize to pixel and RAE spaces and to large-scale text-to-image generation.

Primary area:
generative models
Keywords:
Diffusion, Image Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FJapOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new objective-addition to train flow-matching models. The problem it solves is that during training, flow matching models are trained on the idealized target trajectories but, especially in early phases of training, these models can easily move off the ideal paths they saw during training, which puts them into a out-of-distribution situation leading to a gap between training-time and inference-time. The authors of this paper propose a simple solution: do not train the model on these idealized trajectories only, but instead ask the model to predict a current trajectory direction, and then also use a point on that trajectory and learn a correction towards to actually ground-truth velocity as well. This reminds me of popular predictor-corrector methods for numerical integration like Heun's method. Moreover, they propose a shallow-deep feature alignment loss term that forces earlier shallow model representations to be predictive of later and deeper model representations. The authors show that their additions improve sample efficiency and thus leads to faster convergence.

Critical strengths
  • Clear problem formulation that is well motivated and the proposed solution is simple and intuitive and could therefore actually have an impact on this important area of research.
  • This paper has a clear focus and is a good read, because it is not overloaded with unnecessary background or confusing formalizations.
Critical weaknesses
  • Very little baselines for class-conditioned generation. I understand that comparison might be expensive, but I would have expected more baselines in the main tables beyond REPA. For example: SOAR, DEFAR, SelfFlow, and so on. If they are not a fair comparison, this should be more prominently stated and clarified in the rebuttal. Table 5 is more extensive, but lacks highlighting making it difficult to interpret.
  • The shallow-deep feature alignment is not studied in detail. Which blocks are used? Why are they used? There are details missing, because it is presented as one monolithic addition without any ablations are detailed discussion. Especially for the text-conditioned generation, this feature alignment contributions appears to be more important than the model-induced rollout correction term. This alignment term is never shown in isolation or compared to the conceptually related approaches of SelfFlow and such. More details and careful experiments on this would significantly strengthen this work.
Questions for authors

Clarifications

  • Can you elaborate on the connections of MiTraFlow to well-studied numerical integration methods like Heun's method and beyond? I think these connections need to be clarified and probably added to this work. If these predictor-corrector methods are not related then it should also be clarified.
  • How expensive is a from-scratch training run? If I missed that part in the paper, please simple point me to where I can find it.
  • Can you please provide more details on the shallow-deep feature alignment design and its importance for MiTraFlow? Can one use this feature alignment without the rollout loss component? Which one is more important?

Future work

  • The experiments should be extended by ablations that clearly show the effects of the two proposed loss components (the model-induced rollout correct and the shallow-deep feature alignment).
  • More baselines and/or a more extensive comparison to related methods would significantly strengthen this paper. It is currently presented as an extension to REPA. What about other underlying objectives?
4 comments
AuthorsOpenReview ↗

Response to Reviewer FJap

We thank Reviewer FJap for the positive assessment and for highlighting the clarity, simplicity, and practical motivation of our formulation. Below we address the connection to numerical integration, full training cost, shallow-deep alignment design and component ablations, applicability beyond REPA, and related works.

1. Connections to predictor-corrector integration

We agree that both approaches aim to limit error accumulation during sampling, but they address different sources of error. Heun is an ODE solver: given a trained velocity field, it improves the numerical accuracy with which the sampler follows that field. MiTraFlow instead addresses errors in the learned field through training. Its cross-step optimization teaches the model to correct deviations caused by its own predictions.

The distinction is therefore between more accurately integrating a model and training a model with better corrective ability. MiTraFlow retains the original sampling procedure without additional denoiser evaluations. We have added this discussion to Section 3.4 of the revised paper.

2. Full training cost

Table R14. Training speed and costs for the Large model on NVIDIA H20 GPUs with BF16 and global batch 256. GPU memory denotes peak allocated memory per GPU. Estimated hours are reported as x+yx + y, where xx is the shared 100K REPA initialization time on 8 GPUs and yy is the subsequent training time.

MethodIterationsGPUsSeconds/iterGPU memory (GiB)Estimated hoursFID
REPA700K80.369026.3910.25 + 61.508.43
MiTraFlow400K80.669040.9910.25 + 55.756.91
MiTraFlow400K160.380626.5710.25 + 31.726.91

Under the same x+yx + y breakdown, the final Large model's 200-epoch recipe (1M iterations: 100K REPA initialization on 8 GPUs plus 900K corrective iterations on 16 GPUs) takes an estimated 10.25+95.1510.25 + 95.15 hours, while the XL model's 900K iterations take approximately 112.61112.61 hours in total on 16 H20 GPUs at 2.222.22 iterations/s.

3. Shallow-deep alignment design and component contributions

Alignment design and layer selection

For XL models, our alignment follows Self-Flow's design, including the 8/20 shallow-deep layer pairing, with two adaptations: we use the same MLP architecture as REPA and obtain the deep target from the online model rather than an EMA teacher. This target choice fits our training pipeline: unlike Self-Flow, which already evaluates an EMA teacher, MiTraFlow reuses the forward passes required by corrective learning and avoids an additional teacher evaluation. We also tested EMA-derived deep targets and observed comparable performance.

For the Large model, we compare two layer pairings in a 200K-iteration ablation (Table R15).

Table R15. Alignment-layer ablation on the Large model at 200K iterations.

Shallow/deep layersFIDsFIDFDr6\mathrm{FDr}^6ISPrecisionRecall
7/178.934.6718.55112.390.690.63
8/168.834.6418.53110.920.690.63

We use 8/16 for the Large model. It gives slightly lower FID and sFID, while 7/17 has higher IS; overall, the two pairings yield comparable performance.

Separate component contributions

Table R16. Component comparison on ImageNet 256×256256\times256 at cumulative 400K iterations.

MethodFIDsFIDFDr6\mathrm{FDr}^6ISPrecisionRecall
REPA8.974.8917.20118.10.680.70
MiTraFlow6.914.7715.54133.00.700.79
    w/o corrective loss (alignment only)8.644.9817.87125.40.680.67
    w/o shallow-deep alignment (correction only)7.235.0115.82133.70.690.73

Corrective learning provides the larger FID improvement on ImageNet, and alignment further improves the combined model. The loss-removal control also shows that alignment can be trained without the corrective loss.

We add complementary T2I evaluations following the finetuned model settings in Table 6 of the main paper.

Table R17. T2I quality without classifier-free guidance. Lower FID and higher CLIP are better.

MethodCOCO2014 FIDCOCO2014 CLIPCOCO2017 FIDCOCO2017 CLIP
REPA23.4130.3433.0930.30
MiTraFlow21.6930.5031.1130.47
    w/o corrective loss (alignment only)22.4230.4131.9230.34
    w/o shallow-deep alignment (correction only)22.0630.4831.9730.44

Both components improve FID and CLIP, and their combination gives the best results on both datasets. Alignment supplies representation supervision that complements corrective velocity learning.

AuthorsOpenReview ↗

Response to Reviewer FJap

4. Applicability beyond REPA

The corrective objective depends solely on flow-matching states and velocity predictions, rather than external REPA supervision. In our representation-space experiments (Table 3 of the main paper), RAE (DiTDH-XL\mathrm{DiT}^{\mathrm{DH}}\text{-XL}) applies MiTraFlow directly on top of a non-REPA flow-matching objective (λext=0\lambda_{\mathrm{ext}} = 0) and improves unguided FID from 2.162.16 to 1.861.86 (and IS from 214.8214.8 to 226.5226.5), while PixNerd further demonstrates effectiveness in pixel space (FID 29.07→23.5629.07 \to \mathbf{23.56}). We will clarify this generality in the revised text and distinguish the core corrective objective from optional external or self-alignment losses.

5. Related works and baseline comparisons

Table R18. Controlled comparison with SOAR, DEFAR, and MixFlow under identical DiT-L/2 + REPA-100K initialization settings at cumulative 400K iterations.

MethodFIDsFIDFDr6\mathrm{FDr}^6ISPrecisionRecall
REPA8.974.8917.20118.10.680.70
SOAR8.485.2217.22125.10.690.66
DEFAR†>200> 200----------
MixFlow7.784.5117.39130.00.700.66
MiTraFlow6.914.7715.54133.00.700.79

Note. † DEFAR diverged with FID exceeding 200 at cumulative 200K iterations.

MiTraFlow improves FID, FDr6\mathrm{FDr}^6, IS, and Recall over both SOAR and MixFlow (while MixFlow attains a lower sFID of 4.51 vs. 4.77). SOAR uses detached rollout followed by re-noising, and MixFlow uses slowed-interpolation mixtures for post-training, whereas MiTraFlow controls error exposure and jointly optimizes the earlier prediction and later correction. DEFAR combines directional rectification with frequency compensation, whereas MiTraFlow supervises the full corrective velocity. Under our DiT-L/2 + REPA-100K setup with DEFAR's default weights (λADR=10.0\lambda_{\mathrm{ADR}}=10.0, λFC=1.0\lambda_{\mathrm{FC}}=1.0, α=1.0\alpha=1.0), training became unstable (FID >200> 200 by cumulative 200K iterations), suggesting sensitivity when transferring unmixed rollout (ρ=0\rho=0) and heavily weighted directional losses across backbones and initializations without re-tuning.

In addition to these controlled implementations under our settings, we have updated Table 5 of the revised paper to include the full officially reported results of DEFAR and Self-Flow, and improved its readability by separating model-scale and representation-space blocks, pairing each + MiTraFlow row directly below its baseline, and highlighting both per-block best values (bold) and improvements over the direct baseline (underline).

Thank you for the additional effort in addressing my concerns. I appreciate the added baselines, the explanation of the connections to the predictor-corrector methods, and the additional ablations. Even though I am not fully confident in my assessment, I am still leaning positive on this work and will thus keep my current score.

If this paper is accepted, I would advise adding qualitative comparisons of samples between MiTraFlow, the baselines, and different ablated variants of MiTraFlow to the main paper.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your thoughtful follow-up and continued positive assessment. We appreciate your suggestion and will add qualitative sample comparisons between MiTraFlow, the baselines, and its ablated variants to the main paper.

Reviewer_aYfPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To mitigate exposure bias, a well-studied challenge arising from the discrepancy between ideal training trajectories and model-induced sampling trajectories, this paper puts forward a flow-matching training framework termed MiTraFlow. Specifically, the proposed method constructs controlled rollout states by mixing ground-truth and predicted velocities, and introduces a corrective velocity objective to guide these states toward the paired data endpoint. The primary technical contribution lies in a differentiable rollout mechanism that enables cross-time gradient propagation, allowing earlier velocity predictions and subsequent corrections to be jointly optimized. Additionally, the authors incorporate shallow–deep feature alignment to encourage representation consistency across the constructed transition. Most Importantly, experiments on ImageNet-256 and text-to-image generation demonstrate improvements in generation quality and training efficiency across different representation spaces, supported by ablation studies examining the effects of rollout mixtures, gradient propagation, and feature alignment.

Critical strengths

The paper presents a practical corrective training mechanism that enables cross-time gradient propagation through differentiable model-induced rollouts, with ablation studies supporting the effectiveness of jointly optimizing earlier predictions and subsequent corrections. Another strength is the relatively extensive experimental validation across different model architectures, representation spaces, and generation tasks, demonstrating improvements in generation quality and training efficiency. Particularly, the larger improvements under limited sampling budgets highlight the practical value of the proposed method without introducing additional inference-time computational cost.

Critical weaknesses
  1. First, I have concerns about the novelty of the proposed method. The corrective velocity target follows directly from standard linear interpolation, while the differentiable rollout allows gradients to propagate through model-induced states and their targets. Although this is a reasonable design, I am not fully convinced that it offers a sufficiently distinct contribution over existing trajectory correction methods. The paper discusses SOAR, DEFAR, and MixFlow but lacks controlled comparisons under matched architectures and training budgets. In particular, the MixFlow results are reported under different experimental settings. Since these methods also address exposure bias, more direct comparisons are needed to establish whether the improvements come from MiTraFlow's specific design or trajectory correction in general.

  2. Moreover, although the paper provides useful ablations, the explanation of why the proposed training strategy works remains limited. The derivation relating corrective velocity to previous prediction errors is straightforward, but it is unclear whether the objective actually improves the learned velocity field or reduces accumulated sampling errors. I am also concerned about the sensitivity to the rollout mixture coefficient. As shown in Table 2, FID increases substantially as ρ\rho decreases from 0.9 to 0, suggesting sensitivity to deviations from the ground-truth trajectory. Furthermore, the improvements are not consistent across all settings. In Table 6, rollout correction alone reduces the pretraining GenEval score from 67.1 to 63.7, while the guided FID of DDT-XL slightly increases from 1.26 to 1.29 (Table 5). These results need further discussion, particularly regarding when the proposed correction is beneficial and why it may degrade performance in some cases.

Questions for authors
  1. MiTraFlow seems closely related to existing methods such as SOAR, DEFAR, and MixFlow. Apart from allowing gradients to flow through the rollout states, what are the main differences from these approaches? It would also be helpful to see comparisons using the same backbone, training budget, and sampling settings to better understand the advantages of MiTraFlow.

  2. The authors emphasize the importance of keeping both gradient paths in the corrective loss. However, in Table 2, detaching only the corrective target gives a worse FID (8.63) than detaching the rollout state entirely (8.24). Why does keeping one gradient path lead to worse performance than removing both? Also, have the authors measured whether cross-time gradient propagation actually reduces trajectory errors during sampling?

  3. Table 2 shows that FID increases significantly from 7.23 to 28.20 when ρ\rho changes from 0.9 to 0. This suggests that the method works best when the rollout remains close to the ground-truth trajectory. Could the authors explain why larger model-induced deviations lead to such a large performance drop? Does the choice of ρ=0.9\rho=0.9 work equally well across different architectures and training stages?

  4. Some experimental results seem less convincing. For example, rollout correction alone decreases the pretraining GenEval score from 67.1 to 63.7 in Table 6, while the guided FID of DDT-XL increases from 1.26 to 1.29 in Table 5. Could the authors discuss why the method does not consistently improve performance in these cases? Also, since MiTraFlow requires two denoiser evaluations and backpropagation through the rollout, could the authors report actual GPU-hours or wall-clock training time to support the claimed training efficiency?

3 comments
AuthorsOpenReview ↗

Response to Reviewer aYfP

We thank Reviewer aYfP for the thorough review and for recognizing our differentiable cross-time rollout mechanism, zero inference-cost overhead, and strong gains under limited sampling budgets. We address your four concerns below with controlled comparisons, gradient-path analysis, direct sampling measurements, and training-cost benchmarks.

1. Distinctions from related works

Our contribution is a corrective training design built on the standard state-to-velocity target. The distinctions from related methods are:

  1. SOAR learns correction from detached rollout followed by re-noising. MiTraFlow uses controlled velocity mixing and differentiates through both the rollout state and corrective target.
  2. DEFAR combines directional rectification with frequency compensation. MiTraFlow supervises the full corrective velocity, including direction and magnitude, within the controlled transition.
  3. MixFlow uses slowed-interpolation mixtures for post-training. MiTraFlow constructs an explicit transition from an earlier prediction and jointly optimizes that prediction and the later corrective velocity.

MiTraFlow's distinctive design couples controlled error exposure with joint optimization of the error-producing and error-correcting predictions, using full velocity supervision and gradients through both the rollout state and its target.

Table R7. Controlled comparison with SOAR, DEFAR, and MixFlow under identical DiT-L/2 + REPA-100K initialization settings at cumulative 400K iterations.

MethodFIDsFIDFDr6\mathrm{FDr}^6ISPrecisionRecall
REPA8.974.8917.20118.10.680.70
SOAR8.485.2217.22125.10.690.66
DEFAR†> 200----------
MixFlow7.784.5117.39130.00.700.66
MiTraFlow6.914.7715.54133.00.700.79

Note. † DEFAR diverged with FID exceeding 200 at cumulative 200K iterations.

Using the same controlled setup as the main-paper ablations, MiTraFlow outperforms competing methods on overall generation quality, distributional alignment, and sample diversity, improving FID from SOAR's 8.48 and MixFlow's 7.78 to 6.91, FDr6\mathrm{FDr}^6 from 17.22 / 17.39 to 15.54, IS from 125.1 / 130.0 to 133.0, and Recall from 0.66 to 0.79 (while MixFlow attains a lower sFID of 4.51 vs. 4.77). Under our DiT-L/2 + REPA-100K setup with DEFAR's default weights (λADR=10.0\lambda_{\mathrm{ADR}}=10.0, λFC=1.0\lambda_{\mathrm{FC}}=1.0, α=1.0\alpha=1.0), training became unstable (FID >200> 200 by cumulative 200K iterations), suggesting sensitivity when transferring unmixed rollout (ρ=0\rho=0) and heavily weighted directional losses across backbones and initializations without re-tuning. For a direct reference free of reproduction sensitivity, Table 5 of the revised paper also cites DEFAR's official SiT-XL/2 results (140 epochs: unguided FID 18.45, guided FID 4.14), compared with 6.24 and 1.51 for MiTraFlow (+ REPA-XL, 80 epochs).

AuthorsOpenReview ↗

Response to Reviewer aYfP

2. Cross-time gradient paths and sampling validation

Thank you for pointing out the need to better understand the gradient-path ablation and its effect on actual sampling. Our original ablation omitted the variant that detaches only the prediction branch vθ(x^s,s,c)v_\theta(\widehat{x}_s,s,c) while retaining gradients through the corrective target uscorru_s^{\mathrm{corr}}. We add this variant and measure training–sampling distribution discrepancies for all four settings (also updated in Table 2(b) and Section 4.2 of the revised paper).

Table R8. Gradient-path ablations and sampling-distribution discrepancies on ImageNet, following the experimental settings of Table 2(b) in the main paper. Distribution distances are time-averaged and reported as mean ±\pm sample standard deviation across five evaluation batches. Lower is better.

MethodNormalized W22W_2^2MMD2\mathrm{MMD}^2FID
REPA-700K0.328±0.0060.328 \pm 0.0060.115±0.0030.115 \pm 0.0038.43
MiTraFlow-400K, full gradients0.315±0.004\mathbf{0.315 \pm 0.004}0.105±0.002\mathbf{0.105 \pm 0.002}7.23
Detach prediction branch only (target path only)0.319±0.0050.319 \pm 0.0050.106±0.0020.106 \pm 0.0026.94
Detach uscorru_s^{\mathrm{corr}} only (prediction path only)0.329±0.0050.329 \pm 0.0050.115±0.0020.115 \pm 0.0028.63
Detach x^s\widehat{x}_s and uscorru_s^{\mathrm{corr}}0.334±0.0050.334 \pm 0.0050.117±0.0030.117 \pm 0.0038.24

To quantify the discrepancy between training and actual sampling, we introduce

ED(t)=Ex0∼N(0,I)![D!(pθ(xt∣x0),,pref(xt∣x0))].\mathcal{E}_D(t) =\mathbb{E}_{x_0\sim\mathcal{N}(0,I)}\\!\left[ D\\!\left( p_\theta(x_t\mid x_0),\\, p_{\mathrm{ref}}(x_t\mid x_0) \right) \right].

Here, pθ(xt∣x0)p_\theta(x_t\mid x_0) is the distribution of states reached from a fixed initial noise across class conditions using model-only sampling. The reference pref(xt∣x0)p_{\mathrm{ref}}(x_t\mid x_0) represents the corresponding training-state distribution, constructed as (1−t)x0+tx1(1-t)x_0+t x_1 using real images. We evaluate DD using normalized squared Wasserstein distance and MMD2\mathrm{MMD}^2, compute the distances at 50 recorded sampling times, and average over time. Appendix C.2 provides the complete definitions, evaluation protocol, and time-dependent results.

The results support two conclusions:

  1. Target-path feedback is beneficial: both variants retaining this path improve FID and reduce sampling-distribution discrepancies relative to REPA.
  2. Prediction-path feedback hurts FID in this setting: detaching the prediction branch improves FID from 7.23 to 6.94, with little change in distribution distances, whereas retaining only the prediction branch performs worse than blocking both paths (FID 8.63 vs. 8.24).

3. Rollout input sensitivity and the mixing coefficient

Input displacement and training dynamics

We separate the input displacement from cross-time gradient feedback to examine why larger model-induced deviations degrade training. Let xs=(1−s)x0+sx1x_s=(1-s)x_0+s x_1, et=vθ(xt,t,c)−(x1−x0)e_t=v_\theta(x_t,t,c)-(x_1-x_0), and h=s−th=s-t. We construct

x^s=sg⁡(xs+ahet)+bh[et−sg⁡(et)],\widehat{x}_s =\operatorname{sg}(x_s+ah e_t) +bh\left[e_t-\operatorname{sg}(e_t)\right],

where sg⁡\operatorname{sg} denotes stop-gradient. The forward state is xs+ahetx_s+ah e_t, while its derivative with respect to the earlier velocity prediction is bhIbhI. Thus, aa controls the input displacement at time ss and bb controls cross-time gradient feedback. Standard MiTraFlow with ρ=0.9\rho=0.9 uses a=b=0.1a=b=0.1.

Table R9. Independent coefficient interventions at cumulative 400K iterations.

SettingaabbFID
MiTraFlow0.10.16.91
Larger displacement1.00.139.17
Stronger feedback0.11.012.21

Increasing the input displacement causes severe degradation even when feedback remains small. Stronger feedback also degrades quality, but less severely. The input presented to the model at time ss is therefore an important factor: excessive displacement makes correction harder to learn and can destabilize optimization. Controlled rollout exposes prediction errors at a scale the model can learn to correct.

AuthorsOpenReview ↗

Response to Reviewer aYfP

(Continued from our previous response.) Shared coefficient and additional ablations All main-paper MiTraFlow experiments use ρ=0.9\rho=0.9 across architectures, representation spaces, and training stages, except the explicit coefficient ablations. We provide additional ρ\rho ablations on DDT-XL and PixNerd.

Table R10. Ablations of ρ\rho on DDT-XL at cumulative 300K iterations: 100K REPA pretraining and 200K corrective training. Evaluation uses ImageNet-256, EMA, SDE-250 with a final ODE step, no classifier-free guidance, and 50K samples. The saved model and data configurations differ only in ρ\rho.

ρ\rhoFIDsFIDFDr6\mathrm{FDr}^6ISPrecisionRecall
0.805.604.7813.33145.870.710.66
0.855.524.7113.11147.360.710.66
0.905.534.6212.94147.040.710.66
0.955.814.7213.00145.190.710.67

The FID values at ρ=0.85\rho=0.85 and 0.900.90 are close, while ρ=0.90\rho=0.90 gives the best sFID and FDr6\mathrm{FDr}^6. These results support the shared default as a balanced choice and show that nearby coefficients also train effectively in DDT-XL.

Table R11. Ablations of ρ\rho on PixNerd at 400K iterations. Evaluation uses ImageNet-256, EMA, 50K samples, 100-step Euler sampling, and no CFG.

ρ\rhoFIDsFIDFDr6\mathrm{FDr}^6ISPrecisionRecall
0.9023.569.4733.9060.770.540.67
0.7049.2218.9656.4031.700.470.61

For PixNerd, ρ=0.9\rho=0.9 outperforms 0.70.7 across all reported metrics. Together with the DDT-XL ablations, these results support controlled error exposure and the shared ρ=0.9\rho=0.9 setting used throughout the paper.

4. Non-uniform gains and training efficiency

Performance across evaluation metrics

During large-scale T2I pretraining, while text–image alignment is still forming, applying rollout state perturbations alone can disturb fine-grained compositional binding (GenEval 67.1→63.767.1 \to 63.7). Adding shallow-deep alignment stabilizes semantic representations across the transition, recovering pretraining GenEval to 67.167.1 and raising DPG from 79.879.8 to 81.181.1; once finetuned on high-quality data, rollout correction alone already improves GenEval from 77.277.2 to 79.179.1 (with full MiTraFlow improving both GenEval and DPG over REPA to 77.877.8 and 81.981.9). As supplementary validation of generation quality and text–image alignment, we report FID and CLIP results on COCO2014 and COCO2017—including an alignment-only baseline—following the finetuned model settings in Table 6 of the main paper.

Table R12. T2I quality without classifier-free guidance. Lower FID and higher CLIP are better.

MethodCOCO2014 FIDCOCO2014 CLIPCOCO2017 FIDCOCO2017 CLIP
REPA23.4130.3433.0930.30
MiTraFlow21.6930.5031.1130.47
    w/o corrective loss (alignment only)22.4230.4131.9230.34
    w/o shallow-deep alignment (correction only)22.0630.4831.9730.44

Both components improve FID and CLIP over REPA, and their combination gives the best results on both datasets.

Training time and memory

Table R13. Training costs on NVIDIA H20 GPUs with BF16 and global batch 256. GPU memory denotes peak allocated memory per GPU. Estimated hours are reported as x+yx + y, where xx is the shared 100K REPA initialization time on 8 GPUs and yy is the subsequent training time.

MethodIterationsGPUsSeconds/iterGPU memory (GiB)Estimated hoursFID
REPA700K80.369026.3910.25 + 61.508.43
MiTraFlow400K80.669040.9910.25 + 55.756.91
MiTraFlow400K160.380626.5710.25 + 31.726.91

Despite its higher per-iteration cost, MiTraFlow achieves better FID with fewer iterations and lower estimated training time. Using 16 GPUs further reduces training time and per-GPU memory.

Reviewer_yfSxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MiTraFlow to address exposure bias in flow matching models. The method constructs model induced states by mixing predicted and ground truth velocities and trains the model to correct the resulting errors. It further introduces shallow to deep feature alignment to improve representation learning. Experiments on ImageNet class conditional generation and text to image generation show improvements in generation quality and training efficiency.

Critical strengths

The paper addresses an important problem in flow matching models, namely the discrepancy between training and inference trajectories.

The proposed corrective velocity learning is intuitive and reasonable.

The gradient propagation across different time steps is technically interesting.

The method does not introduce additional inference cost.

The paper provides extensive experiments on ImageNet, including ablation studies, different sampling steps, and representation spaces.

Critical weaknesses

The overall methodology lacks a cohesive motivation. The paper combines corrective velocity learning with shallow to deep feature alignment, but the relationship between these two components is not sufficiently justified. It is unclear why feature alignment is necessary for trajectory correction.

The method is highly sensitive to the mixing coefficient. The best performance is achieved using 90% ground truth velocity, while using only predicted velocity significantly degrades performance. This raises concerns about the discrepancy between training and actual inference, where ground truth velocity is unavailable.

The proposed training strategy always starts from ideal interpolation states and introduces relatively small deviations. It is unclear whether these constructed states accurately represent the states encountered during actual inference. More direct experiments are needed to demonstrate that the method effectively reduces accumulated trajectory errors.

The experimental validation is not sufficiently convincing. Most experiments and ablations are conducted on ImageNet class conditional generation. The improvements on text to image generation are relatively limited and inconsistent. In Table 6, corrective rollout alone reduces GenEval from 67.1 to 63.7 during pretraining. Moreover, standard metrics such as CLIP Score and FID are missing. An alignment only baseline is also needed to better isolate the contributions of the two components.

The comparisons with existing methods are insufficient. More direct comparisons with closely related approaches such as SOAR, DEFAR and Look-Ahead and Look-Back Flows are needed to establish the advantages of the proposed method. Although the authors provide comparisons under approximately matched computational budgets, actual GPU training time and memory consumption should also be reported to better justify the claimed training efficiency.

Questions for authors

The mixing parameter is so important; hard-coded mixing parameter might not be the best.

3 comments
AuthorsOpenReview ↗

Response to Reviewer yfSx

We thank Reviewer yfSx for the constructive feedback and for recognizing the importance of addressing the training–sampling discrepancy, our cross-time gradient design, and our zero inference overhead. We address your concerns below with additional ablations, direct sampling-trajectory measurements, and training-cost comparisons.

1. Coherence of the motivation and the role of feature alignment

Table R1. Component comparison on ImageNet 256×256256\times256.

MethodIterationsFIDsFIDFDr6\mathrm{FDr}^6ISPrecisionRecall
REPA400K8.974.8917.20118.10.680.70
REPA700K8.435.2315.84126.40.680.71
MiTraFlow400K6.914.7715.54133.00.700.79
    w/o corrective loss (alignment only)400K8.644.9817.87125.40.680.67
    w/o shallow-deep alignment (correction only)400K7.235.0115.82133.70.690.73

Thank you for pointing out the need for a clearer motivation. Our primary goal is to reduce the training–sampling mismatch through corrective velocity learning on model-induced states. As shown in Table R1, while shallow-deep alignment alone moderately improves over 400K REPA (FID 8.97→8.648.97 \to 8.64), it does not surpass compute-matched REPA at 700K (8.438.43); the primary gain under matched compute comes from corrective learning (8.43→7.238.43 \to 7.23). Once the dual-state rollout forward passes are computed for corrective learning, shallow-deep alignment reuses these exact states and network evaluations at minimal extra overhead, further improving FID to 6.916.91 and recall to 0.790.79. We will revise the paper to make this hierarchy explicit, presenting corrective learning as the core mechanism and feature alignment as a lightweight complementary regularizer.

2. Sensitivity to the mixing coefficient

We add experiments to examine how ρ\rho affects training and whether gradually increasing the predicted-velocity fraction improves corrective learning.

First, we independently control the rollout displacement and its gradient feedback. Let xs=(1−s)x0+sx1x_s=(1-s)x_0+s x_1, et=vθ(xt,t,c)−(x1−x0)e_t=v_\theta(x_t,t,c)-(x_1-x_0), and h=s−th=s-t. We construct

x^s=sg⁡(xs+ahet)+bh[et−sg⁡(et)],\widehat{x}_s =\operatorname{sg}(x_s+ah e_t) +bh\left[e_t-\operatorname{sg}(e_t)\right],

where sg⁡\operatorname{sg} denotes stop-gradient. The forward state is xs+ahetx_s+ah e_t, while its derivative with respect to the earlier velocity prediction is bhIbhI. Thus, aa controls error exposure and bb controls cross-time gradient feedback. Standard MiTraFlow with ρ=0.9\rho=0.9 uses a=b=0.1a=b=0.1; we increase either coefficient to 1.0 while holding the other fixed.

All variants retain shallow-deep alignment and use REPA-100K initialization. Results are reported at cumulative 400K iterations (Table R2).

Table R2. Independent coefficient interventions at cumulative 400K iterations.

SettingaabbFID
MiTraFlow0.10.16.91
Larger displacement1.00.139.17
Stronger feedback0.11.012.21

Increasing either coefficient degrades quality, with larger rollout displacement causing the greater deterioration. These results show that both error exposure and cross-time feedback affect corrective training.

Without shallow-deep alignment, we also test cosine decay from ρ=0.9\rho=0.9 to 0.70.7 or 0.50.5 over 200K corrective iterations. The schedule to 0.70.7 reaches FID 7.847.84 at cumulative 400K iterations, compared with 7.237.23 for fixed ρ=0.9\rho=0.9 under the same alignment-free setting. The more aggressive schedule to 0.50.5 degrades as ρ\rho approaches 0.50.5, with FID worsening from 8.768.76 at cumulative 200K iterations to 10.5210.52 at 300K.

Fixed ρ=0.9\rho=0.9 therefore gives the best results among the tested settings. Controlled rollout exposes the model to its own prediction errors at a manageable scale, allowing it to learn correction effectively. The sampling metrics in the next response show that this ability transfers to model-only inference and reduces distribution deviations along generated trajectories.

AuthorsOpenReview ↗

Response to Reviewer yfSx

3. Transfer of corrective learning to actual sampling

We introduce a metric to quantify the discrepancy between the state distributions encountered during training and actual sampling. For a given initial noise x0∼N(0,I)x_0 \sim \mathcal{N}(0, I), let pθ(xt∣x0)p_\theta(x_t\mid x_0) denote the distribution of states reached at time tt by sampling from x0x_0 under different class or text conditions using only model predictions, and let pref(xt∣x0)p_{\mathrm{ref}}(x_t\mid x_0) denote the corresponding training-state distribution constructed as (1−t)x0+tx1(1-t)x_0+t x_1 using real images. Taking the expectation over initial noises, we define

ED(t)=Ex0∼N(0,I)![D!(pθ(xt∣x0),,pref(xt∣x0))].\mathcal{E}_D(t) =\mathbb{E}_{x_0\sim\mathcal{N}(0,I)}\\!\left[ D\\!\left( p_\theta(x_t\mid x_0),\\, p_{\mathrm{ref}}(x_t\mid x_0) \right) \right].

We evaluate DD using squared Wasserstein distance normalized by the latent dimensionality and MMD2\mathrm{MMD}^2. We compute ED(t)\mathcal{E}_D(t) at 50 recorded times along actual sampling trajectories and average over time. Appendix C.2 of the revised paper provides the complete metric definitions, evaluation protocol, and step-by-step curves, showing consistently lower deviations across mid-to-late sampling steps on C2I (t∈[0.12,1.0]t\in[0.12,1.0]) and at all 50 recorded steps on T2I.

Table R3. Time-averaged distribution deviations, reported as mean ±\pm sample standard deviation across five evaluation batches. Lower is better.

TaskMetricREPAOurs (Correction only)Reduction
C2INormalized W22W_2^20.328±0.0060.328\pm0.0060.315±0.004\mathbf{0.315\pm0.004}3.70%
C2IMMD2\mathrm{MMD}^20.115±0.0030.115\pm0.0030.105±0.002\mathbf{0.105\pm0.002}9.08%
T2INormalized W22W_2^20.333±0.0210.333\pm0.0210.322±0.021\mathbf{0.322\pm0.021}3.07%
T2IMMD2\mathrm{MMD}^20.097±0.0100.097\pm0.0100.087±0.010\mathbf{0.087\pm0.010}10.34%

Corrective training reduces both metrics in C2I and T2I, with time-averaged MMD reductions of 9.08% and 10.34%. These results show that correction learned through controlled rollout transfers to model-only inference and reduces state-distribution deviations during generation.

4. Text-to-image results and component contributions

We add FID and CLIP evaluations on COCO2014 and COCO2017, including an alignment-only baseline. All comparisons follow the finetuned model settings in Table 6 of the main paper.

Table R4. T2I quality without classifier-free guidance. Lower FID and higher CLIP are better.

MethodCOCO2014 FIDCOCO2014 CLIPCOCO2017 FIDCOCO2017 CLIP
REPA23.4130.3433.0930.30
MiTraFlow21.6930.5031.1130.47
    w/o corrective loss (alignment only)22.4230.4131.9230.34
    w/o shallow-deep alignment (correction only)22.0630.4831.9730.44

Both components improve FID and CLIP over REPA, and their combination gives the best results on both datasets. GenEval and DPG emphasize semantic correctness and composition, which corrective learning does not directly optimize, explaining the smaller gains and occasional decreases on these benchmarks.

AuthorsOpenReview ↗

Response to Reviewer yfSx

5. Related-method comparisons and training efficiency

Table R5. Controlled comparison with SOAR, DEFAR, and MixFlow under identical DiT-L/2 (REPA-100K initialization) settings at cumulative 400K iterations.

MethodFIDsFIDFDr6\mathrm{FDr}^6ISPrecisionRecall
REPA8.974.8917.20118.10.680.70
SOAR8.485.2217.22125.10.690.66
DEFAR†>200> 200----------
MixFlow7.784.5117.39130.00.700.66
MiTraFlow6.914.7715.54133.00.700.79
    w/o shallow-deep alignment (correction only)7.235.0115.82133.70.690.73

Note. † DEFAR diverged with FID exceeding 200 at cumulative 200K iterations.

Corrective learning alone achieves FID 7.23 compared with SOAR's 8.48 and MixFlow's 7.78, and shallow-deep alignment further improves FID to 6.91 (with Recall reaching 0.79 vs. 0.66). Under our DiT-L/2 + REPA-100K setup with DEFAR's default weights (λADR=10.0\lambda_{\mathrm{ADR}}=10.0, λFC=1.0\lambda_{\mathrm{FC}}=1.0, α=1.0\alpha=1.0), training became unstable (FID >200> 200 by cumulative 200K iterations), suggesting sensitivity when transferring unmixed rollout (ρ=0\rho=0) and heavily weighted directional losses across backbones and initializations without re-tuning. For a direct reference free of reproduction sensitivity, Table 5 of the revised paper also cites DEFAR's official SiT-XL/2 results (140 epochs: unguided FID 18.45, guided FID 4.14), compared with 6.24 and 1.51 for MiTraFlow (+ REPA-XL, 80 epochs).

The methods differ in their correction mechanisms:

  1. SOAR uses detached rollout and re-noising. MiTraFlow controls error exposure through velocity mixing and jointly optimizes the earlier prediction and later correction.
  2. DEFAR combines directional rectification with frequency compensation. MiTraFlow supervises the full corrective velocity, including direction and magnitude.
  3. MixFlow uses slowed-interpolation mixtures for post-training. MiTraFlow constructs an explicit transition from an earlier prediction and jointly optimizes that prediction and the later corrective velocity.
  4. Look-Ahead and Look-Back Flows are training-free inference-time samplers that smooth latent trajectories of a frozen velocity model via curvature gating or exponential history averaging, rather than training objectives. Because they modify test-time integration rather than how the velocity field is learned (and are thus orthogonal to training-side correction), we have added a detailed discussion of their conceptual relationship to MiTraFlow in Sections 2 and 3.4 of the revised paper.

Table R6. Training costs on NVIDIA H20 GPUs with BF16 and global batch 256. GPU memory denotes peak allocated memory per GPU. Estimated hours are reported as x+yx + y, where xx is the shared 100K REPA initialization time on 8 GPUs and yy is the subsequent training time.

MethodIterationsGPUsSeconds/iterGPU memory (GiB)Estimated hoursFID
REPA700K80.369026.3910.25 + 61.508.43
MiTraFlow400K80.669040.9910.25 + 55.756.91
MiTraFlow400K160.380626.5710.25 + 31.726.91

Despite its higher per-iteration cost, MiTraFlow achieves better FID with fewer iterations and lower estimated training time. Using 16 GPUs further reduces training time and per-GPU memory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.