acceptodds
Under review as a conference paper at ICLR 2027

Latent Stiffness in MeanFlow Training

Abstract

MeanFlow trains one-step generative models using targets that depend on the network’s own time derivative. We interpret this objective as a semi-gradient bootstrap, analogous to temporal-difference (TD) learning: each update fits a target that subsequent updates can change. We introduce a matrix-free Jacobian probe of stiffness, the derivative channel’s parameter sensitivity relative to the output channel. In the TD interpretation, stiffness scaled by the time gap measures how strongly an output-visible parameter change can alter the bootstrapped target relative to the prediction being fitted. The probe therefore exposes target sensitivity that a small prediction residual need not constrain. Across CIFAR-10 and ImageNet-256 checkpoints with up to 677M parameters, stiffness grows by one to two orders of magnitude, especially at large gaps, while the weighted loss remains bounded and FID improves. Controlled experiments distinguish gradual stiffening without bootstrapping from additional amplification under bootstrapped large-gap training whose onset tracks learning-rate warmup. Released α-Flow and Decoupled MeanFlow checkpoints achieve lower large-gap self-consistency error and better one-step FID than MeanFlow despite retaining high stiffness. Increasing large-gap supervision improves the best CIFAR-10 finetuning FID from 6.03 to 3.95 but degrades an ImageNet-256 transformer. These results connect sampling and learning-rate choices to hidden changes in bootstrap sensitivity and give a diagnostic for studying the training dynamics and transfer limits of one-step generators.

Primary area:
generative models
Keywords:
Flow Matching, MeanFlow, Temporal-difference Learning, Bootstrapping

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9pzVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Presents an interpretation of the MeanFlow objective and formulation in the language of Temporal Difference (TD) learning and propose a Jacobian formulated probe of vector field sensitivity, or ‘stiffness’, with respect to light input and output variation. Authors explain why this factor is hidden from the MeanFlow learning objective, and explore how curricula designed to alter stiffness impact downstream FID scores. They identify high stiffness (or low sensitivity) with more reliable large-gap predictions.

Critical strengths

Strengths:

(1) Interpreting the MeanFlow objective in the language of Temporal Difference learning appears to be a genuinely novel approach.

(2) The resolution of the Jacobian without full formulation appears to be novel (as far as I am aware) and seems like an insightful approach to studying the dynamics of flow fields.

(3) Characterizing new measures of desirable or undesirable latent behavior in flow fields is a valuable goal, and measures of field sensitivity are intriguing.

Critical weaknesses

Weaknesses (see Questions for elaboration):

(1) Swathes of the main body text are readily identifiable as LLM generated. While the AI use statement does disclose this use for “readability”, it in fact has the opposite effect. This weakness obscures the manuscripts content and contribution.

(2) It seems that the stiffness gg is trends similarly to the inverse of the learning rate. This likeness is not mentioned or explored, yet appears to be a genuine source of confounding.

(3) Sensitivities to noise scales and solver budgets are alluded to but are not explored or discussed in the main body.

(4) The main body does not clearly communicate what the dosing/treatment method actually is, or what its expected experimental results would be.

(5) The results presented are mixed, and it is unclear what robust connections if any gg has to training dynamics or outcomes.

(6) The results are strictly confined to MeanFlows, but the probe formulation may not be strictly limited to this setting.

Questions for authors

Questions:

(1) It’s not clear from the main body what exactly hh is. What is g@h=0.90g@h=0.90, and how are readers meant to know? The first reference to the term appears to be line 301-302. It is referred to in many places throughout the text, so it seems important, but it is not clearly defined. What is hh?

(2) Sensitivity to the gg solver budget is mentioned in line 258. How sensitive is gg to this budget? Similarly, how sensitive is gg to the δθ\delta\theta step size?

(3) Finding 1 (lines 295-307) claim that “sensitivity accelerates after learning rate warmup” and peaks before FID and notes a similar result on ImageNet without the warmup period. However, in both of these cases the learning rate is dynamic, and the observed stiffness uptrend is consistent with the inverse of a decreasing learning rate. Is gg just measuring the inverse learning rate or some rescaling of it? Intuitively the learning could be viewed as iteratively refining the field, in which case the we should expect the stiffness gg to be connected with it (ie, the model isn't impacted as much by weight perturbations smaller than the LR, ∂uθ/∂θ≤lr\partial u_\theta / \partial \theta \leq lr). Again here, the δθ\delta\theta size becomes important to document (see question 2). If not, why not? A connected question could be to be rephrase the above around the sensitivity of the integrator method, rather than the LR.

(4) Finding 1 and elsewhere claims that gg increases as generation FID increases. This appears confounded by the lr note in question 3, but more importantly, it’s not actually clear if this claim is true. As the abstract points out, large gap supervision decreased FID performance on ImageNet. Extracting this signal more cleanly would illuminate the connections of gg with training outcomes.

(5) The dosing/treatment processes are not clearly described in the main body, but the setting is fairly clear - transitioning from standard CFM training to MeanFlow training and tracking the magnitude ∣t−r∣|t-r| and the stiffness gg as training continues. The dosing and supervision procedures in section 5 are not clearly described. However, their effects and the authors’ conclusions seem in line with a general intuition: supervising large gaps helps, you cannot remove small gaps altogether, and delayed treatment can destabilize established networks. What is the treatment, and how precisely is it done?

(6) Should readers expect different results for Rectified Linear Flows or other flow-family methods? The probe approach may not be strictly limited to MeanFlow conceptually. This may be best left for future work.

3 comments

We thank the reviewer for the detailed questions and apologize for the poor readability of the submission. We uploaded a revised manuscript, and the section, table and figure numbers below refer to it.

What is hh (Question 1).

h=t−rh=t-r is the time gap between the two times of a MeanFlow pair, and h=1h=1 is one-step generation. gg is largest at large gaps (Tables 3, 6 and 7). The revision defines hh in Sections 1 and 3.1.

Sensitivity to the solver budget and other settings (Question 2, Weakness 3).

This is now in the main text (Section 3.3, Table 5). The solver budget and the probe batch change the absolute value of gg, so every value is a lower bound and we quote medians over three batches. The ratios of medians that our claims rest on change by less than a factor 1.5. The probe has no step size, because it uses exact Jacobian-vector products (Algorithm 1).

Is gg the inverse learning rate (Question 3, Weakness 2).

No. gg is the largest change in ddtuθ\frac{d}{dt}u_\theta, the derivative term of the target, per unit change in the output uθu_\theta. It is computed on a frozen checkpoint, and the learning rate does not enter it. As an analogy, gg is to MeanFlow's bootstrap what sharpness, the largest Hessian eigenvalue, is to gradient descent. Both are properties of the current weights. Gradient descent is guaranteed stable when sharpness is below 2/η2/\eta, which contains the learning rate. The bootstrap is guaranteed to contract when (t−r)g<1(t-r)g<1, which contains only the time gap. In our runs the learning rate never decreases, while gg rises (Table 2, Appendix C).

Does gg rise as FID improves, and how is gg connected to training outcomes (Question 4, Weakness 5).

We claim no relation between gg and the outcome of training as measured by FID. gg rises while FID improves within a run (Table 2), and α\alpha-Flow and Decoupled MeanFlow (DMF), which generate better than MeanFlow, are not less stiff (Table 3). gg is not a quality metric. We claim two links to training dynamics. First, gg grows because of bootstrapped large-gap training (Table 2). Second, the large gaps, where the gain (t−r)g(t-r)g is largest, are the first to break when the target contains an error (Section 5, Fig. 2). Although gg does not track FID, it sets the gain (t−r)g(t-r)g of a bootstrapped target. Many one-step models are trained with such a target, as in TD learning, and the probe can be applied to them (Question 6).

What is the treatment (Question 5, Weakness 4).

gg is not a quality metric, so it does not need a treatment. We focus on the measurement of gg and on its test in Section 5.

Other flow families (Question 6, Weakness 6).

The probe is not limited to MeanFlow, and the way to measure gg is our first contribution (Algorithm 1). It runs on a frozen checkpoint without any training and can be applied to any model whose target contains the network's own output or derivative, such as consistency, shortcut and flow-map models (Section 6).

Reviewer_9pzVOpenReview ↗

Response to Authors

Clarity: My primary weakness remains unaddressed.

The main body text after revision is not much easier on the eyes, and appears to most likely be largely the output of an LLM. The persistent lack of clarity obfuscates the methods, contributions, and experiments.

Relation of gg with the learning rate: I am unconvinced by the authors reasoning.

Just because the learning rate does not enter the calculation of gg does not mean that it is isolated. The learning rate determines the structure and resolution of the loss landscape learned by uθu_\theta. However, with Table 2 and some inline text clarifying constant learning rates in training, I agree that it is unlikely at play in their experiments. I would be surprised if future examinations revealed that gg is independent of learning rate magnitude and schedule.

Contribution: The stated contributions are muddled.

The stated three primary contributions in the introduction appear to be approximate restatements of the same core, that “gg is a neat way to measure flows”. I agree that it could be, if it measured anything other than itself, but the authors do not provide convincing evidence that gg ultimately measures anything other than itself. There are no robust connections with training dynamics, impacts on resolution at inference time, or effects on evaluations revealed. In short, it is simply not clear why gg matters.

At this time, I do not feel that my score has changed.

Contribution. This is an analysis paper. It reads MeanFlow training as a TD bootstrap whose loop gain (t−r)g(t-r)g plays the part of the discount γ\gamma in stability (Table 1, Section 3.2), and it measures this gain on trained models (Algorithm 1). As with γ<1\gamma<1 in TD, a gain below one guarantees that the linearized bootstrap contracts, even with a delayed, averaged or periodically refreshed target (Appendix B). MeanFlow's gain is far beyond this bound, reaching about 10310^3 at large gaps (Section 4), yet training still succeeds. gg is the quantity in this stability condition, not a measure of itself. Its growth comes from bootstrapped large-gap training (Table 2), and once the target lags, the large gaps fail first and one-step FID rises to between 171 and 251, against 12 to 34 online (Section 5, Table 10). The paper never claims that gg improves FID, ranks models or yields a better training recipe.

Learning rate. We agree that gg, like any property of trained weights, depends on the whole recipe, and the paper does not claim otherwise. Its claim is that the rise of gg is not a rescaling of the learning rate. The rise continues after the rate becomes constant, and a tenfold lower rate does not raise gg (Section 4, Appendix C).

Clarity and LLM use. The revision rewrote the main text, defines hh and the control where they first appear (Section 1) and moves the probe into the main text (Algorithm 1). As ICLR 2027 requires, the AI use statement discloses our use of LLMs in writing, and we checked every sentence ourselves. The comment names no sentence that is unclear, and the reviewer is welcome to name one.

Reviewer_AzpdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper brings the temporal-difference (TD) learning to MeanFlow models. It asks if the MeanFlow model is sensitive to the change of objectives when stable and small loss. It introduces scalable stiffness probe g(r,t)=∥JBJA+∥2g(r,t)=\|J_BJ_A^+\|_2 to diagonize the problem. The paper shows that the stiffness probe can brings FID improvements on CIFAR10, but not on ImageNet.

Critical strengths
  1. Interesting and well defined analysis from TD perspective.
  2. Clear motivation and novel analysis probe.
  3. Transparent to limitations
Critical weaknesses
  1. Not work in scale, limited to interpretability across models
  2. The paper only shown limited evidence for the effectiveness of the stiffness probe. The stiffness probe is not well established across various daatsets and models.
Questions for authors
  1. It's an interesting analysis. Have you insvestigate why stiffness probe doesn't work on some model / ImageNet? Is the stffness is a necessary condition for a good model?
  2. Could you vary the time gap to investigate whether lower measured stiffness is associated with more accurate predictions over short intervals?
1 comment

We thank the reviewer. We uploaded a revised manuscript, and the table and figure numbers below refer to it.

Is stiffness necessary for a good model (Question 1).

We do not know, and the paper does not claim it. Every one-step model we probed is stiff at large gaps, and α\alpha-Flow and Decoupled MeanFlow (DMF), which generate better than MeanFlow, are not less stiff (Table 3). The flow-matching models we probed are less stiff (Tables 2 and 3), but they were never trained on large gaps, so they cannot show necessity.

Is lower stiffness associated with more accurate predictions over short intervals (Question 2).

Only in part. At short gaps gg is smallest (Tables 3, 6 and 7) and so is the unweighted training loss (Table 4), but both follow the gap, and the loss compares the network with its own target, not with the true average velocity. We therefore cannot link low stiffness to accuracy. What we can say comes from Section 5. MeanFlow normally computes its target from the current weights. When we compute it from older weights instead, short gaps are barely affected while large gaps fail (Fig. 2).

Not work in scale, limited to interpretability across models (Weakness 1).

Sampling more large-gap pairs did not improve FID on ImageNet, as the reviewer notes. Our contribution is the analysis itself. We show that MeanFlow training is a feedback loop, we give a probe that measures its gain (t−r)g(t-r)g on frozen checkpoints (Algorithm 1), and we find that this gain is far above one at large gaps and grows during training, on CIFAR-10 and on ImageNet-256 (Tables 2, 3 and 7). Section 5 then tests what this gain means (Fig. 2).

Breadth of the evidence (Weakness 2).

We measured gg on two datasets (CIFAR-10 and ImageNet-256), on two architectures (a convolutional U-Net and Transformers), and on three methods, MeanFlow and its follow-ups α\alpha-Flow and DMF (Tables 2, 3 and 7). More importantly, we contribute the way to measure gg. The probe runs on a frozen checkpoint without any training (Algorithm 1), so it can be applied to any model trained with a bootstrapped target.

Reviewer_EqDMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper propose to study the failures of Meanflow training under latent stiffness.

Critical strengths
  • The paper propose an interesting metric to try and explain the recent developments of mean flow follow ups.
  • Understanding why Meanflow is sensitive to sampling or other design choices is interesting and could lead to better designs in the future
Critical weaknesses

My main concern is that the paper don’t actually show evidence that the metric they propose is actually useful:

  • The authors observe that the measured thifteness increase while FID decrease. But actually they just observe that during training, so it could be just an artefact of training, doesn’t guarantee causation. If they want to show a relationship they should compare the 2 metrics for equally trained models having different performances.
  • Improved methods doesn’t have consistent descreased thifteness: So what does that mean? This observation makes it pretty hard to understand if the proposed metrics has any practical use. We would need to see some predictive behiaviour: A change in the probe could lead to a prediction of failures of training. Or some causality intervention

Overall this paper is super hard to parse. Finishing reading the paper I cannot derive a coherent conclusion about the proposed metrics. The paper lacks rigorous and coherent analysis.

Questions for authors

I'm not sure how the authors see their work. What's their conclusion. Reading this paper puzzles me after trying to understand the defined metric that it doesn't seem to correlate with any method improvement.

3 comments

We thank the reviewer and apologize for the poor readability of the submission.We uploaded a revised manuscript, and the section, table and figure numbers below refer to it. We restate what gg measures and what the paper concludes.

What gg is and what we conclude (Question).

An analogy may help. gg is to MeanFlow's bootstrap what sharpness, the largest Hessian eigenvalue, is to gradient descent. Both are properties of the weights that are invisible in the loss and do not rank models. Sharpness says how far gradient descent is from the regime where a step is guaranteed to be stable. gg says how far the bootstrap is from the regime (t−r)g<1(t-r)g<1 where it is guaranteed to contract, that is, how strongly training can amplify an error in its own target. Our first contribution is the way to measure gg. The probe is matrix-free, needs no training and runs on frozen checkpoints (Algorithm 1). It can be applied to any model trained with a bootstrapped target. With it, the revision draws three conclusions.

  1. The loop gain (t−r)g(t-r)g plays the role of the discount γ\gamma in the TD bootstrap (Table 1). TD contracts because γ<1\gamma<1. In MeanFlow the corresponding condition (t−r)g<1(t-r)g<1 fails at large gaps from the first checkpoint, and the gain grows to about 10310^3 while the logged loss stays flat and FID improves (Fig. 1, Tables 2 and 4).

  2. The growth comes from bootstrapped large-gap training (Table 2), and the better methods are not less stiff (Table 3). As the reviewer observes, gg does not correlate with method improvement. It is not a quality metric.

  3. When the target is computed from older weights, the large gaps fail first, at lags far shorter than TD methods use (Fig. 2, Table 10).

Artefact of training (Weakness 1).

We claim no relation between gg and FID. A control with the same recipe, learning rate and steps, trained on instantaneous pairs only, stays between 17 and 70 where MeanFlow reaches 1,147 (Table 2), so the rise is not an effect of training time. Table 3 compares trained models with different performance. α\alpha-Flow and Decoupled MeanFlow (DMF) reach a better one-step FID than MeanFlow and are as stiff or stiffer.

Causal intervention and prediction (Weakness 2).

That α\alpha-Flow and DMF are not less stiff means that stiffness is not the defect they repaired (Table 3).

In standard MeanFlow training the weights are updated at every training step, and the target is always computed from the current weights, so the target changes at every step together with the network. In our intervention (Section 5) we deliberately compute the target from older weights. We keep a copy of the weights that is refreshed only every KK steps and compute the target from this copy, so the target lags behind the network by up to KK steps. If the gain (t−r)g(t-r)g is how strongly an error in the target is amplified, this lag should break the large gaps first.

It does, and at very short lags. TD learning works with a target network that is refreshed every 10410^4 steps (DQN). In MeanFlow on CIFAR-10 a lag of 25 steps already destroys one-step generation, with the large gaps failing first (Fig. 2, Table 10). On ImageNet-256 B/4 a lag of 250 steps destroys every gap within 5000 steps from our 100k checkpoint, and a 400k checkpoint tolerates longer lags (Appendix E). We refer to these for the detailed numbers.

I'm still not convinced what's the purpose of g. The defined quantity doesn't seem to correlate with anything. The authors measure a quantity, but for me this has no use. This quantity just tell me if meanflow models are stiff or not. But why would i care? What's the consequence of having a stiff model or not?

AuthorsOpenReview ↗

An analysis paper, judged by a claim it does not make

This is an analysis paper. It reads MeanFlow training as a TD bootstrap whose loop gain (t−r)g(t-r)g plays the part of the discount γ\gamma in stability (Table 1, Section 3.2), and it measures this gain on trained models (Algorithm 1). As with γ<1\gamma<1 in TD, a gain below one guarantees that the linearized bootstrap contracts, even with a delayed, averaged or periodically refreshed target (Appendix B). MeanFlow's gain is far beyond this bound, reaching about 10310^3 at large gaps (Section 4), yet training still succeeds.

The follow-up sets these findings aside and judges gg by a practical use, but the paper never claims that gg improves FID, ranks models or yields a better training recipe.

The ICLR 2027 reviewer guidelines ask reviewers to judge whether it supports its claims and whether its results are correct. Judging it by whether one reviewer personally cares is contrary to these guidelines, and we have raised this with the AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.