acceptodds
Withdrawn from ICLR 2027

Same Fixed Point, Different Learning Dynamics: Error Transport in Finite-Step Flow Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training objectives for few-step generation can share a solution yet differ in the error feedback they supply during learning. For a frozen teacher, integrated endpoint labels stay fixed, whereas stopped differential labels return a directional derivative of the student's error. We show that an order- horizon error receives diagonal residual gain , with material transport coupling different orders. These unequal gains let one global step size contract some modes and destabilize others in idealized target-space learning. In image networks, the kernel and AdamW reshape these gradient gains before they reach the learned function. Paired training across three image domains favors endpoint supervision; at five sampling steps, CIFAR-10 FID is 7.590 versus 21.133 for stopped-JVP training after 3,000 updates. Endpoint's advantage persists at equal measured time, including label production on shared, reusable training states. Choosing a target thus also chooses how different horizon orders are corrected.

Primary area:
generative models
Keywords:
generative models, flow map learning, consistency models, mean flow, learning dynamics, error transport

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yHH4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

paper analyzes why finite-step flow-learning objectives with the same fixed point can exhibit different learning dynamics, derives horizon-order-dependent error amplification and stability behavior

Critical strengths

The paper provides a clear explanation of why objectives with the same fixed point can have different learning dynamics and derive order-dependent error gains.

The predicted error amplification and stability behavior are validated in the experiments and further supported by image-generation experiments across multiple datasets.

Critical weaknesses

The main k+1k+1 error amplification and stability results are derived in an idealized target-space setting, while the paper acknowledges that the network Jacobian and optimizer can substantially reshape these dynamics.

The image experiments freeze the teacher direction to isolate error feedback, whereas full MeanFlow-style objectives may also learn this direction and introduce additional coupled dynamics. Therefore, the experiments do not fully establish the observed advantage of endpoint supervision.

Questions for authors

The k+1k+1 error amplification and stability results are derived in an idealized target-space setting; under what conditions can these results reliably predict the actual training dynamics after the network Jacobian and optimizer reshape the target-space response?

Reviewer_BJTgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how two different training objectives affect learning finite-step flows from a fixed teacher. The authors compare supervision using fixed labels obtained by integrating the teacher with differential supervision using derivatives of the current student. They analyze how these objectives respond differently to prediction errors, derive theoretical results for different horizon orders, and investigate how these effects are modified by neural network optimization. The analysis is supported by numerical experiments, including image generation on CIFAR-10, FFHQ-64, and AFHQv2-64, where fixed endpoint supervision achieves better results in the studied settings.

Critical strengths

None

Critical weaknesses

1. Poor presentation and insufficient explanation of results

The paper is very difficult to follow. My main concern is not the technical complexity itself, but the lack of clear motivation, intuitive explanations, and interpretation of the results throughout the manuscript.

The Introduction immediately introduces specialized concepts such as differential labels, horizon-order errors, diagonal gains, and material transport, without adequately explaining their meaning or importance. The main contributions are mentioned, but are not clearly motivated or synthesized, making it difficult to understand the overall scientific message.

The presentation problems continue in Section 3, where even central mathematical objects and training objectives are introduced with very limited explanation. For example, the meaning and motivation of the student-dependent label in Eqs. (2), (3), which is fundamental to the paper, are not sufficiently explained. Section 4 follows a similar pattern: the authors introduce mathematical constructions and derive their properties without adequately motivating why these particular quantities are studied. In particular, Theorem 1 introduces formal jets and their spectrum, but the discussion provides limited intuition about why this mathematical characterization is important.

Finally, Section 6 contain many experiments, figures, and numerical measurements. Although the authors discuss individual findings and their immediate implications, I find the broader interpretation and synthesis insufficient. It remains difficult to understand what generalizable knowledge we should extract from the many technical results. The very short Conclusion largely restates the main findings without sufficiently discussing their broader significance.

Overall, the manuscript places too much emphasis on presenting technical calculations and measurements and too little on explaining their motivation, interpretation, and scientific significance. The reader is repeatedly left to reconstruct the scientific argument independently.

2. Unclear novelty and scientific significance

I have serious concerns about the novelty and significance of the contribution. The general observation that different training objectives can share the same desired solution while inducing different learning dynamics is already well known. Previous works have investigated closely related optimization difficulties. In particular, CMT (Hu et al., 2026) analyzes gradient discrepancies between practical consistency-training objectives and oracle regression, while AlphaFlow (Zhang et al., 2026) identifies gradient conflicts in MeanFlow training. The authors briefly discuss these studies in Related Work, but do not sufficiently explain what important questions remain unanswered or how the present paper advances existing understanding.

The main distinctive contribution appears to be the horizon-order characterization of error feedback, including the resulting stability analysis and the role of material transport. However, I do not find its broader significance convincingly established. The main theoretical result is mathematically rather straightforward: the spectral characterization in Theorem 1 follows directly from the triangular structure of the horizon-order operator. Its stability condition applies to an idealized target-space relaxation, rather than directly to practical neural network optimization. Although the authors investigate how the network and optimizer modify the predicted gains, it remains unclear what general or predictive conclusions about practical training dynamics follow from this analysis.

The empirical results are extensive and demonstrate substantial differences between the two supervision methods. The authors also provide gradient diagnostics supporting their proposed mechanism. Nevertheless, it is not clear to what extent the horizon-order analysis explains the observed performance differences, or what important new understanding it provides beyond existing analyses of optimization instability and gradient conflicts. The advantage of endpoint supervision is demonstrated in the studied settings, but the conditions under which this advantage should be expected more generally remain unclear.

Overall, despite the substantial technical effort and empirical evidence, I find the scientific significance of the specific theoretical contribution insufficiently justified. The paper would benefit from a much clearer explanation of what new conceptual understanding its analysis provides, how it advances previous work, and which conclusions extend beyond the particular settings studied.

Questions for authors
  1. Could the authors clarify what the main new scientific insight of their horizon-order error transport analysis is, beyond existing literature? In particular, what generalizable or transferable knowledge does this analysis provide beyond the specific training objectives and experimental settings studied in the paper?
Reviewer_hKtCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper mainly focuses on the analysis of endpoint vs. JVP-based (derivative) objectives in trajectory-based few-step distillation of flow models. Although both share the same theoretical fixed point, the authors show that stopped-JVP targets introduce error feedback with order-dependent gains, potentially affecting optimization stability. Experiments mainly use small scale image generation settings, and report that endpoint supervision achieves better convergence and generation quality.

Critical strengths

The analysis provides an interesting perspective on how the order of flow-map prediction errors with respect to the time horizon affects their contributions to the training residual.

Critical weaknesses

My main concern is that the connection between the theoretical analysis and practical instability is not sufficiently established.

  • Theoretically, higher-order error amplification does not necessarily lead to instability, it could also accelerate the correction of difficult errors, and stability could be improved by tuning the learning rate, as implied in the "idealized relaxation" equation.
  • This order-dependent residual analysis seems to be broadly applicable to trajectory-based methods beyond JVP-based methods. For example, the same residual gain also seems valid in pi-Flow, FastGen-PDD, but empirically these methods are known to be very stable, much more stable than JVP-based methods. Thus the theory about residual gain leading to instability does not sound convincing to me.
  • The experimental comparison is limited. The main experiments focus on fixed endpoint regression vs. a simplified stopped-JVP objective, on small scale dataset and models. Modern distillation methods are not evaluated, and the results appear to favor direct endpoint regression, a very outdated method that is known to be subpar in terms of image sharpness.

In addition, it is unclear how the analysis can guide the design or stabilization of modern few-step distillation methods, which apparently don't primarily rely on endpoint methods.

Questions for authors

Can the authors rigorously show why higher order error amplification should lead to instability in practice, and provide stronger experimental baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.