acceptodds
Under review as a conference paper at ICLR 2027

Lagrangian–Hamiltonian Flows for Video Prediction and Image Generation: A Symplectic Perspective

Abstract

We introduce LHFM, a geometric framework for learning image dynamics that combines Lagrangian image representations with Hamiltonian dynamics. These structures play a central role in classical mechanics, symplectic geometry, and geometric quantization. LHFM represents images as exact Lagrangian graphs and models their evolution through image-dependent Hamiltonian flows. This construction provides a geometric interpretation of image-space dynamics by connecting them to the Hamiltonian evolution of Lagrangian submanifolds. The framework supports both video prediction, our primary application, and image generation. We propose a novel deterministic video prediction method based on the Hamiltonian evolution of Lagrangian image representations. This video variant, LHFM-V, is a recurrent model that achieves the lowest reported FLOP count among the compared recurrent models with similar prediction accuracy. The image variant, LHFM-I, shows that the same construction is compatible with flow matching: in a matched experiment, it attains a lower FID than the flow-matching baseline.

Primary area:
generative models
Keywords:
Flow matching, Generative modeling, Lagrangian submanifolds, Hamiltonian dynamics, Symplectic geometry, Image generation, Video prediction.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tA5pOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Lagrangian–Hamiltonian Flow Matching (LHFM), a geometric framework that represents images as exact Lagrangian graphs and models their evolution through Hamiltonian dynamics. It develops two variants for deterministic video prediction and image generation, demonstrating competitive prediction accuracy with low computational cost and improved FID over a matched flow-matching baseline.

Critical strengths
  1. The paper introduces an interesting geometric perspective that explicitly separates spatial transport from changes in pixel values.
  2. The paper provides theoretical characterization of the proposed representation and its Hamiltonian dynamics.
Critical weaknesses

I should acknowledge that I am not very familiar with Lagrangian graphs, so my concerns are mainly on the evaluation.

  1. The empirical evaluation is limited to Moving MNIST and CIFAR-10, making the generalizability of the approach unclear. The compared baselines are also quite old, excluding several more recent baselines such as SimVPv2.
  2. The comparisons rely on results reported in prior works rather than protocol-matched experiments, limiting the strength of the efficiency and accuracy claims. In addition, the CIFAR-10 metrics reported in the paper are quite poor compared to results of state-of-the-art generation models. For instance, in the official repository of Flow Matching (https://github.com/facebookresearch/flow_matching/tree/main/examples/image), the recorded unconditional FID performance on CIFAR-10 is 2.07 with nfe=50.
  3. The benefits of the Lagrangian–Hamiltonian formulation over simpler transport–source parameterizations are not sufficiently isolated through ablation studies.
Questions for authors

Please see weakness.

1 comment

As the rebuttal period is drawing to a close, the authors have yet to address my concerns. After considering the other reviewers' comments, I am inclined to lean toward rejection.

Reviewer_wCfBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper represents images as exact Lagrangian graphs and derives an image update with transport and source fields. LHFM-V predicts these fields recurrently and integrates them to produce future frames. LHFM-I uses the same parameterization in independent conditional flow matching. The experiments cover ten-frame prediction on Moving MNIST and unconditional generation on CIFAR-10.

Critical strengths
  1. The image graph representation connects Hamiltonian dynamics to an implementable transport-source model. This is the main contribution beyond the existing Hamilton-Jacobi connection and transport-source formulations. The authors also state that every smooth image trajectory admits a Hamiltonian realization and that the decomposition is not unique. These qualifications make the scope of the geometric contribution clear.
  2. The CIFAR-10 comparison uses the same backbone, objective, training budget, and evaluation solver. LHFM-I adds only 2,306 parameters to a model with about 39.6M parameters. FID falls from 3.8241 to 3.5809 at 256 NFE, and gains also appear at 64 and 128 NFE. This gives useful evidence for the proposed parameterization.
Critical weaknesses
  1. The video experiments do not isolate the contribution of the transport-source update. The model also includes temporal attention, future-feature prediction, recurrent feedback, and coarse supervision. Table 3 reports the complete system but has no comparison with a source-only, direct-residual, or ordinary warp-plus-source update using the same architecture. The reported MSE of 18.6 therefore does not show what the proposed update adds. Video evaluation is limited to Moving MNIST, so the practical evidence for this part of the method is still narrow.
  2. The efficiency comparison uses FLOP counts from different papers. LHFM-V reports 13.1G, versus 15.3G for PhyDNet, but the counting protocol is unclear. Appendix B.3 uses 24 stencil recurrences per internal interval. The number of intervals varies with the predicted transport field. It is unclear whether 13.1G includes all this work over the full prediction sequence. The efficiency claim needs consistent accounting of the solver and the rest of the model.
Questions for authors
  1. How do source-only/direct-residual and warp-plus-source updates perform with the same backbone, training budget, and auxiliary losses?
  2. Does 13.1G cover the full ten-observed/ten-predicted sequence? Please specify the profiler, multiply-add convention, and solver operations counted. What are typical subdivision counts and end-to-end runtimes under the same measurement protocol?
  3. How stable are the CIFAR-10 gains and Moving MNIST results across training seeds? The current results use one training seed per setting.
  4. Why is SSIM evaluated with data_range=2 for images in [0,1]? Which Table 3 baselines use the same settings? A consistently evaluated subset would help clarify the comparison.
2 comments
  1. Our computational budget does not allow additional full-training comparisons of source-only/direct-residual and warp-plus-source updates using the same backbone. However, the zero-transport ablation reported in the paper provides a related comparison for image generation. Setting transport to be zero reduces the image velocity to the standard one, recovering the standard flow-matching velocity parameterization without an explicit transport term. This ablation examines the effect of including transport in LHFM-I.

  2. Yes, 13.1G cover the full ten-observed/ten-predicted sequence. The measurement uses fvcore==0.1.5.post20221221, FlopCountAnalysis, batch size 1, FP32, evaluation mode, and inputs and outputs of shape 1×10×1×64×641\times10\times1\times64\times64. One fused multiply-add is counted as one FLOP, following fvcore’s convention. The measured total is 13.123985408G.

  3. Due to computation budget, we are not able to do more experiments.

  4. We retained the legacy SSIM defaults of the referenced evaluation implementation. In scikit-image 0.19.3, omitting data_range for floating-point inputs derives the range from the dtype interval [-1,1], resulting in data_range=2.

Thank you for your response, which has addressed some of my concerns. I understand the computational constraints; however, without additional experiments conducted under the same protocol, I do not believe fairness can be guaranteed.

Reviewer_h3EROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

AI Statement: I do not use AI for generating reviews - that is a abdication of duty, and I do not think LLM generation has a place in academic discourse - LLM generation in papers is plagiarism and poor scholarly practice.

This paper presents a Lagrangian–Hamiltonian Flow Matching approach and then applies it to image scenarios in a reduced format.

Critical strengths

This paper takes standard results from symplectic geometry, and maps images into that symplectic geometric structure. This approach has been handled on a number of occasions, some in more general ways - working in latent spaces rather than pixel spaces, but this paper handles the case as a specific flow over the image pixel surface.

Critical weaknesses

This is a general formalism that has been tackled before. The paper spends significant time elaborating mathematical concepts that then go unused in the actual experiments in the paper, which reduce, basically, to a form of optical flow. This is disappointing, and makes the paper fairly incoherent. It builds the substantial (known) Jacobi-Hamiltonian mathematical framework. It then dives in with symplectic shorthand and undefined terms that are not helpful for the general reader (graph(dS) is not defined for one of many possible examples - yes I know what it means, but you should not believe that every reader has exactly that background - graph means something completely different in most other uses in ML). The maths and context is very unrelated to the actual target of the paper. For example "Whereas a classical phase-space point specifies position and momentum simultaneously, the Heisenberg uncertainty principle prevents a quantum state from being sharply localized in both variable". How is this relevant? Why did the authors even think to mention that? The maths feels like it is textbook reconstruct that does not really relate to the paper application or the audience. The paper then portrays the image as the graph of pixel values at location and applies the Hamiltonian flow to the pixel space image Lagrangian. Lemma 4.1 is presented as a contribution without reference, but it is a (specialisation?) of a standard result. The only new component of it is adding in the specific generating function. That this is used without reference to the underlying literature worries me. The introduction of the probe is extensively described, but is discarded before any model is built.

More importantly, why are we doing all this? What is the point? No Hamiltonian is ever constructed. The Hamiltonian is effectively time dependent and describes dynamics that fits explicitly to the data? What purpose or gain does all this serve?

At the end of the day is a large, rather disconnected, not always relevant, mathematical construct. Sadly most of that is then unused in the actual paper experiments. The experiments reduce to simple transport in pixel space. Applied to images sequences this is fairly useless (it cannot handle temporary occlusion for example), as the method cannot maintain latent information. The application ends up being a form of restricted optical flow. Sadly the paper feels like it is a large well known geometric construct built into a paper looking for a problem. But this large construct mostly collapses on application. Equation 7 is a brightness change equation. This has been used in optical flow methods for decades, and yet this receives no mention. It also feels like the theory constrains nothing. Proposition A.6 makes every path realizable (the probe drops out, and LHFM-I effectively can learn the same probability path as I-CFM).

Moving on to the practice, the experiments are hard to understand in the context of this paper. There is a huge disconnect and it is not clear exactly what is happening in any of the experiments and how they relate to the theory that is established. In image generation, ias described, we have a simple flow matching scenario. But what is the claim about what this construct provides that distinguishes itself from existing flow approaches? Given the lack of constraint in the overall construct it is not obvious to me it ends up being any different to any other flow matching modelling structure?

The background work in this area is extensive, and yet the background work section of the paper is fleeting. There is huge work on Hamiltonian-learning and motion–content literature that receives no comment or discussion including the following (in the Hamitonian and Lagrangian settings - there is also the flow background and optical flow literature)

Greydanus et Al. (2019) Hamiltonian Neural Networks. Toth et Al. (2020) Hamiltonian Generative Networks Cranmer et Al. (2020) Lagrangian Neural Networks. Khan and Storkey (2022) HALO, Hamiltonian latent operators for content–motion separation in image sequences. Allen-Blanchette et Al. (2020) LagNetViP, a Lagrangian network for video prediction.

These are not doing exactly what this paper does - they are more restrictive but in that sense make use of the conservation properties of the Hamiltonian construct more strongly than this paper does.

There are references that seem out of place (Kennard (1927) and Robertson (1929)) that I can't think why an author would actively add to such a paper, and others that appear to misunderstand of the content of the citations.

Questions for authors

I have no questions.

2 comments

We thank the reviewer for the detailed feedback. We address the concerns about notation, the role of the geometric framework, the physical motivation, and related work below.

  1. The graph of an exact one-form is a standard example of a Lagrangian submanifold, we recognize that this notation may be unfamiliar to readers outside symplectic geometry. However it is hard to define every terminology used in this paper.

  2. We encode each image as a graph Lagrangian, whereas a general Hamiltonian flow does not necessarily preserve a global graph representation. Lemma 4.1 proves that it will preserve such property iff the Hamilton–Jacobi compatibility condition is satisfied. This compatibility is what allows the geometric evolution to be expressed in the simpler form used in our model.

Thus, the simplicity of the resulting equation does not mean that the geometric framework is unused, it is exactly the the geometric construction we chose that allows us to reduce the construction to such simple forms. This Lemma 4.1 is obvious to expert, however, we didn't find an explicit proof in existing literature.

  1. The reference to the Heisenberg uncertainty principle was intended solely as conceptual motivation for the importance of Lagrangian submanifold, and our construction echoes the philosophy. We will shorten this discussion and make its motivational role explicit, while explaining the mathematical reason for using graph Lagrangians directly.

  2. We acknowledge that the related-work discussion is insufficient and thank the reviewer for identifying these relevant papers.

Reviewer_h3EROpenReview ↗

Response to rebuttal

"it is hard to define every terminology used in this paper" -> actually it is not that hard if you make it a priority; I know because I very much encourage my students to do this in equally technical settings, and (eventually) to most part they are able to do a good job. I highly recommend making this priority to improve writing quality.

I am not sure any of the main concerns in the paper about the disconnect between the theory, the point, and the experiments are discussed or handled. I do not think I can recommend acceptance at this stage. If there is a real sweet spot to this thinking or approach, then I do commend the authors to push forward with a tighter and clearer explanation and demonstration of what they are doing and saying.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.