acceptodds
Under review as a conference paper at ICLR 2027

HamFormer: A Hamiltonian Flow-based SE(3)-Transformer for Conformation Optimization

Abstract

The Transformer architecture plays a central role in modern deep learning, and its SE(3)-equivariant variants have been widely applied to molecular conformation modeling and optimization. Given its importance, many methods attempt to interpret and improve the Transformer through the lens of dynamics. However, they mainly focus on a single attention head rather than the full Transformer architecture and seldom consider SE(3)-equivariance in their proposed models. In this study, we propose an improved SE(3)-Transformer architecture, called HamFormer, which revisits the design of SE(3)-Transformer through the lens of Hamiltonian flow. In particular, we define a parametrized Hamiltonian flow in the phase space of position and velocity, corresponding to optimizing an energy functional that combines an optimal transport-based potential with a damped momentum. The Lie-Trotter splitting framework of the Hamiltonian flow leads to the proposed HamFormer architecture, whose feedforward computation achieves the optimization step of the proposed energy functional. Furthermore, we combine HamFormer with Equivariant Graph Neural Network (EGNN), build a new model for molecular conformation optimization. Extensive experiments on molecular ground-state conformation prediction show that HamFormer provides a strong and physically interpretable surrogate for classic SE(3)-Transformer and consistently outperforms state-of-the-art competitors.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Hamiltonian flow, SE(3)-equivariant Transformer, Molecular conformation optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eBgdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper revisits the SE(3)-Transformer through the lens of Hamiltonian dynamics. It defines a parametrized Hamiltonian flow on the phase space of positions and velocities, corresponding to minimizing an energy functional that couples an optimal-transport-based potential with a damped momentum term; applying a Lie–Trotter splitting to this flow yields the HamFormer layer, whose feedforward computation is claimed to implement one optimization step of that functional. HamFormer is then combined with EGNN to form a model for molecular ground-state conformation optimization (refinement from an RDKit-initialized conformer), with position, relation, momentum, and scalar-feature states evolving jointly in latent space. Empirically the paper evaluates on QM9 and Molecule3D (random split) using D-MAE, D-RMSE, and C-RMSD against 2D graph models (GINE, GATv2, GPS, GTMGC) and 3D refinement baselines (SE(3)-Transformer, EGNN, ConfOpt variants, WGFormer), and adds a layerwise analysis of latent kinetic/potential energy and architectural ablations on the momentum state.

Critical strengths
  1. The dynamical derivation is a principled way to design an equivariant layer, and it buys real parameter efficiency. Rather than adding another attention variant, the paper derives the layer's computation from a Lie–Trotter splitting of a specified Hamiltonian flow, so each architectural component corresponds to a term in the energy functional (as summarized in Table 1). The payoff is measurable: HamFormer reaches lower errors than WGFormer on QM9 with 16.3M parameters against 78.9M — a ~79% reduction — which supports the claim that second-order phase-space dynamics is a more parameter-efficient inductive bias than the first-order Wasserstein gradient flow used by the closest prior method.

  2. The layerwise energy analysis is an attempt at interpretability that goes beyond accuracy reporting. Figure 3 tracks latent kinetic energy, potential energy, and total Hamiltonian across six layers, showing an overall dissipative trend with a transient kinetic surge at the stage boundary (layer H4), and the ablation that zeroes the learned momentum at inference gives a sizeable degradation (test C-RMSD 0.654 → 0.714 Å on Molecule3D). Together these are at least a coherent story linking the architecture to the dynamics it was derived from, and the momentum ablation is a genuine, falsifiable test of the design choice.

Critical weaknesses
  1. The empirical gains are small and reported without any measure of uncertainty, which is exactly the regime where the claimed advantage is hardest to defend. On QM9 the headline comparison is test C-RMSD 0.198 vs. 0.206 Å for WGFormer (about a 4% relative improvement), with D-MAE and D-RMSE differences of similar magnitude; on Molecule3D the reported improvements are likewise in the third decimal place. No standard deviations, confidence intervals, number of seeds, or paired significance tests are given, and the baseline numbers are taken from Wang et al. (2025) rather than re-run. Since HamFormer and WGFormer share the same evaluation protocol and differ by a modest architectural change, seed-level variance could plausibly account for the gap. A reviewer cannot distinguish a real improvement from run-to-run noise here.

  2. The baseline set is narrow relative to the current conformation-prediction literature, and the task is restricted to refinement. All 3D baselines belong to the WGFormer/ConfOpt lineage, plus EGNN and SE(3)-Transformer; there is no comparison with recently published equivariant generative or refinement models for conformers that would establish where HamFormer sits in 2026. The setting is also refinement-only — the model starts from an RDKit conformer and improves it — so the results do not speak to de novo conformer generation, yet the abstract's claim of "consistently outperforming state-of-the-art competitors" is stated without that qualification. Finally, the physical interpretation is currently descriptive: a monotone decrease of a latent scalar labelled "potential energy" and a transient rise labelled "kinetic energy" do not by themselves demonstrate that the network implements a Hamiltonian flow, since the quantities are learned latent projections with no calibration against actual molecular energy.

Questions for authors
  1. Please report mean ± std over at least three seeds for HamFormer and WGFormer on both datasets, together with a paired significance test on per-molecule C-RMSD. If the 0.198 vs. 0.206 gap is within seed variance, how should the claim of consistent outperformance be restated?

  2. Can you add at least one strong recent 3D conformer model outside the WGFormer lineage as a baseline under the same protocol? If not, please state explicitly that the comparison is restricted to that family.

  3. Is the latent "potential energy" in Figure 3 calibrated in any way against true molecular energy (e.g. correlation with DFT energy on the test set)? Without such calibration, what evidence supports reading the layerwise curve as physical dissipation rather than as an unconstrained learned scalar?

  4. The method is evaluated as a refinement step from an RDKit conformer. How much of the final accuracy is attributable to the initial conformer, and how does the model behave for molecules with large initial RMSD (e.g. the worst decile)? Does the advantage over WGFormer grow or shrink there?

  5. The momentum ablation (ρ = 0) degrades accuracy noticeably. Is the learned momentum interpretable in physical units, or is it better described as a learned memory term? A clarification would help calibrate the strength of the "Hamiltonian" claim.

Reviewer_K3bvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes HamFormer, a model that takes a coarse 3D shape of a molecule, generated instantly by the off-the-shelf chemistry toolkit RDKit, and refines it toward conformer that defines its true lowest-energy 3D shape.

Novelty: Each Transformer layer is modeled on a ball rolling with momentum across an energy landscape (a "Hamiltonian" system) wherein each layer does two things:

  1. a "kick": balanced attention (Sinkhorn attention) updates a hidden momentum;
  2. a "drift": a small tied layer uses that momentum to update the atom features. A learned fraction of each layer's momentum, called "Momentum memory" is passed to the next layer. These augmented Transformer blocks alternate with EGNN blocks, which perform all of the actual 3D coordinate updates. When trained with such architecture to learn lowest energy 3D conformation of a molecule, the Hamformer architecture slightly outperforms WGFormer (closest prior model in terms of architecture) on QM9 and Molecule3D dataset across most metrics while consuming about 5× fewer parameters.

Stated limitation: The authors say the implemented model no longer guarantees the physical properties, such as energy conservation or energy decrease, that motivate the design.

Critical strengths
  1. A sensible new idea. Carrying a "momentum memory" across layers, so earlier layers can influence later ones, is a reasonable and interesting addition to molecule-refinement models.
  2. The order of operations inside each layer comes from one physics-inspired recipe rather than ad-hoc choices.
  3. A much smaller model, that matches or slightly beats the previous best model, WGFormer, with roughly 5× fewer parameters (16.3M vs. 78.9M)
Critical weaknesses
  • The design is motivated by a physical system whose energy behaves in a controlled way. The authors themselves state that the implemented model guarantees neither energy conservation nor energy decrease. Yet the abstract says the network "achieves the optimization step of the proposed energy functional". * The energy plot (Figure 3) does not clearly support its conclusions. The paper describes the energy as "dissipating" toward a stable state [p.8 lines 420–425]. But, the plotted energy rises between some layers (H1→H2 and H5→H6).
  • Fig. 3 (p.9) compares the energy from one layer to the next. However, each layer has its own parameters (p.5 L251) and recomputes its pair relations (p.5 L252–256). The FFN and the EGNN stage also change the features between layers.
  • (Question) Is the plotted energy the same function at every layer, or a different one per layer? If different, what does a rise or fall between layers mean?
  • The momentum experiment does not show that momentum helps. The authors remove the momentum only at test time, from a model trained with it [p.8 line 429; Table 6, p.21].

Few related concerns:

  • The momentum "carry" is allowed to become negative [p.5 lines 264–266], which no longer behaves like friction.
  • It is unclear where the improvement comes from. In the authors' own setup, EGNN alone already achieves most of the improvement over older baselines: 0.697 vs. 0.799 Å on Molecule3D [Table 5, p.21; Table 3, p.8]. - The HamFormer blocks add about 0.04 Å, and the margin over WGFormer is about 0.01 Å. * (Question) Author should make head-to-head comparison with WGFormer-style block inside the same pipeline with a similar parameter count.
Questions for authors

(Clarification) Physics vs. implementation: Since the final model does not guarantee energy conservation or decrease, which specific benefit of the Hamiltonian view remains in the trained model? Is there an experiment that shows it? (Clarification) Momentum, trained properly: How does a model trained without momentum from the start compare with the full model? What values does the learned momentum carry in each layer, and are any negative? (Clarification) Energy plot: Since each layer has its own parameters, are the energy values in Figure 3 comparable across layers? Why does energy increase between some layers if the system is described as dissipative? (Clarification) Source of the gains: How does a WGFormer block perform inside exactly the same EGNN pipeline, with a similar number of parameters? Can each new component be removed separately? (Clarification) Reliability: Can you report results averaged over several training runs with standard deviations, given that some improvements are below 0.01 Å? (Clarification) Efficiency: How do training and inference time compare with WGFormer and EGNN, given the extra Sinkhorn balancing steps? (Clarification) Pair memory: How does the pair information passed from layer to layer change across layers? Does it keep the distance information consistent, or can its effect reverse from one layer to the next? What value of the Sinkhorn temperature (ε_OT) is used? (Clarification) Geometry in the Transformer: Why doesn't the second Transformer stage use the refined 3D coordinates produced by the first EGNN stage?

Reviewer_BZveOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes HamFormer, a Hamiltonian-inspired architecture for molecular ground-state conformation refinement. It augments invariant atom features with persistent latent momentum and evolving pair relations. A multi-head entropic optimal-transport potential supplies an interaction force, while a learned kinetic gradient converts updated momentum into feature motion, following a kick–drift reference structure. HamFormer stages alternate with EGNN stages that refine Cartesian coordinates while preserving SE(3)-equivariance. The model is trained using distance and aligned-coordinate losses. On QM9 and the random split of Molecule3D, it achieves competitive or improved geometric accuracy with 16.3M parameters, compared with 78.9M for WGFormer. The theoretical guarantees concern the reference dynamics and coordinate equivariance; the practical network does not generally inherit the reference Hamiltonian's conservation or dissipation properties.

Critical strengths

S1. Substantial parameter efficiency. The model achieves competitive geometric accuracy with 16.3M parameters, compared with 78.9M for WGFormer, a reduction of approximately 79%. Parameter efficiency is the clearest empirical benefit of the proposed architecture. S2. An explicit and inspectable reference construction. The paper derives the entropic-transport potential gradient and couples it to a learned kinetic gradient. Under fixed parameters and relations, exact transport, the stated regularity assumptions, and no momentum decay (γ = 0, or r = 1), the kick–drift update is a well-defined symplectic Euler step. The treatment of transport normalization, symmetry, and tied weights makes the connection between the energy formulation and the core updates reasonably transparent. S3. Equivariance of the complete coordinate predictor. The SE(3)-equivariance argument covers invariant latent features and relations, EGNN coordinate updates, centering, and centroid restoration. It provides a sound geometric property for the molecular predictor.

Critical weaknesses

W1. The physical interpretation is stronger than the guarantees of the implemented model. The authors acknowledge that normalization, approximate transport, signed retention, evolving relations, and the additional FFN remove the reference guarantees. However, Section 4.3 interprets latent-energy changes as dissipation toward a stable ground-state equilibrium. Figure 3 is non-monotonic, and parameters and relations vary across layers. These observations do not establish evolution under a common physical energy or convergence to a molecular ground state. W2. The ablations do not isolate the proposed mechanisms. Table 5 removes the HamFormer block and the P&T branch together. Table 6 sets momentum retention to zero only at inference, which demonstrates that the trained model relies on that pathway rather than establishing its advantage over a model trained without momentum. These controls do not distinguish the contributions of ordinary momentum, transport-based interactions, persistent relations, and the specific Hamiltonian weight constraints. Separately trained, capacity-controlled variants would strengthen the attribution of the gains. W3. The robustness of the modest accuracy gains remains unclear. Test C-RMSD decreases from 0.206 to 0.198 Å on QM9 and from 0.664 to 0.654 Å on Molecule3D, while Molecule3D D-RMSE increases from 0.652 to 0.656 Å. Table 4 reports one random seed, without repeated-run uncertainty. The results are promising, but their sensitivity to training variability is not quantified. W4. Computational efficiency is not established by parameter count alone. The reduction in parameters is substantial, but the complexity analysis retains quadratic dependence on atom count, dense pair-relation storage, and iterative Sinkhorn normalization. No measured training time, inference latency, throughput, or peak-memory comparison is reported. W5. The demonstrated generalization scope is limited. The evaluation covers QM9 and the random split of Molecule3D. It supports usefulness for the evaluated molecular setting, but does not establish generalization to unseen chemical scaffolds or substantially larger systems. The conclusion acknowledges the latter direction. More explicit scope qualification would be helpful; size-stratified analysis could further characterize where the method helps.

Questions for authors
  1. Interpretation of the energy analysis (W1). In Figure 3, are all layers evaluated using a common energy function, or each layer's own parameters and relations? Given the non-monotonic curves and the implementation caveats, what supports interpreting these changes as dissipation toward a stable molecular ground state? Please clarify the interpretation or provide evidence connecting the latent-energy diagnostic to geometric error or physical energy.
  2. Attribution to the Hamiltonian design (W2). Could you provide a separately trained no-momentum control and a capacity-controlled generic momentum variant that removes the Hamiltonian weight tying while keeping the surrounding refinement pipeline comparable? These controls would help distinguish the effects of persistent momentum from those of the particular energy-derived maps.
  3. Training variability (W3). How stable are the reported improvements across independent training runs, particularly the small differences relative to WGFormer?
  4. Measured computational cost (W4). What are the inference latency, throughput, and peak memory relative to WGFormer under comparable hardware, batch sizes, and molecular-size distributions?
  5. Scope and molecular size (W5). Within the existing test sets, how does performance vary with molecular size or flexibility? Are any scaffold-based results available?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.