acceptodds
Under review as a conference paper at ICLR 2027

VideoMD: Generating Protein Dynamics with Continuous Molecular Videos

Abstract

Understanding continuous protein dynamics is fundamental to elucidating biological function. Molecular dynamics (MD) simulations capture such dynamics at atomistic resolution, but long-timescale simulations remain computationally expensive. Deep generative models offer a promising alternative, yet most rely on discrete structural abstractions, where continuous motion is mediated by predefined nodes, tokens, or geometric features. This can create bottlenecks for smooth conformational modeling and increase computational overhead for large proteins. Here, we propose VideoMD, a conditional molecular video generation framework that represents protein dynamics as visual molecular videos and models them directly in continuous visual space. Unlike discrete structural abstractions, molecular videos provide a natural dynamic representation: molecular shape, spatial distance, and relative motion are captured as observable spatiotemporal cues along the evolving trajectory. To evaluate generated dynamics in this space, we introduce Molecular Video Evaluation (MoViE), a unified structure-aware protocol assessing structural validity, ensemble statistics, and temporal dynamics. Under MoViE, extensive experiments show that \VideoMD achieves the best performance among the evaluated generative baselines on several ensemble and temporal metrics, including the strongest pairwise-RMSD correlation and a 73.82% relative improvement in latent autocorrelation MAE, while remaining competitive on selected structural-validity metrics. Using a structure-conditioned decoder, generated videos can be converted into coordinate trajectories that retain informative conformational and dynamical statistics. Further analysis shows nearly constant video-generation time across the tested protein sizes at fixed resolution and clip length, highlighting molecular videos as a scalable representation for efficient protein dynamics modeling.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein Dynamics, Molecular Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NBYaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies protein MD trajectory generation and proposes VideoMD, which renders each trajectory into a fixed-resolution, fixed-length 2D video tensor and fine-tunes a pretrained image-to-video diffusion model (Wan + LoRA) to generate future frames conditioned on the first rendered frame and a simulation-protocol text prompt. The main claims are competitive ensemble/temporal statistics on ATLAS under a new video-based protocol (MoViE) and inference cost largely decoupled from protein length.

Critical strengths
  1. A new formulation of MD trajectory generation as conditional video prediction (Sec. 3.1). Representing a trajectory as a fixed-resolution, fixed-length video tensor V∈RT×C×H×WV\in\mathbb{R}^{T\times C\times H\times W} and modeling pθ(x1:T−1∣x0,c)p_\theta(x_{1:T-1}\mid x_0,c) is a clean reframing that lets the method reuse pretrained video generators rather than building a molecular-specific backbone.

  2. Evidence that generated videos carry recoverable motion information (Table 2). Under a frozen structure-conditioned decoder, generated videos outperform repeated-first-frame and shuffled-input controls across five coordinate metrics, supporting that temporal content in the video is informative beyond the static initial frame.

  3. A video-native, structure-aware evaluation protocol. Two CLIP-based predictors estimate structural invalidity and pairwise RMSD from rendered videos alone, with reported predictor error and threshold-classification metrics, covering structural validity, ensemble statistics, and temporal dynamics.

Critical weaknesses

W1 . The decoder is trained on paired rendered MD videos and coordinates, uses initial CαC_\alpha, sequence, ANM priors, and geometric/dynamical losses, and is frozen before evaluation (App. F, F.4). It does not access Wan VAE/DiT features (App. F.1). No weak-decoder ablation, no generator-quality sweep, and no error decomposition are reported. Table 2's controls (repeated first frame, shuffled input) show video content matters, but cannot quantify how much capability comes from video learning versus decoder priors. The paper itself states the gap "reflects the combined effects of video generation and recovery" (Sec. 4.5).

W2 . Input is one fixed-view frame with NewCartoon rendering (Sec. 3.2, App. A.1). Side chains are not rendered, so drug-binding/active-site conformational changes are largely invisible. No multi-view input, no occlusion analysis, and no site-specific metric are reported. The paper acknowledges the single-view depth limitation (App. F.3) but does not test whether multi-view snapshots would help.

W3 . No training wall-clock time or GPU-hours are reported, and no comparison with baselines' training cost.

W4. All generation uses a fixed spatial size; rendering uses 2048×20482048\times2048 then a content-aware crop; MoViE predictors use 224×224224\times224. No resolution ablation is reported: no comparison of 1280×7201280\times720 vs. lower/higher resolution, no per-length-bin resolution sensitivity, and no quality–cost trade-off curve. This matters because (i) coordinate/graph-based baselines operate in coordinate space and have no rendering-resolution issue, making the comparison asymmetric; (ii) at a fixed grid, a 51-residue and a 724-residue protein occupy very different pixel footprints, yet Fig. 3c bins by length without controlling for resolution; (iii) the "size-decoupled" efficiency claim (Fig. 3c, Table 14) holds only at fixed resolution—attention/VAE cost typically scales with pixel/token count, and no such scaling is reported.

Overall, casting MD as video learning is interesting and offers a new paradigm, but I am not convinced the video representation is necessary. Because a separately trained decoder converts the 2D video back into coordinates, it is hard to tell whether gains come from the video carrying better dynamical information or from the decoder obtaining a stronger representation. Coordinate/GNN baselines' latents can also contain reasonable dynamical information—they are just less human-interpretable—so the paper does not establish what video adds beyond such latents or why image space is better for MD. Video also introduces a resolution constraint absent in coordinate models, and training efficiency relative to GNN baselines is not reported. I do think video has potential and can encode 3D information, but only a single-view snapshot is used, making this potential hard to see. I would encourage the authors to add (a) an explicit discussion of when and why video modeling should be preferred over latent coordinate representations, and (b) mechanism-level experiments isolating the video contribution.

Questions for authors
  1. Can you provide a weak-decoder or decoder-frozen-vs-retrained ablation to separate generator contribution from decoder priors? Alternatively, sweep generator quality (e.g., no-LoRA, fewer steps) and report whether coordinate metrics track video quality monotonically?
  2. Can you report training cost (GPU-hours/wall-clock) for VideoMD and baselines, and a matched-hardware end-to-end timing from first frame to final coordinates, including decoder/refinement/backmapping?
  3. Can you report a resolution ablation (e.g., 640×360640\times360, 1280×7201280\times720, higher) on MoViE and coordinate-recovery metrics, together with inference time, and compare against coordinate baselines at matched quality rather than matched nominal setting?
  4. Would multi-view input improve coordinate recovery or MoViE? At minimum, can you quantify the occluded-residue fraction in the input view?
  5. Can you provide any energy-based or thermodynamic check (free-energy surface comparison) to support physical plausibility?
  6. How sensitive are results to the content-aware crop and rendering choices (NewCartoon, coloring, material), and to the single fixed camera?
Reviewer_NXPCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a model which generates protein dynamics using molecular videos instead of molecular dynamics simulation. The proposed model can generate long-timesclae simulations without being restrictive with computational expense. Basically, the proposed model is contioned on the first rendered frame (rendered by VMD from 1996) and text prompt with simulation protocol. They use Wan2.1-I2V-14B backbone is fine-tuned with LoRA using flow matching. The authors also propose MoViE to evaluate outputs in pixel space. It uses two CLIP predictors. One for per-frame for structural validation, the second for the pairwise predictor to estimate aliigned RMSD. On the ATLAS 100 ns benchmark (82 test proteins), VideoMD is compared against MDGen, AlphaFolding, ConfRover and PVB. It reports the best RMSD Pearson r (0.711), Joint PCA W2 and latent autocorrelation MAE (73.8% relative improvement). But, the method is far behind on all structural-validity metrics.

Critical strengths
  • Representing trajectories as fixed-size rendered videos is very useful for leveraging pretrained video models and allow a computationally feasible simulations of long-scale MD.
  • Provided model can be scaled for generating higher number of residues with same inference time.
  • Generated videos beat both controls on all five metrics. This is the strongest evidence that the outputs are not just static appearance plus the decoder’s priors, and it is independent of MoViE evaluation.
  • VideoMD’s Joint PCA W2 slightly improves across length bins (Table 16). ConfRover’s worsens from 11.25 to 12.24. Generation time also stays near constant while ConfRover runs out of memory at 724 residues. Together these support the claim that the representation does not penalise larger systems, at least within the tested range.
  • VideoMD’s latent autocorrelation values at lags 1, 5, 10 and 20 (Table 15) are close to MD1. Its MAE of 0.0167 is near the MD2-vs-MD1 floor of 0.0071, whereas the baselines show faster decay (MDGen, ConfRover). This holds within the MoViE feature space. This is also a thing that authors note.
Critical weaknesses
  • The main weakness is evaluating the model using their own, newly proposed evaluator. The validation predictor’s MAE is larger than the pass thresholds it is used with. For clash, the MAE is 0.0111 against a threshold of 0.002. For breaks, 0.0103 against 0.005 (Tables 11 and 6). The MD oracle itself passes only 95.6% on clash, although its true 99th-percentile clash ratio is 0. The table 2 provides a geometric evaluation but there is no way to validate the usefullnes of the evaluator MoViE.
  • The predictors were trained on renders of real/synthetically corrupted structures. Artifacts in generated video are out of distribution for them. But, the rendered baseline coordinates are not. The paper acknowledges this but does not test it.
  • There is no matched, evaluator-independent comparison with baselines, and no uncertainty estimates available.
  • The main experiments use same one ATLAS prompt, and there is no quantitative ablation. There is a comparison of ATLAS and CATH2 but only one prompt. It is therefore unclear whether the gains come from the video representation or from the pretrained backbone.
  • Under MoViE, VideoMD has the lowest Clash pass rate among working methods (50.12). Its Overall pass rate is 45.00, compared with MDGen’s 69.22. In coordinate space, raw decoded Cα validity is only 26.8% and rises to 97.4% only after restrained OpenMM refinement (Table 7). The physically valid structures are thus largely produced by refinement rather than by the generator. This weakens the claim of competitive structural plausibility. One can argue that decoder is good for OpenMM refinement, but still the outputs are not raw outputs.
  • Each clip covers 100 ns in 49 frames, roughly 2 ns per frame. Faster motions are therefore not represented. The paper also does not show any extension beyond a single clip, such as autoregressive rollout. The introduction motivates the work with long-timescale transitions, but the experiments do not test horizons longer than the training data.
Questions for authors
  • Can you report coordinate-space metrics for MDGen, ConfRover and PVB on the same 82-protein test set used for VideoMD in Table 2?
  • How many samples per system are generated? Providing bootstrap confidence intervals for RMSD would be great.
  • Is the PCA fit per system or globally?
  • Why does AlphaFolding receive 0 on all validity metrics and no score on the dynamics and statistics?
  • Are the coordinates in Table 2 raw decoder outputs or OpenMM-refined outputs? As raw decoder outputs are pretty low (TAble 7), I assume outputs are refined ones.
Reviewer_jcHBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes VideoMD, which represents MD trajectories as rendered protein videos and generates them using a pretrained video diffusion model. A model is fine-tuned with LoRA to generate 49-frame trajectories from an initial structure and a text description of the simulation protocol. The paper also introduces MoViE, a learned evaluation framework for estimating structural validity and RMSD from rendered frames. Generated videos can further be decoded back to CαC_\alpha trajectories using a separate structure-conditioned decoder.

Critical strengths
  1. Representing MD trajectories as videos and using pretrained video models for trajectory generation is an interesting direction. The rendering pipeline is also clearly documented.

  2. Conditioning generation on force field, solvent, ionic strength, and simulation time through text is potentially useful, although the current experiments do not yet show that the model learns these factors independently.

Critical weaknesses
  1. The main motivation is that priors learned from natural videos may transfer to molecular motion. However, previous work, e.g., Kang et al. (ICML 2025, How Far is Video Generation from World Model: A Physical Law Perspective), has questioned whether video diffusion models learn general physical rules rather than case-specific visual patterns. This concern is consistent with Table 1: VideoMD performs well on several global feature-space metrics but has the lowest Clash Pass among the non-degenerate baselines

  2. The paper claims to model protein dynamics, but it does not evaluate implied timescales, metastable states, transition probabilities, or free-energy landscapes. Instead, the main temporal evaluation uses PCA projections of features from a learned RMSD predictor. These metrics can measure temporal similarity in the evaluator's representation, but they do not establish physical dynamical fidelity. If the 100 ns ATLAS trajectories are too short or remain mostly within one basin, this should be tested in smaller systems.

  3. The structural-validity predictor errors are too large relative to the thresholds in Table11. So the structural validity is somewhat meaningless. Please report the actual clash, break, and Ramachandran statistics directly from coordinates

  4. There is a possible evaluation bias because the MoViE predictors are trained on rendered real structures, whereas VideoMD produces generated pixels. VideoMD performs best on metrics based on the RMSD predictor's feature space, e.g., PCA, PC similarity, but performs much worse on the structural-validity metrics, e.g., Clash, break. I wonder whether generated artifacts affect the evaluator.

  5. The same 49 frames therefore represent 100 ns for ATLAS and 1000 ns for CATH2. At the same time, force field and ionic conditions also change. It is also tricky to choose the lagtime, and how do authors choose the lagtime? This would directly test whether the time condition has a consistent physical meaning.

  6. Methods such as ITO take a fundamentally different approach: they learn a transition operator, where the lag time is explicit and repeated application is meaningful. VideoMD instead generates a fixed-length joint video distribution. This is not necessarily a weakness, but the connection and differences should be stated clearly.

Questions for authors

The same as weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.