acceptodds
Under review as a conference paper at ICLR 2027

PhyMo: A Physical-Field Modality for Multimodal AI4Physics

Abstract

Multimodal learning is emerging as a powerful paradigm for AI for Physics (AI4Physics), where predicting physical systems requires the joint interpretation of heterogeneous observations, measurements, and domain knowledge. However, existing approaches typically represent physical quantities and governing equations as generic numerical or textual tokens, overlooking the physical constraints that determine their spatiotemporal interactions. To address this limitation, we introduce the **physical-field modality** and propose **PhyMo**, a physics-grounded multimodal framework that organizes heterogeneous measurements through PDE-associated operators. PhyMo follows a three-stage learning procedure: the physical-field encoder is first pretrained through field reconstruction under PDE residual supervision, its representations are subsequently aligned with visual embeddings in a shared latent space, and the fused multimodal representations are finally processed by corresponding downstream prediction heads. Experiments on five datasets spanning diverse physical environments show that PhyMo achieves state-of-the-art performance, compared to the strongest baseline on each dataset, demonstrating the superiority of PhyMo on multimodal representation learning in AI4Physics.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
physical-field modality, multimodal representation learning, AI4Physics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KmdpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates AI for physical questions, including embedding physical fields, representation learning, and task generalization. Specifically, it proposes a new physical-field modality for embedding physical fields as an individual modality. The multimodal framework improves the learning and alignment of the physical-field encoder. Finally, the paper proposes sufficient theoretical demonstrations.

Critical strengths

The question of AI4Physics is significant, and the proposed methods are interesting.

Critical weaknesses

There is a lack of demonstrations of the differences from related works. The improvements compared with related works and the intuition behind the method design are unclear.

The formulation or methods are introduced directly, without enough evidence or support from theoretical phenomena. For instance, how can the following statement be supported: “The key observation is that a broad class of physical laws can be formulated in terms of partial differential equations”?

Questions for authors

There is a lack of supporting results or visualizations for the relevant descriptions. For instance: “As a result, the model may learn statistical correlations from the observed data while failing to adequately exploit the governing physical constraints (Nie et al. (2023)).”

Related works deserve to be split into individual scientific questions.

The paper lacks basic mathematical formatting, such as punctuation in equations.

The font in the figure is too small compared with that in the main paper, and the figure contains too much content, making it hard to understand.

The overall improvement is too weak, averaging less than 1% across datasets.

Why does the “without field” ablation in Table 2 perform much worse than 8/10 of the related works?

Does the limited number of figures hamper understanding of the proposed question and its comparison with related works?

It would be interesting to conduct the following experimental analyses: 1) Use the same input and network while only reducing the PDE loss to verify the extent of the physical constraints. 2) Use the current framework with inaccurate equations and parameters to verify whether the model makes incorrect inferences when the physical prior fails.

How much data is used in the multimodal framework? Is it essential to use three stages to learn representations?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely thank the reviewer for the careful reading and the constructive comments. We have revised the paper according to your advice, and answered each of your question in turn below.

W1: There is a lack of demonstrations of the differences from related works. The improvements compared with related works and the intuition behind the method design are unclear.

Answer: Thank you very much for raising this question. As illustrated in Section 2, the difference between related works and ours are twofolds: (1) early physical-system models predominantly use a single structured physical-data representation, while PhyMo integrates complementary visual and physical-field modalities, providing richer information for prediction. (2) recent multimodal approaches primarily learn statistical correspondences among numerical observations, while PhyMo explicitly constrains a dedicated physical-field representation through PDE-structured inputs and PDE-supervised reconstruction. As a result, the improvement arises from both greater observational coverage through multimodal integration and stronger physical grounding through governing-equation supervision, allowing predictions to exploit observable patterns together with the dynamics underlying them. Following your advice, we have blue-highlighted the revised content in Section 2. Please kindly review it.

W2: The formulation or methods are introduced directly, without enough evidence or support from theoretical phenomena. For instance, how can the following statement be supported: “The key observation is that a broad class of physical laws can be formulated in terms of partial differential equations”?

Answer: Thank you for raising this question. According to the textbooks and well-established researches, lots of physical laws can be formulated in terms of PDEs. For example, heat conduction is described by the heat equation [1], fluid motion by the continuity and Navier–Stokes equations [2] and electromagnetic fields by Maxwell’s equations [3]. These examples span distinct branches of physics and share a well-established PDE-based framework, which justifies our claim above.

Q1: There is a lack of supporting results or visualizations for the relevant descriptions. For instance: “As a result, the model may learn statistical correlations from the observed data while failing to adequately exploit the governing physical constraints (Nie et al. (2023)).”

Answer: Thank you for raising this question. We have already provided the supporting work in the citaion. Take the cited Nie et al. [4] for example. They report that SUNSET’s forecasts exhibit temporal lags, following past observations rather than anticipating future events, and underestimate fluctuations under cloudy conditions, which exactly illustrates how fitting observed trends can coexist with inadequate modeling of cloud dynamics. Mechanistically, prediction losses penalize errors in observed values but do not explicitly penalize violations of the differential relationships governing their evolution. Thus, learning statistical correlations does not necessarily ensure physical consistency.

Q2: Related works deserve to be split into individual scientific questions.

Answer: Thank you for raising this question. Our related work focuses on three aspects: (1) the development of existing approaches, (2) their limitations, and (3) how our work differs from them. Unfortunately, we cannot include the discussion of scientific questions which may be out of the scope of our paper in Section 2, due to the space limit.

Q3: The paper lacks basic mathematical formatting, such as punctuation in equations.

Answer: Thank you for raising this question. We have improved the mathematical formatting throughout the paper (e.g., equation (1)). Please kindly review them.

Q4: The font in the figure is too small compared with that in the main paper, and the figure contains too much content, making it hard to understand.

Answer: Thank you for raising this question. Unfortunately, we have to use a compact figure to summarize the framework, minimize repetition with the main text, and use the available space efficiently, due to the space limit.

Q5: The overall improvement is too weak, averaging less than 1% across datasets.

Answer: Thank you for raising this question. The average gain of our model against other baselines across datasets is actually 6.932%, with detailed information listed in the table below:

DatasetsMean R2R^2 gain (%)
SKIPP’D2.759
Folsom0.662
NREL3.719
MeteoNet2.291
Boreas25.229
Mean R2R^2 gain across datasets6.932

Although the gains vary across datasets, the proposed PhyMo generally achieves a significant improvement across datasets, especially the 25.229% gain on Boreas.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q6: Why does the “without field” ablation in Table 2 perform much worse than 8/10 of the related works?

Answer: Thank you for raising this question. The ‘w/o field’ ablation in Table 2 only perform worse than 6/10 of the baselines (i.e., RF, LSTM, TabM, TSMixer, xLSTM and BiMamba). As discussed in Section 4.3, this ablation removes the entire physical-field modality, leaving an image-only predictor. Images capture observable appearance but cannot fully recover the complementary physical states and dynamics supplied by the field modality. Its lower performance than baselines exploiting numerical or temporal observations is therefore consistent with this information loss. Restoring the physical-field modality improves performance from 82.90 to 84.80, surpassing all baselines on SKIPP’D and supporting the benefit of integrating complementary visual and physical information.

Q7: Does the limited number of figures hamper understanding of the proposed question and its comparison with related works?

Answer: Thank you for raising this question. Section 2 has already provided a textual comparison that clarifies the key differences between PhyMo and related work, which we believe is suffcient for the goal of illustrating the related work in Section 2.

Q8: It would be interesting to conduct the following experimental analyses: 1) Use the same input and network while only reducing the PDE loss to verify the extent of the physical constraints. 2) Use the current framework with inaccurate equations and parameters to verify whether the model makes incorrect inferences when the physical prior fails.

Answer: Thank you for raising this question. We follow your advice and implement the additional ablation experiments on SKIPP’D.

For 1), we replace the PDE-residual loss with a single MSE loss, with the rest of components unchanged. The detailed results are listed as the table below, which illustrates the effectiveness of the proposed PDE loss and the underlying physical constraints:

ConfigurationR2R^2
Original84.80 ± 0.14
w/o PDE-residual loss83.29 ± 0.29

For 2), we replace the residual loss of the the diffusion–reaction:

roriginal=∂tR^−D∇2R^−pR^r_{\mathrm{original}}=\partial_t\hat R-D\nabla^2\hat R-p\hat R

with

$ r_{\mathrm{incorrect}}=\partial_t\hat R-p\hat R$,

and the results are shown as the table below, which also illustrates the importance of the correct PDE constraints applied to the model:

ConfigurationR2R^2
Original84.80 ± 0.14
Incorrect PDE residual loss84.31 ± 0.84

Q9: How much data is used in the multimodal framework? Is it essential to use three stages to learn representations?

Answer: Thank you for raising this question. As stated in Appendix B.1, we follow the official training/test partition for each evaluated dataset. For example, we use 2014–2015 for training and 2016 for testing on Folsom.

Moreover, we believe it’s necessary to employ the three-stage training paradigm illustrated in the paper. We have to obtain the PDE encoder for physical field modality following the procedure in Stage 1. Then, we need to address the alignment issue between visual and physical representation in Stage 2. Finally, we train task-specific prediction heads for various downstream AI4Physics problems in Stage 3. This ordering establishes physical structure before cross-modal alignment and downstream adaptation, giving each training objective a distinct role. To better support our claim, we have also implemented additional ablation experiments in Section 4.3, which removes stage 1 by using a pretrained encoder. Please kindly review the blue-highlighted content.

References

[1] Fourier J, Freeman A. The analytical theory of heat[M]. New York: Dover, 1955.

[2] Landau L D, Lifshitz E M. Course of theoretical physics[M]. Elsevier, 2013.

[3] Maxwell J C. VIII. A dynamical theory of the electromagnetic field[J]. Philosophical transactions of the Royal Society of London, 1865 (155): 459-512.

[4] Nie Y, Li X, Scott A, et al. A SKy Images and Photovoltaic Power Generation Dataset for Short-term Solar Forecasting. arXiv 2022[J]. arXiv preprint arXiv:2207.00913.

Dear reviewer,

Thank you again for your thoughtful comments. We have provided our responses and clarifications in the rebuttal, and we would greatly appreciate it if you could take a moment to review them, since there is only 3 days left. Your feedback is very valuable to us, and we hope our responses can help address your concerns. Thank you for your time and consideration.

Best regards,

Authors of Submission 11324

Reviewer_vqrVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new physical-field modality to achieve more accurate 'forecasting'. It also proposes a pretraining and fine-tuning combination to train the entire model on the physical field more effectively.

Critical strengths

The entire method pipeline seems simple yet effective, maybe easy to follow. Also, the figure is clear to read, and most of the writing is clear.

Critical weaknesses

The implementation details of the physical field are not clear.

The experiments are not satisfactory, especially the ablation about proving the effectiveness of the proposed pipeline.

Questions for authors
  1. It is not clear about implementing the physical field. In the submission, the physical field seems to be defined as the combination of the physical quantities and the corresponding PDEs. However, to my knowledge, most datasets used may not provide the corresponding PDEs. So, it is important for the author to provide the derivation of the corresponding PDEs.

  2. Continuing from the previous question. In my current understanding, it seems like the submission uses 'general domain PDEs'. For example, the Temperature Field uses a fixed formulated temperature PDE. Is that correct?

  3. The necessity of the proposed method has not been sufficiently discussed. The motivation of the proposed method is the introduction of PDE. So, do we need multistage pretraining? It would be better for the authors to provide no pretraining experiments in the ablation.

  4. Please provide the details about the current ablation study. Does 'w/o field' mean removing the field during training or just during inference?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely thank the reviewer for the careful reading and the constructive comments. The questions mostly concern the presentation and the ablation experiments, which are exactly the aspects we care about, and we answer each in turn below.

[W1] Thank you very much for raising this issue. We apologize for not sufficiently illustrating the details of the proposed physical field modality, which may cause some misunderstandings. We have revised the corresponding content according to your questions below (i.e., Q1 and Q2) in Section 3.2. Please kindly review it.

[W2] Thank you for this helpful suggestion. Following your feedback and those of Reviewer SSX8 and Reviewer Kmdp, we have added ablation experiments in Section 4.3 to examine the effects of removing PDE-operator tokens and replacing the PDE-residual loss, and to assess the contribution of Stage 1 pretraining. The new results and corresponding analyses are blue-highlighted in Section 4.3.

[Q1 & Q2] Thank you very much for raising these questions. Our model indeed uses a fixed set of PDEs, such as temperature PDE, as domain-informed priors, where datasets do not necessarily need to explicitly provide them. This is motivated by the observation that these AI4Physics datasets follow physical rules and thus have implicitly governed by these underlying PDEs [1-3]. Even if some of the PDEs do not govern the datasets, stage 1 of PhyMo would be reduced to conventional data-driven regression, where the model can still learn predictive relationships directly from the observed quantities, retaining the expressive capacity of its regression backbone. We have blue-highlighted the revised content in Section 3.2, according to the illustration above. Please kindly review it.

[Q3] Thank you very much for raising this question. The main motivation of using multistage-training paradigm is that there is no reliable pre-trained encoder available. To better justify our claim, we have followed your advice and implement additional ablation experiment on SKIPP’D where we replace the original pretraining process by using an existing classical pre-trained PDE encoders, as shown in the table below.

ConfigurationR2R^2
Original84.80 ± 0.14
w/o pretraining in stage 182.06 ± 0.99

Specifically, we use the classical DINOv2 [4] encoder, which is pretrained on a large number of physics-related datasets (e.g., Flowers-102 [5] and NYU Depth V2 [6]), with the rest of the components of PhyMo unchanged. As shown in the table above, the performance is significantly reduced, which supports the effectiveness of three-stages training design and is consistent with our motivation in Section 1 to develop representations tailored to physical systems.

[Q4] Thank you for raising this question. We apologize for not sufficiently illustrating the ablation study, which may cause your misunderstanding. ‘w/o’ is the abbreviation for ‘without’, and ‘w/o field’ indeed means removing the physical field modality during training and inference. We have blue-highlighted the revised content in Section 4.3. Please kindly review it.

References

[1] Kochkov D, Yuval J, Langmore I, et al. Neural general circulation models for weather and climate[J]. Nature, 2024, 632(8027): 1060-1066.

[2] Negrini E, Liu Y, Yang L, et al. A multimodal pde foundation model for prediction and scientific text descriptions[J]. arXiv preprint arXiv:2502.06026, 2025.

[3] Huang H, Zhang B, Ren J, et al. A multimodal foundation model of multi-physics for chemical reactor design[J]. Chemical Engineering Science, 2025: 122998.

[4] Oquab M, Darcet T, Moutakanni T, et al. Dinov2: Learning robust visual features without supervision[J]. arXiv preprint arXiv:2304.07193, 2023.

[5] Nilsback M E. An automatic visual flora-segmentation and classification of flower images[D]. Oxford University, 2009.

[6] Silberman N, Hoiem D, Kohli P, et al. Indoor segmentation and support inference from rgbd images[C]//European conference on computer vision. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012: 746-760.

Dear reviewer,

Thank you again for your thoughtful comments. We have provided our responses and clarifications in the rebuttal, and we would greatly appreciate it if you could take a moment to review them, since there is only 3 days left. Your feedback is very valuable to us, and we hope our responses can help address your concerns. Thank you for your time and consideration.

Best regards,

Authors of Submission 11324

Reviewer_SSX8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a new multimodal approach for AI4Physics that combines image embeddings and a novel physical-field modality. The presented framework, PhyMo, builds a representation that respects the physical constraints enforced by PDEs and achieves state-of-the-art performance on multiple forecasting benchmarks across multiple contexts involving physical fields. Its main contributions are the introduction of a new physical-field modality that incorporates physical information through PDE tokens, its combination with image inputs in a setting that admits heterogeneous data combinations, and a demonstration of the theoretical and empirical effectiveness of PhyMo. The experiments presented in this paper are comprehensive, well-motivated and their results are clearly conveyed.

Critical strengths
  • The paper presents an original and effective way of combining the pretrained embeddings of a physics-field encoder with PDE constraints with image representations.
  • The paper achieves state of the art performance on multiple physics-field regression benchmarks across multiple domains.
Critical weaknesses
  • The only major issue with this paper is the presentation. There are several concepts or results that are not properly introduced, obstructing readability. Concretely, the notions of early and late fusion and image span are never defined, the R^2 metric is used before being introduced, the global loss for stage 1 is missing, and Section 4.4 is noticeably harder to read than the rest of the paper. Furthermore, there are several points where the descriptions imply that the model works only for weather forecasting while being generally presented as a more general physics framework.
Questions for authors
  • Have you tried removing the PDE-operator tokens and the PDE-residual loss as an extra ablation test?
  • How does your model compare to others in the literature in terms of parameter size and inference/training time?
  • How are the PDE operator tokens encoded?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely thank the reviewer for the careful reading and the constructive comments. The questions mostly concern the presentation of the paper, which is exactly the aspects we care about, and we answer each in turn below.

[W1] Thank you very much for raising this issue. We have revised the paper according to your reviews. Specifically, early fusion is defined as concatenating modality embeddings before prediction, late fusion as combining modality-specific predictions, and image span as the number of input image frames. The global loss for stage 1 is illustrated as equation (3). Also, we have introduced R2R^2 before first use, and explicitly presented the complete objective as shown in the blue-highlighted content in Section 3.2, Section 4.1, Section 4.3 and Section 4.4.

However, as illustrated in Section 1, Section 3.1, Section 4.1 and Appendix B.1, the proposed PhyMo is a unified framework for multimodal AI4Physics, and our evaluation has already included various scopes of physics-related downstream tasks, including photovoltaic power, solar irradiance, meteorological prediction, and vehicle-speed prediction.

[Q1] Thank you very much for raising this question. We follow your advice and implement the additional ablation experiment on SKIPP’D, shown as the table below:

ConfigurationR2R^2
Original84.80 ± 0.14
w/o PDE-operator tokens84.06 ± 0.12
w/o PDE-residual loss83.29 ± 0.29

Specifically, ‘w/o PDE-operator tokens’ means removing the PDE-operator tokens only, and ‘w/o PDE-residual loss’ is achieved by replacing the PDE-residual loss with a single MSE loss. The results above illustrate that both PDE-operator tokens and PDE-residual loss play an important role in the proposed PhyMo. We have blue-highlighted the revised content in Section 4.3. Please kindly review it.

[Q2] Thank you very much for raising this question. Following your advice, we have measured parameter counts, training time, and inference time for PhyMo and all 10 baselines on SKIPP’D at the 15-minute horizon, shown as the table below:

ModelsParameters (M)Training (min)Inference (ms)
LASSON/A0.14 ± 0.0112.36 ± 0.08
RFN/A5.96 ± 0.1042.45 ± 5.33
LSTM0.0370.28 ± 0.0712.73 ± 0.09
TabM0.0670.66 ± 0.0912.94 ± 0.08
TSMixer0.0071.54 ± 0.1713.76 ± 0.08
xLSTM0.3184.79 ± 0.2422.69 ± 0.09
BiMamba0.48624.32 ± 0.9220.51 ± 0.10
AstroCLIP1.3195.69 ± 0.0718.11 ± 0.10
Maven0.25336.44 ± 0.725.82 ± 0.02
AION-10.58217.35 ± 0.2614.58 ± 0.03
PhyMo1.77124.35 ± 0.5219.37 ± 0.07

[Q3] Thank you for raising this question. As illustrated in Section 3.2 and Figure 1, each PDE operator, such as ∂t\partial_t, ∂x\partial_x, ∂xx\partial_{xx}, ∂xy\partial_{xy}, or ∇2\nabla^2, is assigned a distinct token ID and mapped to a learnable dd-dimensional vector through an operator embedding table. These operator embeddings are combined with the corresponding physical measurements and processed by the Transformer-based physical-field encoder. The embeddings are learned jointly with the encoder during Stage 1, with field reconstruction and PDE-residual supervision grounding the resulting representations in the governing dynamics. We have revised the corresponding content according to the illustration above in Section 3.2.

Dear reviewer,

Thank you again for your thoughtful comments. We have provided our responses and clarifications in the rebuttal, and we would greatly appreciate it if you could take a moment to review them, since there is only 3 days left. Your feedback is very valuable to us, and we hope our responses can help address your concerns. Thank you for your time and consideration.

Best regards,

Authors of Submission 11324

Thank you very much for your thorough response. I am satisfied with the answers you presented to my question and the associated edits that you included in the manuscript. My only remaining concerns are the following:

  • I believe that your "w/o stage 1" ablation test does not cleanly isolate the contribution of Stage 1 pretraining since you are plugging in another pretrained encoder into your framework instead of just skipping pretraining altogether. It should be made clear in the text what is actually tested by working with the DINOv2 encoder.
  • I believe that the presentation could still be improved. Relevant concepts are now introduced, but the way they are discussed is quite awkward to read. A greater effort could be made to improve readability

Given these concerns, I will keep my original score for your paper.

Thank you very much for your response. The "w/o stage 1" ablation experiment is designed to investigate the contribution of the original architectural design of Stage 1, as suggested by Reviewer vqrV. Directly removing the entire Stage 1 does not seem feasible, since our architecture requires at least one encoder to support the subsequent training of the prediction heads. Given that the primary purpose of Stage 1 is to train the proposed PDE encoder, we therefore decided to replace it with another pretrained encoder (i.e., DINOv2). We have revised the corresponding descriptions accordingly. In addition, we have refined the explanations of relevant concepts in Sections 4.1 and 4.3 to further improve readability.

Thank you again for your response and your dedicated work.

Comments on the paper

3 comments

Dear Reviewers,

Thank you again for your reviews. We have carefully addressed all the concerns raised in the reviews and conducted all requested additional experiments, including ablations on PDE-operator tokens, PDE-residual loss, Stage 1 pretraining, and incorrect PDE constraints. We would also like to emphasize that the concerns raised so far primarily focus on presentation and experimental validation, without identifying fundamental issues regarding the novelty or logic of the proposed model.

As the rebuttal deadline is now less than two days, we would sincerely appreciate it if you could take another look at our responses since we really put a lot of effort into them.

Thank you for your time and consideration!

Authors of Submission 11324

Dear all,

Sorry to bother you again. But to be honest, I do feel frustrated about this situation, as I'm really not sure whether anyone has ever read my rebuttal yet. Initially, I was genuinely excited about the motivation and the actual experiment results of this paper. Although the initial scores were not very good, I still insisted on doing my best to work on this rebuttal. The main weaknesses and questions were merely about the presentation and the ablation experiments, and I thought I could address these concerns through this rebuttal. Nevertheless, no one has replied to me so far.

Now, there is only 1 day left. I would like to try my last chance to humbly ask for a response from any reviewer or AC, because I honestly don't know what else I can do now.

Best regards

Dear AC, SAC, and PC,

The deadline for the rebuttal phase is now approaching, and we have already followed all of the reviewers' suggestions, which are mainly about implementing additional ablation experiments and improving the presentation. Despite the effort we’ve made to achieve the positive acknowledgement from Reviewer SSX8, none of the other reviewers has ever responded to our rebuttals. We really put a lot of effort into this rebuttal, with good wishes to engage in a meaningful discussion. We hope you can consider our manuscript and rebuttals, as we really try our best, but are still unable to connect the remaining reviewers throughout the whole rebuttal phase.

Best regards,

Authors of Submission 11324

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.