acceptodds
Withdrawn from ICLR 2027

Vision Through Fields: Learning Real-Sim-Real Reasoning for MLLM Understanding of Multiphysics Combustion Dynamics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have made notable progress in image and video understanding, yet their ability to reason about physical processes remains limited. To systematically study this problem, we design a comprehensive multiphysics physical reasoning benchmark grounded in complex engineering scenarios, covering combustion dynamics across diverse environments. The benchmark spans a hierarchy from intuitive physical perception to engineering level reasoning. Evaluations on this benchmark reveal a substantial gap between visual perception and physical state understanding in current MLLMs. To bridge this gap, we propose Vision Through Fields (VTF), a framework for learning real-sim-real physical reasoning, where MLLMs first ground visual observations using aligned simulation fields provided at observed timestamps, reason over field states, field evolution, and cross field relations, and finally transfer the resulting understanding back to visual decisions. To make this reasoning process learnable, we decompose it into real-to-sim grounding, simulation field reasoning, and sim-to-real transfer, and train these capabilities with structured supervision derived from physical fields. This learned real-sim-real reasoning raises mean benchmark accuracy from 42.50% to 54.85% with an MLLM backbone. More broadly, our results highlight the value of simulation derived physical fields as reusable supervision for field guided MLLM physical reasoning. All datasets and code will be made publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Physical Understanding, Multimodal Large Language Models, Chain of Thought, Real-to-Sim-to-Real, Combustion Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Pv2eOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes to train MLLM to answer questions about combustion simulation. The authors benchmarked several MLLMs which show chance performance. To resolve it, they generates several pair scene-QA data and SFT the Qwen models which result in significant increases.

Critical strengths
  1. This paper is clearly written and easy to follow. The motivation and concepts are delivered clearly.
  2. Several models are benchmarked making the evaluation more sound.
Critical weaknesses
  1. Misleading title: The "Real-Sim-Real" statement usually means real-to-sim reconstruction or generation and then real-to-sim refers to retargeting or domain adaptation, and neither is relevant to the scope of this paper, which focuses on VQA. Thus, this paper reduces to a domain-specific VQA problem focusing on enabling VQA for combustion-relevant information. At the same time, the technical novelty is relatively small and to the reviewer, the main contribution is the dataset, whose size is relatively limited and small.
  2. Train-val-test split: In L297 the authors mention obtaining 67k scene-clip pairs, and then split to 7:2:1 (train/validation/test), without mentioning if the validation and testing sets are held-out or not. From the context, it seems that the clips are not held out (7 scenarios and 228 cases), making the results less convincing as the testing data may have leaked to the model during training. Can the authors elaborate more about the dataset split setup?
  3. Base models' performance are chance: In Table 1, most performance are likely ~25%, even for GPT-5.5 and Gemini-3.5 the best accuracies are just 33%. Can the authors describe and specify how the inputs are provided to the model?
  4. When do we need this benchmark? The use case assumes access to the field simulation, and when one ran the simulation, the simulation did not have the problem of consistency or ordering problem. In real-world, it is hard to think of a need that we require MLLM to answer such questions and so it is hard to imagine a use case when an MLLM has to answer the coherence, selection, and R2S tasks.
Questions for authors

I have all the questions listed above. It would be helpful to elaborate more about the setup, the contribution, and the title naming.

Reviewer_YNRyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce a benchmark for evaluating the mllm's ability of understanding combustion dynamics. Additionally, based on the dataset, authors propose Vision Through Fields (VTF), which fine-tunes MLLMs using four auxiliary tasks involving scene–field correspondence, field evolution, temporal coherence, and transfer from fields to visual decisions. During inference, the model receives scene images along with field visualizations and a field-guided reasoning prompt. Experiments shows that this pipeline and the fine-tuned model improve the accuracy on the benchmark.

Critical strengths

The paper uses simulation to develop a dataset pairing combustion scene renderings with synchronized physical field visualizations. Based on these, authors evaluate MLLMs across temporal, state, and environmental understanding tasks. The comparisons show that fine-tuning and with field-augmented prompting provide useful evidence, model’s ability to understand combustion dynamics can be improved.

Critical weaknesses
  1. Limited novelty and restricted evidence for the reasoning framework. In my opinion, the improve accuracy of fine-tuned model on the benchmark mainly comes from the task-specific knowledge. The paper fail to show any results that can prove the effectiveness of VTF across tasks in other domains.
  2. The inference setting relies heavily on extra information that may be unavailable. Full VTF reasoning needs aligned visualization of the physical field in the observed scenes. However, given an arbitrary video with fire burning, it's hard to get the corresponding physcial field information. Therefore, the usability of the model is doubted.
Questions for authors
  1. How well does the model perform if only the field visualization is provied during reasoning? This determines whether the model is only reading the supplied field.
  2. The training only uses a simple tunnel geometry, while each unseen environment appears to contain only one simulation case. Could authors clarify whether the generalization holds across different physical conditions like fire locations, in these unseen geometries?
  3. How are the simulation fields constructed and aligned with observations in the NIST experiment? If given an arbitrary fire video, how to get these fields?
  4. What are the individual contributions of R2S, NFS, FCJ, and S2R? Is there any ablation experiments?
Reviewer_sQMuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies physical understanding in multimodal large language models through combustion and ventilation scenarios. It introduces a benchmark containing 228 simulation cases across seven environments, with six tasks covering temporal consistency, physical state perception, and environmental assessment. The proposed Vision Through Fields framework fine-tunes an MLLM through four tasks connecting rendered observations with temperature, gas concentration, and velocity fields. At inference, VTF receives both scene images and aligned simulation fields at observed timestamps. The reported Qwen2.5-VL-7B accuracy increases from 42.50% to 54.85% overall and from 41.88% to 53.79% across six environments excluded from training. An additional evaluation on NIST fire imagery reports an improvement from 38.26% to 49.04%, also with auxiliary simulation fields. The paper further examines temporal alignment, rendering choices, and model size.

Critical strengths
  1. The paper develops a useful setting for studying physical understanding beyond simple object interactions. Combustion involves geometry-dependent transport, ventilation, and interacting physical quantities. Pairing scene renderings with synchronized fields offers a concrete way to study what visual models can infer about these processes. The tunnel case matrix and the numerical checks in Appendix C give this resource a stronger engineering basis than a collection of loosely specified synthetic examples.

  2. The transfer results provide encouraging evidence that the supervision is useful. Training is restricted to tunnel cases, while six other environments are held out. The scene-only tuned checkpoint already improves their mean accuracy from 41.88% to 48.77%, and adding fields increases it further to 53.79%. This is a useful result: the improvement cannot be attributed entirely to supplying extra information at test time. Transfer to real camera images without real-image fine-tuning is another positive observation, within the stated assumption that aligned auxiliary simulation fields are available.

Critical weaknesses
  1. The construction of the environmental-assessment evaluation is insufficiently specified. Sections 3.2–3.3 define E1 as selecting the best exhaust strategy and E2 as comparing environmental outcomes using a case-level steady-state temperature criterion. However, the reported dataset contains 222 tunnel cases and only six cases across the six other environments, apparently one per geometry. It is unclear how alternative strategies, comparison pairs, and their reference outcomes are constructed for these environments. Appendix E.2 states that the real-image labels come from NIST measurements, but does not show how those measurements determine individual answers. This matters because environmental assessment is a central contribution. Without representative labeled examples and task-level counts, I cannot judge whether these questions test the intended capability. Moreover, 1,782 clips from six trajectories provide much less coverage of independent physical conditions than the clip count might suggest.

  2. The experiments do not sufficiently establish the added value of the particular four-task training design. The zero-shot, CoT, SFT, and full-VTF comparisons are useful and show complementary benefits from tuning and field inputs. Nevertheless, there is no fields-only control or conventional fine-tuning baseline using comparable field inputs, supervision volume, and training budget. These comparisons matter because the supplied maps can directly expose information relevant to the answers, such as source location and flow direction.

The current evidence supports the usefulness of a tuned model with physical-field evidence. It is less conclusive about what the proposed R2S/NFS/FCJ/S2R decomposition contributes beyond simpler ways of learning from the same data. The temporal-shuffling experiment demonstrates sensitivity to alignment, but does not resolve this attribution question.

Questions for authors
  1. Clarification of benchmark construction: Does each non-tunnel environment contain only one independent simulation configuration? Please provide a complete E1 and E2 example from one such environment, including the input, candidate strategies or comparison clips, underlying physical outcomes, and label calculation. How many questions and distinct configurations contribute to each task?

  2. Clarification of the real-image evaluation: The cited NIST study describes sprinklered and unsprinklered mockup experiments. How are the exhaust-related P2 and E1 tasks adapted to this setting, and which measurements determine the E1/E2 answers? Please specify the independent experiments and question counts used.

  3. Clarification of simulation inputs: How are the NIST simulation fields configured and aligned with the observations? Which physical parameters are assumed known when generating them, particularly fire location and ventilation conditions that may also be prediction targets?

  4. Evidence that could change my assessment: How does VTF compare with a fields-only model and a conventional field-conditioned fine-tuning baseline under comparable training budgets? These comparisons would help determine whether the proposed training decomposition and joint use of scenes and fields provide additional value.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.