- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors present a study consisting of two experiments in the context of multimodal world models. The study tackles the question of whether different sensors produce similar physical responses, and whether these responses can be reused when actions are combined in a new order. In their first experiment, the authors assess whether different sensors can make the same physical predictions based on the Cluster Haptic dataset. In their second experiment, the authors study whether accurate physical predictions imply generalization to unseen action orders, based on an elastoplastic system. The authors claim that their results distinguish cross-sensor consistency, multimodal fusion, and generalization to new action orders as separate questions when evaluating multimodal physical representations. Nonetheless, the connection between the conclusion, experiments, and questions is not well-communicated, as this paper lacks a clear storyline. This makes it hard to judge, whether there is an actual contribution in this work.
Critical strengths
I am afraid I was unable to indentify a critical strengths that would support acceptance.
Critical weaknesses
W1: Limited significance in experiments, based on...
- ...the type of data: The paper examines models trained on data from only two datasets. As a result, the study lacks relevance for the research community.
- ...the type of models: The presented model architectures differ depending on the types of data, which makes it impossible to draw a conclusion for a specific model architecture. It would of course, be possible to make general statements about multimodal world models from different architectures, but two datasets are not enough for such an empirical study.
W2: Unclear contributions
- The authors conduct two disjoint experiments in the context of multi-modal world models. They claim that the results of these experiments distinguish cross-sensor consistency, multimodal fusion, and generalization to new action orders as separate questions in evaluating multimodal physical representations. However, it remains undiscussed, whether this is a new finding for the community.
- In the abstract, the paper highlights that "similar representations do not necessarily imply that the models make the same physical predictions, or that those representations can be reused when actions are combined in a new order." The authors then introduce the paper as asking the same question from a perspective of physical responses, which raises the expectation that representations and physical responses are somewhat compared or their difference is clearly discussed. We do not see such a comparison or discussion in the paper.
W3: Strongly lacking presentation
- Figures lack explanations of axes (example: Fig. 2 (b): no labels on the y-axis) and some presentation decisions leave ambiguity (example Fig. 3 (c): the MSE-values vary in x per group, does this mean the x-axis carries values or separates by group?)
- The referencing shows some issues, for which we highlight examples: (i) In prominent places (introduction, related work) there is a lack of peer-reviewed papers in the references (e.g., (Shang et al., 2026; Tan et al., 2026; Jin et al., 2026) are all 2026 arxiv preprints); (ii) Some statements in related work lack a clear reference (e.g., "Cross-modal interchangeability asks whether different sensors support a common task.")
- At several points, it is unclear what the paper quantifies (examples: using unexplained "gap" in the abstract; NMMSE mentioned but without equation)
Questions for authors
- Q1: What is the concrete gap the paper aims to solve?
- Q2: Have there been similar studies that cover different datasets or systems?
- Q3: What is the impact of your findings?
3 comments
Response to Reviewer Humn (2/2): scope of the experiments, presentation
W1. Two datasets and different model architectures.
Data. We agree that one real dataset and one simulator limit how far the findings generalize, and we narrowed the claims accordingly. A second real dataset would be the most direct answer, and we could not complete one within the discussion period.
Models. You are right that the models differ between the two settings. We draw no conclusion about one architecture across them, and the paper no longer makes general statements about multimodal world models. Each finding is a comparison inside one setting, in which the data are fixed and one thing changes: the training objective of the encoders in the first study, and the dynamics model in the second. The models differ because the tasks differ: the first maps a scan condition to a response, and the second advances a state through time.
Why two studies in one paper? Both ask whether a model can predict a situation that it was not shown. In the first study the new situation is a way of scanning a surface that the sensor did not observe, and in the second it is a new order of two force pulses. Each setting was chosen for what it lets us control. Cluster Haptic records sound and acceleration at the same time and provides the measured response under 80 scan conditions, so we can give the two sensors different scan conditions and check every prediction against a measurement. The simulator lets us decide which action orders appear in training, which targets each model receives, and which physical parameter each observation cannot see.
Within these two settings, the revision adds evidence.
- The action-order result no longer rests on one pulse pair. We built six versions of the experiment, in which each pair of pulses is the unseen pair twice, and trained every model again on each. With the same training targets, the step-by-step model is more accurate than the whole-sequence model with every seed in all six versions (Figure 6). The same experiment showed that training the step model on every time step helps in only four of the six versions, and the paper now says so.
- The Cluster Haptic result does not depend on which scan conditions each sensor observes. We tried all 1,716 ways of dividing the 13 observed conditions between the two sensors. The two sensors' predictions agree in every case, and the acceleration error stays between 0.68 and 0.72. The audio error varies more, from 0.72 to 1.21 (Appendix B.1).
- Neural encoders are evaluated next to the linear ones on the real data (Section 3.5).
- In the simulator, the predictions from each of two observations, and from both combined, are evaluated through one frozen dynamics model on training, validation, and unseen orders (Table 2, Appendix C.6). This connects sensors and action order within one system. The two studies still share no data or model, and Section 5 states this as a limitation.
W3. Presentation.
Figures. All main figures were redrawn with plain labels. In Figure 2b the rows are now named (parallel, perpendicular, and normal acceleration) and the color scale is labelled, and Figure 2c shows 95% intervals. Your question about Figure 3 concerns the panel with the error of each model per group of sequences, which is now Figure 5b. Its x-axis only separates the groups (Training, Validation, Test) and carries no values; the small horizontal offsets inside a group separate the three training runs. Its y-axis is numeric and labelled: the error divided by the error of a baseline that is not told which oscillator it predicts. Panel c also has a labelled numeric axis.
References. The introduction and related work now rest on peer-reviewed work (Radford et al., ICML 2021; Kornblith et al., ICML 2019; Huh et al., ICML 2024; Lee et al., ICRA 2019; Wi et al., CoRL 2022). The 2026 preprints are kept only as recent examples, and the sentence on cross-modal interchangeability that lacked a reference is removed.
Definitions. NMMSE is Equation 3 in the main text and is explained in words: the error relative to predicting an average surface. The "4.5x gap" is now written as the ratio of two distances between the sensors' predictions, for different surfaces and for the same surface, with interval [2.3, 9.0].
Storyline and language. The introduction starts from what the usual checks on embeddings leave open, describes each study in one paragraph, and ends with three points. The general problem-setup section is gone; each study now introduces its task, its model, and its measures where they are needed. Section 5 says what the two studies show together. We rewrote the whole paper in plainer language, with fewer numbers in the text, and moved formal statements and implementation details to the appendix.
Dear authors, thank you for the rebuttal. While the rebuttal addresses some points about the presentation issues I raised, I still find it hard to read. I also agree with the other reviewers, that considering only a single dataset does not establish sufficient significance.
Response to Reviewer Humn (1/2): the gap, the contribution, prior work, impact
Thank you for the review. The revised PDF marks new experiments and analyses in blue, and the paper is rewritten in plainer language. We agree that the submission did not make clear what was new or why it matters. This comment answers Q1, W2, Q2, and Q3, and a second one answers W1 and W3.
Q1. What gap does the paper address?
Models that predict physical behavior from several sensors are usually judged by how close the sensors' embeddings are. This leaves two things unmeasured: whether close embeddings come with accurate predictions, and whether the model still predicts correctly when familiar actions arrive in a new order. The paper measures each of them in a setting where predictions can be compared with a measured or simulated response.
W2. What is the contribution?
Two controlled comparisons, each with one finding. Neither was in the submission.
1. Does closeness of embeddings tell us that predictions are accurate? (Section 3.5) This is the comparison between representations and responses that you found missing. A model gets a few scans of a new surface, as sound or as acceleration, and predicts the surface's vibration under other scan conditions. We train neural encoders in three ways: to predict the vibration, to align the two sensors with a contrastive loss, or both. We then score every encoder twice on 19 test surfaces: by standard alignment measures on its embeddings, and by its error against the measured vibration.
R@1 is how often the nearest embedding from the other sensor belongs to the same surface (chance 0.053). The error is relative to predicting an average surface. The encoder trained only for alignment finds the matching surface at least as often, and its predictions are less accurate in every run. CKA is much lower for it, in line with its error. The finding is therefore specific to retrieval: here it did not tell us how accurate the physical predictions were.
2. Does a step-by-step dynamics model generalize better to new action orders, or was it only trained with more information? (Section 4) A simulated oscillator is pushed by sequences of force pulses. One model predicts one time step at a time, and the other maps the whole sequence to the response. The test uses two pulses in an order never seen in training. In the submission, the step model was trained on the true state at every time step and the other model was not, so richer supervision could have explained its advantage. Both now get the same targets, with about the same number of parameters and the same training budget. The test error is 0.0177 ± 0.0041 for the step model and 0.614 ± 0.181 for the whole-sequence model. We then repeated the experiment for five other choices of the unseen pulse pair. With every seed and in all six versions the step model is more accurate; on average the other model's error is 8 to 36 times larger (Figure 6). The difference in supervision therefore does not explain the advantage. The models still differ in architecture and in how easily they train, so we do not attribute the whole gap to step-by-step prediction.
You also asked whether the separation of the three questions is a new finding. That separation organized the submission, but no comparison supported it. The revision replaces it with the two findings above.
Q2. Have there been similar studies on other datasets or systems?
Yes, and the submission did not cite them. Evaluating multimodal representations through physical tasks is established: Lee et al. (ICRA 2019) learn a joint vision and touch representation and evaluate it by policy learning on peg insertion, and Wi et al. (CoRL 2022) estimate the state of deformable objects from vision and touch and predict their dynamics. Generalization to new combinations of familiar parts has been studied mostly in language (Lake and Baroni, ICML 2018). Related work now describes these studies. What we add to them is the two comparisons above.
Q3. What is the impact?
Two consequences for practice. If a multimodal model is meant to predict physical behavior, retrieval accuracy between its sensor embeddings should be reported together with the error against measurements; in our experiment retrieval did not go down while the error went up. And when dynamics models are compared on new combinations of actions, they should get the same training targets and the same rule for choosing checkpoints. Our own submission shows why: one result that we had reported depended on the checkpoint rule and is removed, while the gap on unseen orders remained in all six versions.
These conclusions hold for the systems we tested: one real dataset and one simulator.