acceptodds
Under review as a conference paper at ICLR 2027

Beyond Multimodal Alignment: Shared Physical Representations Across Sensors and Action Orders

Abstract

Multimodal world models are often evaluated by whether different sensors produce similar representations. Yet representational similarity alone does not establish whether different sensory observations support consistent physical predictions or whether the resulting representations can be reused when familiar actions appear in new orders. We study these questions through the physical responses predicted by a model. On the Cluster Haptic dataset, audio and acceleration independently recover surface-specific behavior from disjoint observations: predictions from the two sensors are 4.5× closer for the same surface than for different surfaces on average, while both outperform a population-level prediction. We then study action-order generalization in a controlled elastoplastic system. Shared-step dynamics fit observed action programs less accurately than a whole-program predictor, but generalize substantially better to unseen action orders, with the ranking reversing on both held-out transitions across three independent initializations. Combining complementary free-decay and hysteresis observations further improves response prediction, with diagonal Gaussian beliefs giving the lowest errors among the evaluated fusion models. These results characterize cross-sensor consistency, multimodal fusion, and generalization to new action orders as distinct properties of shared physical representations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal representation learning, world models, cross-sensor consistency, predictive representations, sensor fusion, compositional generalization, physical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_HumnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors present a study consisting of two experiments in the context of multimodal world models. The study tackles the question of whether different sensors produce similar physical responses, and whether these responses can be reused when actions are combined in a new order. In their first experiment, the authors assess whether different sensors can make the same physical predictions based on the Cluster Haptic dataset. In their second experiment, the authors study whether accurate physical predictions imply generalization to unseen action orders, based on an elastoplastic system. The authors claim that their results distinguish cross-sensor consistency, multimodal fusion, and generalization to new action orders as separate questions when evaluating multimodal physical representations. Nonetheless, the connection between the conclusion, experiments, and questions is not well-communicated, as this paper lacks a clear storyline. This makes it hard to judge, whether there is an actual contribution in this work.

Critical strengths

I am afraid I was unable to indentify a critical strengths that would support acceptance.

Critical weaknesses

W1: Limited significance in experiments, based on...

  • ...the type of data: The paper examines models trained on data from only two datasets. As a result, the study lacks relevance for the research community.
  • ...the type of models: The presented model architectures differ depending on the types of data, which makes it impossible to draw a conclusion for a specific model architecture. It would of course, be possible to make general statements about multimodal world models from different architectures, but two datasets are not enough for such an empirical study.

W2: Unclear contributions

  • The authors conduct two disjoint experiments in the context of multi-modal world models. They claim that the results of these experiments distinguish cross-sensor consistency, multimodal fusion, and generalization to new action orders as separate questions in evaluating multimodal physical representations. However, it remains undiscussed, whether this is a new finding for the community.
  • In the abstract, the paper highlights that "similar representations do not necessarily imply that the models make the same physical predictions, or that those representations can be reused when actions are combined in a new order." The authors then introduce the paper as asking the same question from a perspective of physical responses, which raises the expectation that representations and physical responses are somewhat compared or their difference is clearly discussed. We do not see such a comparison or discussion in the paper.

W3: Strongly lacking presentation

  • Figures lack explanations of axes (example: Fig. 2 (b): no labels on the y-axis) and some presentation decisions leave ambiguity (example Fig. 3 (c): the MSE-values vary in x per group, does this mean the x-axis carries values or separates by group?)
  • The referencing shows some issues, for which we highlight examples: (i) In prominent places (introduction, related work) there is a lack of peer-reviewed papers in the references (e.g., (Shang et al., 2026; Tan et al., 2026; Jin et al., 2026) are all 2026 arxiv preprints); (ii) Some statements in related work lack a clear reference (e.g., "Cross-modal interchangeability asks whether different sensors support a common task.")
  • At several points, it is unclear what the paper quantifies (examples: using unexplained "gap" in the abstract; NMMSE mentioned but without equation)
Questions for authors
  • Q1: What is the concrete gap the paper aims to solve?
  • Q2: Have there been similar studies that cover different datasets or systems?
  • Q3: What is the impact of your findings?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Humn (1/2): the gap, the contribution, prior work, impact

Thank you for the review. The revised PDF marks new experiments and analyses in blue, and the paper is rewritten in plainer language. We agree that the submission did not make clear what was new or why it matters. This comment answers Q1, W2, Q2, and Q3, and a second one answers W1 and W3.

Q1. What gap does the paper address?

Models that predict physical behavior from several sensors are usually judged by how close the sensors' embeddings are. This leaves two things unmeasured: whether close embeddings come with accurate predictions, and whether the model still predicts correctly when familiar actions arrive in a new order. The paper measures each of them in a setting where predictions can be compared with a measured or simulated response.

W2. What is the contribution?

Two controlled comparisons, each with one finding. Neither was in the submission.

1. Does closeness of embeddings tell us that predictions are accurate? (Section 3.5) This is the comparison between representations and responses that you found missing. A model gets a few scans of a new surface, as sound or as acceleration, and predicts the surface's vibration under other scan conditions. We train neural encoders in three ways: to predict the vibration, to align the two sensors with a contrastive loss, or both. We then score every encoder twice on 19 test surfaces: by standard alignment measures on its embeddings, and by its error against the measured vibration.

Trained toRetrieval R@1CKAError on measured responses
Predict the response0.237 ± 0.0390.8820.713 ± 0.002
Align the two sensors0.285 ± 0.0330.3260.785 ± 0.015

R@1 is how often the nearest embedding from the other sensor belongs to the same surface (chance 0.053). The error is relative to predicting an average surface. The encoder trained only for alignment finds the matching surface at least as often, and its predictions are less accurate in every run. CKA is much lower for it, in line with its error. The finding is therefore specific to retrieval: here it did not tell us how accurate the physical predictions were.

2. Does a step-by-step dynamics model generalize better to new action orders, or was it only trained with more information? (Section 4) A simulated oscillator is pushed by sequences of force pulses. One model predicts one time step at a time, and the other maps the whole sequence to the response. The test uses two pulses in an order never seen in training. In the submission, the step model was trained on the true state at every time step and the other model was not, so richer supervision could have explained its advantage. Both now get the same targets, with about the same number of parameters and the same training budget. The test error is 0.0177 ± 0.0041 for the step model and 0.614 ± 0.181 for the whole-sequence model. We then repeated the experiment for five other choices of the unseen pulse pair. With every seed and in all six versions the step model is more accurate; on average the other model's error is 8 to 36 times larger (Figure 6). The difference in supervision therefore does not explain the advantage. The models still differ in architecture and in how easily they train, so we do not attribute the whole gap to step-by-step prediction.

You also asked whether the separation of the three questions is a new finding. That separation organized the submission, but no comparison supported it. The revision replaces it with the two findings above.

Q2. Have there been similar studies on other datasets or systems?

Yes, and the submission did not cite them. Evaluating multimodal representations through physical tasks is established: Lee et al. (ICRA 2019) learn a joint vision and touch representation and evaluate it by policy learning on peg insertion, and Wi et al. (CoRL 2022) estimate the state of deformable objects from vision and touch and predict their dynamics. Generalization to new combinations of familiar parts has been studied mostly in language (Lake and Baroni, ICML 2018). Related work now describes these studies. What we add to them is the two comparisons above.

Q3. What is the impact?

Two consequences for practice. If a multimodal model is meant to predict physical behavior, retrieval accuracy between its sensor embeddings should be reported together with the error against measurements; in our experiment retrieval did not go down while the error went up. And when dynamics models are compared on new combinations of actions, they should get the same training targets and the same rule for choosing checkpoints. Our own submission shows why: one result that we had reported depended on the checkpoint rule and is removed, while the gap on unseen orders remained in all six versions.

These conclusions hold for the systems we tested: one real dataset and one simulator.

AuthorsOpenReview ↗

Response to Reviewer Humn (2/2): scope of the experiments, presentation

W1. Two datasets and different model architectures.

Data. We agree that one real dataset and one simulator limit how far the findings generalize, and we narrowed the claims accordingly. A second real dataset would be the most direct answer, and we could not complete one within the discussion period.

Models. You are right that the models differ between the two settings. We draw no conclusion about one architecture across them, and the paper no longer makes general statements about multimodal world models. Each finding is a comparison inside one setting, in which the data are fixed and one thing changes: the training objective of the encoders in the first study, and the dynamics model in the second. The models differ because the tasks differ: the first maps a scan condition to a response, and the second advances a state through time.

Why two studies in one paper? Both ask whether a model can predict a situation that it was not shown. In the first study the new situation is a way of scanning a surface that the sensor did not observe, and in the second it is a new order of two force pulses. Each setting was chosen for what it lets us control. Cluster Haptic records sound and acceleration at the same time and provides the measured response under 80 scan conditions, so we can give the two sensors different scan conditions and check every prediction against a measurement. The simulator lets us decide which action orders appear in training, which targets each model receives, and which physical parameter each observation cannot see.

Within these two settings, the revision adds evidence.

  • The action-order result no longer rests on one pulse pair. We built six versions of the experiment, in which each pair of pulses is the unseen pair twice, and trained every model again on each. With the same training targets, the step-by-step model is more accurate than the whole-sequence model with every seed in all six versions (Figure 6). The same experiment showed that training the step model on every time step helps in only four of the six versions, and the paper now says so.
  • The Cluster Haptic result does not depend on which scan conditions each sensor observes. We tried all 1,716 ways of dividing the 13 observed conditions between the two sensors. The two sensors' predictions agree in every case, and the acceleration error stays between 0.68 and 0.72. The audio error varies more, from 0.72 to 1.21 (Appendix B.1).
  • Neural encoders are evaluated next to the linear ones on the real data (Section 3.5).
  • In the simulator, the predictions from each of two observations, and from both combined, are evaluated through one frozen dynamics model on training, validation, and unseen orders (Table 2, Appendix C.6). This connects sensors and action order within one system. The two studies still share no data or model, and Section 5 states this as a limitation.

W3. Presentation.

Figures. All main figures were redrawn with plain labels. In Figure 2b the rows are now named (parallel, perpendicular, and normal acceleration) and the color scale is labelled, and Figure 2c shows 95% intervals. Your question about Figure 3 concerns the panel with the error of each model per group of sequences, which is now Figure 5b. Its x-axis only separates the groups (Training, Validation, Test) and carries no values; the small horizontal offsets inside a group separate the three training runs. Its y-axis is numeric and labelled: the error divided by the error of a baseline that is not told which oscillator it predicts. Panel c also has a labelled numeric axis.

References. The introduction and related work now rest on peer-reviewed work (Radford et al., ICML 2021; Kornblith et al., ICML 2019; Huh et al., ICML 2024; Lee et al., ICRA 2019; Wi et al., CoRL 2022). The 2026 preprints are kept only as recent examples, and the sentence on cross-modal interchangeability that lacked a reference is removed.

Definitions. NMMSE is Equation 3 in the main text and is explained in words: the error relative to predicting an average surface. The "4.5x gap" is now written as the ratio of two distances between the sensors' predictions, for different surfaces and for the same surface, with interval [2.3, 9.0].

Storyline and language. The introduction starts from what the usual checks on embeddings leave open, describes each study in one paragraph, and ends with three points. The general problem-setup section is gone; each study now introduces its task, its model, and its measures where they are needed. Section 5 says what the two studies show together. We rewrote the whole paper in plainer language, with fewer numbers in the text, and moved formal statements and implementation details to the appendix.

Dear authors, thank you for the rebuttal. While the rebuttal addresses some points about the presentation issues I raised, I still find it hard to read. I also agree with the other reviewers, that considering only a single dataset does not establish sufficient significance.

Reviewer_CnRpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that multimodal physical representations should be evaluated by their ability to predict physical behavior, not only by embedding alignment. It explores this through two studies. On the Cluster Haptic dataset, audio and acceleration observations are each mapped into a shared representation, which is used to predict a surface’s vibration response under scan conditions that neither sensor observed. Predictions from the two sensors agree more for the same surface than for different surfaces, and both outperform an average-surface baseline. In a simulated elastoplastic system, a model that learns step-by-step dynamics generalizes better to unseen action orders than a whole-sequence predictor. In the same simulation, combining complementary observations improves prediction. The authors also give a theoretical example in which observations of single actions do not determine the effect of an unseen action order.

Critical strengths

The paper asks a relevant question: whether agreement between sensors reflects shared physical content, which matters for multimodal and tactile learning. Testing each sensor through its predictions under held-out conditions, with wrong-surface and average-surface controls, is a reasonable design. The experiments are carefully controlled. Evidence and test conditions do not overlap, test data are held back until the models are fixed, and development results are reported alongside test results. The authors also state the main limitations and confounds of their design openly, and the simulator is described in detail.

Critical weaknesses

W1

The paper is difficult to read. The writing is dense and abstract, and the paper introduces new terms for its components, such as “compiler” and “executor”. Are these meant as an encoder and a decoder? If so, why introduce new terminology instead of the standard terms? Many key sentences state relations between abstract concepts instead of saying what was done and found. For example, the introduction states that “An ambiguity construction separates recovery of entity information from identification of an unseen composition law”. At this point the reader does not know what is constructed, what a “composition law” is, or what it means to separate one from the other. The sentence appears to mean that knowing an object’s properties does not determine the effect of a new action order, but the paper does not say this plainly. Some formal definitions in Section 3 are never used in the experiments, and the real-data model itself is described only briefly.

W2 The paper does not clearly state its contributions or how it differs from prior work. Its motivating claim, that multimodal world models are often evaluated by whether different sensors produce similar representations, is given without any citation. The related work does not explain what existing methods lack, and the conclusion restates the results without a clear takeaway.

W3 The paper does not show what its prediction-based test adds over alignment. The title and abstract present the work as going beyond multimodal alignment, but no experiment measures alignment or compares it with the proposed test. The paper also gives no theoretical support for the test. Its only formal result, Proposition 1, addresses a different question about action orders.

W4 The evidence is too narrow to support the paper’s framing around multimodal world models. The real-data study uses one dataset, a small test set and simple linear models, and the simulation is a small, idealized system. The authors acknowledge this scope, but it limits what the results show about the learned multimodal world models that motivate the paper.

Questions for authors

Q1 Can you show, on the same data, a case where an alignment measure and your prediction-based test disagree? Which of the two better reflects accuracy on measured responses? In the real-data setup, how does comparing the two sensors’ predictions differ from comparing their learned codes directly?

Q2 The real-data errors in Table 2 and Figure 2c are hard to interpret without a reference point. What is the best accuracy the model could reach, and how much do the results vary across test surfaces?

Q3 In the comparison of Section 5.2, the step model receives much richer supervision than the sequence baseline, so its advantage may come from the supervision rather than from the step structure. Does the advantage remain when both models receive the same supervision?

Q4 The real-data study and the simulation share no data or models. How should readers combine their results? Is there any system in which cross-sensor agreement and generalization to new action orders are measured together?

4 comments
AuthorsOpenReview ↗

Response to Reviewer CnRp (1/3): same supervision, terminology, contributions

Thank you for the detailed review. The revised PDF marks new experiments and analyses in blue, and the paper is rewritten in plainer language. This comment answers Q3, W1, and W2; the next two answer W3 and Q1, then Q2, Q4, and W4.

Q3. Does the step model keep its advantage when both models get the same supervision?

Yes. In the submission, the step model was trained on the true state at every time step, and the whole-sequence model only on the response after the last pulse. We now also train the step model on that final response only. It is run from rest through the whole force sequence, up to 360 steps, and the loss is taken on the last 60 steps, so it never sees a true intermediate state. Both models have about 17,900 parameters and the same minibatches, optimizer, number of updates, and checkpoint rule.

Model and training targetsTest MSE on the unseen orders AB, BAError of the predicted difference between AB and BA
Step model, final response only0.0177 ± 0.00410.223 ± 0.093
Whole-sequence model, final response only0.614 ± 0.1810.952 ± 0.240
Step model, every time step0.0033 ± 0.00020.126 ± 0.029

Values are mean ± s.d. over three random seeds. In the last column, predicting no difference between the two orders gives 1. With the same targets, the step model's error is lower with every seed and for both orders, by factors of 52, 39, and 20.

Is this specific to the pair A, B? We repeated the experiment for five other choices of the unseen pulse pair, with all models retrained (Figure 6). With every seed and in all six versions, the step model has the lower test error; averaged over seeds, the whole-sequence error is 8 to 36 times larger. Training on every time step helps in four of the six versions and hurts in the two where the training sequences combine only the two pulses that push in the same direction. A recurrent model that receives the force at every time step but keeps a learned hidden state did not train in 3 of 18 runs and was less accurate than the step model in 14 of the other 15, so receiving the force at every step does not explain the result by itself.

How the result was obtained. Training through up to 360 steps needed stabilizing: we initialized the step model's last layer to zero and clipped the gradient norm at 1. The same settings change the whole-sequence model little (0.614 with them, 0.551 ± 0.103 without; Appendix C.4). The rule for choosing checkpoints changed one of our claims. The submission chose between the checkpoints at 15,000 and 30,000 updates, and the whole-sequence model then fit the training sequences best, which we described as a reversal between training and test ranking. With checkpoints evaluated every 250 updates its training error is no longer the lowest, so we removed that claim. The gap on unseen orders holds under both rules.

What the comparison does not show. It removes the difference in targets. It does not show that feeding back the predicted state is the cause, because the two models also differ in architecture and in how easily they train (Section 5).

W1. Readability and terminology.

We agree. "Compiler" meant an encoder, and "executor" meant a decoder in the first study and a dynamics model in the second. The paper now says encoder, decoder, step model, and whole-sequence model, and introduces each with an example. Your reading of the quoted sentence was right, and Section 4.1 now makes its point in plain words: knowing how a system responds to A alone and to B alone does not tell us how it responds to AB and BA. Proposition 1 in Appendix A gives the formal example, and the unused formal definitions are removed. The real-data model now has Sections 3.1 to 3.3 for its task, its two parts, and its measures.

W2. Contributions, prior work, takeaway.

Contributions. The introduction ends with three points, one for each finding.

Motivating claim. It is now tied to cited practice: retrieval after contrastive training (Radford et al., 2021), CKA (Kornblith et al., 2019), and nearest-neighbor comparison (Huh et al., 2024).

What existing work leaves open (now stated in related work). Alignment scores are computed without any physical measurement. Evaluating multimodal representations through a physical task is established (Lee et al., ICRA 2019; Wi et al., CoRL 2022) and is not our idea, but a task result does not show how well the modalities are aligned. Our first study measures both for the same encoders. Generalization to new combinations has been studied mostly in language, and our second study poses it for a physical system.

Takeaway. Section 5 gives two recommendations, and the conclusion ends with the practice they share: evaluate physical representations on predictions, against measurements, and under conditions not seen in training.

AuthorsOpenReview ↗

Response to Reviewer CnRp (2/3): alignment measures and our prediction-based test

W3 and Q1. You asked three things. How does comparing predictions differ from comparing codes? Can an alignment measure and our test disagree on the same data? Which of the two better reflects accuracy on measured responses? Parts (a) to (c) answer them in this order, and (d) gives the limits.

(a) In the linear model of the submission, comparing predictions is the same as comparing codes. The decoder is affine on the 67 scan conditions on which we score, G(θ)=b+WθG(\theta)=b+W\theta. The distance between two sensors' predictions is therefore (θa−θb)⊤W⊤W(θa−θb)/Z(\theta_a-\theta_b)^\top W^\top W(\theta_a-\theta_b)/Z, a fixed function of the two codes. The eigenvalues of W⊤WW^\top W are 0.755, 0.849, and 0.902, so the three code directions count almost equally. In that model, the distance between predictions tells us nothing that the codes do not, and the paper now says so (Section 3.5, where the codes are called coordinates). What our evaluation adds there is the comparison with measurements that were held back, and two controls: the prediction for a different surface and the prediction for an average surface. You are also right that Proposition 1 is about action order. We have no theoretical result for the test itself. The identity above is the only formal statement, and it shows equivalence in the linear case.

(b) With learned embeddings, an alignment measure and our test disagree. We replaced each ridge-regression encoder by a small MLP that outputs a 32-dimensional embedding h. A linear layer, shared by the two sensors being compared, maps h to θ, and the frozen decoder is unchanged. We trained in three ways: to predict the response, to align the two sensors (symmetric InfoNCE between their embeddings of the same surface), or both (alignment weight λ). The alignment-only encoder gets its linear layer afterwards by ridge regression. Each setting has three seeds, checkpoints are chosen by the error on validation surfaces, and alignment is measured on h for the 19 test surfaces.

Training objectiveRetrieval R@1CKAAgreement of predictionsError on measured responses (NMMSE)
Prediction only0.237 ± 0.0390.8829.80.713 ± 0.002
Prediction + alignment, λ = 10.307 ± 0.0200.8518.00.717 ± 0.001
Alignment only0.285 ± 0.0330.3262.30.785 ± 0.015

R@1 is how often the nearest embedding from the other sensor belongs to the same surface (chance 0.053). Agreement of predictions is the measure of our test: how many times closer the two sensors' predictions are for the same surface than for different surfaces. NMMSE is the error relative to predicting an average surface, averaged over both sensors. Figure 4 shows every run, and Table 6 in Appendix B.2 gives all settings and scores under two rules for choosing checkpoints.

Judged by retrieval, the alignment-only encoder looks at least as good as the prediction-trained one, and adding the alignment loss raises retrieval further. Judged by our test, it is clearly worse. Its predictions from the two sensors agree much less (2.3 against 9.8), and its error on measured responses is higher in every run (0.774 to 0.802, against 0.711 to 0.714).

(c) Which better reflects accuracy? Retrieval does not follow the error. The agreement of predictions does: over the five training settings (including λ = 0.1 and 10) it falls in the order in which the error rises, 9.8, 8.8, 8.0, 6.5, 2.3 against 0.713, 0.715, 0.717, 0.721, 0.785. Among the embedding scores, the cosine similarity between the two embeddings of the same surface follows the same order, and CKA separates the alignment-only encoders without ordering the other settings. Two cautions apply. The predictions are computed after the linear layer, which was trained toward the response in every setting except alignment only, so their agreement is closer to the target by construction. And the four settings with a prediction loss differ in error by less than 0.01. No score replaces the check against measurements.

(d) Limits. The retrieval difference is small: 0.285 against 0.237 is about four of 76 retrieval trials per model. It is higher with all three seeds under this checkpoint rule and with two of three under the other, so the safe reading is that retrieval is not lower. The linear layers of the other encoders are trained together with them, while the alignment-only encoder gets its layer afterwards. The experiment therefore compares how the measures rank models and is not a causal test of the alignment loss. After seeing the test results, we also fitted a separate ridge layer for each encoder of every model. The gap in error remains (0.787 ± 0.033 against 0.713 to 0.720), so sharing the layer does not explain it.

Relevant to W4: the MLP encoders also lower the audio error from 0.860 and 0.830 to 0.732 and 0.745.

AuthorsOpenReview ↗

Response to Reviewer CnRp (3/3): reference points, combining the two studies, scope

Q2. What is the best achievable accuracy, and how much do surfaces vary?

Prediction fromError (NMMSE)95% interval over surfacesSurfaces better than the average-surface prediction
Audio, group P00.860[0.72, 1.09]11/19
Audio, group P10.830[0.68, 1.07]13/19
Acceleration, group P00.687[0.61, 0.80]18/19
Acceleration, group P10.687[0.61, 0.80]18/19
Best-fit coordinate for each test surface0.672[0.60, 0.78]19/19

NMMSE is the error relative to predicting an average surface, so 1 means no gain. P0 and P1 are the two groups of scan conditions that a sensor can receive. The last row is the best that this model can reach: for each test surface we compute the coordinate that best fits its own measured responses and decode it. It uses the test targets, so it is a ceiling for the frozen decoder and not a predictor. Acceleration (0.687) is within 0.016 of it. Audio is not close, and its intervals include 1; on one surface its error is 2.3 times that of the average-surface prediction. The paper now says that the gain from audio is a pooled estimate that is not reliable across surfaces. The intervals come from resampling the 19 test surfaces 2,000 times, and Figure 3 plots every test surface.

A second reference is the other recording of the same scan. Used directly as the prediction it gives 0.822, and after passing through the decoder 0.688. Repeated recordings differ considerably in this dataset. We have not separated measurement noise from detail that the decoder discards.

The two sensors' predictions are 4.5 times closer for the same surface than for different surfaces, with interval [2.3, 9.0]. Across the 19 surfaces, how closely the two sensors agree on a surface has a Spearman correlation of 0.10 with the error on that surface. We report this descriptively and do not infer that agreement and accuracy are independent.

Q4. How should readers combine the two studies, and is there one system in which both are measured?

Combining them. They are two separate findings about two kinds of situations that a model was not shown: a scan condition that the sensor did not observe, and an order of actions that training did not contain. Neither result is evidence for the other. Section 5 states what they share: in each study a measure that is convenient to compute (closeness between the sensors, or the error on training sequences) looked fine for a model that was inaccurate on the withheld test.

One system. Yes, within the simulator. The oscillator can be observed in two ways, free decay and hysteresis. We now evaluate the prediction from each observation, and from both combined, through the same frozen step model, on training, validation, and the unseen orders AB and BA (Table 2, Appendix C.6). On the unseen orders:

Observation usedError relative to no observationError of the predicted difference between AB and BA
None (prior only)1.0000.374
Free decay0.5490.297
Hysteresis0.4070.220
Both combined0.3190.182
Oracle coordinate0.1860.121

The two single-observation values in the last column are new, and the re-run that produced them reproduces the earlier numbers.

We also measured whether the two single-observation predictions agree, using the agreement test of Section 3.3 adapted to the simulator. On the unseen orders they do: the distance is 0.522 for the same oscillator, 1.239 for different oscillators, and 0.706 to the prior prediction. On training and validation sequences they do not, because the same-oscillator distance exceeds the distance to the prior prediction.

Four limits apply. Each observation misses one physical parameter, so the result does not show that either one is sufficient by itself or that the two are interchangeable. The agreement analysis was added afterwards, with an adapted normalization and without separate sets of observed conditions. The combination was run once. The real-data study and the simulator still use different data and models, which Section 5 states as a limitation.

W4. Narrow evidence.

We agree, and the limitation remains: one real dataset, 19 test surfaces, one simulator. Within that scope the revision adds neural encoders next to the linear ones, uncertainty over surfaces, a ceiling, and two robustness checks. First, we tried all 1,716 ways of dividing the 13 observed scan conditions between the two sensors. The two sensors' predictions agree in every case and the acceleration error stays between 0.68 and 0.72; the audio error ranges from 0.72 to 1.21 (Appendix B.1). Second, the action-order experiment now covers six choices of the unseen pulse pair (Figure 6). The paper no longer speaks about multimodal world models in general, and Section 5 states the scope.

Thank you for responding to my concerns. The additional experiments and clarifications strengthen the paper, especially the comparison with the same supervision for both models, which addresses my concern about the step model’s advantage. However, I still find the writing difficult to follow and the main claims unclear. I believe the paper needs substantial changes to its structure and explanations, beyond language edits. The experiments also remain limited to one real dataset and one simulated system, and I am still concerned about how well the findings generalize to other settings. I will reassess my score during the discussion with the area chair and other reviewers.

Reviewer_e5FHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies multimodal alignment for physical sensors. The questions studied are 1) if similar representations make the same physical predictions, and 2) if the action order changes the representation. Studies are done on the Cluster Haptic dataset under different settings. The authors identify different factors such as cross-sensor consistency and multimodal fusion to assess physical representations.

Critical strengths

The authors study their exact physical setup thoroughly.

Critical weaknesses
  1. The writing is very difficult to parse for me. It was hard to understand what questions are being solved, the drawbacks with previous methods, the baselines, or the study. The paper is dense on physical systems without an introduction of it to a general ML audience.

  2. I am not sure if this study is useful for a wider ML audience, the core members of ICLR. There was not much spoken in terms of drawbacks of multimodal alignment, or any relevant baselines from multimodal literature. This paper seems better suited for a physics journal, since the machine learning aspect of this paper is very narrow.

Questions for authors

Not a question, but it may help to revisit the writing of this paper, and what questions are solved from a Machine Learning point-of-view. What would be in this study for the wider machine learning audience?

2 comments

Thank you for the review. We agree with the main points: the submission was hard to read, did not state its machine-learning questions, and had no baseline from the multimodal literature. We rewrote the paper and uploaded a revised PDF in which new experiments and analyses are marked in blue.

1. The questions, from a machine-learning point of view

The first question is about multimodal representation learning. When two sensors observe the same object, people often check that their embeddings are close. Does that closeness tell us that the model predicts the object's physical behavior accurately? In our task, a model gets a few scans of a new surface from one sensor (sound or acceleration) and predicts the surface's measured vibration under 67 other scan conditions.

The second question is about compositional generalization. Can a learned dynamics model predict what happens when familiar actions are applied in a new order? In our task, a simulated oscillator is pushed by force pulses A, B, and C. The model sees these pulses in some orders during training and is tested on two pulses that were never next to each other.

The paper contributes no result about the physical systems themselves. They are there to supply measured or simulated answers against which predictions can be checked.

2. A drawback of multimodal alignment, with a baseline from the multimodal literature

The revision adds the comparison with contrastive multimodal learning that the submission lacked (Section 3.5). We train neural encoders in three ways: to predict the response, to align the two sensors with a symmetric InfoNCE loss as in CLIP, or both. We then score them with standard alignment measures and with their error against measured responses.

Trained toRetrieval R@1CKAError on measured responses
Predict the response0.237 ± 0.0390.8820.713 ± 0.002
Align the two sensors0.285 ± 0.0330.3260.785 ± 0.015

R@1 is how often the nearest embedding from the other sensor belongs to the same surface. The error is relative to predicting an average surface. The two rows show the drawback. The encoder trained only for alignment is at least as good at retrieval, and its predictions are less accurate in every run. Choosing between the two by retrieval would select the less accurate encoder. The finding is specific to retrieval: CKA is much lower for the alignment-only encoder, in line with its error. We trained our own encoders and did not evaluate pretrained multimodal models.

3. Generalization to a new order of actions

Here the revision removes a flaw in our original comparison (Section 4). One model predicts one time step at a time, and the other maps the whole action sequence to the response. In the submission the first received more detailed training targets. Both now receive the same targets, with about the same number of parameters and the same training budget. On the unseen orders, the test error is 0.0177 ± 0.0041 for the step model and 0.614 ± 0.181 for the whole-sequence model. The step model is also more accurate with every seed in five further versions of the experiment that use other unseen pulse pairs (Figure 6). The more detailed targets therefore do not explain the difference.

4. What is in the study for a wider machine-learning audience?

Two implications are relevant to machine-learning model evaluation beyond these particular testbeds, although our empirical findings remain limited to the systems studied. For multimodal representation learning: a retrieval score between embeddings can stay the same or rise while the predictions made from those embeddings get worse, so a representation that will be used for prediction should also be checked against measurements. For sequence models: architectures that are compared on new combinations of familiar actions should receive the same training targets. With that control, the model that advances a state step by step still generalized much better than the model that maps the whole sequence.

5. Presentation

The introduction now starts from what the usual checks on embeddings leave open. Each physical system is introduced in everyday terms before any notation: a probe sliding over a surface while a microphone and an accelerometer record, and a mass on a spring that deforms permanently when pushed hard. Each study now introduces its task, its model, and its measures where they are needed, and invented terms are replaced by plain ones (encoder, decoder, step model). The figures were redrawn with plain labels, and the one formal statement is in the appendix. Related work now covers contrastive alignment (Radford et al., 2021), representation similarity (Kornblith et al., 2019; Huh et al., 2024), and multimodal learning for physical tasks (Lee et al., 2019; Wi et al., 2022), and says what each leaves open.

The evidence covers one real dataset and one simulator, and the conclusions are stated within that scope.

Dear authors, I thank you for the rebuttal. The paper is better structured. However, I concur with my fellow reviewers. I believe that a study on one dataset is not adequate, and your own response says that there are baselines to compare with (https://openreview.net/forum?id=Gt3XdDPBdM&noteId=YlEvGbXqpy) which are not yet compared quantitatively with. I still feel that the writing falls below the bar of an ICLR paper (subjective opinion). I will update my review during the AC-Reviewer discussion period.

Comments on the paper

1 comment
Area_Chair_tv4yOpenReview ↗

Reminder: Please Review Author Rebuttals and Participate in Discussions

Dear Reviewers,

Thank you for your time on the reviews. The authors have now submitted their rebuttals, and I encourage you to read their responses and participate in the discussion.

Please consider whether the rebuttals address your concerns and, where clarification is still needed, engage with the authors through the discussion forum. I also encourage you to exchange perspectives with your fellow reviewers and update your review or rating if appropriate.

Your participation will help ensure a thorough and fair evaluation of each paper. Thank you again for your time and effort.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.