acceptodds
Under review as a conference paper at ICLR 2027

Post-Training Multimodal Embedding Models towards Physical Dynamics Awareness

Abstract

Physical AI needs to connect what is observed with what can happen next, but pixel-level prediction couples learning physical dynamics with reproducing potentially irrelevant appearance details. Predicting learned representations can ease this burden while retaining the state information needed to anticipate change. Embedding models built on multimodal large language models (MLLMs) offer a promising starting point, with our experiments indicating greater utility as physical state encoders than same-size MLLMs. These embedding models combine vision–language processing with reusable representations, yet their relevance-oriented training does not directly target physical transitions. We use soft tokens and learned-query pooling to learn physical state representations alongside a frozen backbone's native retrieval embeddings, then look beyond task-specific adaptation to explore cross-task predictive post-training of a shared, dynamics-aware encoder. Frozen-backbone adaptation delivers stronger overall planning performance than LeWorldModel with fewer trainable parameters, while cross-task post-training improves the shared encoder's utility for world modeling, supporting the potential of a dynamics-aware representation foundation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Representation Learning, Multimodal Large Language Models, World Models, Latent Dynamics, Parameter-Efficient Adaptation, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_RN68OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether relevance-oriented multimodal embedding models can be adapted into useful physical state representations for image-based planning. It introduces (1) task-specific adaptation with learnable soft tokens and learned-query pooling over a frozen visual representation, with DoRA updates to the language component, and (2) a cross-task predictive post-training objective that aligns a representation of the current image conditioned on a state/action description with the embedding of a future image. Experiments cover PushT, OGBench-Cube, and TwoRoom, plus PointMaze, where the encoder is held out from post-training but the downstream planning setup uses target-environment training data. The paper also reports linear probes for physical attributes. The authors find gains from both task-specific adaptation and cross-task post-training, although the scope of the transfer and efficiency claims requires qualification.

Critical strengths
  • The paper targets a meaningful gap between semantic image representations and state representations useful for control, and tests the representations in closed-loop planning rather than relying only on static similarity metrics.
  • The evaluation compares frozen embeddings, instruction-tuned backbones, and a prior world-model baseline, and includes both task-specific adaptation and cross-task post-training variants. Appendix D also contains a relevant text ablation: for the 2B model, state-plus-action descriptions reach 85.56 average planning success, versus 79.33 with state descriptions and 76.44 with image pairs only, although the effect varies by task.
  • The proposed adaptation is compact in trainable parameters and preserves an image-only interface for the downstream planner, which could make the approach useful when full policy retraining is undesirable.
Critical weaknesses
  • On the 2B model, the full state/action descriptions improve the three-task mean over state-only and image-pairs-only training. However, the ablation is limited to one model scale, and image-pairs-only already matches the full method on TwoRoom (96.00%). No shuffled, mismatched, or matched non-physical text control is reported, so it remains unclear whether the gain requires correct physical semantics rather than additional textual conditioning. These results support a narrower simulator-planning claim, not broad physical-dynamics awareness or transfer to real visual environments.
  • Table 5’s PointMaze column should be interpreted as encoder transfer under a target-specific downstream setup. Appendix A.5 says PointMaze is excluded from encoder post-training, but uses 1,600 target-environment trajectories (160,000 frames) and target-environment training data for action-normalization statistics. Thus the full planning pipeline is not zero-shot in PointMaze. The reported gains over the un-post-trained encoders are modest, with standard deviations of 6.11 and 6.00 across three seeds; no LeWM PointMaze result is given. The split and exact role of the 1,600 trajectories should be stated, with a target-trained downstream-predictor control.
  • The LeWM comparison is not input-matched: Appendix A.4 retains LeWM’s three-frame context, while the proposed TwoRoom models use one frame. This may affect the comparison in either direction and should be controlled or explicitly qualified. In Table 5, the cross-task average also hides per-task regressions: post-trained 2B and 8B score 82.67 and 87.33 on PushT, below LeWM’s 91.33, even though their three-task averages are higher. The broad summary should acknowledge this task dependence.
  • The “parameter-efficient” count is a fine-tuning count, not an inference-cost comparison: Table 3 reports trainable parameters on PushT and excludes the frozen 2B/8B backbone.
Questions for authors
  1. In the PointMaze protocol, how are the 1,600 target-environment trajectories divided and used? Are any trajectories or derived statistics shared with evaluation cases? Please compare against a predictor trained on the same target data with the un-post-trained encoder, and describe this as encoder transfer rather than full-pipeline zero-shot transfer if appropriate.
  2. Appendix D already compares image pairs, state descriptions, and state-plus-action descriptions on the 2B model. Can you add shuffled/mismatched or matched non-physical text controls, and repeat the key ablation at 8B, to test whether semantically correct interaction information is necessary?
  3. Can you provide a matched-context LeWM comparison (one versus three frames) and per-task results alongside averages, especially the PushT regressions in Table 5?
  4. What are inference latency, peak memory, and compute for the frozen 2B/8B backbones and adapted models, compared with LeWM on the same hardware?
3 comments
AuthorsOpenReview ↗

Response to Reviewer RN68 (1/3)

We thank the reviewer for the thoughtful feedback and address the concerns below.

W1. On the 2B model, ... the ablation is limited to one model scale, ... it remains unclear whether the gain requires correct physical semantics rather than additional textual conditioning. These results support a narrower simulator-planning claim, not broad physical-dynamics awareness or transfer to real visual environments.

Q2. Appendix D already compares image pairs, state descriptions, and state-plus-action descriptions on the 2B model. Can you add shuffled/mismatched or matched non-physical text controls, and repeat the key ablation at 8B, to test whether semantically correct interaction information is necessary?

Correct action–transition correspondence improves overall planning when text format and action distribution are held fixed. A fixed action permutation within each task preserves current state descriptions, image pairs, text format, and action distribution. At each model size, both conditions use the same encoder training budget and fresh downstream models trained for ten epochs. Planning success (%) is:

ConfigurationPushT (↑)TwoRoom (↑)Cube (↑)Average (↑)
Qwen3-VL-Embedding-2B, Matched actions82.67 ± 3.0696.00 ± 0.0078.00 ± 7.2185.56
Qwen3-VL-Embedding-2B, Shuffled actions64.67 ± 5.0394.67 ± 4.1680.00 ± 3.4679.78
Qwen3-VL-Embedding-8B, Matched actions87.33 ± 3.0694.67 ± 2.3184.00 ± 6.0088.67
Qwen3-VL-Embedding-8B, Shuffled actions68.67 ± 7.0294.00 ± 3.4681.33 ± 4.1681.33

Matched actions improve average planning by 5.78/7.33 points for 2B/8B, with gains across all three 8B tasks. Because the shuffled condition retains the text format and action distribution, the results show that predictive alignment benefits from descriptions that agree with the observed transition. The 2B model scores slightly lower on Cube with matched actions. This difference is consistent with ConfigWM's finding that Cube is less sensitive than PushT to query mismatch under visual distraction [1], suggesting that action descriptions may offer limited additional guidance for this configuration. We have added this comparison of matched and shuffled actions to Appendix E.2, Table 31.

The original manuscript (Appendix D.2, Table 23) already reports the ablation using only image pairs: state/action descriptions improve PushT and Cube, while image pairs alone match their TwoRoom performance. Together, the two comparisons separate the benefit of adding descriptions from the benefit of preserving their correspondence to physical interactions.

For transfer to real observations, we compare the original encoders and those post-trained on simulation, keeping both frozen and using identical linear probes on 1,024 AgiBot trajectories. Probes use all six end-effector coordinates, decoding current state from one image and observed displacement from two:

EncoderCurrent state MSE (↓)Current state r (↑)Observed motion MSE (↓)Observed motion r (↑)
Qwen3-VL-Embedding-2B, original0.3280.8120.8390.063
Qwen3-VL-Embedding-2B, post-trained0.3110.8230.8350.089
Qwen3-VL-Embedding-8B, original0.3020.8280.8380.087
Qwen3-VL-Embedding-8B, post-trained0.2520.8590.8210.147

MSE uses standardized targets, and r is Pearson correlation. With the encoders frozen and the probe protocol fixed, post-training on simulation alone improves access to both robot state and observed motion information in real images at both model sizes. The motion result extends the transfer evidence beyond decoding the current configuration to representing change across observations. This evaluation is added in Appendix H, Table 39.

Reference:

[1] Yijun Shen, Delong Chen, Xianming Hu, Kai Zhang, and Pascale Fung. World Models with Configurable Perception. arXiv:2610.27914, 2026.

AuthorsOpenReview ↗

Response to Reviewer RN68 (2/3)

W2. Table 5’s PointMaze column should be interpreted as encoder transfer under a target-specific downstream setup. ... Thus the full planning pipeline is not zero-shot in PointMaze. The reported gains over the un-post-trained encoders are modest ...; no LeWM PointMaze result is given. The split and exact role of the 1,600 trajectories should be stated, with a target-trained downstream-predictor control.

Q1. In the PointMaze protocol, how are the 1,600 target-environment trajectories divided and used? Are any trajectories or derived statistics shared with evaluation cases? Please compare against a predictor trained on the same target data with the un-post-trained encoder, and describe this as encoder transfer rather than full-pipeline zero-shot transfer if appropriate.

The 1,600 PointMaze trajectories are used to train the downstream models, while both the original and post-trained encoders remain frozen. Encoder post-training uses only PushT, Cube, and TwoRoom. Accordingly, we now explicitly describe this evaluation as encoder transfer in Section 5.2.

Evaluation uses fresh simulator rollouts, with initial and goal configurations generated by the evaluation seeds. The action normalization statistics used in these rollouts are computed from the downstream training trajectories and kept fixed. We have clarified these training and evaluation details in Appendix A.5.

We do not report a LeWM result on PointMaze because LeWM has not released a checkpoint for this environment. The original submission already included the requested comparison with the original encoder in Table 5. Original and adapted encoders receive freshly trained downstream models on the same target data. Higher mean success at both scales shows that experience from other environments adds useful information beyond downstream training specific to the target environment.

W3. The LeWM comparison is not input-matched: Appendix A.4 retains LeWM’s three-frame context, while the proposed TwoRoom models use one frame. ... In Table 5, the cross-task average also hides per-task regressions: ... The broad summary should acknowledge this task dependence.

Q3. Can you provide a matched-context LeWM comparison (one versus three frames) and per-task results alongside averages, especially the PushT regressions in Table 5?

In the experiments reported in the original submission, both LeWM and our models already received a single observed image at each TwoRoom planning step. The wording referring to three frames could cause confusion, while the experimental setting was already consistent with LeWM's Appendix D (“Predictor Architecture”), which specifies a history length of one for TwoRoom. We have clarified the description in Section 3.2 and Appendix A.4 accordingly.

Results for each task were reported alongside the averages in the original submission (Tables 3 and 5). The comparison with LeWM is indeed task dependent: the shared post-trained encoders remain below LeWM on PushT, while the shared 8B encoder exceeds it on Cube. However, soft tokens and learned-query pooling can surpass LeWM on PushT with fewer trainable parameters while retaining native retrieval, extending a general embedding backbone to physical planning. Cross-task post-training also improves PushT and Cube planning over the original encoders at both model sizes. Both shared encoders achieve higher mean planning success than the original encoders across all four physical parameter scenarios and exceed LeWM in mean success in the Cube friction scenario (see the second part of Q4 below and Section 6, Table 7). These results support the prospect of a dynamics-aware representation foundation: heterogeneous physical experience can strengthen one encoder for multiple tasks.

AuthorsOpenReview ↗

Response to Reviewer RN68 (3/3)

W4. The “parameter-efficient” count is a fine-tuning count, not an inference-cost comparison: Table 3 reports trainable parameters on PushT and excludes the frozen 2B/8B backbone. Without total inference memory, latency, and compute against a similarly capable encoder or LeWM, the deployment-efficiency claim is not established.

Q4. What are inference latency, peak memory, and compute for the frozen 2B/8B backbones and adapted models, compared with LeWM on the same hardware?

The original submission already reports the comparison of trainable parameter counts in Table 3. To measure inference costs, we benchmark complete planning calls for all models on the same hardware, using the same precision, resolution, and planning settings. Each planning call includes encoding of current and goal images and the full CEM action search. We report median and p95 latency, peak allocated memory, and profiled GFLOPs:

DeploymentPlanning median (ms)Planning p95 (ms)Planning peak memory (GiB)Planning GFLOPs
Qwen3-VL-Embedding-2B, Last-token baseline549.36577.494.0113189.54
Qwen3-VL-Embedding-2B, Soft tokens + learned-query pooling546.99552.774.0213378.49
Qwen3-VL-Embedding-8B, Last-token baseline601.61612.6515.2324871.50
Qwen3-VL-Embedding-8B, Soft tokens + learned-query pooling603.19613.1515.2515788.67
DINOv2 ViT-L/14 (CLS)484.76489.330.6182840.06
DINOv2 ViT-L/14 (patch + pooling)487.07498.230.6202840.57
LeWM450.46455.630.0572522.80

GFLOPs are lower bounds because profiling covers only part of the computation. Task-specific adaptation trains fewer parameters than LeWM while keeping planning latency and peak memory close to its own last-token baseline. These measurements are added in Appendix I, Table 40.

Q4 (continued). Can the main findings also be reproduced across more seeds and held-out dynamics or visual domains?

The original manuscript (Appendix A.3, Table 9) already specifies three paired planning seeds and 50 common cases per seed. To test whether adaptation continues to improve planning under changed interaction dynamics, we vary mass/inertia and friction in Cube, and inertia and action gain in PushT. Mass, inertia, and Cube friction use factors of 0.5 or 2, while PushT action gain uses 0.8 or 1.2. Encoders, adaptation modules, projectors, and predictors remain frozen, with the original action normalization and no descriptions of the changed parameters. We use the same number of seeds and cases for each condition, with a budget of 50 steps.

Soft tokens and learned-query pooling use S = Q = 32. Planning success (%) is:

Model / configurationCube mass/inertia (↑)Cube friction (↑)PushT inertia (↑)PushT action gain (↑)
Qwen3-VL-Embedding-2B, Last-token baseline71.0069.3357.0054.67
Qwen3-VL-Embedding-2B, Soft tokens + learned-query pooling76.6777.3376.3375.67
Qwen3-VL-Embedding-2B, Cross-task post-training75.6776.3368.0062.33
Qwen3-VL-Embedding-8B, Last-token baseline68.3369.0066.3362.33
Qwen3-VL-Embedding-8B, Soft tokens + learned-query pooling78.6777.6777.0075.67
Qwen3-VL-Embedding-8B, Cross-task post-training76.3378.0070.3371.33
LeWM76.0075.0071.6770.33
DINOv2 ViT-L/14 (patch + pooling)76.3375.3377.0075.00

Every measured adapted configuration improves its corresponding native encoder in mean success for each scenario. Both adaptation strategies retain their planning advantage across these four scenarios without updating the deployed models. All four adaptations also exceed both external baselines on Cube friction. We have added these results to Section 6, Table 7. Appendix F.1 gives the deployment protocol, and Appendix F.2, Tables 32–35 gives all results for each condition. For real image transfer, see the AgiBot results in this response, W1/Q2, and Appendix H, Table 39.

Reviewer_xfFHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the potential of multimodal embedding models as physical state encoders for world modeling. It first demonstrates that Qwen3-VL-Embedding models generally outperform their instruction-tuned counterparts in downstream planning tasks. The authors then introduce two adaptation strategies: task-specific adaptation using soft tokens and learned-query pooling, and cross-task predictive post-training using language-conditioned future-state alignment. Experiments demonstrate improvements in planning performance and physical-state prediction.

Critical strengths
  1. Exploring multimodal embedding models for physical world modeling is a meaningful research direction, and the comparison between embedding and instruction-tuned models provides useful empirical insights.

  2. The proposed methods improve downstream performance with relatively limited trainable parameters.

  3. The paper evaluates planning performance, physical-state probing, and cross-environment transfer, with encouraging results.

Critical weaknesses

1. Insufficient validation of physical dynamics awareness. The experiments primarily evaluate observable state variables and downstream planning performance, without directly assessing intrinsic physical properties such as mass, friction, or changes in contact dynamics. Although the authors define dynamics awareness in terms of action-conditioned state prediction, the current evidence is insufficient to establish whether the learned representations capture generalizable physical dynamics beyond the evaluated environments.

2. Limited methodological novelty. The proposed approaches primarily combine established techniques, including soft prompting, learned-query pooling, LoRA, and predictive feature alignment. The paper does not sufficiently distinguish its core methodological contribution from existing predictive representation learning approaches. This limits the significance of the technical contribution and weakens the case for acceptance.

Questions for authors
  1. What exactly constitutes physical dynamics awareness in this work? The term "physical dynamics" may suggest understanding intrinsic physical properties such as mass, friction, and material characteristics, or physical processes such as combustion. However, the experiments primarily evaluate observable state variables, including object positions and gripper states, alongside downstream planning performance. While these are relevant to physical dynamics, they do not necessarily demonstrate an understanding of underlying physical laws. Could the authors clarify the scope of physical dynamics awareness and consider evaluating generalization under changes in physical properties, such as object mass or friction?

  2. Why do multimodal embedding models outperform instruction-tuned MLLMs? The paper attributes the advantage to relevance-oriented representation learning. However, the results do not establish whether this advantage comes from the embedding objective, training data, or representation extraction strategy. Could the authors provide stronger evidence supporting this explanation?

  3. What is the key methodological novelty? The proposed methods combine established techniques, including soft prompting, learned-query pooling, LoRA, and future-feature prediction. Could the authors better distinguish the proposed approach from existing predictive representation learning methods and clarify its primary technical contribution?

  4. How generalizable are the learned physical representations? Cross-task post-training is evaluated on only three relatively constrained environments, with PointMaze used for out-of-distribution testing. Moreover, some physical-state probing results do not improve consistently. Could the authors provide stronger evidence of generalization to unseen physical interactions, dynamics, or real-world environments?

5 comments
AuthorsOpenReview ↗

Response to Reviewer xfFH (1/3)

We thank the reviewer for the thoughtful feedback and address the concerns below.

W1. Insufficient validation of physical dynamics awareness. ... without directly assessing intrinsic physical properties such as mass, friction, or changes in contact dynamics. ... the current evidence is insufficient to establish whether the learned representations capture generalizable physical dynamics beyond the evaluated environments.

Q1. What exactly constitutes physical dynamics awareness in this work? ... Could the authors clarify the scope of physical dynamics awareness and consider evaluating generalization under changes in physical properties, such as object mass or friction?

We test transfer under changes in physical parameters across four scenarios while keeping the deployed models frozen. The suite covers Cube mass/inertia and friction, and PushT inertia and action gain. Mass, inertia, and Cube friction use factors of 0.5 or 2, while PushT action gain uses 0.8 or 1.2. Encoders, adaptation modules, projectors, and predictors remain frozen, with the original action normalization and no descriptions of the changed parameters. Each condition uses three paired evaluation seeds, 50 common cases per seed, and a budget of 50 steps. Soft tokens and learned-query pooling use S = Q = 32. Planning success (%) is:

Model / configurationCube mass/inertia (↑)Cube friction (↑)PushT inertia (↑)PushT action gain (↑)
Qwen3-VL-Embedding-2B, Last-token baseline71.0069.3357.0054.67
Qwen3-VL-Embedding-2B, Soft tokens + learned-query pooling76.6777.3376.3375.67
Qwen3-VL-Embedding-2B, Cross-task post-training75.6776.3368.0062.33
Qwen3-VL-Embedding-8B, Last-token baseline68.3369.0066.3362.33
Qwen3-VL-Embedding-8B, Soft tokens + learned-query pooling78.6777.6777.0075.67
Qwen3-VL-Embedding-8B, Cross-task post-training76.3378.0070.3371.33
LeWM76.0075.0071.6770.33
DINOv2 ViT-L/14 (patch + pooling)76.3375.3377.0075.00

Both adaptation strategies achieve higher mean planning success than their native embeddings in each of the four scenarios. This includes the mass and friction changes requested here. On Cube friction, all four adapted configurations also exceed both LeWM and DINOv2 in mean success. The results relative to the native encoders show that adaptation remains useful when interaction conditions change at deployment.

The operational definition is clarified in the Introduction. We have added these results to Section 6, Table 7. Appendix F.1 gives the deployment protocol, and Appendix F.2, Tables 32–35 gives all results for each condition.

Physical dynamics awareness is operationalized through a representation's usefulness for predicting action-conditioned change. We complement these transfer tests with action-conditioned probes that directly assess this predictive information, as reported in our response to Reviewer UhFM, W5/Q2, and Appendix E.1, Table 29.

W2. Limited methodological novelty. The proposed approaches primarily combine established techniques, including soft prompting, learned-query pooling, LoRA, and predictive feature alignment. The paper does not sufficiently distinguish its core methodological contribution from existing predictive representation learning approaches. This limits the significance of the technical contribution and weakens the case for acceptance.

Q3. What is the key methodological novelty? The proposed methods combine established techniques, including soft prompting, learned-query pooling, LoRA, and future-feature prediction. Could the authors better distinguish the proposed approach from existing predictive representation learning methods and clarify its primary technical contribution?

Our contribution connects multimodal embeddings trained for retrieval with world modeling through learning from physical interaction. Predictive supervision reshapes feature extraction on a frozen backbone while preserving native retrieval (Section 4). This makes physical specialization a way to expand a general embedding model's capabilities, extending the reach of multimodal pretraining into physical interaction. Language brings heterogeneous controls into a shared encoder's predictive learning objective (Section 5). The broader prospect is a dynamics-aware representation foundation that turns physical experience into shared knowledge for world modeling across environments.

AuthorsOpenReview ↗

Response to Reviewer xfFH (2/3)

Q2. Why do multimodal embedding models outperform instruction-tuned MLLMs? The paper attributes the advantage to relevance-oriented representation learning. However, the results do not establish whether this advantage comes from the embedding objective, training data, or representation extraction strategy. Could the authors provide stronger evidence supporting this explanation?

We test whether the embedding advantage extends beyond last-token extraction by comparing both frozen 8B Qwen models using image token means from the final layer, centered using the training set. Both models use the same downstream architecture, training data, budget of ten epochs, and planning cases. Planning success (%) is:

ModelPushT (↑)TwoRoom (↑)Cube (↑)Average (↑)
Qwen3-VL-8B-Instruct76.00 ± 11.1493.33 ± 2.3170.67 ± 6.4380.00
Qwen3-VL-Embedding-8B78.67 ± 1.1594.67 ± 1.1571.33 ± 6.4381.56

The embedding model achieves higher mean planning success on all three tasks when both models use image token mean pooling. Its advantage therefore extends beyond the native last-token interface.

Learning representations for relevance matching motivates our choice of embedding models as a starting point. The objective and pretraining data are coupled in the released models. This comparison directly addresses the representation extraction part of the question. We have added these results to Appendix G.3, Table 38, and summarized the finding in Section 3.2.

We complement this extraction comparison with evidence from a second model family, presented in our response to Reviewer UhFM, W3/Q5, and Appendix G.2, Table 37.

AuthorsOpenReview ↗

Response to Reviewer xfFH (3/3)

Q4. How generalizable are the learned physical representations? Cross-task post-training is evaluated on only three relatively constrained environments, with PointMaze used for out-of-distribution testing. Moreover, some physical-state probing results do not improve consistently. Could the authors provide stronger evidence of generalization to unseen physical interactions, dynamics, or real-world environments?

We address these three aspects in turn.

(1) Unseen physical interactions: The physical parameter tests in W1/Q1 above show that both adaptation strategies retain their planning advantage across four scenarios with interaction conditions unseen during training. The original submission also reports improved planning with both post-trained encoders in PointMaze, an environment excluded from encoder post-training (original Table 5).

(2) Dynamics: To complement the tests with changed parameters, we examine whether cross-task post-training improves the predictive information available in the representations themselves. We compare the original and post-trained Qwen3-VL-Embedding-2B encoders using linear probes of physical state changes over five steps. Both encoders are frozen during probing and supply last-token representations with 2048 dimensions. Each probe receives image representations concatenated with recorded numerical actions. Both encoders use the same image contexts, action sequences, data splits, and fitting protocols. MSE uses standardized targets, and r is Pearson correlation.

TaskModelMSE (↓)Pearson r (↑)
PushTQwen3-VL-Embedding-2B, original0.2750.826
PushTQwen3-VL-Embedding-2B, post-trained0.2050.875
TwoRoomQwen3-VL-Embedding-2B, original0.0580.970
TwoRoomQwen3-VL-Embedding-2B, post-trained0.0540.971
CubeQwen3-VL-Embedding-2B, original0.3170.782
CubeQwen3-VL-Embedding-2B, post-trained0.2460.828

Cross-task post-training lowers MSE and raises Pearson r on all three tasks. With both probes receiving the same recorded actions, these results show that post-training makes information about physical change more accessible in the image representations. We have added these probes to Section 3.2, with full results in Appendix E.1, Table 29.

(3) Real-world environments: We freeze encoders post-trained exclusively on the three simulated tasks and evaluate AgiBot recordings from 1,024 real robot trajectories. All models share the same images, splits grouped by trajectory, and linear probe procedure. Targets retain all six Cartesian end-effector coordinates of the two arms. The current state probe uses one image, while the observed motion probe decodes displacement from the difference between two observed image embeddings five recorded steps apart.

ModelCurrent state MSE (↓)Current state r (↑)Observed motion MSE (↓)Observed motion r (↑)
Qwen3-VL-Embedding-2B, original0.3280.8120.8390.063
Qwen3-VL-Embedding-2B, post-trained0.3110.8230.8350.089
Qwen3-VL-Embedding-8B, original0.3020.8280.8380.087
Qwen3-VL-Embedding-8B, post-trained0.2520.8590.8210.147

Post-training on simulation alone improves both state and observed motion decoding from real robot images at both scales. Because no real image data enter this post-training, these results support transferring physical representation learning across the simulation-to-real visual gap.

We have added the real image results to Appendix H, Table 39, with an overview in Section 6. Together, these evaluations show that the learned representations provide information about action consequences and remain useful across interaction conditions, downstream environments, and real robot observations.

Reviewer_UhFMOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

the paper studies if MLLM based multimodal embeddings models can be good state encoders for action conditioned latent style world models. It proposes 2 adaptations via soft tokens and learned query pooling for a parameter efficient adaptation recipe. Experiments are done on 3 simulation tasks with qwen3 based backbone and the 3 findings are: 1. embedding models beat base MLLM'S 2. soft tokens and learned query pooling provide good adaptation recipe for frozen backbones. 3. cross task posttraining improves the shared encoder. Both soft prompts and pooling improve results on planning, with a small gain on the held-out PointMaze.

Critical strengths
  1. well motivated problem: can MLLM models which are retrieval trained, also provide state embeddings for latent world models ?
  2. Reasonable recipe for adaptation design, to preserve the backbone capabilities
  3. average planning improves over the baseline, particularly with the native last token baseline.4. detailed experimental setup for reproducibility
Critical weaknesses
  1. paper is framed around 'physcial dynamics aware' but the tests are done on 3 simple simulation environment with no clutter, easier background and state-action space. The framing and claims around a 'dynamics aware representations' are overstated than the experiments done
  2. for tasks and benchmarks: some tasks like twotoom are nearly saturated, so difficult to isolate the relevance of method
  3. claims of MLLM embedding models being better than MLLM are overstated, the experiments are done on only one Qwen-3 based backbone. Paper would benefit with trying out the same recipe on a different backbone to claim generality.
  4. Missing external baselines: currently the only external baseline is LeWM, there isn't any vision-only encoder based baseline which are doing good in current papers. It is recommended to add frozen vision only encoder(dino vjepa style) ones in the same predictor and planning settings to see the advantages of MLLLM based embedding models better.
  5. the current probes don't measure dynamics, this is a big missing point given the paper's theme is around predicting the change.
  6. It is not clear from the results, if probe quality tracks the planning better as well. The results are mixed for different tasks here. Hence, difficult to conclude. Also the paper states the results and numbers but analyis around when probes predict planning success is limited in the currenrt state.
  7. Finding 1, 2 gains are mixed, gains are higher on pushT but the model is worse on tworoom at 2b. there isn't explanation for why embedding training helps as well.
Questions for authors
  1. How would the method compare with vision-only encoders like dino and vjepa2 style ones under identical conditions.
  2. Can authors add dynamics probes and further analysis relationship between probe quality and success in planning?
  3. Why does adapted 8B model not outperform the adapter 2B one ?
  4. Can you extend the method on more visually realistic and harder benchmarks or real robot data?
  5. Can you try to extend the experiments to one more MLLM based embedder family, to see if the adapters generalize there ?
6 comments
AuthorsOpenReview ↗

Response to Reviewer UhFM (1/4)

We thank the reviewer for the thoughtful feedback and address the concerns below.

W1. paper is framed around 'physcial dynamics aware' but the tests are done on 3 simple simulation environment with no clutter, easier background and state-action space. The framing and claims around a 'dynamics aware representations' are overstated than the experiments done

Q4. Can you extend the method on more visually realistic and harder benchmarks or real robot data?

Dynamics awareness has a concrete predictive meaning here: retaining visual information that helps predict the changes produced by actions. We test whether physical adaptation learned in simulation also improves representations of real robot observations.

On AgiBot World 2026, original and post-trained encoders stay frozen and share images, data splits, and ridge fitting. One image predicts six current end-effector coordinates, while the difference of two observed image embeddings decodes observed motion. MSE uses standardized coordinate targets, and both MSE and Pearson r average equally over all six coordinates.

Encoder / configurationCurrent state MSE (↓)Current state r (↑)Observed motion MSE (↓)Observed motion r (↑)
Qwen3-VL-Embedding-2B, original0.3280.8120.8390.063
Qwen3-VL-Embedding-2B, Cross-task post-training0.3110.8230.8350.089
Qwen3-VL-Embedding-8B, original0.3020.8280.8380.087
Qwen3-VL-Embedding-8B, Cross-task post-training0.2520.8590.8210.147

Both scales improve both measurements using the same probe protocol. Post-training on simulation alone makes state and observed motion information more accessible in real robot images.

We have added this real image transfer evaluation to Section 6 and Appendix H, Table 39.

We complement these real image results with the action-conditioned probes and planning analysis in our responses to W5 and W6/Q2 below (Appendix E.1, Tables 29–30). Our response to W2 below further evaluates transfer under changed physical parameters (Section 6, Table 7).

W2. for tasks and benchmarks: some tasks like twotoom are nearly saturated, so difficult to isolate the relevance of method

We further test whether adaptation continues to improve planning under changed interaction dynamics by varying mass/inertia and friction in Cube, and inertia and action gain in PushT. Mass, inertia, and Cube friction use factors of 0.5 or 2, while PushT action gain uses 0.8 or 1.2. Encoders, adaptation modules, projectors, and predictors remain frozen, with the original action normalization and no descriptions of the changed parameters. Each condition uses three paired evaluation seeds, 50 common cases per seed, and a budget of 50 steps.

Soft tokens and learned-query pooling use S = Q = 32. Planning success (%) is:

Model / configurationCube mass/inertia (↑)Cube friction (↑)PushT inertia (↑)PushT action gain (↑)
Qwen3-VL-Embedding-2B, Last-token baseline71.0069.3357.0054.67
Qwen3-VL-Embedding-2B, Soft tokens + learned-query pooling76.6777.3376.3375.67
Qwen3-VL-Embedding-2B, Cross-task post-training75.6776.3368.0062.33
Qwen3-VL-Embedding-8B, Last-token baseline68.3369.0066.3362.33
Qwen3-VL-Embedding-8B, Soft tokens + learned-query pooling78.6777.6777.0075.67
Qwen3-VL-Embedding-8B, Cross-task post-training76.3378.0070.3371.33
LeWM76.0075.0071.6770.33
DINOv2 ViT-L/14 (patch + pooling)76.3375.3377.0075.00

Every measured adapted configuration improves its corresponding last-token baseline in mean success for each scenario. Both adaptation strategies therefore retain their planning advantage across these four scenarios with the original deployments frozen. All four adaptations exceed both LeWM and DINOv2 patch pooling in mean planning success under Cube friction shifts. This makes the adaptation benefit visible beyond the nearly saturated TwoRoom comparison.

We have added these results to Section 6, Table 7. Appendix F.1 gives the deployment protocol, and Appendix F.2, Tables 32–35 gives the complete results for each condition.

AuthorsOpenReview ↗

Response to Reviewer UhFM (2/4)

W3. claims of MLLM embedding models being better than MLLM are overstated, the experiments are done on only one Qwen-3 based backbone. Paper would benefit with trying out the same recipe on a different backbone to claim generality.

Q5. Can you try to extend the experiments to one more MLLM based embedder family, to see if the adapters generalize there ?

Beyond Qwen, we compare the native representations of VLM2Vec-Full and its backbone, Phi-3.5-vision-instruct. We also evaluate the soft tokens and learned-query pooling of Section 4 on VLM2Vec-Full with S = Q = 32. The backbone weights remain frozen in all configurations, which use the full task data, a training budget of one epoch, and the same planning cases.

Model / configurationPushT (%) (↑)Cube (%) (↑)Average (↑)
Phi-3.5-vision-instruct, Last-token baseline56.00 ± 5.2972.00 ± 5.2964.00
VLM2Vec-Full, Last-token baseline58.67 ± 8.3374.00 ± 3.4666.33
VLM2Vec-Full, Soft tokens + learned-query pooling70.67 ± 5.0378.00 ± 10.0074.33

The native VLM2Vec-Full encoder achieves higher mean success than its MLLM backbone on both tasks. Soft tokens and learned-query pooling further improve PushT and Cube success by 12.00 and 4.00 percentage points, raising the mean across the two tasks from 66.33% to 74.33%. The same frozen-backbone adaptation improves planning in a second embedding model family.

We have added these results to Appendix G.2, Table 37. Section 3.2 also summarizes the comparison of native encoders.

W4. Missing external baselines: currently the only external baseline is LeWM, there isn't any vision-only encoder based baseline which are doing good in current papers. It is recommended to add frozen vision only encoder(dino vjepa style) ones in the same predictor and planning settings to see the advantages of MLLLM based embedding models better.

Q1. How would the method compare with vision-only encoders like dino and vjepa2 style ones under identical conditions.

We compare DINOv2 ViT-L/14 with each approach at the same representation granularity. All downstream world models are trained for ten epochs, use the same predictor architecture and states with 192 dimensions, and are evaluated on common planning cases.

For token representations, we compare the soft tokens and learned-query pooling of Section 4 with DINOv2 patch features. Both use 32 learned queries, with S = Q = 32 for our adaptation.

ModelConfigurationPushT (%) (↑)TwoRoom (%) (↑)Cube (%) (↑)Average (↑)
DINOv2 ViT-L/14Patch + pooling96.00 ± 2.0092.00 ± 3.4678.00 ± 3.4688.67
Qwen3-VL-Embedding-2BSoft tokens + learned-query pooling94.67 ± 1.1595.33 ± 2.3182.67 ± 5.0390.89
Qwen3-VL-Embedding-8BSoft tokens + learned-query pooling92.67 ± 3.0693.33 ± 3.0680.00 ± 2.0088.67

Frozen-backbone adaptation matches or exceeds DINOv2 patch pooling in overall planning success.

The transfer results also favor frozen-backbone adaptation over DINOv2 patch pooling. In the complete comparison across four scenarios presented in our response to W2 above and Section 6, Table 7, both frozen-backbone adaptations exceed DINOv2 patch pooling in mean success on Cube mass/inertia, Cube friction, and PushT action gain. The 8B adaptation matches it on PushT inertia. The comparison with DINOv2 therefore favors frozen-backbone adaptation both in overall performance on the standard tasks and in planning under changed interaction dynamics.

For global representations, we compare the cross-task post-trained encoders of Section 5, using last-token embeddings, with DINOv2's CLS output.

ModelConfigurationPushT (%) (↑)TwoRoom (%) (↑)Cube (%) (↑)Average (↑)
DINOv2 ViT-L/14CLS73.33 ± 6.1196.67 ± 2.3178.67 ± 7.5782.89
Qwen3-VL-Embedding-2BCross-task post-training82.67 ± 3.0696.00 ± 0.0078.00 ± 7.2185.56
Qwen3-VL-Embedding-8BCross-task post-training87.33 ± 3.0694.67 ± 2.3184.00 ± 6.0088.67

Both cross-task post-trained encoders outperform DINOv2's CLS baseline in overall planning success.

We have added the token comparison to Section 4.2, Table 3, and the global comparison to Section 5.2, Table 5. Appendix G.1, Table 36 provides the complete results and protocol, and Section 6, Table 7 reports transfer under changed physical parameters.

AuthorsOpenReview ↗

Response to Reviewer UhFM (3/4)

W5. the current probes don't measure dynamics, this is a big missing point given the paper's theme is around predicting the change.

We test this directly with linear probes of physical state changes over five steps, keeping the encoders frozen. Image-only uses image representations, Action-only uses the five recorded numerical actions, and Joint concatenates both. PushT and Cube use image history, while TwoRoom uses the current frame. Numeric current state and future images are excluded. We compare the 2B MLLM and embedding configurations with LeWM. MSE uses standardized targets.

TaskModel / configurationImage-only MSE (↓)Action-only MSE (↓)Joint MSE (↓)Joint Pearson r (↑)
PushTQwen3-VL-2B-Instruct0.5450.5050.3010.804
PushTQwen3-VL-Embedding-2B, Last-token baseline0.5120.5050.2750.826
PushTQwen3-VL-Embedding-2B, Soft tokens + learned-query pooling0.3760.5050.1910.885
PushTQwen3-VL-Embedding-2B, Cross-task post-training0.3650.5050.2050.875
PushTLeWM0.3830.5050.1980.880
TwoRoomQwen3-VL-2B-Instruct0.9090.0750.0580.970
TwoRoomQwen3-VL-Embedding-2B, Last-token baseline0.9030.0750.0580.970
TwoRoomQwen3-VL-Embedding-2B, Soft tokens + learned-query pooling0.8980.0750.0540.971
TwoRoomQwen3-VL-Embedding-2B, Cross-task post-training0.8960.0750.0540.971
TwoRoomLeWM0.9040.0750.0540.972
CubeQwen3-VL-2B-Instruct0.6820.3320.2970.795
CubeQwen3-VL-Embedding-2B, Last-token baseline0.6860.3320.3170.782
CubeQwen3-VL-Embedding-2B, Soft tokens + learned-query pooling0.5360.3320.2230.845
CubeQwen3-VL-Embedding-2B, Cross-task post-training0.5790.3320.2460.828
CubeLeWM0.5520.3320.2270.840

Both adaptation strategies make action consequences more accessible in the representations. Relative to the native embedding, both lower Joint MSE and raise Joint Pearson r on all three tasks. Frozen-backbone adaptation also outperforms LeWM in both metrics on PushT and Cube.

Joint prediction improves on either input alone for every model and task, showing that visual representations supply predictive state information beyond the recorded actions. All joint predictors also improve on a persistence prediction of no change, whose MSE is 0.935 on PushT, 1.015 on TwoRoom, and 0.963 on Cube.

We have added these probes to Section 3.2. Appendix E.1, Table 29 contains the probe definition, input comparisons, and results.

W6. It is not clear from the results, if probe quality tracks the planning better as well. The results are mixed for different tasks here. Hence, difficult to conclude. Also the paper states the results and numbers but analyis around when probes predict planning success is limited in the currenrt state.

Q2. Can authors add dynamics probes and further analysis relationship between probe quality and success in planning?

Our response to W5 above presents the requested dynamics probes and complete results, which we have added to Section 3.2 and Appendix E.1, Table 29.

The static probes in the original manuscript (Section 3.2) assess how well the representations expose physical state information relevant to action-conditioned prediction and comparison with visual goals. We correlate static and dynamic probe MSE with planning success within each task across five configurations: MLLM, Last-token baseline, Soft tokens + learned-query pooling, Cross-task post-training, and LeWM. Each configuration is one observation in this Pearson correlation analysis. Negative values indicate that lower probe error is associated with higher planning success.

TaskModelsStatic MSE vs. planning success (Pearson r)Dynamic MSE vs. planning success (Pearson r)
PushT5−0.982−0.950
TwoRoom5−0.946+0.354
Cube5−0.736−0.870

The original static probes consistently track planning performance across all three tasks. Lower static MSE is associated with higher planning success in both manipulation and navigation, supporting the original choice of state decoding as a diagnostic of representations relevant to planning.

Dynamic probe MSE is strongly negatively correlated with planning success on PushT and Cube. This relationship does not hold on TwoRoom, where dynamic errors cluster at 0.054–0.058. The original static probes are therefore better indicators of planning utility across tasks.

We have added this comparison and analysis to Appendix E.1, Table 30.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

In the revised manuscript, additions are highlighted in blue, and deletions are marked in red with strikethrough.

  1. Evaluation overview. Added a brief overview of the new predictive and transfer evaluations (Abstract and Section 1). Corresponding comments: UhFM W1/Q4; xfFH W1/Q1 and Q4.

  2. Dynamics probes and planning analysis. Added action-conditioned state change probes and analysis of the relationship between static/dynamic probe errors and planning success (Section 3.2; Appendix E.1, Tables 29–30). Corresponding comments: UhFM W5, W6/Q2; xfFH Q4.

  3. Transfer under changed physical parameters. Added four physical parameter transfer evaluations and their protocols (Section 6, Table 7; Appendices F.1–F.2, Tables 32–35). Corresponding comments: UhFM W1 and W2; xfFH W1/Q1 and Q4; RN68 Q4.

  4. Backbone scale in transfer. Added a comparison and discussion of backbone scale within the physical parameter transfer evaluation (Appendix F.2). Corresponding comment: UhFM Q3.

  5. Real robot transfer. Added current state and observed motion probes, action-conditioned future state prediction, and input comparisons on AgiBot recordings, with a main text overview (Section 6; Appendices H.1–H.2, Tables 39–41). Corresponding comments: UhFM W1/Q4; xfFH Q4; RN68 W1/Q2 and Q4, follow-up question 1, and subsequent discussion.

  6. Vision-only baselines. Added DINOv2 patch and CLS comparisons for the token and global representations, respectively (Section 4.2, Table 3; Section 5.2, Table 5; Appendix G.1, Table 36). Corresponding comments: UhFM W4/Q1.

  7. Additional model family. Added the native comparison of VLM2Vec-Full and Phi-3.5-vision-instruct and results for soft tokens and learned-query pooling on VLM2Vec-Full, with corresponding training protocol details (Section 3.2; Appendix G.2, Table 37; Appendix A.3). Corresponding comments: UhFM W3/Q5 and W7; xfFH Q2.

  8. Alternative representation extraction. Added an image token mean pooling comparison (Section 3.2; Appendix G.3, Table 38). Corresponding comments: xfFH Q2; UhFM W7.

  9. Shared language interface. Clarified how one encoder accommodates controls with different dimensions and semantics (Section 5.1). Corresponding comments: xfFH W2/Q3.

  10. Action description ablation. Added comparisons of matched and shuffled action descriptions at both model scales (Section 5.3; Appendix E.2, Table 31). Corresponding comments: RN68 W1/Q2.

  11. PointMaze transfer protocol. Clarified encoder transfer, use of target data, action normalization statistics, and evaluation case generation (Section 5.2; Appendix A.5). Corresponding comments: RN68 W2/Q1.

  12. TwoRoom observation context. Corrected the description of the input of a single image used in the experiments (Section 3.2; Appendix A.4). Corresponding comments: RN68 W3/Q3.

  13. Planning costs. Added measurements of latency, peak memory, and profiled computation for complete planning calls (Appendix I, Table 42). Corresponding comments: RN68 W4/Q4, follow-up question 2, and subsequent discussion.

  14. Limitations and trade-offs. Added a dedicated discussion of the scope of real robot evaluation, including readouts trained on real data and the remaining validation of closed-loop planning. Clarified trainable parameter efficiency and the trade-off between planning performance, shared multimodal functionality, and deployment memory (Appendix J). Corresponding comments: RN68 follow-up questions 1–2 and subsequent discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.