- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether relevance-oriented multimodal embedding models can be adapted into useful physical state representations for image-based planning. It introduces (1) task-specific adaptation with learnable soft tokens and learned-query pooling over a frozen visual representation, with DoRA updates to the language component, and (2) a cross-task predictive post-training objective that aligns a representation of the current image conditioned on a state/action description with the embedding of a future image. Experiments cover PushT, OGBench-Cube, and TwoRoom, plus PointMaze, where the encoder is held out from post-training but the downstream planning setup uses target-environment training data. The paper also reports linear probes for physical attributes. The authors find gains from both task-specific adaptation and cross-task post-training, although the scope of the transfer and efficiency claims requires qualification.
Critical strengths
- The paper targets a meaningful gap between semantic image representations and state representations useful for control, and tests the representations in closed-loop planning rather than relying only on static similarity metrics.
- The evaluation compares frozen embeddings, instruction-tuned backbones, and a prior world-model baseline, and includes both task-specific adaptation and cross-task post-training variants. Appendix D also contains a relevant text ablation: for the 2B model, state-plus-action descriptions reach 85.56 average planning success, versus 79.33 with state descriptions and 76.44 with image pairs only, although the effect varies by task.
- The proposed adaptation is compact in trainable parameters and preserves an image-only interface for the downstream planner, which could make the approach useful when full policy retraining is undesirable.
Critical weaknesses
- On the 2B model, the full state/action descriptions improve the three-task mean over state-only and image-pairs-only training. However, the ablation is limited to one model scale, and image-pairs-only already matches the full method on TwoRoom (96.00%). No shuffled, mismatched, or matched non-physical text control is reported, so it remains unclear whether the gain requires correct physical semantics rather than additional textual conditioning. These results support a narrower simulator-planning claim, not broad physical-dynamics awareness or transfer to real visual environments.
- Table 5’s PointMaze column should be interpreted as encoder transfer under a target-specific downstream setup. Appendix A.5 says PointMaze is excluded from encoder post-training, but uses 1,600 target-environment trajectories (160,000 frames) and target-environment training data for action-normalization statistics. Thus the full planning pipeline is not zero-shot in PointMaze. The reported gains over the un-post-trained encoders are modest, with standard deviations of 6.11 and 6.00 across three seeds; no LeWM PointMaze result is given. The split and exact role of the 1,600 trajectories should be stated, with a target-trained downstream-predictor control.
- The LeWM comparison is not input-matched: Appendix A.4 retains LeWM’s three-frame context, while the proposed TwoRoom models use one frame. This may affect the comparison in either direction and should be controlled or explicitly qualified. In Table 5, the cross-task average also hides per-task regressions: post-trained 2B and 8B score 82.67 and 87.33 on PushT, below LeWM’s 91.33, even though their three-task averages are higher. The broad summary should acknowledge this task dependence.
- The “parameter-efficient” count is a fine-tuning count, not an inference-cost comparison: Table 3 reports trainable parameters on PushT and excludes the frozen 2B/8B backbone.
Questions for authors
- In the PointMaze protocol, how are the 1,600 target-environment trajectories divided and used? Are any trajectories or derived statistics shared with evaluation cases? Please compare against a predictor trained on the same target data with the un-post-trained encoder, and describe this as encoder transfer rather than full-pipeline zero-shot transfer if appropriate.
- Appendix D already compares image pairs, state descriptions, and state-plus-action descriptions on the 2B model. Can you add shuffled/mismatched or matched non-physical text controls, and repeat the key ablation at 8B, to test whether semantically correct interaction information is necessary?
- Can you provide a matched-context LeWM comparison (one versus three frames) and per-task results alongside averages, especially the PushT regressions in Table 5?
- What are inference latency, peak memory, and compute for the frozen 2B/8B backbones and adapted models, compared with LeWM on the same hardware?
3 comments
Response to Reviewer RN68 (2/3)
W2. Table 5’s PointMaze column should be interpreted as encoder transfer under a target-specific downstream setup. ... Thus the full planning pipeline is not zero-shot in PointMaze. The reported gains over the un-post-trained encoders are modest ...; no LeWM PointMaze result is given. The split and exact role of the 1,600 trajectories should be stated, with a target-trained downstream-predictor control.
Q1. In the PointMaze protocol, how are the 1,600 target-environment trajectories divided and used? Are any trajectories or derived statistics shared with evaluation cases? Please compare against a predictor trained on the same target data with the un-post-trained encoder, and describe this as encoder transfer rather than full-pipeline zero-shot transfer if appropriate.
The 1,600 PointMaze trajectories are used to train the downstream models, while both the original and post-trained encoders remain frozen. Encoder post-training uses only PushT, Cube, and TwoRoom. Accordingly, we now explicitly describe this evaluation as encoder transfer in Section 5.2.
Evaluation uses fresh simulator rollouts, with initial and goal configurations generated by the evaluation seeds. The action normalization statistics used in these rollouts are computed from the downstream training trajectories and kept fixed. We have clarified these training and evaluation details in Appendix A.5.
We do not report a LeWM result on PointMaze because LeWM has not released a checkpoint for this environment. The original submission already included the requested comparison with the original encoder in Table 5. Original and adapted encoders receive freshly trained downstream models on the same target data. Higher mean success at both scales shows that experience from other environments adds useful information beyond downstream training specific to the target environment.
W3. The LeWM comparison is not input-matched: Appendix A.4 retains LeWM’s three-frame context, while the proposed TwoRoom models use one frame. ... In Table 5, the cross-task average also hides per-task regressions: ... The broad summary should acknowledge this task dependence.
Q3. Can you provide a matched-context LeWM comparison (one versus three frames) and per-task results alongside averages, especially the PushT regressions in Table 5?
In the experiments reported in the original submission, both LeWM and our models already received a single observed image at each TwoRoom planning step. The wording referring to three frames could cause confusion, while the experimental setting was already consistent with LeWM's Appendix D (“Predictor Architecture”), which specifies a history length of one for TwoRoom. We have clarified the description in Section 3.2 and Appendix A.4 accordingly.
Results for each task were reported alongside the averages in the original submission (Tables 3 and 5). The comparison with LeWM is indeed task dependent: the shared post-trained encoders remain below LeWM on PushT, while the shared 8B encoder exceeds it on Cube. However, soft tokens and learned-query pooling can surpass LeWM on PushT with fewer trainable parameters while retaining native retrieval, extending a general embedding backbone to physical planning. Cross-task post-training also improves PushT and Cube planning over the original encoders at both model sizes. Both shared encoders achieve higher mean planning success than the original encoders across all four physical parameter scenarios and exceed LeWM in mean success in the Cube friction scenario (see the second part of Q4 below and Section 6, Table 7). These results support the prospect of a dynamics-aware representation foundation: heterogeneous physical experience can strengthen one encoder for multiple tasks.
Response to Reviewer RN68 (3/3)
W4. The “parameter-efficient” count is a fine-tuning count, not an inference-cost comparison: Table 3 reports trainable parameters on PushT and excludes the frozen 2B/8B backbone. Without total inference memory, latency, and compute against a similarly capable encoder or LeWM, the deployment-efficiency claim is not established.
Q4. What are inference latency, peak memory, and compute for the frozen 2B/8B backbones and adapted models, compared with LeWM on the same hardware?
The original submission already reports the comparison of trainable parameter counts in Table 3. To measure inference costs, we benchmark complete planning calls for all models on the same hardware, using the same precision, resolution, and planning settings. Each planning call includes encoding of current and goal images and the full CEM action search. We report median and p95 latency, peak allocated memory, and profiled GFLOPs:
| Deployment | Planning median (ms) | Planning p95 (ms) | Planning peak memory (GiB) | Planning GFLOPs |
|---|---|---|---|---|
| Qwen3-VL-Embedding-2B, Last-token baseline | 549.36 | 577.49 | 4.011 | 3189.54 |
| Qwen3-VL-Embedding-2B, Soft tokens + learned-query pooling | 546.99 | 552.77 | 4.021 | 3378.49 |
| Qwen3-VL-Embedding-8B, Last-token baseline | 601.61 | 612.65 | 15.232 | 4871.50 |
| Qwen3-VL-Embedding-8B, Soft tokens + learned-query pooling | 603.19 | 613.15 | 15.251 | 5788.67 |
| DINOv2 ViT-L/14 (CLS) | 484.76 | 489.33 | 0.618 | 2840.06 |
| DINOv2 ViT-L/14 (patch + pooling) | 487.07 | 498.23 | 0.620 | 2840.57 |
| LeWM | 450.46 | 455.63 | 0.057 | 2522.80 |
GFLOPs are lower bounds because profiling covers only part of the computation. Task-specific adaptation trains fewer parameters than LeWM while keeping planning latency and peak memory close to its own last-token baseline. These measurements are added in Appendix I, Table 40.
Q4 (continued). Can the main findings also be reproduced across more seeds and held-out dynamics or visual domains?
The original manuscript (Appendix A.3, Table 9) already specifies three paired planning seeds and 50 common cases per seed. To test whether adaptation continues to improve planning under changed interaction dynamics, we vary mass/inertia and friction in Cube, and inertia and action gain in PushT. Mass, inertia, and Cube friction use factors of 0.5 or 2, while PushT action gain uses 0.8 or 1.2. Encoders, adaptation modules, projectors, and predictors remain frozen, with the original action normalization and no descriptions of the changed parameters. We use the same number of seeds and cases for each condition, with a budget of 50 steps.
Soft tokens and learned-query pooling use S = Q = 32. Planning success (%) is:
| Model / configuration | Cube mass/inertia (↑) | Cube friction (↑) | PushT inertia (↑) | PushT action gain (↑) |
|---|---|---|---|---|
| Qwen3-VL-Embedding-2B, Last-token baseline | 71.00 | 69.33 | 57.00 | 54.67 |
| Qwen3-VL-Embedding-2B, Soft tokens + learned-query pooling | 76.67 | 77.33 | 76.33 | 75.67 |
| Qwen3-VL-Embedding-2B, Cross-task post-training | 75.67 | 76.33 | 68.00 | 62.33 |
| Qwen3-VL-Embedding-8B, Last-token baseline | 68.33 | 69.00 | 66.33 | 62.33 |
| Qwen3-VL-Embedding-8B, Soft tokens + learned-query pooling | 78.67 | 77.67 | 77.00 | 75.67 |
| Qwen3-VL-Embedding-8B, Cross-task post-training | 76.33 | 78.00 | 70.33 | 71.33 |
| LeWM | 76.00 | 75.00 | 71.67 | 70.33 |
| DINOv2 ViT-L/14 (patch + pooling) | 76.33 | 75.33 | 77.00 | 75.00 |
Every measured adapted configuration improves its corresponding native encoder in mean success for each scenario. Both adaptation strategies retain their planning advantage across these four scenarios without updating the deployed models. All four adaptations also exceed both external baselines on Cube friction. We have added these results to Section 6, Table 7. Appendix F.1 gives the deployment protocol, and Appendix F.2, Tables 32–35 gives all results for each condition. For real image transfer, see the AgiBot results in this response, W1/Q2, and Appendix H, Table 39.
Response to Reviewer RN68 (1/3)
We thank the reviewer for the thoughtful feedback and address the concerns below.
Correct action–transition correspondence improves overall planning when text format and action distribution are held fixed. A fixed action permutation within each task preserves current state descriptions, image pairs, text format, and action distribution. At each model size, both conditions use the same encoder training budget and fresh downstream models trained for ten epochs. Planning success (%) is:
Matched actions improve average planning by 5.78/7.33 points for 2B/8B, with gains across all three 8B tasks. Because the shuffled condition retains the text format and action distribution, the results show that predictive alignment benefits from descriptions that agree with the observed transition. The 2B model scores slightly lower on Cube with matched actions. This difference is consistent with ConfigWM's finding that Cube is less sensitive than PushT to query mismatch under visual distraction [1], suggesting that action descriptions may offer limited additional guidance for this configuration. We have added this comparison of matched and shuffled actions to Appendix E.2, Table 31.
The original manuscript (Appendix D.2, Table 23) already reports the ablation using only image pairs: state/action descriptions improve PushT and Cube, while image pairs alone match their TwoRoom performance. Together, the two comparisons separate the benefit of adding descriptions from the benefit of preserving their correspondence to physical interactions.
For transfer to real observations, we compare the original encoders and those post-trained on simulation, keeping both frozen and using identical linear probes on 1,024 AgiBot trajectories. Probes use all six end-effector coordinates, decoding current state from one image and observed displacement from two:
MSE uses standardized targets, and r is Pearson correlation. With the encoders frozen and the probe protocol fixed, post-training on simulation alone improves access to both robot state and observed motion information in real images at both model sizes. The motion result extends the transfer evidence beyond decoding the current configuration to representing change across observations. This evaluation is added in Appendix H, Table 39.
Reference:
[1] Yijun Shen, Delong Chen, Xianming Hu, Kai Zhang, and Pascale Fung. World Models with Configurable Perception. arXiv:2610.27914, 2026.