- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces ObjDream, an object-centric vision-language-action framework centered on object-level understanding and future object state prediction.
ObjDream uses a spatio-temporal object encoder to organize visual observations into temporal object slots, where persistent object identity and dynamic object state are represented through separate channels.
These structured object representations are inserted into the token sequence of the vision-language model, where identity-guided routing modulates VLM attention to the corresponding object tokens.
The framework further introduces an action-conditioned object-level forward dynamics module, which predicts compact future object states relevant to manipulation during training and explicitly supervises object state evolution under robot actions without requiring future prediction at inference.
Inter-object relations are modeled from object features and pairwise geometry to provide structured context for action generation, and a relation token, together with prospective object tokens, is appended to the object block.
Critical strengths
-
The identity and state factorization, combined with a steering-only attention bias, is a consequential design choice here. Persistent identity is built once at episode initialization from a frozen CLIP phrase embedding and a frozen DINOv2 visual reference and is then cached, while the dynamic channel carries appearance, geometry, and time. Identity-guided routing turns the cached identity into a gated additive bias on the attention logits and deliberately leaves keys, values, and all other logits untouched, so the pretrained VLM is nudged rather than rewired.
-
The automatic offline annotation pipeline and the training-only future prediction branch make the whole approach reproducible and cheap at deployment time. Masks, geometry, visibility, and phrase-region correspondences are produced by Qwen3-VL-8B phrase extraction, Grounding DINO localization, and SAM2 mask propagation, so no manual frame-wise labeling is needed and the same recipe transfers across LIBERO, RoboTwin 2.0, and the real robot.
Critical weaknesses
-
The headline claim is stronger than the evidence supports. The paper is framed around improving generalization under distribution shift, yet on LIBERO-Plus, the benchmark built specifically to measure that, ObjDream scores 84.4 overall against 85.7 for π0.5, and it loses to π0.5 on exactly the geometric shifts its object representation ought to handle best, namely camera viewpoint, robot initialization, and object layout. On RoboTwin 2.0 Hard, it ties EventVLA and remains far behind methods with large-scale robot pretraining. In the real world, it reaches 55 against 53 for π0.5 under out-of-distribution evaluation on 80 trials, which the paper itself describes as comparable.
-
Effect sizes are small, unmeasured, and come with costs that are never accounted for. LIBERO is saturated, with ObjDream at 98.6 against 98.1 for X-VLA and 97.8 for OA-WAM, and the LIBERO-Plus ablation rows all sit within about two points of each other, with the w/o ObjRel variant at 84.1 essentially matching the full model at 84.4. The paper provides no standard deviations, no multiple seeds, and no significance tests, and the real-world study uses only 20 trials per task, so readers cannot tell which of the five proposed components matter.
Questions for authors
-
Why does ObjDream fall behind π0.5 on camera viewpoint, robot initialization, and object layout specifically, the three geometric perturbation categories, when the object representation is explicitly geometric?
-
What happens when a scene contains more than four task-relevant objects, or when the Qwen3-VL-8B phrase parser extracts the wrong phrases at episode initialization?
-
Can you state the achievable control frequency in the main text and discuss the roughly two-fold latency increase over π0.5, including how much of it is the object encoder versus the routing and action head?
2 comments
Author Response to Reviewer xvbg (Part 2/2)
We thank the reviewer again for the constructive feedback. We address the remaining questions below.
Q2: Robustness to more than four task-relevant objects and phrase errors
A3: Thank you for raising these important questions. Regarding object count, ObjDream supports a configurable number of object slots, with the capacity determined by each experimental setting. Simulation experiments use up to four task-relevant non-robot slots (Appendix A.2, p. 19, lines 1009–1011), while real-world experiments use eight non-robot slots and one robot slot, particularly to accommodate multi-object tabletop sweeping (Appendix E.1, p. 35, lines 1870–1876). This demonstrates support for larger object sets through slot configuration. However, capacity remains fixed within each setting, and scenes exceeding it may leave relevant objects without explicit object-slot representations. Increasing capacity requires object encoder adaptation and incurs additional computational cost. We clarified the real-world configuration in Appendix E.1 and updated Appendix F (p. 38, lines 2043–2048) as follows:
The current implementation uses a fixed number of task-relevant non-robot object slots, which limits its scalability to scenes containing more relevant objects than the available slot capacity. Future work will explore dynamic slot allocation to accommodate varying numbers of objects while controlling the additional inference cost.
To evaluate robustness to incorrect object initialization, we conducted additional experiments on all ten LIBERO-Long tasks (Appendix A.4, pp. 21–22, lines 1128–1152), using the same checkpoint and 50 rollouts per task (500 per condition). All conditions share identical initial states and evaluation seeds. We test missing or incorrect object phrases and swapped cached DINO references. Results are summarized below (Table 10, p. 22, lines 1135–1143).
| Initialization condition | Success rate (%) ↑ | ΔSR (pp) ↓ |
|---|---|---|
| Clean initialization | 97.2 | — |
| One missing phrase | 95.4 | 1.8 |
| One incorrect phrase | 95.8 | 1.4 |
| Two incorrect phrases | 94.0 | 3.2 |
| Swapped DINO references | 93.6 | 3.6 |
As shown in the table, these initialization errors reduce success rates by 1.4–3.6 percentage points, with performance remaining at least 93.6% across all tested conditions. The original task instructions and full visual observations remain available to the policy, providing additional context when object initialization is imperfect. These results indicate that ObjDream maintains reasonable robustness to the evaluated phrase and identity-reference errors. We have added the experimental protocol and results to Appendix A.4 and clarified the current object capacity limitation in Appendix F.
Q3: Control frequency and inference latency
A4: Thank you for raising this important efficiency concern. We have added an Inference latency and control frequency paragraph at the end of Sec. 4.3 (p. 10, lines 502–507). In real-world deployment, ObjDream executes action chunks of length 16 at a nominal frequency of 20 Hz, with policy replanning performed after each chunk. On a single NVIDIA RTX 4090 with batch size 1, the mean inference latency is 177.4 ms per action chunk, measured over 200 synchronized runs after 50 warm-up iterations. Under the same timing protocol, requires 84.6 ms per action chunk, corresponding to an approximately 2.1$\times$ latency increase.
The object encoder takes 38.3 ms. The VLM forward pass, including object-token injection and identity-guided routing, takes 68.1 ms. The four-step flow-matching action head takes 70.2 ms, accounting for the largest share of the inference latency. The higher end-to-end cost partly reflects the object-processing pathway and architectural differences.
We added the following clarification to Sec. 4.3:
Inference latency and control frequency. In real-world deployment, ObjDream executes action chunks of length 16 at a nominal frequency of 20 Hz, with policy replanning performed after each chunk. On a single NVIDIA RTX 4090 with batch size 1, the mean inference latency is 177.4 ms per action chunk, measured over 200 synchronized runs after 50 warm-up iterations. The object encoder, VLM forward pass with object-token injection and identity-guided routing, and four-step flow-matching action head take 38.3 ms, 68.1 ms, and 70.2 ms, respectively. The action head accounts for the largest share of the inference latency.
Author Response to Reviewer xvbg (Part 1/2)
We sincerely thank the reviewer for their careful evaluation and constructive feedback. We address the concerns below. For ease of review, we have highlighted the corresponding revisions in blue in the updated manuscript.
A1: Thank you for raising this important concern. One plausible contributing factor is the object encoder's pretraining setting. The encoder is pretrained exclusively on clean LIBERO demonstrations, which may limit its robustness to unseen camera viewpoints and geometric variations. We deliberately avoid incorporating LIBERO-Plus data into training to preserve the zero-shot evaluation protocol and prevent test-set leakage (Appendix A.1, p. 17, Table 4, lines 865–872). Another consideration is that ObjDream uses visual features and 2D object geometry, including masks, bounding boxes, and centers, without explicitly estimating 3D object poses. While 3D pose information could provide additional spatial cues, online pose estimation would introduce additional computational overhead and potential estimation errors. As discussed in Appendix F (p. 38, lines 2048–2050), incorporating richer 3D spatial information is a potential direction for future work.
As shown in Table 1 (p. 8, lines 379–393), ObjDream achieves a trial-weighted average of 74.3% across the three geometric perturbation categories, ranking third among the 11 evaluated methods, behind OA-WAM (84.3%) and π0.5 (79.5%). To further assess the contributions of ObjDream under geometric perturbations, we reproduce the relevant columns from the revised Table 2 (p. 9, lines 433–448), reporting success rates (%) averaged over three independent policy training seeds:
Compared with the same-backbone baseline, ObjDream improves success rates by 0.9, 2.3, and 2.6 percentage points under camera viewpoint, robot initialization, and object layout perturbations, respectively. Although ObjDream does not achieve the highest performance in these categories, these results indicate positive contributions under geometric shifts. Furthermore, ObjDream achieves the best results among the evaluated methods under language (90.6%), lighting (98.4%), and background (98.5%) perturbations, highlighting its strengths under semantic and appearance variations.
A2: Thank you for pointing out that the component-level improvements should be supported by uncertainty estimates and an analysis of inference cost. We have expanded Table 2 (p. 9, lines 433–448) to three independent training seeds for the backbone and all key ablation variants, reporting mean success rates and standard deviations. On LIBERO-Plus Overall, ObjDream achieves 84.3 ± 0.22%, versus 80.8 ± 0.38% for the Backbone baseline, 83.6 ± 0.40% without ObjRel, 82.8 ± 0.35% without ObjDyn, and 81.9 ± 0.16% without both. Further controlled comparisons show 82.6 ± 0.05% without factorized injection and 82.6 ± 0.03% without identity-guided routing. These results clarify the relative contributions of the main components, while the three-run evaluation does not establish statistical significance for every small improvement. Individual seed-level results are provided in Appendix D.1 (p. 33, Table 20, lines 1729–1744).
We also acknowledge the limited number of real-world trials (20 per task under each ID/OOD setting) and have clarified the evaluation protocol in Sec. 4.3 (pp. 9–10, lines 481–492) and Appendix E.3 (p. 37, lines 1980–1987). To account for the computational cost, we have added the measured inference latency and module-level breakdown in Sec. 4.3 (p. 10, lines 502–507), discussed further in A4 below.