acceptodds
Under review as a conference paper at ICLR 2027

ObjDream: Guiding Vision-Language-Action Models with Object-Level Understanding and Future State Prediction

Abstract

Vision-language-action (VLA) models have emerged as a powerful paradigm for robotic manipulation, enabling robots to generate actions from visual observations and language instructions. However, most existing VLA models still operate on image-level visual representations inherited from pretrained vision-language backbones, while lacking fine-grained object understanding for manipulation. As a result, their robustness can degrade under out-of-distribution object layouts, clutter, and background variations. To address these limitations, we introduce ObjDream, an object-centric VLA framework centered on object-level understanding and future object state prediction. ObjDream uses a spatio-temporal object encoder to organize visual observations into temporal object slots, where persistent object identity and dynamic object state are represented through separate channels. These structured object representations are inserted into the token sequence of the vision-language model (VLM), where identity-guided routing modulates VLM attention to the corresponding object tokens. In addition, an action-conditioned object dynamics module predicts compact future object states relevant to manipulation during training, explicitly supervising object state evolution under robot actions without requiring future prediction at inference. Experiments on LIBERO, LIBERO-Plus, and RoboTwin 2.0 show that ObjDream improves generalization under diverse distribution shifts and scene perturbations. Real-world robot experiments further demonstrate that ObjDream transfers to physical manipulation settings with novel object arrangements and background changes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Object-Centric Representation, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xvbgOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces ObjDream, an object-centric vision-language-action framework centered on object-level understanding and future object state prediction.

ObjDream uses a spatio-temporal object encoder to organize visual observations into temporal object slots, where persistent object identity and dynamic object state are represented through separate channels.

These structured object representations are inserted into the token sequence of the vision-language model, where identity-guided routing modulates VLM attention to the corresponding object tokens.

The framework further introduces an action-conditioned object-level forward dynamics module, which predicts compact future object states relevant to manipulation during training and explicitly supervises object state evolution under robot actions without requiring future prediction at inference.

Inter-object relations are modeled from object features and pairwise geometry to provide structured context for action generation, and a relation token, together with prospective object tokens, is appended to the object block.

Critical strengths
  • The identity and state factorization, combined with a steering-only attention bias, is a consequential design choice here. Persistent identity is built once at episode initialization from a frozen CLIP phrase embedding and a frozen DINOv2 visual reference and is then cached, while the dynamic channel carries appearance, geometry, and time. Identity-guided routing turns the cached identity into a gated additive bias on the attention logits and deliberately leaves keys, values, and all other logits untouched, so the pretrained VLM is nudged rather than rewired.

  • The automatic offline annotation pipeline and the training-only future prediction branch make the whole approach reproducible and cheap at deployment time. Masks, geometry, visibility, and phrase-region correspondences are produced by Qwen3-VL-8B phrase extraction, Grounding DINO localization, and SAM2 mask propagation, so no manual frame-wise labeling is needed and the same recipe transfers across LIBERO, RoboTwin 2.0, and the real robot.

Critical weaknesses
  • The headline claim is stronger than the evidence supports. The paper is framed around improving generalization under distribution shift, yet on LIBERO-Plus, the benchmark built specifically to measure that, ObjDream scores 84.4 overall against 85.7 for π0.5, and it loses to π0.5 on exactly the geometric shifts its object representation ought to handle best, namely camera viewpoint, robot initialization, and object layout. On RoboTwin 2.0 Hard, it ties EventVLA and remains far behind methods with large-scale robot pretraining. In the real world, it reaches 55 against 53 for π0.5 under out-of-distribution evaluation on 80 trials, which the paper itself describes as comparable.

  • Effect sizes are small, unmeasured, and come with costs that are never accounted for. LIBERO is saturated, with ObjDream at 98.6 against 98.1 for X-VLA and 97.8 for OA-WAM, and the LIBERO-Plus ablation rows all sit within about two points of each other, with the w/o ObjRel variant at 84.1 essentially matching the full model at 84.4. The paper provides no standard deviations, no multiple seeds, and no significance tests, and the real-world study uses only 20 trials per task, so readers cannot tell which of the five proposed components matter.

Questions for authors
  • Why does ObjDream fall behind π0.5 on camera viewpoint, robot initialization, and object layout specifically, the three geometric perturbation categories, when the object representation is explicitly geometric?

  • What happens when a scene contains more than four task-relevant objects, or when the Qwen3-VL-8B phrase parser extracts the wrong phrases at episode initialization?

  • Can you state the achievable control frequency in the main text and discuss the roughly two-fold latency increase over π0.5, including how much of it is the object encoder versus the routing and action head?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer xvbg (Part 1/2)

We sincerely thank the reviewer for their careful evaluation and constructive feedback. We address the concerns below. For ease of review, we have highlighted the corresponding revisions in blue in the updated manuscript.

W1 & Q1: Geometric robustness under distribution shifts

A1: Thank you for raising this important concern. One plausible contributing factor is the object encoder's pretraining setting. The encoder is pretrained exclusively on clean LIBERO demonstrations, which may limit its robustness to unseen camera viewpoints and geometric variations. We deliberately avoid incorporating LIBERO-Plus data into training to preserve the zero-shot evaluation protocol and prevent test-set leakage (Appendix A.1, p. 17, Table 4, lines 865–872). Another consideration is that ObjDream uses visual features and 2D object geometry, including masks, bounding boxes, and centers, without explicitly estimating 3D object poses. While 3D pose information could provide additional spatial cues, online pose estimation would introduce additional computational overhead and potential estimation errors. As discussed in Appendix F (p. 38, lines 2048–2050), incorporating richer 3D spatial information is a potential direction for future work.

As shown in Table 1 (p. 8, lines 379–393), ObjDream achieves a trial-weighted average of 74.3% across the three geometric perturbation categories, ranking third among the 11 evaluated methods, behind OA-WAM (84.3%) and π0.5\pi_{0.5} (79.5%). To further assess the contributions of ObjDream under geometric perturbations, we reproduce the relevant columns from the revised Table 2 (p. 9, lines 433–448), reporting success rates (%) averaged over three independent policy training seeds:

VariantCam. (%)Robot (%)Layout (%)
Backbone baseline65.473.679.0
ObjDream (Full)66.375.981.6

Compared with the same-backbone baseline, ObjDream improves success rates by 0.9, 2.3, and 2.6 percentage points under camera viewpoint, robot initialization, and object layout perturbations, respectively. Although ObjDream does not achieve the highest performance in these categories, these results indicate positive contributions under geometric shifts. Furthermore, ObjDream achieves the best results among the evaluated methods under language (90.6%), lighting (98.4%), and background (98.5%) perturbations, highlighting its strengths under semantic and appearance variations.

W2: Statistical reliability, component contributions, and inference cost

A2: Thank you for pointing out that the component-level improvements should be supported by uncertainty estimates and an analysis of inference cost. We have expanded Table 2 (p. 9, lines 433–448) to three independent training seeds for the backbone and all key ablation variants, reporting mean success rates and standard deviations. On LIBERO-Plus Overall, ObjDream achieves 84.3 ± 0.22%, versus 80.8 ± 0.38% for the Backbone baseline, 83.6 ± 0.40% without ObjRel, 82.8 ± 0.35% without ObjDyn, and 81.9 ± 0.16% without both. Further controlled comparisons show 82.6 ± 0.05% without factorized injection and 82.6 ± 0.03% without identity-guided routing. These results clarify the relative contributions of the main components, while the three-run evaluation does not establish statistical significance for every small improvement. Individual seed-level results are provided in Appendix D.1 (p. 33, Table 20, lines 1729–1744).

We also acknowledge the limited number of real-world trials (20 per task under each ID/OOD setting) and have clarified the evaluation protocol in Sec. 4.3 (pp. 9–10, lines 481–492) and Appendix E.3 (p. 37, lines 1980–1987). To account for the computational cost, we have added the measured inference latency and module-level breakdown in Sec. 4.3 (p. 10, lines 502–507), discussed further in A4 below.

AuthorsOpenReview ↗

Author Response to Reviewer xvbg (Part 2/2)

We thank the reviewer again for the constructive feedback. We address the remaining questions below.

Q2: Robustness to more than four task-relevant objects and phrase errors

A3: Thank you for raising these important questions. Regarding object count, ObjDream supports a configurable number of object slots, with the capacity determined by each experimental setting. Simulation experiments use up to four task-relevant non-robot slots (Appendix A.2, p. 19, lines 1009–1011), while real-world experiments use eight non-robot slots and one robot slot, particularly to accommodate multi-object tabletop sweeping (Appendix E.1, p. 35, lines 1870–1876). This demonstrates support for larger object sets through slot configuration. However, capacity remains fixed within each setting, and scenes exceeding it may leave relevant objects without explicit object-slot representations. Increasing capacity requires object encoder adaptation and incurs additional computational cost. We clarified the real-world configuration in Appendix E.1 and updated Appendix F (p. 38, lines 2043–2048) as follows:

The current implementation uses a fixed number of task-relevant non-robot object slots, which limits its scalability to scenes containing more relevant objects than the available slot capacity. Future work will explore dynamic slot allocation to accommodate varying numbers of objects while controlling the additional inference cost.

To evaluate robustness to incorrect object initialization, we conducted additional experiments on all ten LIBERO-Long tasks (Appendix A.4, pp. 21–22, lines 1128–1152), using the same checkpoint and 50 rollouts per task (500 per condition). All conditions share identical initial states and evaluation seeds. We test missing or incorrect object phrases and swapped cached DINO references. Results are summarized below (Table 10, p. 22, lines 1135–1143).

Initialization conditionSuccess rate (%) ↑ΔSR (pp) ↓
Clean initialization97.2—
One missing phrase95.41.8
One incorrect phrase95.81.4
Two incorrect phrases94.03.2
Swapped DINO references93.63.6

As shown in the table, these initialization errors reduce success rates by 1.4–3.6 percentage points, with performance remaining at least 93.6% across all tested conditions. The original task instructions and full visual observations remain available to the policy, providing additional context when object initialization is imperfect. These results indicate that ObjDream maintains reasonable robustness to the evaluated phrase and identity-reference errors. We have added the experimental protocol and results to Appendix A.4 and clarified the current object capacity limitation in Appendix F.

Q3: Control frequency and inference latency

A4: Thank you for raising this important efficiency concern. We have added an Inference latency and control frequency paragraph at the end of Sec. 4.3 (p. 10, lines 502–507). In real-world deployment, ObjDream executes action chunks of length 16 at a nominal frequency of 20 Hz, with policy replanning performed after each chunk. On a single NVIDIA RTX 4090 with batch size 1, the mean inference latency is 177.4 ms per action chunk, measured over 200 synchronized runs after 50 warm-up iterations. Under the same timing protocol, π0.5\pi_{0.5} requires 84.6 ms per action chunk, corresponding to an approximately 2.1$\times$ latency increase.

The object encoder takes 38.3 ms. The VLM forward pass, including object-token injection and identity-guided routing, takes 68.1 ms. The four-step flow-matching action head takes 70.2 ms, accounting for the largest share of the inference latency. The higher end-to-end cost partly reflects the object-processing pathway and architectural differences.

We added the following clarification to Sec. 4.3:

Inference latency and control frequency. In real-world deployment, ObjDream executes action chunks of length 16 at a nominal frequency of 20 Hz, with policy replanning performed after each chunk. On a single NVIDIA RTX 4090 with batch size 1, the mean inference latency is 177.4 ms per action chunk, measured over 200 synchronized runs after 50 warm-up iterations. The object encoder, VLM forward pass with object-token injection and identity-guided routing, and four-step flow-matching action head take 38.3 ms, 68.1 ms, and 70.2 ms, respectively. The action head accounts for the largest share of the inference latency.

Reviewer_USyZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ObjDream, an object-centric VLA framework for robust robotic manipulation. It separates persistent object identity from dynamic object state, injects these representations into a VLM through object tokens and identity-guided attention, and predicts action-conditioned future object states as auxiliary training supervision. Experiments on LIBERO, LIBERO-Plus, RoboTwin 2.0, and real-robot tasks show strong standard performance and competitive robustness under environmental perturbations.

Critical strengths
  1. ObjDream presents a coherent object-centric VLA design that combines identity-state factorization, identity-guided attention, and future object-state supervision. This provides a principled way to incorporate structured object information into action generation.

  2. The paper evaluates the method across LIBERO, LIBERO-Plus, RoboTwin 2.0, and real-robot tasks. The broad evaluation demonstrates strong standard manipulation performance and competitive robustness under several environmental perturbations.

Critical weaknesses
  1. The empirical gains are limited relative to the method’s complexity. ObjDream underperforms π0.5\pi_0.5 on LIBERO-Plus overall, while most component-level improvements in Table 2 are only 0.3–1.9 points and lack multi-seed results or uncertainty estimates.

  2. The ablations do not cleanly isolate the main contributions. In particular, the paper lacks a simple object-token baseline with neither identity/state factorization nor identity-guided routing, making it unclear whether the gains come from the proposed mechanisms or simply from additional object features and supervision.

Questions for authors
  1. Can the authors report multi-seed results or confidence intervals for the backbone and key ablations in Table 2? Since most component-level gains are below two percentage points, their statistical stability would materially affect the assessment.

  2. Can the authors provide a controlled 2×22\times2 ablation of unified versus factorized object tokens and standard versus identity-guided attention? In particular, the missing unified-token-without-routing configuration is necessary to determine whether the gains come from the proposed mechanisms or simply from additional object features and supervision.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer USyZ

We sincerely thank the reviewer for the careful evaluation of our manuscript and the constructive suggestions. We have addressed these concerns as detailed below. For ease of review, we have highlighted the corresponding revisions in blue in the updated manuscript.

W1 & Q1: Multi-seed evaluation and statistical reliability

A1: Thank you for highlighting the need for more reliable ablation evidence. Following your suggestion, we have extended the experiments to three independent training seeds for the backbone baseline, the full model, and all key ablation variants. We now report mean success rates and standard deviations across training runs in the revised Table 2 (p. 9, lines 433–448). The relevant results are summarized below.

VariantLIBERO-Long (%)LIBERO-Plus Overall (%)
Backbone baseline94.9 ± 0.5080.8 ± 0.38
w/o ObjRel & ObjDyn95.1 ± 0.1281.9 ± 0.16
w/o ObjDyn95.5 ± 0.6482.8 ± 0.35
w/o ObjRel96.5 ± 0.4283.6 ± 0.40
ObjDream (Full)97.1 ± 0.3184.3 ± 0.22

On LIBERO-Plus, the full model outperforms the Backbone baseline by 3.5 percentage points, while achieving a 2.2 percentage point improvement on LIBERO-Long. Removing ObjDyn reduces the LIBERO-Plus overall success rate from 84.3% to 82.8%, while removing ObjRel yields 83.6%. Removing both components further reduces performance to 81.9%. On LIBERO-Plus, the full model consistently outperforms these variants across all three training seeds, providing stronger evidence for the stability of the observed improvements and the complementary contributions of relation modeling and future object state supervision. We have updated Table 2 and Sec. 4.2 (Ablation and Analysis) accordingly. The seed-level results for all ablation variants on LIBERO-Long and LIBERO-Plus Overall are additionally provided in Appendix D.1 (p. 33, Table 20, lines 1729–1744).

W2 & Q2: Controlled ablations of factorized object injection and identity-guided routing

A2: Thank you for this valuable suggestion. We have added the requested unified-token-without-routing configuration and organized the ablations into a controlled 2×2 comparison of unified versus factorized object injection and standard attention versus identity-guided routing. Unified tokens are constructed through a single projection of the object encoder's identity and state features, while factorized tokens use separate projections. Across these configurations, we retain the same pretrained object encoder, object-token count, relation modeling, future object state supervision, backbone initialization, training data, and optimization budget. This allows us to evaluate the contributions of token factorization and identity-guided routing under matched conditions.

The relevant results from the revised Table 2 (p. 9, lines 433–448) are summarized below. LIBERO-Plus overall success rates (%) are reported as mean ± standard deviation over three independent training seeds.

Object tokensAttention mechanismAblation variantOverall (%)
UnifiedStandardw/o Fact. Injection & Id. Routing81.9 ± 0.25
UnifiedIdentity-guidedw/o Fact. Injection82.6 ± 0.05
FactorizedStandardw/o Id. Routing82.6 ± 0.03
FactorizedIdentity-guidedObjDream (Full)84.3 ± 0.22

Starting from unified tokens with standard attention (81.9%), introducing identity-guided routing increases the overall success rate to 82.6%, while adopting factorized injection also increases it to 82.6%. Combining both mechanisms achieves 84.3%, outperforming the configuration without either mechanism by approximately 2.4 percentage points. These results support the complementary contributions of factorized object injection and identity-guided routing.

To further examine the benefits of object-level information, we also introduce a simple Object Token Baseline using the same backbone, pretrained object encoder, training data, and optimization budget. This baseline directly injects unified object tokens with standard attention, without factorized injection, identity-guided routing, ObjRel, ObjDyn, or auxiliary object-state supervision. It achieves 81.0 ± 0.17% on LIBERO-Plus, compared with 80.8 ± 0.38% for the Backbone baseline and 84.3 ± 0.22% for the full model. The marginal improvement from direct object-token injection highlights the importance of effectively integrating and supervising object-level information. Together, these controlled comparisons provide further empirical support for the proposed factorized object injection, identity-guided routing, and auxiliary object-state supervision. The complete results and configuration details are included in the revised Table 2 and Sec. 4.2 (pp. 8–9, lines 422–463), with additional ablation details in Appendix D.1 (pp. 32–33, lines 1727–1761).

We sincerely thank you again for your constructive feedback. Following your suggestions, we have provided additional experimental results and clarifications in our rebuttal and revised manuscript.

If time permits, we would greatly appreciate any further feedback you may have on our responses. We would be happy to clarify any remaining questions or concerns.

Thank you again for your time and consideration.

The response substantially improves the reliability and attribution of the results, but it does not change my main concern about the magnitude of the contribution. The controlled 2×22\times2 ablation shows that factorized injection and identity-guided routing individually improve LIBERO-Plus by only 0.7 points, while the complete system requires several additional foundation models, a separately pretrained object encoder, modified attention, auxiliary prediction modules, and approximately twice the reported inference latency of the real-robot baselines. Moreover, the full model remains below π0.5\pi_0.5 on LIBERO-Plus overall, and the real-world OOD advantage is only marginal. I therefore acknowledge that the method is empirically sound, but I do not find the demonstrated benefit sufficiently consequential relative to the system complexity, and I maintain my rating of 2.

Reviewer_EcQ2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ObjDream, an object-centric VLA framework that factorizes object slots into persistent identity and dynamic state, injects these slots into a VLM token sequence, uses identity-guided attention routing, and adds an action-conditioned future object-state prediction branch as auxiliary training supervision. The method is evaluated on LIBERO, LIBERO-Plus, a fixed 8-task RoboTwin 2.0 subset, and four real-world manipulation tasks.

Critical strengths

The separation between persistent object identity and time-varying object state is a sensible design choice, and the paper makes a reasonable attempt to carry this factorization from the object encoder into the VLM interface rather than treating object tokens as a single undifferentiated embedding.

Using future object-state prediction only as auxiliary supervision during training, while avoiding explicit future-state decoding at inference, is a practical design decision that avoids some of the inference overhead of video/world-model-style VLA approaches.

The empirical scope is broad: standard LIBERO, zero-shot LIBERO-Plus perturbations, RoboTwin 2.0 Easy/Hard, and real-world ID/OOD experiments. The perturbation-focused evaluation is more informative than nominal success rates alone.

Critical weaknesses

The core novelty is narrower than the framing suggests. Dual-state slot models already separate identity from dynamic appearance, and object-centric tokens/routing for VLA policies are an active line of work. The proposed identity-guided routing appears to be an additive attention bias computed from a cached identity reference, with a learned gate, on top of otherwise standard VLM attention. That is a reasonable engineering choice, but the paper does not convincingly isolate why this specific routing mechanism is necessary rather than simply adding more object tokens or a standard cross-attention adapter. The claim that identity-guided routing is a key contributor is not cleanly supported by the ablation: in Table 2, the full model’s overall LIBERO-Plus score (88.6) is actually slightly lower than the “w/o ObjRel” variant (88.9). The authors need to address this directly rather than describing the components as complementary in general terms.

The comparison story is not fully fair or fully convincing. The paper compares against π0.5, which benefits from large-scale robot pretraining, while ObjDream is presented as not relying on such pretraining. This makes the LIBERO-Plus aggregate comparison difficult to interpret: ObjDream is second overall and loses to π0.5 on the aggregate, yet the paper emphasizes robustness gains. Conversely, on real-world OOD, ObjDream is only comparable to π0.5 in aggregate (55 vs 53) and is worse on button pressing under ID (80 vs 90). The paper needs a same-backbone, same-data, same-training-budget baseline with object tokens but without the proposed routing/future-state machinery. Without that control, the gains are hard to attribute.

The ablation evidence is weak and lacks statistical grounding. Several improvements are small (e.g., LIBERO-Long 97.4 vs 96.6; LIBERO-Plus overall 88.6 vs 88.9 for a variant), and no confidence intervals, seed variance, or per-task variability are reported. For a method paper making robustness claims under distribution shift, reporting only mean success rates over what appear to be single checkpoints is not enough. The real-world protocol is also under-specified: number of rollouts per task, OOD definition, success criteria, whether baselines were tuned under the same budget, and whether the same checkpoint was used across ID/OOD are not clearly stated in the main text.

The manuscript presentation has gaps that affect assessability. In the version I received, key tables (e.g., Table 1 and Table 16) are incomplete/truncated, and the real-world table reports aggregate numbers without per-task rollout counts. This may be a PDF extraction issue, but the authors should ensure complete tables and a clear evaluation protocol in the revision. Also, the claim that future object-state prediction captures “nontrivial” state changes is supported by a small analysis on three LIBERO tasks; it is useful but not sufficient to justify the generality of the auxiliary objective.

Questions for authors

In Table 2, the full model’s overall LIBERO-Plus score (88.6) is lower than the “w/o ObjRel” variant (88.9). How do the authors reconcile this with the claim that relation modeling and future object-state supervision provide complementary gains? Please report the exact aggregation and whether this difference is within run-to-run variance.

For the real-world experiments, please state the number of rollouts per task under ID and OOD, the exact OOD shifts, the success criterion, and whether all methods used the same demonstration data, training budget, and checkpoint-selection protocol. The current aggregate averages are difficult to interpret without these details.

Can the authors provide a same-backbone baseline with object tokens but without identity-guided routing and without future-state supervision, trained with the same data and optimization budget? This is the control needed to attribute the reported gains to the proposed mechanism rather than to additional object-level parameters or longer training.

What exactly does identity-guided routing add beyond a standard attention bias/adapter? Since keys and values are computed from the full contextualized tokens and the identity term only modifies logits before softmax, please quantify the gain from the routing bias alone versus simply adding the cached identity as an extra token or using standard cross-attention.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer EcQ2 (Part 1/3)

We sincerely thank the reviewer for the careful evaluation of our manuscript and the constructive suggestions. We address the concerns below. For ease of review, we have highlighted the corresponding revisions in blue in the updated manuscript.

W1, W3 & Q1: Clarification of LIBERO-Plus ablation results, score aggregation, and statistical reliability

A1: Thank you for highlighting these concerns. We first clarify that the original Table 2 results of 88.6% (Full) and 88.9% (w/o ObjRel) refer to the Sensor Noise category, not LIBERO-Plus Overall. For clarity, we reproduce the relevant columns and two rows from the originally submitted Table 2 (p. 9, lines 433–444), whose caption explicitly identifies Overall as the trial-weighted LIBERO-Plus average:

VariantLIBERO-Plus Noise (%)LIBERO-Plus Overall (%)
w/o ObjRel88.984.1
ObjDream (Full)88.684.4

The corresponding Overall scores are 84.4% and 84.1%, respectively. For ObjDream and its ablations, Overall is the trial-weighted average across 10,030 evaluation episodes: 1,599 Camera, 1,550 Robot, 1,525 Layout, 1,537 Language, 1,142 Light, 1,076 Background, and 1,601 Noise. Thus, Sensor Noise accounts for only 16.0% of the benchmark.

To assess statistical reliability, we extended the ablation study to three independent training seeds and report mean success rates and standard deviations. The full model achieves 84.3 ± 0.22% on LIBERO-Plus Overall, compared with 83.6 ± 0.40% without ObjRel, an average improvement of approximately 0.6 percentage points based on unrounded results. The gains are positive across all three seeds (0.3, 0.9, and 0.7 percentage points). For Sensor Noise, the corresponding results are 88.2 ± 0.62% and 88.1 ± 0.98%. This small difference relative to run-to-run variability indicates that the original single-run reversal is not a consistent degradation.

Removing ObjDyn reduces Overall to 82.8%, while removing both ObjRel and ObjDyn further reduces it to 81.9%, supporting their complementary contributions. We have updated Table 2 (p. 9, lines 433–448) and Sec. 4.2 with three-seed results and component analyses, and clarified the aggregation protocol and individual seed-level results in Appendix D.1.

W3 & Q2: Real-world evaluation protocol and fairness

A2: Thank you for your suggestion. The real-world evaluation protocol was originally detailed in Appendix E, and we have now incorporated the key information into Sec. 4.3 (pp. 9–10, lines 481–492). All methods use the same demonstration data (100 demonstrations per task, 400 in total) and follow their respective training protocols, with detailed training configurations provided in Appendix E.2 (pp. 35–37, lines 1887–1978). Each method is evaluated over 20 trials per task under both ID and OOD settings (80 trials per setting). OOD evaluation covers background, lighting, object layout, and clutter, with five trials per type per task and matched perturbation configurations across methods. For each method, the same trained checkpoint is used for both ID and OOD evaluation without additional fine-tuning. We have also clarified the task-specific success criteria in Appendix E.3 (p. 38, Table 22, lines 1999–2006).

The revised paragraph in Sec. 4.3 (Real-world experiments, pp. 9–10, lines 481–492) reads as follows:

We evaluate ObjDream on four real-world tasks: button pressing, block pick-and-place, two-fruit placement, and tabletop sweeping, covering contact interaction, object manipulation, and long-horizon control. We compare against π0.5\pi_{0.5} and ControlVLA. All methods use the same 100 demonstrations per task (400 in total), with baselines following their official implementations and respective training protocols. Each method is evaluated over 20 trials per task under both in-distribution (ID) and out-of-distribution (OOD) settings, yielding 80 trials per setting. OOD shifts include background, lighting, object layout, and clutter variations, with 5 trials per perturbation type for each task. The same perturbation configurations are applied to all methods for each task. For each method, the same trained checkpoint is used for ID and OOD evaluation without additional fine-tuning. A trial is considered successful only upon completing the full task objective: pressing the target button, placing the block into the designated box gap, placing both fruits into the basket, or sweeping all target blocks into the collection box, respectively. Further training and evaluation details are provided in Appendix E.

AuthorsOpenReview ↗

Author Response to Reviewer EcQ2 (Part 2/3)

We thank the reviewer again for the constructive feedback. We continue our responses to the remaining concerns below.

W2 & Q3: Controlled object token baseline under matched training conditions

A3: Thank you for this important suggestion. We have added an Object Token Baseline using the same backbone, pretrained initialization, training data, and policy optimization budget. It directly injects unified object tokens without factorized injection, identity-guided routing, ObjRel, ObjDyn, or auxiliary object-state supervision. All variants use three independent training seeds.

Revised Table 2 (p. 9, lines 433–448) reports success rates (%) as mean ± standard deviation across three training seeds.

VariantLIBERO-LongLIBERO-Plus Overall
Backbone baseline94.9 ± 0.5080.8 ± 0.38
Object Token Baseline95.9 ± 0.9981.0 ± 0.17
ObjDream (Full)97.1 ± 0.3184.3 ± 0.22

The Object Token Baseline improves LIBERO-Plus by only approximately 0.2 percentage points over the Backbone baseline. The full model achieves 84.3 ± 0.22%, exceeding the Object Token Baseline by approximately 3.3 percentage points. Together with the component ablations in Table 2, these results support the contributions of factorized injection, identity-guided routing, and auxiliary object-state supervision. Detailed ablation configurations are provided in Sec. 4.2 (pp. 8–9, lines 422–463) and Appendix D.1 (pp. 32–33, lines 1727–1761).

W1 & Q4: Contribution of identity-guided routing compared with alternative attention mechanisms

A4: Thank you for raising this important question. Identity-guided routing matches cached identity references with contextualized queries to compute a gated additive bias for valid object tokens. The bias adjusts attention logits before softmax, while standard Keys and Values are computed from contextualized tokens, preserving dynamic object information. The mechanism is detailed in Sec. 3.2 (p. 6, lines 270–285) and Appendix B.1 (pp. 23–24, lines 1232–1271).

We added controlled comparisons using the same backbone, training data, and policy optimization budget across three independent training seeds. Revised Table 2 (p. 9, lines 433–448) reports mean ± standard deviation (%).

VariantLIBERO-LongLIBERO-Plus Overall
w/o Id. Routing94.7 ± 2.0282.6 ± 0.03
Cached Id. Tokens95.1 ± 1.4582.9 ± 0.27
Standard Cross Attention94.9 ± 1.3082.8 ± 0.29
ObjDream (Full)97.1 ± 0.3184.3 ± 0.22

Removing identity-guided routing reduces the LIBERO-Plus overall success rate by approximately 1.7 percentage points. Introducing cached identity representations as additional tokens achieves 82.9%, while using a standard cross-attention adapter achieves 82.8%. The full model outperforms these alternatives by approximately 1.4 and 1.5 percentage points, respectively. These results support identity-guided routing allocation (Sec. 4.2, p. 9, lines 465–477).

AuthorsOpenReview ↗

Author Response to Reviewer EcQ2 (Part 3/3)

W4: Presentation clarity and expanded future object state prediction analysis

A5: Thank you for bringing the possible table-display issue to our attention. We carefully re-examined the rendered pages of our original submission and found no missing or truncated rows or columns in Tables 1 and 16 (pp. 8 and 31). We appreciate that PDF rendering or extraction may differ across viewers. The original Table 16 is now Table 21 (p. 34) in the revised manuscript. For convenience, we reproduce the original Table 1 below.

Original Table 1: LIBERO and LIBERO-Plus success rates (%).

MethodSpatialObjectGoalLongAvgCam.RobotLayoutLang.LightBGNoiseOverall
OpenVLA84.788.479.253.776.50.83.528.523.08.134.815.215.6
OpenVLA-OFT97.698.497.994.597.156.431.974.279.588.793.375.869.6
UniVLA96.596.895.692.095.21.846.231.969.669.081.021.242.9
π096.898.895.885.294.213.86.068.958.885.081.479.053.6
π0.598.898.298.092.496.975.477.585.785.696.994.689.785.7
AVA-VLA97.499.497.497.698.055.525.974.185.695.588.978.070.1
X-VLA98.298.697.897.698.123.489.771.875.788.296.062.771.4
WorldVLA87.696.283.460.081.80.127.938.041.643.717.110.925.0
VLA-JEPA96.299.697.295.897.263.367.185.185.495.693.666.379.5
OA-WAM98.999.097.495.997.880.589.682.885.396.595.975.683.9
ObjDream99.299.698.297.498.666.275.881.290.698.498.588.684.4

We also clarified real-world evaluation protocols in Sec. 4.3 and Appendix E.3.

We expanded the future object state prediction analysis from three to ten LIBERO tasks spanning Spatial, Object, Goal, and Long using the same checkpoint without additional training (20 trajectories, 320 prediction origins). We evaluate Unclamped subset (excluding targets beyond available annotations) and a Displacement subset (centroid changes exceeding 1% of the image diagonal). On the Unclamped subset, future prediction reduces center error from 5.58% to 1.36%, while improving box IoU from 0.652 to 0.717 and mask IoU from 0.538 to 0.617. On the Displacement subset with unclamped targets, center error decreases from 12.28% to 1.98%, while box and mask IoU improve from 0.352 and 0.282 to 0.586 and 0.479, respectively. The per-task results show improvements across all ten tasks in all three metrics on this subset. Paired trajectory-bootstrap 95% confidence intervals further support these reconstruction improvements.

We further examine action conditioning using the same trained model. On the Displacement subset with unclamped targets, removing action information increases center error from 1.98% to 2.27%, while using mismatched actions increases it to 2.52%. These results provide additional evidence of meaningful object-state reconstruction and sensitivity to action conditioning. The expanded protocol, per-task results, confidence intervals, and action-conditioning analysis are provided in Appendix C (pp. 28–32, Tables 15–19).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions for the Area Chairs and Reviewers

Dear ACs and Reviewers,

We sincerely appreciate the time and effort you have devoted to evaluating our paper. We are encouraged that the reviewers recognized several aspects of our work:

  • Methodology. Reviewers EcQ2 and USyZ recognized the coherent design combining persistent object identity, dynamic object state, identity-guided routing, and future object state supervision. Reviewer xvbg further highlighted the identity-guided routing mechanism, which introduces an additive attention bias while preserving the standard contextualized keys and values.

  • Training and Implementation. Reviewers EcQ2 and xvbg acknowledged the practical benefits of training-only future object state prediction. Reviewer xvbg also highlighted the automatic offline annotation pipeline and its reproducibility.

  • Experimental Evaluation. Reviewers EcQ2 and USyZ recognized the broad evaluation across LIBERO, LIBERO-Plus, RoboTwin 2.0, and real-world manipulation tasks.

We also sincerely thank all reviewers for their constructive suggestions, which have helped us strengthen the experimental evidence and improve the manuscript. In addition to our detailed point-by-point responses, we summarize the major revisions made according to each reviewer's concerns below.

To Reviewer EcQ2:

  1. We clarified the original LIBERO-Plus ablation results, distinguishing Sensor Noise scores from the trial-weighted Overall scores, and provided the exact aggregation protocol.

  2. We extended the key ablations to three independent training seeds, reporting mean success rates and standard deviations. ObjDream achieves 84.3 ± 0.22% on LIBERO-Plus Overall, compared with 80.8 ± 0.38% for the backbone baseline.

  3. We introduced a same-backbone Object Token Baseline and controlled comparisons with cached identity tokens and standard cross-attention to further examine the contributions of the proposed mechanisms.

  4. We clarified the real-world evaluation protocol, including demonstration data, training configurations, ID/OOD trial counts, perturbation settings, checkpoint usage, and task-specific success criteria.

  5. We expanded the future object state prediction analysis to ten LIBERO tasks and added per-task results, confidence intervals, and action-conditioning experiments.

To Reviewer USyZ:

  1. We conducted three-seed evaluations for the backbone baseline, full model, and all key ablation variants to provide stronger evidence of statistical reliability, with individual seed-level results reported in Appendix D.1.

  2. We added the requested controlled 2×2 ablation comparing unified versus factorized object injection and standard attention versus identity-guided routing. The full model achieves 84.3 ± 0.22%, compared with 81.9 ± 0.25% when both mechanisms are removed.

  3. We introduced a simple Object Token Baseline achieving 81.0 ± 0.17% under matched training conditions, providing additional evidence for the importance of effective object-information integration and supervision.

To Reviewer xvbg:

  1. We discussed possible factors affecting geometric robustness, including clean-only encoder pretraining and the use of 2D object geometry, and reported three-seed results relative to the same-backbone baseline.

  2. We provided three-seed ablation results and uncertainty estimates to further examine the contributions of individual components, while acknowledging the limited statistical evidence for some small improvements.

  3. We clarified the configurable object-slot capacity, including eight non-robot object slots in real-world experiments, and evaluated robustness to incorrect object initialization across all ten LIBERO-Long tasks, with 500 rollouts per condition.

  4. We added real-world inference latency and control-frequency details, including 20 Hz action execution, 177.4 ms inference latency per action chunk, and a module-level latency breakdown covering the object encoder, VLM forward pass, and action head.

We have incorporated these revisions into the main manuscript and appendix, particularly Secs. 4.2–4.3 and Appendices A, C, D, E, and F. Appendix F discusses fixed object-slot capacity and identifies richer 3D spatial information as a direction for future work. For ease of review, we have highlighted the corresponding modifications in blue in the revised manuscript.

We hope these additional experiments, controlled comparisons, and clarifications help address the reviewers' concerns and provide stronger empirical support for the contributions of ObjDream. We sincerely appreciate your time and consideration. We welcome any further questions or suggestions that could help improve our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.