- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Z-RoboSTAR, a System 2 framework that uses a dynamically updated symbolic scene graph to synchronize scene beliefs, task progress, and high-level planning. Action-outcome evaluation and global visual reconciliation update the graph, after which task progress is reassessed, allowing previously completed subtasks to be reopened when environmental changes invalidate them. The paper also introduces Z-RoboSTAR-Bench, covering 57 tabletop scenes with six intervention types, and Z-RoboSTAR-27B, trained using symbolic-execution trajectories. Evaluation spans symbolic execution, human-executed actions, and physical deployment with a fixed controller.
Critical strengths
-
Well-motivated treatment of persistent state under environmental changes. The framework addresses an important distinction between successfully executing the latest action and maintaining progress toward the overall goal. Synchronizing scene beliefs and task progress allows the system to reconsider completed subtasks when their conditions no longer hold. .
-
Evaluation across execution settings provides useful evidence of system-level effectiveness. The shared semantic action interface supports comparisons across symbolic, human, and VLA execution. The physical experiment holds the low-level controller fixed and reports 7/11 successful trials for Z-RoboSTAR-27B versus 0/11 for direct VLA execution on the evaluated desk-organization task.
Critical weaknesses
-
The methodological advance over existing stateful agents is insufficiently established. Skill composition through language-generated control programs has precedents in Code as Policies [1], persistent world-state maintenance in Statler [2], and scene-graph-based planning with feedback in SayPlan [3]. More directly, Thea [4] maintains a persistent symbolic scene graph by combining observation-derived updates with verified action outcomes. These approaches are not identical to Z-RoboSTAR, but they make the distinction from existing stateful, feedback-driven orchestration important. I acknowledge the explicit synchronization of scene beliefs and task progress, particularly reopening invalidated goals. However, its additional value over these approaches is not sufficiently isolated experimentally. A matched comparison with a faithfully adapted Statler [2] or Thea [4], controlling the high-level model, observations, skill interface, and interaction budget where feasible, would strengthen the novelty claim.
-
The benchmark’s additional diagnostic value relative to existing evaluation suites needs clearer demonstration. RoboCerebra [5] already targets System 2 planning, reflection, and memory in long-horizon manipulation, while VLA-Arena [6] includes long-horizon tasks and dynamic perturbations. These are relevant comparison points for establishing what Z-RoboSTAR-Bench adds. Section 2.3 discusses related benchmarks, and the proposed progress-triggered interventions and completed-subgoal rollback are potentially valuable distinctions. However, the paper should more systematically compare these capabilities and show which failure modes its protocol exposes that existing evaluations do not adequately capture. Moreover, the reported cross-mode ranking agreement concerns four models operating within Z-RoboSTAR; it supports model selection within this framework, but does not yet establish that the benchmark reliably distinguishes different agent architectures.
Questions for authors
-
What specific capability or state-consistency property does Z-RoboSTAR provide beyond the persistent-state and graph-based orchestration mechanisms in [1–4]? Can you provide a matched comparison with a representative method such as Statler [2] or Thea [4], preserving its defining capabilities? In particular, does explicitly recomputing task progress from the revised graph improve recovery over an agent that already maintains a dynamic scene representation and updates its task record from feedback?
-
Could you provide a structured comparison with RoboCerebra [5] and VLA-Arena [6], covering partial observability, intervention triggers, invalidation of completed goals, and separation of high-level reasoning from motor execution? A concrete example demonstrating an additional diagnostic finding would help establish the benchmark’s contribution. Is any cross-mode consistency evidence available for different agent frameworks, rather than only different models within Z-RoboSTAR?
References
[1] Jacky Liang et al. Code as Policies: Language Model Programs for Embodied Control. arXiv:2209.07753, 2022.
[2] Takuma Yoneda et al. Statler: State-Maintaining Language Models for Embodied Reasoning. IEEE International Conference on Robotics and Automation (ICRA), 2024. arXiv:2306.17840.
[3] Krishan Rana et al. SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning. Conference on Robot Learning (CoRL), 2023. arXiv:2307.06135.
[4] Qi Wang et al. Towards the Harness of Embodied Agents. arXiv:2608.11246, 2026.
[5] Songhao Han et al. RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation. NeurIPS, 2025. arXiv:2506.06677.
[6] Borong Zhang et al. VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models. arXiv:2512.22539, 2025.
5 comments
Response to Reviewer nh79
W2&Q2. Benchmark's value and cross-mode consistency
Reviewer: Could you provide a structured comparison with RoboCerebra [3] and VLA-Arena [4], covering partial observability, intervention triggers, invalidation of completed goals, and separation of high-level reasoning from motor execution?"
We agree that prior benchmarks already cover memory, dynamic perturbations, and long-horizon execution. Our emphasis is twofold: (1) progress-triggered interventions that invalidate previously satisfied goals during sustained long-horizon execution, and (2) task complexity arising from temporal and spatial dependencies. We will add the following comparison to the appendix.
| Dimension | RoboCerebra [3] | VLA-Arena [4] | Z-RoboSTAR-Bench |
|---|---|---|---|
| Temporal and spatial dependencies | -- | -- | Define the length of action prerequisites, the number of hidden objects, and the occlusion depth to create dependencies across steps. |
| Intervention timing | Random disturbances and observation mismatch. | Moving obstacles and distractors evolve with simulator time. | Six standardized intervention types with scene-specific triggers based on task progress and current state, providing diverse and reproducible perturbations. |
| Invalidation of completed goals | Disturbance recovery | Preserving conditions, such as keeping contents inside a container. | Explicitly undoing completed subgoals and evaluating their restoration. |
| Execution modes | System 2 planning with a shared System 1 VLA. | Simulation and real-robot evaluation of VLAs. | Symbolic, human, and VLA execution through a common semantic action interface. |
Refs:
[3] RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation.
[4] VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models.
Reviewer: Is any cross-mode consistency evidence available for different agent frameworks, rather than only different models within Z-RoboSTAR?
We conduct cross-mode (symbolic and human) evaluation on Thea w/ our scene-graph backend and Thea w/o scene graph. The results are shown below. Both execution modes show the same ordering, with Thea using our scene-graph backend outperforming Thea without a scene graph, supporting cross-mode ranking consistency between these two configurations.
| Framework | Symbolic SR (%) | Human SR (%) |
|---|---|---|
| Thea w/ our scene-graph backend | 56.6 | 41.3 |
| Thea w/o scene graph | 16.2 | 3.4 |
Dear Reviewer nh79,
Thank you again for your valuable time and constructive feedback.
As the rebuttal period is nearing its end, we would like to kindly check whether our previous response has addressed your main concerns. If there are any aspects that you feel need further clarification or improvement, please let us know, and we would be more than happy to provide additional explanations or revisions.
We sincerely appreciate your thoughtful review and support.
Best regards,
Submission29512 Authors
Response to Reviewer nh79
Thank you for your insightful feedback. We clarify our contribution through comparison with other agents (W1&Q1), provide a benchmark comparison (W2&Q2), and demonstrate cross-mode consistency for different agent frameworks (W2&Q2).
W1&Q1. Comparison with other agents
We compare Z-RoboSTAR with two existing embodied-agent frameworks, Thea [1] and Statler [2], using their official implementations with benchmark-specific adaptations.
(1) Thea is a general-purpose embodied-agent harness combining tool use, scene-graph memory, and post-action evaluation, whereas our Z-RoboSTAR focuses on leveraging a scene graph as a persistent shared-state interface, not only memory, to synchronize scene understanding, task progress, and planning, with particular emphasis on integrating global scene-belief reconciliation to sustain manipulation under interventions. For Thea, the public release provides the scene-graph interface and harness-side update/query mechanisms but does not include a complete scene-graph backend, so we implement this interface using our symbolic scene-graph backend. We retain Thea's agent loop, memory, and official evaluator, and denote this adaptation as Thea w/ our scene-graph backend. We also include a configuration using Thea’s released NullSceneGraph implementation, denoted as Thea w/o scene graph. (2) Statler maintains explicit world-state memory through a reader that generates action programs and a writer that updates stored state. For Statler, we retain the official state reader/writer, recursive function generator, and prompt structure with domain-specific examples. We extend its original pick-and-place interface with our semantic action schemas. Generated programs execute through our symbolic execution and receive feedback after each primitive action. We denote this adaptation as Statler w/ extended actions.
We run both baselines with GPT-5.5, Claude Opus 4.7, and Z-RoboSTAR-27B. The evaluation suite contains 57 tabletop source scenes, each with one nominal condition and six intervention conditions, yielding 399 rollouts per configuration (57 nominal and 342 intervention episodes). The tables below report the results. We include Z-RoboSTAR's existing Table 1 results as reference rows marked †.
Conclusion: compared with Thea, our advantage is reflected in the scene graph as a shared interface to control task progress after visual conflict reconciliation. The results show that under interventions, Z-RoboSTAR improves the reported SR over this baseline by 27.2, 7.7, and 26.8 percentage points, and NGP by 11.6, 4.2, and 14.7 points, with GPT-5.5, Claude Opus 4.7, and Z-RoboSTAR-27B, respectively. Thus, promptly recomputing task progress from the revised scene graph allows after an intervention invalidated subgoals to be reopened for subsequent planning.
Refs:
[1] Towards the Harness of Embodied Agents.
[2] Statler: State-Maintaining Language Models for Embodied Reasoning.