acceptodds
Under review as a conference paper at ICLR 2027

Z-RoboSTAR: Building and Benchmarking Robotic System 2 Agents for Reasoning in Sustained Long-Horizon Manipulation

Abstract

Sustainable long-horizon robotic manipulation requires a deliberative System 2 agent that can reason, plan, and sustain progress toward a goal as the world evolves. Complex multi-step manipulation changes the world state, while interventions may invalidate prior beliefs, requiring System 2 to continually track progress from partial observations. We present Z-RoboSTAR, a harness that enables vision-language models (VLMs) to act as System 2 agents, centered on a dynamically updated symbolic scene graph that serves as a persistent shared-state interface to synchronize scene understanding, task progress, and planning, for failure recovery and resolving reconciliation. To efficiently compare open- and closed-source VLMs for our System 2 harness, we construct Z-RoboSTAR-Bench, an offline open-loop benchmark and evaluation engine spanning 57 tabletop scenes with diverse spatiotemporal dependencies and interventions. The model ranking is preserved under human-executed closed-loop evaluation, supporting the benchmark's relevance to real-world agent performance. Beyond evaluation, we develop Z-RoboSTAR-27B through trajectory-level distillation to improve its robustness over prolonged execution. When deployed on a Galaxea R1 Lite with a fine-tuned System 1 controller, Z-RoboSTAR-27B achieves a 66.9% task success rate on our real-world long-horizon manipulation tasks, ranking second only to GPT-5.5. These results demonstrate effective System 2 coordination of complex, long-horizon manipulation in real-world scenes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied System 2, Vision-Language Models, Long-Horizon Tasks, Closed-Loop Reasoning, Hierarchical Robot Control

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_nh79OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Z-RoboSTAR, a System 2 framework that uses a dynamically updated symbolic scene graph to synchronize scene beliefs, task progress, and high-level planning. Action-outcome evaluation and global visual reconciliation update the graph, after which task progress is reassessed, allowing previously completed subtasks to be reopened when environmental changes invalidate them. The paper also introduces Z-RoboSTAR-Bench, covering 57 tabletop scenes with six intervention types, and Z-RoboSTAR-27B, trained using symbolic-execution trajectories. Evaluation spans symbolic execution, human-executed actions, and physical deployment with a fixed π0.5\pi_{0.5} controller.

Critical strengths
  1. Well-motivated treatment of persistent state under environmental changes. The framework addresses an important distinction between successfully executing the latest action and maintaining progress toward the overall goal. Synchronizing scene beliefs and task progress allows the system to reconsider completed subtasks when their conditions no longer hold. .

  2. Evaluation across execution settings provides useful evidence of system-level effectiveness. The shared semantic action interface supports comparisons across symbolic, human, and VLA execution. The physical experiment holds the low-level controller fixed and reports 7/11 successful trials for Z-RoboSTAR-27B versus 0/11 for direct VLA execution on the evaluated desk-organization task.

Critical weaknesses
  1. The methodological advance over existing stateful agents is insufficiently established. Skill composition through language-generated control programs has precedents in Code as Policies [1], persistent world-state maintenance in Statler [2], and scene-graph-based planning with feedback in SayPlan [3]. More directly, Thea [4] maintains a persistent symbolic scene graph by combining observation-derived updates with verified action outcomes. These approaches are not identical to Z-RoboSTAR, but they make the distinction from existing stateful, feedback-driven orchestration important. I acknowledge the explicit synchronization of scene beliefs and task progress, particularly reopening invalidated goals. However, its additional value over these approaches is not sufficiently isolated experimentally. A matched comparison with a faithfully adapted Statler [2] or Thea [4], controlling the high-level model, observations, skill interface, and interaction budget where feasible, would strengthen the novelty claim.

  2. The benchmark’s additional diagnostic value relative to existing evaluation suites needs clearer demonstration. RoboCerebra [5] already targets System 2 planning, reflection, and memory in long-horizon manipulation, while VLA-Arena [6] includes long-horizon tasks and dynamic perturbations. These are relevant comparison points for establishing what Z-RoboSTAR-Bench adds. Section 2.3 discusses related benchmarks, and the proposed progress-triggered interventions and completed-subgoal rollback are potentially valuable distinctions. However, the paper should more systematically compare these capabilities and show which failure modes its protocol exposes that existing evaluations do not adequately capture. Moreover, the reported cross-mode ranking agreement concerns four models operating within Z-RoboSTAR; it supports model selection within this framework, but does not yet establish that the benchmark reliably distinguishes different agent architectures.

Questions for authors
  1. What specific capability or state-consistency property does Z-RoboSTAR provide beyond the persistent-state and graph-based orchestration mechanisms in [1–4]? Can you provide a matched comparison with a representative method such as Statler [2] or Thea [4], preserving its defining capabilities? In particular, does explicitly recomputing task progress from the revised graph improve recovery over an agent that already maintains a dynamic scene representation and updates its task record from feedback?

  2. Could you provide a structured comparison with RoboCerebra [5] and VLA-Arena [6], covering partial observability, intervention triggers, invalidation of completed goals, and separation of high-level reasoning from motor execution? A concrete example demonstrating an additional diagnostic finding would help establish the benchmark’s contribution. Is any cross-mode consistency evidence available for different agent frameworks, rather than only different models within Z-RoboSTAR?

References

[1] Jacky Liang et al. Code as Policies: Language Model Programs for Embodied Control. arXiv:2209.07753, 2022.

[2] Takuma Yoneda et al. Statler: State-Maintaining Language Models for Embodied Reasoning. IEEE International Conference on Robotics and Automation (ICRA), 2024. arXiv:2306.17840.

[3] Krishan Rana et al. SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning. Conference on Robot Learning (CoRL), 2023. arXiv:2307.06135.

[4] Qi Wang et al. Towards the Harness of Embodied Agents. arXiv:2608.11246, 2026.

[5] Songhao Han et al. RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation. NeurIPS, 2025. arXiv:2506.06677.

[6] Borong Zhang et al. VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models. arXiv:2512.22539, 2025.

5 comments
AuthorsOpenReview ↗

Response to Reviewer nh79

Thank you for your insightful feedback. We clarify our contribution through comparison with other agents (W1&Q1), provide a benchmark comparison (W2&Q2), and demonstrate cross-mode consistency for different agent frameworks (W2&Q2).

W1&Q1. Comparison with other agents

Reviewer: What specific capability or state-consistency property does Z-RoboSTAR provide beyond the persistent-state and graph-based orchestration mechanisms in [1–4]?"

We compare Z-RoboSTAR with two existing embodied-agent frameworks, Thea [1] and Statler [2], using their official implementations with benchmark-specific adaptations.

(1) Thea is a general-purpose embodied-agent harness combining tool use, scene-graph memory, and post-action evaluation, whereas our Z-RoboSTAR focuses on leveraging a scene graph as a persistent shared-state interface, not only memory, to synchronize scene understanding, task progress, and planning, with particular emphasis on integrating global scene-belief reconciliation to sustain manipulation under interventions. For Thea, the public release provides the scene-graph interface and harness-side update/query mechanisms but does not include a complete scene-graph backend, so we implement this interface using our symbolic scene-graph backend. We retain Thea's agent loop, memory, and official evaluator, and denote this adaptation as Thea w/ our scene-graph backend. We also include a configuration using Thea’s released NullSceneGraph implementation, denoted as Thea w/o scene graph. (2) Statler maintains explicit world-state memory through a reader that generates action programs and a writer that updates stored state. For Statler, we retain the official state reader/writer, recursive function generator, and prompt structure with domain-specific examples. We extend its original pick-and-place interface with our semantic action schemas. Generated programs execute through our symbolic execution and receive feedback after each primitive action. We denote this adaptation as Statler w/ extended actions.

We run both baselines with GPT-5.5, Claude Opus 4.7, and Z-RoboSTAR-27B. The evaluation suite contains 57 tabletop source scenes, each with one nominal condition and six intervention conditions, yielding 399 rollouts per configuration (57 nominal and 342 intervention episodes). The tables below report the results. We include Z-RoboSTAR's existing Table 1 results as reference rows marked †.

Conclusion: compared with Thea, our advantage is reflected in the scene graph as a shared interface to control task progress after visual conflict reconciliation. The results show that under interventions, Z-RoboSTAR improves the reported SR over this baseline by 27.2, 7.7, and 26.8 percentage points, and NGP by 11.6, 4.2, and 14.7 points, with GPT-5.5, Claude Opus 4.7, and Z-RoboSTAR-27B, respectively. Thus, promptly recomputing task progress from the revised scene graph allows after an intervention invalidated subgoals to be reopened for subsequent planning.

ModelFrameworkNominal SR (%)Nominal NGP (%)Intervention SR (%)Intervention NGP (%)
GPT-5.5Thea w/ our scene-graph backend80.789.850.382.9
GPT-5.5Thea w/o scene graph26.375.314.660.1
GPT-5.5Statler w/ extended actions20.968.80.044.1
GPT-5.5Z-RoboSTAR†84.296.377.594.5
Claude Opus 4.7Thea w/ our scene-graph backend61.489.547.981.8
Claude Opus 4.7Thea w/o scene graph17.552.32.940.2
Claude Opus 4.7Statler w/ extended actions9.840.02.336.1
Claude Opus 4.7Z-RoboSTAR†63.290.655.686.0
Z-RoboSTAR-27BThea w/ our scene-graph backend71.991.244.875.5
Z-RoboSTAR-27BThea w/o scene graph50.888.65.357.8
Z-RoboSTAR-27BStatler w/ extended actions15.461.74.547.9
Z-RoboSTAR-27BZ-RoboSTAR†78.994.471.690.2

Refs:

[1] Towards the Harness of Embodied Agents.

[2] Statler: State-Maintaining Language Models for Embodied Reasoning.

AuthorsOpenReview ↗

Response to Reviewer nh79

W2&Q2. Benchmark's value and cross-mode consistency

Reviewer: Could you provide a structured comparison with RoboCerebra [3] and VLA-Arena [4], covering partial observability, intervention triggers, invalidation of completed goals, and separation of high-level reasoning from motor execution?"

We agree that prior benchmarks already cover memory, dynamic perturbations, and long-horizon execution. Our emphasis is twofold: (1) progress-triggered interventions that invalidate previously satisfied goals during sustained long-horizon execution, and (2) task complexity arising from temporal and spatial dependencies. We will add the following comparison to the appendix.

DimensionRoboCerebra [3]VLA-Arena [4]Z-RoboSTAR-Bench
Temporal and spatial dependencies----Define the length of action prerequisites, the number of hidden objects, and the occlusion depth to create dependencies across steps.
Intervention timingRandom disturbances and observation mismatch.Moving obstacles and distractors evolve with simulator time.Six standardized intervention types with scene-specific triggers based on task progress and current state, providing diverse and reproducible perturbations.
Invalidation of completed goalsDisturbance recoveryPreserving conditions, such as keeping contents inside a container.Explicitly undoing completed subgoals and evaluating their restoration.
Execution modesSystem 2 planning with a shared System 1 VLA.Simulation and real-robot evaluation of VLAs.Symbolic, human, and VLA execution through a common semantic action interface.

Refs:

[3] RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation.

[4] VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models.

Reviewer: Is any cross-mode consistency evidence available for different agent frameworks, rather than only different models within Z-RoboSTAR?

We conduct cross-mode (symbolic and human) evaluation on Thea w/ our scene-graph backend and Thea w/o scene graph. The results are shown below. Both execution modes show the same ordering, with Thea using our scene-graph backend outperforming Thea without a scene graph, supporting cross-mode ranking consistency between these two configurations.

FrameworkSymbolic SR (%)Human SR (%)
Thea w/ our scene-graph backend56.641.3
Thea w/o scene graph16.23.4

Dear Reviewer nh79,

Thank you again for your valuable time and constructive feedback.

As the rebuttal period is nearing its end, we would like to kindly check whether our previous response has addressed your main concerns. If there are any aspects that you feel need further clarification or improvement, please let us know, and we would be more than happy to provide additional explanations or revisions.

We sincerely appreciate your thoughtful review and support.

Best regards,

Submission29512 Authors

Reviewer_xExPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Z-RoboSTAR, a System 2 harness for sustainable long-horizon manipulation, where previously completed subgoals may be invalidated by external disturbances or by the policy’s own subsequent actions. The framework maintains a symbolic scene graph that represents objects, their current states, and their relationships. This graph is dynamically updated during task execution: action-outcome evaluation checks whether a planned action achieved its intended effect, while visual reconciliation examines current observations against the maintained scene belief to identify additional or conflicting changes. The updated graph informs task-progress monitoring and subsequent planning. Building on this harness, the paper introduces Z-RoboSTAR-Bench, a real-world benchmark for long-horizon tabletop manipulation, and evaluates different VLMs within the same harness. The benchmark assesses task completion, execution efficiency, and recovery from changes that invalidate prior beliefs or completed subgoals.

Critical strengths
  1. The paper’s focus on sustainable long-horizon manipulation is well motivated. Human interventions or unintended effects of the policy’s own actions can disrupt the scene and invalidate previously completed subgoals. Studying how a policy can recover from these disruptions and maintain progress toward the overall goal is valuable.
  2. The paper develops a benchmark grounded in real-world tabletop scenarios and presents a practical harness that integrates planning, execution, and feedback for long-horizon manipulation. The benchmark covers diverse tabletop scenes and multiple types of human intervention.
Critical weaknesses

This paper couples a harness framework with a benchmark, but important aspects of both contributions remain unclear.

  1. As a harness framework, its advantages over existing approaches are insufficiently demonstrated. Most comparisons vary the VLM while keeping Z-RoboSTAR fixed. The ablations demonstrate that components such as the scene graph are important to the system, but do not demonstrate whether the proposed framework improves upon existing scene-graph and feedback-driven replanning methods. It remains unclear how much benefit comes from Z-RoboSTAR’s particular design and which aspects account for that benefit.
  2. As a benchmark, its broader applicability and evaluation protocol remain unclear. The experiments demonstrate model comparison within Z-RoboSTAR, but leave unclear how alternative harnesses could be evaluated under comparable conditions. Furthermore, the paper provides only high-level descriptions of task and intervention categories, with insufficient detail on scene configurations and intervention protocols to assess reproducibility. For example, an object relocation may cause failure because the low-level policy cannot generalize to the new position, even when the high-level decision is correct. The paper does not explain how intervention conditions are standardized or how execution failures are distinguished from System 2 reasoning failures. Consequently, it is difficult to determine what the reported scores measure and how reliably they support comparisons across methods.
Questions for authors
  1. Section 5.1 introduces three evaluation modes, with the second relying on human execution. Why does this mode produce different results from symbolic execution? If the human serves as an expert System 1, I would expect them to execute System 2’s planned actions correctly. Where do the remaining errors come from?
  2. The framework includes several modules, such as the monitor, planner, and visual reconciler. Are these implemented using separate VLMs (e.g., is each module instantiated as a separate GPT-5.5)? If so, how are their roles distinguished? Are different system prompts used to assign these roles?
  3. Following the previous question, if each module uses the model being evaluated, would this introduce substantial inference costs and waiting times, particularly for closed-source models? Using a model such as GPT-6 as a policy can already be slow and costly, would requiring it to serve so many roles further increase this overhead? For open-source models, would deploying these modules require additional computational resources, and could the full framework still run on a single machine?
  4. Finally, if the same model is evaluated across different roles, is it sufficient to assess its performance mainly through task-completion metrics? The final results combine errors arising from multiple roles, making it difficult to determine where each model struggles. Would more fine-grained metrics help distinguish performance across roles and identify the main bottleneck for each model?
5 comments
AuthorsOpenReview ↗

Response to Reviewer xExP

Thank you for your insightful comments. We add comparisons with other agent harnesses to better assess Z-RoboSTAR's advantages (W1), clarify the benchmark interface and intervention protocol (W2), distinguish execution modes (Q1) and model roles (Q2), and provide overhead analysis (Q3) and additional metrics (Q4).

W1. Comparison with other harness frameworks

Reviewer: "As a harness framework, its advantages over existing approaches are insufficiently demonstrated."

We compare Z-RoboSTAR with two existing embodied-agent frameworks, Thea [1] and Statler [2], using their official implementations with our benchmark-specific adaptations.

(1) Thea is a general-purpose embodied-agent harness combining tool use, scene-graph memory, and post-action evaluation, whereas our Z-RoboSTAR focuses on leveraging a scene graph as a persistent shared-state interface, not only memory, to synchronize scene understanding, task progress, and planning, with particular emphasis on integrating global scene-belief reconciliation to sustain manipulation under interventions. For Thea, the public release provides the scene-graph interface and harness-side update/query mechanisms but does not include a complete scene-graph backend, so we implement this interface using our symbolic scene-graph backend. We retain Thea's agent loop, memory, and official evaluator, and denote this adaptation as Thea w/ our scene-graph backend. We also include a configuration using Thea’s released NullSceneGraph implementation, denoted as Thea w/o scene graph. (2) Statler maintains explicit world-state memory through a reader that generates action programs and a writer that updates stored state. For Statler, we retain the official state reader/writer, recursive function generator, and prompt structure with domain-specific examples. We extend its original pick-and-place interface with our semantic action schemas. Generated programs execute through our symbolic execution and receive feedback after each primitive action. We denote this adaptation as Statler w/ extended actions.

We run both baselines with GPT-5.5, Claude Opus 4.7, and Z-RoboSTAR-27B. The evaluation suite contains 57 tabletop source scenes, each with one nominal condition and six intervention conditions, yielding 399 rollouts per configuration (57 nominal and 342 intervention episodes). The tables below report the results. We include Z-RoboSTAR's existing Table 1 results as reference rows marked †. Z-RoboSTAR achieves higher reported SR and NGP than Thea and Statler, especially under intervention, exhibiting an advantage in global scene-belief reconciliation.

ModelFrameworkNominal SR (%)Nominal NGP (%)Intervention SR (%)Intervention NGP (%)
GPT-5.5Thea w/ our scene-graph backend80.789.850.382.9
GPT-5.5Thea w/o scene graph26.375.314.660.1
GPT-5.5Statler w/ extended actions20.968.80.044.1
GPT-5.5Z-RoboSTAR†84.296.377.594.5
Claude Opus 4.7Thea w/ our scene-graph backend61.489.547.981.8
Claude Opus 4.7Thea w/o scene graph17.552.32.940.2
Claude Opus 4.7Statler w/ extended actions9.840.02.336.1
Claude Opus 4.7Z-RoboSTAR†63.290.655.686.0
Z-RoboSTAR-27BThea w/ our scene-graph backend71.991.244.875.5
Z-RoboSTAR-27BThea w/o scene graph50.888.65.357.8
Z-RoboSTAR-27BStatler w/ extended actions15.461.74.547.9
Z-RoboSTAR-27BZ-RoboSTAR†78.994.471.690.2

Refs:

[1] Towards the Harness of Embodied Agents.

[2] Statler: State-Maintaining Language Models for Embodied Reasoning.

AuthorsOpenReview ↗

Response to Reviewer xExP

W2. Benchmark applicability and detailed intervention protocols

Reviewer: “As a benchmark, its broader applicability and evaluation protocol remain unclear.”

Z-RoboSTAR-Bench separates the task environment, including the System 1 action executor and the scoring protocol, from the agent policy, making it readily applicable to other agent frameworks. Specifically, a compiled task instance specifies the initial scene, goal, semantic action interface, observation projection, and intervention rules. An agent adapter receives the permitted observations from the environment and returns either an action with grounded arguments or a stop decision. The System 1 action executor executes the action, while the environment applies interventions when their triggering conditions are met and records outcomes for scoring. After the task stops or is complete, the scoring protocol computes and records the final evaluation metrics. As mentioned in W1, we have accommodated other agent frameworks (i.e., Thea and Statler) in our benchmark, demonstrating interface compatibility with symbolic execution.

Reviewer: “Furthermore, the paper provides only high-level descriptions of task and intervention categories, with insufficient detail on scene configurations and intervention protocols to assess reproducibility.”

We will open-source our benchmark for reproducibility. For scene configurations, six task families include buffet-tray sorting, cosmetics storage, desk organization, dining-table organization, fridge-condiment loading, and toy organization. Each family contains 7–10 source-scene variants with different temporal and spatial dependencies (Sec. 4), yielding 57 tabletop scenes in total. Detailed intervention protocols are summarized below.

IDEventTrigger ψλ\psi_\lambdaState update δλ\delta_\lambda
I1I_1Action failureFirst otherwise successful execution of each action type; at most once per action name.Suppress the action effect, leave GtG_t unchanged, and return typed failure feedback. The agent must retry or replan.
I2I_2State regressionFirst achieved a reversible node state, such as an opened container, after the minimum-step guard.Reset one node attribute (e.g., open, closed, or moved_aside) to its prior value.
I3I_3Completed-subgoal rollbackCurrently satisfied object-placement goal fact.Remove the satisfied ON/INSIDE relation and relocate the object to a valid non-goal staging surface.
I4I_4Wrong relocationCurrently satisfied object-inside-the-container goal fact.Update the object's location relative to another visible, plausible, but wrong container.
I5I_5Added occlusionEligible mid-episode blocker-target pair whose occlusion is resolvable by an available action schema.Move a goal-relevant object behind a blocker and add an active OCCLUDES edge; recovery may require open or move_aside followed by placement restoration.
I6I_6Added objectA source object's local goal becomes satisfied, or a designated container reaches its capacity condition.Add an object node and staging relation, then extend the active goal by an inherited placement rule or activate an existing capacity-bound goal.
AuthorsOpenReview ↗

Response to Reviewer xExP

Q1. Why human execution differs from symbolic execution

Reviewer: “If the human serves as an expert System 1, I would expect them to execute System 2’s planned actions correctly. Where do the remaining errors come from?”

Even if the human (System 1) executes every command correctly, the agent (System 2) may make mistakes, such as misidentifying an object, failing to observe a state change, incorrectly judging the preceding outcome, or terminating while task conditions remain unsatisfied. That is, System 2’s planned actions are not guaranteed to be correct. In symbolic execution, the executor applies state transitions and supplies a symbolic observation projection. In human execution, the agent instead reasons from newly acquired camera observations. The symbolic condition therefore reduces visual and physical uncertainty, but the human condition retains it. Thus, in our Figure 4, the results reveal the gap. However, they (human and symbolic) remain consistent.

Q2&Q3. Separate VLMs and their inference overhead.

Reviewer: The framework includes several modules, such as the monitor, planner, and visual reconciler. Are these implemented using separate VLMs (e.g., is each module instantiated as a separate GPT-5.5)? If so, how are their roles distinguished? Are different system prompts used to assign these roles?"

For different modules, we use separate context to the same VLM. The task monitor tracks subtask progress and checks whether the task is complete, the planner selects the next action, the evaluator compares before/after observations to judge whether the action succeeded, and the visual reconciler proposes scene-graph updates based on observed changes. We provide two example prompts below.

  • Task monitor: You are the task-level planner and progress monitor for an embodied agent...
  • Visual reconciler: Audit the entire known scene from the AFTER_ACTION_IMAGE camera views...Report only changes supported by direct, unambiguous visual evidence. Every change needs confidence and camera-specific evidence...

Reviewer: “If each module uses the model being evaluated, would this introduce substantial inference costs and waiting times, particularly for closed-source models?”

We acknowledge that multiple VLMs increase end-to-end latency. Appendix C describes asynchronous communication between System 1 and System 2. System 1 executes VLA chunks at 15 Hz, with 16 control steps per chunk and a default eight-chunk semantic execution window. Specifically for System 2, we use compacted context and lower reasoning budgets for post-action evaluation and visual reconciliation while running the planner asynchronously through the shared scene-state interface to synchronize information, which can reduce inference latency introduced by multi-module collaboration.

Q4. Fine-grained metrics

Reviewer: “Would more fine-grained metrics help distinguish performance across roles and identify the main bottleneck for each model?”

We agree. We therefore supplement SR and NGP with role-specific diagnostics: Action Executability Rate (AER) measures the executability of planner actions, action-outcome accuracy and false-success rate assess the evaluator’s completion judgments, and graph-correction precision assesses global visual reconciliation. For the latter two modules, we use AI-assisted annotations with partial human review. We report two models with two System 1 executors to demonstrate that all roles work well under different settings. These metrics can provide additional insight into why SR varies.

Model / System 1 executorTotal judgmentsAction-outcome accuracy ↑False-success rate ↓Graph precision ↑
Z-RoboSTAR-27B, human execution192/27095.8% (184/192)1.9% (1/54)68.0%
GPT-5.5, VLA execution28/3889.3% (25/28)15.8% (3/19)58.3%
Reviewer_6XKnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Z-RoboSTAR, a System 2 framework for continuously maintaining scene beliefs and task progress during long-horizon robotic manipulation under environmental changes and external interventions. It uses a dynamically updated symbolic scene graph as shared state, revising scene beliefs, task progress, and subsequent plans through action-outcome evaluation and global visual reconciliation. A shared semantic action interface connects high-level reasoning to low-level execution, allowing symbolic executors, human operators, and VLA controllers to follow the same command structure. The paper also introduces Z-RoboSTAR-Bench, comprising 57 tabletop scenes and six intervention types, to compare the state-tracking and task-execution capabilities of different VLMs. Finally, real-robot experiments combining the trajectory-distilled Z-RoboSTAR-27B or GPT-5.5 with a π0.5 controller demonstrate the framework’s applicability to long-horizon manipulation.

Critical strengths
  1. Clear problem formulation and a novel method aligned with it

    The paper clearly identifies how environmental changes can invalidate both scene beliefs and previously completed subtasks during long-horizon execution. In particular, it emphasizes that individual action success does not necessarily imply progress toward the overall goal, framing the problem as jointly maintaining scene beliefs and task progress. To address this problem, it introduces a dynamic scene graph as shared state and updates task progress and planning through action-outcome evaluation and global visual reconciliation. Integrating these components into a unified state-maintenance process closely connects the proposed method to the problem formulation and provides a concrete methodological contribution.

  2. Experiments covering diverse state changes and real-robot execution

    The benchmark includes 57 scenes and six intervention types, evaluating both long action sequences and responses to changing world states. Reporting task success, normalized goal progress, and intervention recovery provides complementary perspectives on execution performance. The shared action interface connecting symbolic execution, human execution, and VLA-based robot execution is also useful. In particular, comparing high-level reasoning configurations while holding the low-level controller fixed provides concrete evidence of the framework’s practical applicability.

Critical weaknesses
  1. Limitations in temporal modeling

    Long-horizon planning requires remembering past actions and changes in addition to understanding the current state. The paper addresses this through a dynamically updated scene graph, task-progress state, and bounded interaction history. However, the graph primarily represents the current state, and the paper does not sufficiently explain how the temporal ordering and causal relationships of past changes are preserved and used. It is also unclear what information the separate interaction history retains and for how long. Consequently, it is difficult to assess whether the framework can reliably retain and use information acquired much earlier when it becomes relevant to later decisions.

  2. Reliability and comparability of the long-horizon benchmark

    During long rollouts, small differences in observations or actions can alter subsequent trajectories and affect final scores and model rankings. However, repeated evaluations under identical conditions and confidence intervals are not clearly reported, making the stability of the results difficult to assess. Furthermore, because interventions are triggered by task progress, a model’s action ordering can affect the intervention timing, the scene state, and the remaining execution budget. For example, if an intervention occurs immediately after pen placement, a model that first organizes the books and trash encounters it at a different stage of completion from a model that handles the pen first. IRR does not control for these differences, making it difficult to separate recovery capability from differences in the conditions induced by action ordering.

Questions for authors
  1. When rollouts are repeated with different evaluation random seeds under the same scene and intervention conditions, what are the distributions and confidence intervals of SR, NGP, and IRR? Do the reported performance differences and model rankings remain stable across these repeated runs?

  2. Since interventions are triggered by task progress, different action orderings may lead to different progress levels, scene states, and remaining execution budgets at the time of intervention; how are these differences handled? In particular, when comparing IRR, how can the effects of recovery capability be distinguished from those of the conditions under which each model encounters the intervention?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6XKn

Thank you for your positive feedback. We clarify the complementary roles of the scene graph and interaction history (W1), provide a history-window ablation (W1) and repeated evaluations with confidence intervals (W2&Q1), and examine how intervention-trigger conditions affect IRR (W2&Q2).

W1. Persistent state and bounded interaction history

Reviewer: "However, the graph primarily represents the current state, and the paper does not sufficiently explain how the temporal ordering and causal relationships of past changes are preserved and used."

Z-RoboSTAR maintains complementary forms of task context: the scene belief GtG_t, task-progress state qtq_t, and recent interaction history HtH_t, as described in Sec. 3.1. Temporal ordering is provided by the ordered interaction history HtH_t, for example, opening a drawer, observing a pen inside, and then grabbing it. The current history window retains the 16 most recent interaction records. Causal relationships and dependencies are supported by the continually updated scene belief GtG_t, which is constructed from accumulated evidence, as described in Sec. 3.2. For example, opening a drawer and observing a pen inside updates GtG_t to record the pen’s containment relation. Closing the drawer then updates its state to closed, making the pen unobservable without removing that relation solely because it is not visible. This illustrates the scene graph’s role as persistent state memory. Even after the opening interaction leaves the history window, the pen’s location remains in GtG_t as the last supported belief for later decisions, unless contradicted by subsequent evidence, such as reopening the drawer and observing that the pen is no longer inside. Thus, Z-RoboSTAR can reliably retain and use information acquired much earlier when it becomes relevant to later decisions.

Reviewer: “It is also unclear what information the separate interaction history retains and for how long.”

The bounded interaction history retains the 16 most recent interaction records. We specify this setting in the revised manuscript (Sec. 3.1). We further compare windows of 16 and 32 while keeping the remaining configuration fixed, using six scenes spanning all six task categories, seven conditions per scene, and three repeated runs per condition. The results for the two models below show modest, mixed changes when increasing the window size. Increasing the history window does not consistently improve performance, suggesting that access to extended interaction records alone may be insufficient to improve task completion.

ModelHistory windowSR (%) ↑NGP (%) ↑IRR (%) ↑
GPT-5.51686.597.488.9
GPT-5.53288.196.687.0
Z-RoboSTAR-27B1672.493.872.6
Z-RoboSTAR-27B3271.194.871.4

Note: SR and NGP include nominal and intervention conditions. IRR includes only triggered intervention rollouts.

AuthorsOpenReview ↗

Response to Reviewer 6XKn

W2&Q1. Stability of the results

Reviewer: “When rollouts are repeated with different evaluation random seeds under the same scene and intervention conditions, what are the distributions and confidence intervals of SR, NGP, and IRR?”

We evaluate repeated executions under fixed scene and intervention configurations, i.e., six scenes spanning all six task categories, seven conditions per scene, and three repeated runs per condition. The results across the three runs are:

ModelSR (%)NGP (%)IRR (%)
GPT-5.595.2, 81.0, 90.599.6, 96.5, 99.2100.0, 76.5, 88.2
Z-RoboSTAR-27B71.4, 61.9, 71.494.3, 94.5, 91.970.6, 58.8, 70.6
Claude Opus 4.752.4, 52.4, 61.983.0, 82.0, 86.652.9, 56.2, 64.7

We estimate 95% percentile confidence intervals using 10,000 bootstrap resamples of the three executions within each fixed scene–condition combination:

ModelSR, 95% CI (%)NGP, 95% CI (%)IRR, 95% CI (%)
GPT-5.588.9 [82.5, 95.2]98.4 [97.2, 99.5]88.2 [80.4, 96.1]
Z-RoboSTAR-27B68.3 [61.9, 74.6]93.6 [90.7, 96.0]66.7 [58.8, 74.5]
Claude Opus 4.755.6 [49.2, 61.9]83.9 [81.1, 86.7]58.0 [51.0, 65.3]

Note: SR and NGP include nominal and intervention conditions. IRR includes only triggered intervention rollouts. Three runs are independent repeated invocations. Z-RoboSTAR-27B uses a temperature of 1.0, and temperature is not explicitly specified for GPT-5.5 or Claude Opus 4.7.

In conclusion, the ordering GPT-5.5 > Z-RoboSTAR-27B > Claude Opus 4.7 is preserved for all three metrics in each run.

W2&Q2. Intervention timing and interpretation of IRR

Reviewer: “In particular, when comparing IRR, how can the effects of recovery capability be distinguished from those of the conditions under which each model encounters the intervention?”

Thank you for raising this point. In our evaluation, interventions follow predefined, scene-specific rules shared across models, rather than being tied to a fixed action such as placing a pen. To partially control for differences in intervention conditions, we identify a matched subset in the existing evaluation results. Under the completed-subgoal rollback intervention, nine scenes have identical recorded post-intervention state and trigger steps across eight models, aligning the state and remaining step budget within each scene. The results are summarized below.

ModelSuccessful rolloutsIRR (%) ↑
GPT-5.59/9100.0
Z-RoboSTAR-27B8/988.9
Gemini 3.1 Pro8/988.9
Claude Opus 4.77/977.8
DeepSeek-V4 Pro5/955.6
GLM-5.24/944.4
GPT-5.43/933.3
Qwen3.6-Plus3/933.3
Qwen3.7-Plus1/911.1

Results show that differences in recovery performance persist on the subset matched for post-intervention state and trigger step, suggesting that these two factors alone do not explain the observed performance differences in our evaluation.

Thank you for the response. I will maintain my original rating.

Reviewer_mz3dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a System 2 robotic agent harness centered on a symbolic scene graph used to synchronize scene understanding, task-progress monitoring, and action planning. The conceptual core of the proposed approach is maintaining task progress as a revisable state: previously completed subgoals can become invalid when the environment changes.

To evaluate the proposed harness, the authors construct Z-RoboSTAR-Bench, a benchmark with 57 tabletop scenes and six intervention variants per task. They evaluate multiple open and closed VLMs, and claim to distill their own 27B System 2 model.

Experimental results with symbolic actions characterize the performance of the diverse models evaluated. Additional experiments on a Galaxea R1 Lite with a π0.5 controller show the best results reach 81.8% success using GPT-5.5 as System 2, while the distilled Z-RoboSTAR-27B obtains 63.6% SR.

Critical strengths

This paper addresses a particularly important problem in embodied AI: long-horizon task progress is not necessarily monotonic and not properly evaluated in current benchmarks.

The proposed harness based on persistent scene graph, global visual reconciliation, and reconsidering of task progress form a coherent solution to the problem and the proposed benchmark with intervention helps to more in-depth test system behaviour.

The paper also includes multiple evaluation settings from symbolic action execution, to human action execution, and real-world action control with a low-level in-robot controller.

Critical weaknesses

W1. The contribution of the persistent scene graph is not sufficiently validated

Persistent scene graphs are not new, and previous embodied systems have explored state tracking, structured memory, and replanning. However, the paper makes no comparison to other SG approaches. The potentially novelty here is the explicit synchronization of scene belief, evidence, and non-monotonic task progress. But, the experiments shown do not establish if the gains claimed actually result from the scene graph representation, the progress monitor, global visual reconciliation, or simply using a structured agent loop.

W2. The benchmark's relationship to real-world closed-loop behavior is also not well supported

The core results shown use symbolic semantic transitions and previously collected observations. Although the model iteratively receives some feedback, this evaluation does not fully expose it to errors caused by actual physical execution or to the observation distributions generated by arbitrary new action trajectories. The paper does report preserved rankings across symbolic and human-executed evaluation for four models. This is encouraging, but a weak criterion for establishing predictive validity.

W3. Task-progress state is maintained, but its correctness is not evaluated

This is an important omission due to the paper claims. The paper evaluates final task completion and normalized goal progress, but these focus on outcomes. They do not directly establish if the agent's internal progress estimates were correct throughout execution. An agent could, for example, complete a task while holding incorrect progress beliefs, or fail despite accurately identifying that progress had regressed.

W4. Many reporting and reproducibility details seem to be missing

The paper claims their model as part of its first contribution, and mention distillation for that in the abstract, but provides no detail about the data generation and training process for Z-RoboSTAR-27B. The text points to Appendix A, but that deals with "evaluation platform and execution protocol". Details on data sources, training/evaluation separation, distillation objectives, and model-training budget are critical.

Questions for authors
  • What is the actual key contribution you'd like to claim here? The paper would be much stronger if more specific in its claims and evidence, instead of claiming harness architecture, model training, and distinct benchmarks all at once.

  • Why not compare to other scene-graph approaches if this is the key point of the proposed harness? Do the scene-graph and task-monitoring ablation gains persist on exactly matched trials?

  • How accurately does "System 2" detect progress regression immediately after an unexpected intervention?

  • What portion of the claimed performance gain is from just persistent memory versus explicit task-progress recomputation?

  • How would the distilled model generalize to previously unseen tasks or scene compositions?

  • In what form and under what license would the benchmark be released?

2 comments
AuthorsOpenReview ↗

Response to Reviewer mz3d

Thank you for these questions. They help us clarify our key contribution and scope (W1&Q1, W2), comparison with other scene-graph agents (Q2), ablation study (Q2, Q3, Q4), and our distilled details (Q5).

W1&Q1. Our key contribution

Reviewer: What is the actual key contribution you'd like to claim here? The paper would be much stronger if more specific in its claims and evidence, instead of claiming harness architecture, model training, and distinct benchmarks all at once.

Our key contribution is the proposal of a System 2 harness, Z-RoboSTAR, for sustained manipulation under interventions, using a scene graph as a persistent shared-state interface to synchronize scene understanding, task progress, and planning. Specifically, an evaluator assesses the success of the preceding action, a global scene-belief reconciliation revises the maintained state from new evidence, and task progress is reassessed from that revised state before subsequent planning. This allows a previously completed subgoal to be reactivated when its supporting condition no longer holds.

The benchmark, Z-RoboSTAR-Bench, provides a way to evaluate this capability through diverse temporal and spatial dependencies and kinds of progress-triggered interventions. The distilled model provides an implementation of the framework with an adapted backbone. We will revise the presentation to make this hierarchy clear, with the harness mechanism as the primary contribution and the benchmark and model adaptation supporting its evaluation. The distilled model, Z-RoboSTAR-27B, instantiates our framework with a smaller backbone post-trained (SFT) on agentic trajectories from stronger models, with the goal of achieving comparable task success rates in less time (nearly 50%) for practicality. We will provide more details about the model when we release its weights.

W2. The benchmark's relationship to real-world closed-loop behavior

Reviewer: The benchmark's relationship to real-world closed-loop behavior is also not well supported.

Thank you for raising this point. We think that the current decoupling of System 2, System 1, and even the System 0 controller actually makes it difficult to pass certain closed-loop information, which you may be interested in (such as infeasible IK), from System 0 through System 1 back to System 2. This is because our interface operates at the subtask level and has not yet moved down to the end-effector/pose level, as in the recent impressive advances of GPT-6 Astra and Claude Fable 5.2. Therefore, within the scope of this paper, where the goal is for System 2 to plan effectively and provide subtasks to System 1, our benchmark’s open-loop symbolic execution can align well with real-world closed-loop execution, as reported in our evaluation.

Q2. Comparison with other scene-graph agents and our module ablation

Reviewer: Why not compare to other scene-graph approaches if this is the key point of the proposed harness?

We have added comparisons with Thea [1] and Statler [2], using their official implementations with our benchmark-specific adaptations. For Thea, we retain its agent loop, memory, and official evaluator. Since its public release provides the scene-graph interface but not a complete backend, we implement that interface using our symbolic scene-graph backend. We also evaluate its released NullSceneGraph configuration. For Statler, we retain its state reader/writer, recursive function generator, and prompt structure, while extending its action interface to cover our benchmark's semantic actions.

Each configuration is evaluated on 57 source scenes, with one nominal and six intervention conditions per scene, yielding 399 rollouts. The following table reports intervention SR / NGP (%):

ModelStatler w/ extended actionsThea w/o scene graphThea w/ our scene-graph backendZ-RoboSTAR
GPT-5.50.0 / 44.114.6 / 60.150.3 / 82.977.5 / 94.5
Claude Opus 4.72.3 / 36.12.9 / 40.247.9 / 81.855.6 / 86.0
Z-RoboSTAR-27B4.5 / 47.95.3 / 57.844.8 / 75.571.6 / 90.2

With the backbone held fixed within each row, Z-RoboSTAR improves intervention SR over Thea with our scene-graph backend by 27.2, 7.7, and 26.8 percentage points, respectively. These results support the complete framework beyond providing access to a persistent scene graph.

[1] Towards the Harness of Embodied Agents. [2] Statler: State-Maintaining Language Models for Embodied Reasoning.

Reviewer: Do the scene-graph and task-monitoring ablation gains persist on exactly matched trials?

The results in Table 4 are obtained from exactly matched trials, with identical initial scene states and task goals, the same model, System 1 executor, and runtime configuration, identical intervention targets, types, and triggering conditions, and the same execution budgets and scoring rules.

AuthorsOpenReview ↗

Response to Reviewer mz3d

Q3. Detecting progress regression after an intervention

Reviewer: How accurately does "System 2" detect progress regression immediately after an unexpected intervention?

The framework reassesses task progress after supported scene revisions are committed and before the next action is planned. For example, if new evidence shows that a previously placed object has left its required container, the corresponding placement goal can be reopened even when the latest action on another object succeeded. This depends on the change being observable and correctly incorporated into the scene belief.

On nine completed-subgoal rollback scenes with identical post-intervention states and trigger steps across models, GPT-5.5 recovers in 9/9 rollouts and Z-RoboSTAR-27B in 8/9.

ModelSuccessful rolloutsIRR (%) ↑
GPT-5.59/9100.0
Z-RoboSTAR-27B8/988.9
Gemini 3.1 Pro8/988.9
Claude Opus 4.77/977.8
DeepSeek-V4 Pro5/955.6
GLM-5.24/944.4
GPT-5.43/933.3
Qwen3.6-Plus3/933.3
Qwen3.7-Plus1/911.1

Separately, the visual audit reports graph-correction precision of 68.0% for Z-RoboSTAR-27B with human execution and 58.3% for GPT-5.5 with VLA execution.

Model / System 1 executorTotal judgmentsGraph precision ↑
Z-RoboSTAR-27B, human execution192/27068.0%
GPT-5.5, VLA execution28/3858.3%

Q4. Persistent memory versus task-progress recomputation

Reviewer: What portion of the claimed performance gain is from just persistent memory versus explicit task-progress recomputation?

We think that the Thea configurations in W1&Q1 can help establish the value of the scene-graph backend: adding it increases intervention SR by 35.7, 45.0, and 39.5 percentage points for GPT-5.5, Claude Opus 4.7, and Z-RoboSTAR-27B, respectively. Thus, persistent structured state already provides a substantial benefit.

Z-RoboSTAR achieves further gains over Thea with our scene graph backend, showing that the complete framework provides additional value when a dynamic scene representation is already available. However, this comparison also changes the surrounding update and planning mechanisms. It therefore does not isolate how much of the additional gain comes specifically from task-progress recomputation.

The distinction is operational: persistent memory retains and updates facts about the scene, while progress recomputation checks whether those facts still support each goal's completion.

Q5. Generalization of the distilled model

Reviewer: How would the distilled model generalize to previously unseen tasks or scene compositions?

Thank you for raising this point. We evaluate the distilled model using both layout-held-out and task-held-out splits, where one task used for testing is excluded, and one scene layout for each other tasks is excluded from the distillation training data. This is intended to test whether the model can apply its acquired planning and state-tracking capabilities to unseen tasks and spatial layouts. We will specify this in our revised manuscript.

Q6. Benchmark release

Reviewer: In what form and under what license would the benchmark be released?

We plan to release Z-RoboSTAR-Bench as a public, versioned repository containing the scene and task configurations, intervention protocols, execution code, and evaluation scripts needed to reproduce our results. Our code will be released under the MIT License, with third-party components retaining their original licenses.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely appreciate your time and constructive feedback on our manuscript.

We are encouraged by the reviewers' positive assessments, including their recognition of the importance of the problem we address (mz3d, 6XKn, xExP, nh79), their remarks on the paper's novel framework and benchmark (mz3d, 6XKn, xExP, nh79), and extensive experiments covering diverse execution (mz3d, 6XKn, xExP, nh79).

Following the discussion, three reviews (of four) confirmed that our responses addressed their questions and concerns. We thank Reviewers xExP and nh79 for raising their rating to 3, and all reviewers for their feedback, which helped clarify the details and benefits of our approach. An additional review (mz3d) first became visible to us on November 16 at the end of the discussion. We have submitted our response and hope that our clarifications and additional evidence address the reviewer’s concerns.

In response to the reviewers' comments, we provided the following clarifications and additional evaluations:

  • Compare with existing agent harnesses (xExP, nh79, mz3d). We added Thea and Statler comparisons across three VLM backbones, 57 scenes, and seven conditions per scene. Z-RoboSTAR improves intervention SR over Thea with our scene-graph backend by 7.7–27.2 percentage points, supporting the complete framework’s benefit.

  • Ensure stability of results and conduct more ablation studies (6XKn, xExP, mz3d). We added a history-window ablation, repeated evaluations with 95% confidence intervals, a recovery analysis on nine scenes matched for post-intervention state and trigger step, and more metrics to evaluate our methods' effectiveness. These additional evaluations strengthen the evidence for the stability of our results and provide a more comprehensive assessment of our framework.

  • Benchmark protocols and applicability (xExP, nh79). We detailed the six intervention protocols, compared the benchmark with RoboCerebra and VLA-Arena, and evaluated two Thea configurations in symbolic and human execution, observing consistent ordering across both modes. We will open-source our benchmark and agent to the community.

Thank you for your time and consideration.

Best regards,

Submission29512 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.