acceptodds
Under review as a conference paper at ICLR 2027

GenieMaster: An Embodied Continual Harness for Persistent Multi-Robot Work

Abstract

Persistent robot work requires general reasoning, responsive manipulation, and reliable recovery as tasks and environments change. Frontier agents provide high-level reasoning, while learned policies provide continuous, closed-loop control from current observations. We present GenieMaster, an embodied continual harness for persistent multi-robot work. At its core is the Embodied Execution Protocol (EEP), an interface through which the agent interacts with the physical world. Through EEP, the agent invokes robot capabilities and guides policies with spatial cues, object references, and detailed language. We co-design the guidance interface and policy training so that policies can translate agent-provided guidance into robot actions. The agent uses physical feedback as evidence for task progression, guidance revision, and recovery, while retained task state supports coordination across robots and resumption after interruptions. GenieMaster supports continual harness evolution through asynchronous reflection and memory consolidation, using execution experience to update guidance, assessment criteria, and orchestration strategies. Across six real-world tasks, GenieMaster achieves 78.67% macro-averaged success versus 38.83% for the Dual-system baseline based on a vision–language model. Further evaluations show improved retrieval of unseen products with object-reference conditioning, while a convenience-store deployment illustrates ongoing task orchestration over a 24-hour window as new requests arrive.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Agents, Agent Harnesses, Vision-Language-Action Models, Multimodal Guidance, Continual Harness Evolution, Multi-Robot Coordination

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_g6GKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GenieMaster, an embodied continual harness for persistent multi-robot task execution. Its core component, the Embodied Execution Protocol (EEP), connects high-level agent reasoning with robot policies through capability invocation, multimodal guidance, execution feedback, and task-state continuity. The framework further incorporates asynchronous reflection and memory consolidation to improve subsequent task execution without updating model parameters. Experiments across six real-world tasks demonstrate 78.67% average success, compared with 38.83% for a dual-system baseline. Additional evaluations show improvements in unseen-object retrieval, LIBERO-Pro performance, and experience-based orchestration, alongside a 24-hour convenience-store deployment.

Critical strengths

1.The paper addresses an important challenge in embodied intelligence: connecting high-level agent reasoning with continuous robot control and persistent multi-robot task execution.

2.EEP provides a structured interface for capability invocation, multimodal policy guidance, physical feedback, and task continuity.

3.Experiments across six tasks and multiple robot platforms demonstrate substantial improvements over conventional VLA and dual-system baselines.

4.The integration of asynchronous reflection and persistent memory offers a promising approach to improving task orchestration without modifying model parameters.

Critical weaknesses

1.EEP primarily integrates established components, including hierarchical planning, tool invocation, multimodal policy conditioning, execution monitoring, and memory-based reflection. Its fundamental distinction from existing embodied-agent harnesses and hierarchical VLA systems remains insufficiently established.

2.GenieMaster differs from the dual-system baseline in agent backbone, guidance modalities, feedback mechanisms, and orchestration capabilities. Consequently, the large performance improvements cannot be attributed to EEP or continual adaptation individually.

3.The experience-learning evaluation uses only five rounds of seven trials. Performance is non-monotonic, dropping from 71.4% to 42.9% in the fourth round, and no repeated independent learning trajectories or memory-free controls are reported.

4.The capability ablation reports execution time only for successful trials, while success rates vary substantially. The reported time reductions therefore do not establish an equivalent improvement in overall task efficiency.

5.Although a 24-hour convenience-store deployment is demonstrated, quantitative evidence regarding task throughput, intervention frequency, recovery reliability, and long-term multi-robot coordination is limited.

Questions for authors

1.What is the fundamental technical novelty of EEP compared with existing embodied-agent harnesses, hierarchical VLA systems, and tool-based robotic orchestration frameworks?

2.Can the authors provide controlled ablations isolating the contributions of multimodal guidance, success/error detection, task continuity, and experience memory under the same agent and policy backbones?

3.How robust is the proposed continual adaptation mechanism across multiple independent runs, and how does it compare with memory-free reflection or fixed-memory baselines?

4.Can the authors report end-to-end execution time, agent inference overhead, failure recovery costs, and efficiency metrics averaged over both successful and unsuccessful trials?

5.Can the authors provide quantitative results from the 24-hour deployment, including completed tasks, failures, human interventions, robot utilization, and recovery success rates?

2 comments

We thank the reviewer for the thoughtful feedback and concrete suggestions. We address the five questions below.

Q1. Technical novelty of EEP

Thank you for this important question. GenieMaster co-designs the agent–policy guidance interface and policy training; EEP connects guidance, execution, and physical feedback.

Agent→policy. Training and deployment accommodate agent-provided visual and textual guidance, improving instruction following and zero-shot generalization to unseen products. Temporal-offset augmentation pairs earlier guidance frames with current observations to accommodate guidance latency and reuse, improving success by 9 percentage points.

Policy→agent. With model parameters fixed, feedback supports progression and recovery, while reflection adapts guidance and orchestration. Shelf-selection and waist-adjustment strategies emerge from EEP-supported execution and reflection rather than manual specification.

As foundation models advance, we hope to encourage low-level policy data collection and training designed for agentic systems, jointly optimizing the agent harness and policy.

Q2. Component ablations

Thank you for suggesting clearer ablations. Section 4.3 cumulatively evaluates tools, agent visual guidance, and control allocation. We add three ablations and planning-failure attribution; GM and Dual-system use matched policy-training conditions.

MethodOrchestrationVisual guidanceMemorySuccess % ↑Trials ending in planning failure (%) ↓
Vanilla VLANone——18.8377.17
Dual-systemVLM subtask updates——38.8343.17
GM (full)Agent + Detector✓✓78.676.00
GM: VLM orchestrationVLM subtask updates✓✓58.0036.00
GM: no visual guidanceAgent + Detector—✓45.006.00
GM: no memoryAgent + Detector✓—66.0017.00

Closed-loop orchestration. Retained progress and detector-driven transitions/recovery naturally support task continuity. Replacing GM orchestration with VLM subtask updates, using the same policy, visual-guidance mechanism, and historical memory, raises planning failures from 6% to 36%. This gap highlights the limitations of the fine-tuned VLM planner in our store setting, with over a thousand SKUs and dozens of shelf locations. Correctly executing the wrong subtask need not achieve the goal.

Guidance and memory. Separate ablations retain the full GM agent and policy. Memory removal excludes historical experience but preserves current task progress; see Q3.

Q3. Robustness of continual adaptation

Thank you for this suggestion. We conducted five independent runs from the same initial memory, each spanning five rounds, with a control retaining no cross-trial memory. With 10 trials per group per round, the two groups total 500 trials across all runs, including failures and timeouts.

Memory settingRound 1Round 2Round 3Round 4Round 5
Accumulated memory4.00 ± 5.4822.00 ± 10.9544.00 ± 8.9448.00 ± 19.2454.00 ± 19.49
No cross-trial memory8.00 ± 8.378.00 ± 4.472.00 ± 4.474.00 ± 5.4810.00 ± 10.00

Success rate (%), reported as mean ± sample standard deviation across five independent runs.

Q4. End-to-end efficiency and overhead

Extending the successful-trial timings in Section 4.3 and Appendix B, including retries, the table averages execution times and VLM calls over all trials.

ConfigurationSuccess rate (%)Mean end-to-end time (s)Mean VLM calls
C1251921.75122.1
C240752.4057.4
C38338.463.2
C49418.942.1
C59715.381.5

C5 uses 3.6 s of inference per task (2.8 s blocking execution), with 0.08 recoveries per task averaging 3.4 s each.

Q5. Quantitative results from the 24-hour deployment

24-hour outcomes. Of 264 tasks—96 retrieval, 65 restocking, 29 cleaning, 74 patrols—253 completed and 11 failed (4 retrieval, 6 restocking, 1 cleaning).

Intervention/recovery. Human intervention occurred 8 times across 6 tasks, mainly for unrecoverable floor drops. Autonomous recovery completed the affected subtask in 13/18 events (72.2%).

Utilization. Task execution, including patrols, occupied 78%/57% of the 24-hour window for A/B. A’s higher priority yields more patrol assignments when both robots are idle and no customers are present.

Reviewer_g6GKOpenReview ↗

To auhtors

Thank you for your reply. I will keep my score. I have no further questions.

Reviewer_WmKmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GenieMaster connects agent planning, robot policies, physical feedback, and task state through the Embodied Execution Protocol (EEP). Policy training supports spatial guidance and product references. Across six physical tasks, macro-averaged success is 78.67%, versus 38.83% for the Dual-system baseline (Table 1, p.6).

Critical strengths
  1. Each method has 100 randomized-layout trials per task, with matched policy-training conditions, across manipulation, coordination, and interruption recovery (Table 1, p.6).

  2. Adding object references raises success from 83% to 94% with the agent and policy checkpoint fixed (C3–C4). Temporal-offset training adds 9 percentage points. Separately trained policies with matched demonstrations and training budgets achieve 23/25 versus 5/25 on novel products; this tests reference-conditioned training together with inference guidance (pp.7–8; Appendix B).

Critical weaknesses
  1. Five rounds of seven trials lack a frozen-memory control. The change from 2/7 to 6/7 is insufficient to establish reliable continual improvement (Fig.3c, pp.7–9).

  2. EEP schemas, operative prompts, numerical training settings, and common time/retry limits are underspecified. Detector sample counts and the separation between training and evaluation data are also unclear (pp.3–8; Table 8, p.20).

Questions for authors
  1. Can a frozen-memory control or independent task sequences confirm adaptation? Please also report the detector evaluation split and the number of completion boundaries assessed.

  2. What common time/retry limits and numerical training settings were used? Please provide sample sizes and absolute success rates for the temporal-offset ablation.

1 comment

Thank you for the additional results. The five independent runs with a no-cross-trial-memory control substantially strengthen the evidence for continual adaptation. The final-round mean success of 54% versus 10% addresses much of my concern about the original small experiment. The component ablations and timings over all trials also help clarify the contributions and practical costs of the system.

I remain supportive of acceptance and will retain my score of 3 (Weak acceptance). My remaining concerns are the reproducibility details raised in my review: the detector’s training/evaluation split and number of completion boundaries evaluated; the common time and retry limits and numerical training settings; and the sample sizes and absolute success rates for the temporal-offset ablation.

Please include these details, along with the EEP schemas and operative prompts, in the revision. They would make the stronger experimental evidence easier to assess and reproduce.

Reviewer_6jXnOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces an embodied harness, GenieMaster, which builds on Embodied Execution Protocol (similar to MCP) to have agents control robots. The authors train policies on demonstrations from this embodied agent and show asynchronous reflection allows the agent to better accomplish robotic tasks. Simulation experiments are done with Libero-pro and real experiments are done with AGIBOT in a store environment.

Critical strengths

The simulated and real evaluations seem thorough and testing against baselines also seems thorough. There is a substantial improvement in the Libero-pro benchmark over strong baselines e.g. CaP-X and pi0.5. For real-world experiments, there are multiple tasks across multiple embodiments tested in a convenience store with many products.

Critical weaknesses

The paper doesn’t go into depth about what aspect of the reflection helps to increase performance. The design for the orchestration is also not ablated and there doesn't seem to be a systematic study into the continuously improving aspect of the embodied harness.

Questions for authors

How does the harness improve itself? What are the types of failures in the libero-pro benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.