acceptodds
Under review as a conference paper at ICLR 2027

OmniAct: A Framework for Long-Horizon Cyber-Physical Embodied Agents

Abstract

Embodied agents deployed in everyday environments must carry out extended tasks across physical and digital domains while respecting user requirements and responding to unexpected execution outcomes. This requires coordinating robotic skills and digital tools with context management and timely feedback. We present **OmniAct**, a framework for long-horizon cyber-physical embodied agents that integrates multimodal planning, hierarchical memory, and asynchronous visual monitoring. A shared skill interface supports task decomposition and execution across heterogeneous tools and robotic policies. Sensory, episodic, and reflective memory organize recent observations, task events, and reusable experience, while visual monitoring enables interruption and replanning during physical execution. Constraints derived from earlier user dialogue are retained in reflective memory and supplied to the monitor, allowing historical requirements to guide stopping decisions. Across 40 manipulation and navigation tasks on two robotic platforms, OmniAct achieves overall end-to-end success rates of 63.3% and 65.0%, respectively, exceeding the evaluated planning, agent-harness, and composed baselines. Ablations support the contributions of memory organization and timely intervention. Further evaluations show improved constraint satisfaction across six planner backbones and substantially reduced input-context growth over 270 interaction rounds. These results demonstrate the value of coordinating memory and execution feedback for long-horizon embodied tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Agents, Cyber-Physical Systems, Long-Horizon Execution, Hierarchical Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_375ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OmniAct combines a planner, three layers of memory, and a visual monitor. It lets an agent use robot skills and digital tools within the same task. The monitor can flag a problem during execution, and the system then stops the action and returns control to the planner. The planner also turns earlier user instructions into saved rules and passes the relevant ones to the monitor, so earlier requests can guide later stopping decisions. On 40 robot tasks, OmniAct reaches success rates of 63.3% for manipulation and 65.0% for navigation. The paper also tests memory design, monitoring, and the handling of long interaction histories.

Critical strengths
  1. The paper addresses a useful problem: a robot should remember earlier user requirements while acting and recovering from errors. Tests cover two robot platforms and tasks that combine physical actions with digital tools. The comparisons include systems with existing memory and monitoring methods, which makes them more useful than comparisons with basic planners alone.

  2. Several component tests have useful controls. In Table 3, “Flat E/R” keeps the same total memory budget but combines two memory layers. It scores 56.7%, compared with 63.3% for “Full.” In Table 4A, “Delayed” and “Online” both observe execution as it happens, but differ in when they intervene. These tests help show the value of memory structure and timely interruption.

Critical weaknesses
  1. The lead over the strongest competing systems is small, and its reliability is unclear. In Table 1, OmniAct improves manipulation success from 60.0% to 63.3% and navigation success from 61.7% to 65.0%. Each gain amounts to two more successful trials out of 60. Each task is repeated three times, but no uncertainty estimates are given for these success rates. The results are encouraging, but they do not yet establish a clear advantage over the strongest baselines. Results for each task and a comparison that accounts for repeated trials would help.

  2. Table 3 and Table 1 compare OmniAct with A-MEM, which is useful. Still, two tests leave some uncertainty about what the proposed memory system adds. Passing saved user rules to the monitor seems to be an important part of the method. However, Table 4B does not clearly state whether DoReMi and CaM receive the same earlier instructions or saved rules. Without that detail, the difference could partly reflect which information each system receives. The longer-history tests also compare OmniAct only with uncompressed history. Those histories are built by adding unrelated events between relevant instructions and later requests. This is a useful test, but the A-MEM comparison in Table 3 uses independent manipulation episodes, not these long histories. Comparisons with other memory methods on the long histories would better show whether OmniAct’s memory design offers an advantage there. Fewer planner input tokens alone do not establish better task performance or lower total cost. The reported token counts are estimates of the planner’s input only. They do not include the language-model calls that compress and consolidate the memory.

Questions for authors
  1. Do DoReMi and CaM in Table 4B receive the same historical instructions or saved rules as OmniAct? If not, can you compare them with the same inputs?

  2. Can you provide results for each task and uncertainty estimates for Table 1? Appendix D says each task runs under varied initial conditions. Were these conditions matched across methods?

  3. Table 2 is labelled as a simulation experiment, but no simulator is described. Its “Full” row (63.3% manipulation, 65.0% navigation) is identical to OmniAct’s real-robot results in Table 1, while Appendix D.1 says the two use distinct trial sets. What simulator and task setup were used, and how do these episodes relate to the real robot tests?

Reviewer_mWYkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OmniAct, a framework for long-horizon cyber-physical embodied agents. It integrates a multimodal semantic planner, a shared skill interface, hierarchical memory (sensory/episodic/reflective), and asynchronous visual monitoring with interruption. The planner can store historical user constraints in reflective memory and supply them to the visual monitor during execution. The authors evaluate on 40 tasks across UR5e manipulation and Keenon navigation, comparing with SayCan, Code as Policies, RoboBrain2, OpenClaw, Hermes, and composed memory/monitoring baselines. They report improvements in task completion, constraint satisfaction, and context growth.

Critical strengths
  • The system integration is comprehensive: real robots, IoT, APIs, visual monitoring, and hierarchical memory are placed in a single planning–execution–recovery loop, giving the work clear system-level value.
  • The design that passes historical constraints from reflective memory to the monitor is well motivated, and the instruction-consistent stopping results in Table 4 support it.
Critical weaknesses
  • The main evaluation is relatively small: 20 tasks per platform with 3 repetitions each, assessed by human evaluation. Confidence intervals, annotator protocol, and inter-rater agreement are missing, making it difficult to judge robustness.
  • Baseline fairness needs clearer explanation: most systems share Gemini-3.1-Pro, but RoboBrain2 keeps its own 32B backbone. The degree to which composed baselines adapt memory/monitoring also affects conclusions. Some component ablations are reported “in simulation,” and their relationship to the real-robot main results should be clarified.
Questions for authors
  1. What is the exact human-evaluation protocol? Were there multiple annotators, blind evaluation, and agreement statistics?
  2. Are all main-table real-robot results from real execution? How does the “in simulation” Table 2 relate to the main results?
  3. Can the authors provide confidence intervals per task group and clarify whether baselines use the same skill/tool interface and constraint-injection mechanism?
Reviewer_zwxnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OmniAct links sensory/episodic/reflective memory with asynchronous visual monitoring for robot and digital-tool tasks. Across 20 manipulation and 20 navigation tasks, repeated three times, it reports 63.3% and 65.0% success; separate studies examine memory, interruption, and historical rules (pp.4–9).

Critical strengths
  1. Two physical backends and composed memory/monitoring baselines test a useful integration (Table 1, p.6).

  2. Online intervention reaches 76.2% versus 36.9% for Delayed. Within-runtime memory comparisons add evidence; equal budget is explicit only for Flat E/R (Tables 3–4A, p.8).

Critical weaknesses
  1. Historical stopping reaches 70.8% versus CaM’s 37.5%, but baseline histories and monitoring resources are unspecified, leaving attribution unresolved (Table 4B, pp.8,17).

  2. The strongest composition trails by only 2/60 successes per platform, without paired results or uncertainty. Configuration details and Table 2’s simulation setting remain unclear (pp.6–8,15–17).

Questions for authors
  1. What historical rules and monitoring resources did DoReMi/CaM receive, and can existing controls separate organization from rule supply?

  2. Please provide paired per-task outcomes, memory/monitor settings, and Table 2’s environment; do these records substantiate the incremental gain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.