- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OmniAct combines a planner, three layers of memory, and a visual monitor. It lets an agent use robot skills and digital tools within the same task. The monitor can flag a problem during execution, and the system then stops the action and returns control to the planner. The planner also turns earlier user instructions into saved rules and passes the relevant ones to the monitor, so earlier requests can guide later stopping decisions. On 40 robot tasks, OmniAct reaches success rates of 63.3% for manipulation and 65.0% for navigation. The paper also tests memory design, monitoring, and the handling of long interaction histories.
Critical strengths
-
The paper addresses a useful problem: a robot should remember earlier user requirements while acting and recovering from errors. Tests cover two robot platforms and tasks that combine physical actions with digital tools. The comparisons include systems with existing memory and monitoring methods, which makes them more useful than comparisons with basic planners alone.
-
Several component tests have useful controls. In Table 3, “Flat E/R” keeps the same total memory budget but combines two memory layers. It scores 56.7%, compared with 63.3% for “Full.” In Table 4A, “Delayed” and “Online” both observe execution as it happens, but differ in when they intervene. These tests help show the value of memory structure and timely interruption.
Critical weaknesses
-
The lead over the strongest competing systems is small, and its reliability is unclear. In Table 1, OmniAct improves manipulation success from 60.0% to 63.3% and navigation success from 61.7% to 65.0%. Each gain amounts to two more successful trials out of 60. Each task is repeated three times, but no uncertainty estimates are given for these success rates. The results are encouraging, but they do not yet establish a clear advantage over the strongest baselines. Results for each task and a comparison that accounts for repeated trials would help.
-
Table 3 and Table 1 compare OmniAct with A-MEM, which is useful. Still, two tests leave some uncertainty about what the proposed memory system adds. Passing saved user rules to the monitor seems to be an important part of the method. However, Table 4B does not clearly state whether DoReMi and CaM receive the same earlier instructions or saved rules. Without that detail, the difference could partly reflect which information each system receives. The longer-history tests also compare OmniAct only with uncompressed history. Those histories are built by adding unrelated events between relevant instructions and later requests. This is a useful test, but the A-MEM comparison in Table 3 uses independent manipulation episodes, not these long histories. Comparisons with other memory methods on the long histories would better show whether OmniAct’s memory design offers an advantage there. Fewer planner input tokens alone do not establish better task performance or lower total cost. The reported token counts are estimates of the planner’s input only. They do not include the language-model calls that compress and consolidate the memory.
Questions for authors
-
Do DoReMi and CaM in Table 4B receive the same historical instructions or saved rules as OmniAct? If not, can you compare them with the same inputs?
-
Can you provide results for each task and uncertainty estimates for Table 1? Appendix D says each task runs under varied initial conditions. Were these conditions matched across methods?
-
Table 2 is labelled as a simulation experiment, but no simulator is described. Its “Full” row (63.3% manipulation, 65.0% navigation) is identical to OmniAct’s real-robot results in Table 1, while Appendix D.1 says the two use distinct trial sets. What simulator and task setup were used, and how do these episodes relate to the real robot tests?