- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes REA, a three-level runtime comprising executable skills (System 0), a router (System 1), and an agent for skill authoring or completion judgment (System 2). Its main components are development-set route memory, a boundary-triggered judge, and a fixed escalation ladder. REA-M achieves 95.9% success on 3,000 LIBERO-Pro episodes versus 94.7% for the strongest single-author baseline, with additional evaluations on two simulators and a real humanoid.
Critical strengths
- The paper addresses the latency introduced by online agent calls during skill execution. REA provides a common runtime for skill execution and routing, allowing different configurations to be evaluated on the same tasks, seeds, and execution budgets.
- The LIBERO-Pro experiments cover 60 tasks with 50 held-out seeds per task. The paper reports paired statistical tests and ablations of route memory and the judge gate. It also examines the additional execution time associated with judge-based recovery.
Critical weaknesses
- Experimental Evaluation and Component design.
The contribution of the escalation ladder is unclear. Table 1 reports at least 95.7% success for route memory only (=0), compared with 95.9% for REA-M. In addition, route memory is fitted on development episodes of the same 60 tasks, while evaluation on unseen tasks is not provided. The results mainly support per-task skill selection, with limited evidence for the benefit of escalation or routing on new tasks.
The latency experiments do not isolate asynchronous scheduling. In Table 3, the blocking advisor makes 1.71 online model calls per episode, whereas REA-M makes none. This demonstrates the cost of online inference but does not establish an advantage over a non-blocking advisor with a matched inference budget.
The contribution of the judge gate is difficult to isolate from re-runs and replay variance. On SS-center, Fix and Evo are byte-identical, yet REA-V succeeds in 9/12 episodes compared with 4/12 for REA-M (Table 7). Table 11 uses REA-M as a proxy for the first-run outcome, making the attribution of rescued failures and replay variance approximate. A more direct analysis of the first-run outcomes is needed.
The experimental evaluation is insufficient. The RoboTwin 2.0 experiment does not evaluate the routing method used on LIBERO-Pro. Appendix J states that RoboTwin has no skill library. Instead, REA uses a reflex model as System 0, and the router hands single decisions to a deliberate model. Therefore, this experiment does not test the route memory and escalation ladder used in the main experiments.
- The authors employ a substantial amount of specialized terminology throughout the paper. Apart from terms that are standard and universally recognized in the field, most lack explicit definitions or supporting references, making the paper difficult to comprehend.
For example, the abstract uses "best single-agent baseline" for the 94.7% result, while Table 1 evaluates frozen programs from a single skill author. Figure 1 uses "first/other/first" to describe the escalation ladder without explaining this notation in the caption, although Section 3.2 specifies a different order when three candidates are available. The use of "asynchronous" also needs clarification, since Algorithm 1 waits for JUDGE to return before the next run.
Questions for authors
- Table 1 reports at least 95.7% success for route memory only (=0). What is the actual success rate when this configuration is evaluated independently? How does route memory perform on unseen tasks?
- Can the authors compare REA with an online advisor that does not block skill execution, using the same number of System 2 calls and inference budget?
- How much of the improvement from the judge gate comes from rescued failures, re-runs, and replay variance? Can the authors report the actual first-run outcomes instead of using REA-M as a proxy?
- Since RoboTwin 2.0 has no skill library, which parts of the routing method are evaluated in this experiment? Can the authors explain how these results support the conclusions drawn from LIBERO-Pro? completion labels and retry-matched baselines, and demonstrate the same routing mechanism across platforms?