- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RoboAware studies how embodied coding agents can select between modular robotic skills and frozen end-to-end policies. It introduces the P5 skill schema, State-Locked Counterfactual Branching (SCB), and Execution-Aware Learning (EAL). The approach uses same-state counterfactual executions to train a value-based coordinator that selects the policy family at semantic skill boundaries. Experiments are conducted on LIBERO-Pro, RoboSuite, and RoboTwin.
Critical strengths
-
Same-state counterfactual branching addresses a genuine limitation of embodied policy coordination: an executed action does not reveal how an alternative policy would have performed from the same physical state. This makes the motivation for SCB compelling and gives EAL a meaningful training signal.
-
The empirical gains are substantial, and internal ablations help isolate their sources. On RoboTwin, RoboAware achieves 90.0% success, compared with 53.8% without paired counterfactual outcomes and 41.9% using only modular execution. These comparisons provide stronger evidence than benchmarking against standalone policies alone.
Critical weaknesses
W1. Evaluation does not establish generalization to unseen task families.
The coordinator is trained using counterfactual branches collected from scene seed 0 and evaluated on seeds 1–10 of the same underlying task pool. While this protocol measures robustness to variations in initial conditions, it does not establish whether the learned responsibility coordinator generalizes to unseen tasks, manipulation primitives, or substantially different environments. Since the principal contribution is learning when different execution families should be selected, task-level generalization is essential for distinguishing transferable execution awareness from task-specific responsibility memorization.
W2. Strong dependence on privileged simulation and exact state restoration.
State-Locked Counterfactual Branching requires restoring identical physical states and obtaining privileged outcome information, including object-pose and perception-based reward signals. These capabilities are generally unavailable in physical robotic systems. Although the learned coordinator does not require such information at deployment, the paper provides no physical-robot validation or evaluation under approximate state restoration. Consequently, the applicability of the learning framework outside privileged simulators remains uncertain.
W3. Potential confounding factors in the counterfactual ablation.
The comparison between RoboAware and RoboAware-UO shows large improvements from using paired counterfactual outcomes. However, counterfactual branching may also change the amount, diversity, and quality of training experience available to the coordinator. The paper does not sufficiently isolate the benefit of same-state pairing from the benefit of collecting additional execution trajectories. A matched-interaction-budget comparison with unpaired but equally diverse trajectories would be particularly informative.
W4. The primary baseline comparison does not fully isolate coordinator quality.
RoboAware has access to a hybrid execution interface containing both modular and end-to-end policy families, whereas several baselines operate with more restricted action interfaces. Additionally, some baselines are evaluated without their usual cross-episode memory, and certain end-to-end policies are incompatible with the bimanual RoboTwin setting. Therefore, the aggregate performance advantage combines contributions from the available execution tools, hierarchical decomposition, and learned family selection. More matched-interface comparisons would strengthen the attribution to the proposed coordinator.
W5. Dependence on manually specified P5 stage contracts.
The P5 schema provides predefined semantic boundaries, typed inputs, postconditions, and admissible policy families. These structured interfaces are useful but require substantial domain knowledge and careful specification. The system's success may depend on the quality of these manually designed contracts rather than solely on learned execution responsibility. The paper does not evaluate robustness to incorrect stage boundaries, imperfect postconditions, or previously unseen skill schemas.
W6. Restricted family selection and limited interruption capabilities.
The coordinator chooses between only two broad execution families: modular and end-to-end. It does not explicitly optimize over multiple heterogeneous controllers with different capabilities. Furthermore, responsibility can change only at P5 decision boundaries, and some end-to-end invocations may span several stages before returning control. This limits its ability to detect and correct failures during long or contact-sensitive atomic executions.
W7. Limited characterization of computational and interaction costs.
The paper reports approximately 24 GPU-hours for coordinator training and an additional day for counterfactual data collection. However, counterfactual collection requires repeated execution of expensive coding-agent-generated programs and environment rollouts. A complete accounting of coding-model calls, simulation steps, execution latency, and total training cost would make it easier to assess whether the learned coordinator offers a favorable trade-off relative to simpler responsibility-selection methods.
Questions for authors
-
How does the coordinator perform on completely unseen tasks or manipulation primitives, with task families held out during counterfactual data collection?
-
How would EAL behave when exact simulator restoration or privileged pose-based reward signals are unavailable? Can training rely on approximate restoration or outcome-only feedback without substantially degrading performance?
5 comments
Point-by-point response to Reviewer s5xQ [2/2]
W5. About P⁵ boundaries
As specified in Table 2, each modular API is assigned to exactly one P⁵ stage, making its intended stage boundary unambiguous. The E2E API is the explicit exception that can span several stages. Appendix B provides the execution steps, postconditions, and return conditions for each API. The admissible modular and /FastWAM execution families are designed for broad coverage of object-centric manipulation, so these contracts are reusable across tasks. We will clarify this design and discuss stage-boundary re-estimation, imperfect postconditions, and previously unseen skill schemas as extensions for future work.
W6. Two-family selection and interruption during atomic execution
The modular family exposes a complete modular composition with separate perception, grasp-pose proposal, motion, and gripper components, while the coding agent dynamically selects an available end-to-end backend through in-context reasoning. Thus, the two-family abstraction organizes multiple execution capabilities. The retained end-to-end modules are screened offline for reliable closed-loop execution and provide internal feedback and correction during a call. Our coordinator learns when to schedule these capabilities to complete longer tasks and recover from failures. Responsibility is reconsidered when a call returns at a P⁵ decision boundary; finer-grained interruption within an atomic call remains an extension.
W7. Coding-model calls, simulation steps, latency, and total training cost
We report offline collection and training costs separately from deployment latency, including coding-model usage and simulator interactions.
| Stage | Measurement | Value |
|---|---|---|
| Coordinator fitting | Hardware | 4 × A100, 80 GB each |
| Coordinator fitting | Wall-clock time | ~ 6 hours |
| Coordinator fitting | GPU-hours | ~ 24 |
| Counterfactual collection | Wall-clock time on the same hardware pool | ~ 1 day |
| Full collection-and-training pipeline | GPU-hours | ~ 120 |
| Full collection-and-training pipeline | Coding-model calls | ~ 4×10⁴ |
| Full collection-and-training pipeline | Input & output tokens | ~ 3×10⁸ & ~ 2×10⁷ |
| Full collection-and-training pipeline | Simulator control steps | ~ 3×10⁶ |
| Deployment | Coordinator inference per decision (ms) | ~ 50 |
| Deployment | Mean episode execution time, RoboAware (s) | ~ 150 |
| Deployment | Mean episode execution time, training-free selection (s) | ~ 180 |
These measurements distinguish the one-time cost of counterfactual experience from deployment overhead. Deployment uses the frozen coordinator without further counterfactual branching.
Once again, thank you for your careful review. If most of your concerns are addressed, would you please consider increasing the recommendation of the paper?
Thank you for the detailed rebuttal and additional experiments. However, I am not convinced that the responses fully address my concerns. In particular, the generalization of the learned coordinator to genuinely unseen tasks and skills remains insufficiently established, and the independent contribution of same-state counterfactual pairing is not yet fully clear. Moreover, the method still relies heavily on manually defined P5 stage boundaries, and its robustness to imperfect task decomposition remains unexplored. Overall, while the additional results are encouraging, I do not believe they sufficiently address the limitations regarding generalization, methodological contribution, and practical applicability. Therefore, I maintain my original rating.
Point-by-point response to Reviewer s5xQ [1/2]
We sincerely thank the reviewer for the comments and review. We provide a point-by-point response to address your concerns below.
W1. Held-out-task generalization
Please see Held-out generalization for the three-level evaluation (within-domain RoboTwin held-out tasks, cross-benchmark LIBERO-Pro2, and real-world deployment). We will update the results in the revision.
W2.1 State restoration and binary outcome
We further test whether training itself can relax exact restore and dense rewards on RoboSuite, keeping the same EAL objective, model, and evaluation protocol. (1) Approximate restore: instead of exact simulator snapshot restore, we calculate and apply the EEF inverse transformation from the recorded state. As the surrounding environment may already have changed slightly, the branch start is only approximately the locked state. (2) Outcome-only reward: we keep the same training target but drop dense / pose-based reward terms, retaining only the binary terminal task outcome.
Success rates remaining comparable on RoboSuite, supporting that EAL does not require perfect restore or dense privileged rewards to remain effective.
W2.2 Physical validation beyond simulation
EAL is designed for Sim2Real transfer: during training we deliberately exploit available privileged simulator information (exact state restoration, dense rewards, and counterfactual branching) to fit the coordinator. At deployment the frozen coordinator uses only onboard observations and never requires state restoration, privileged rewards, or further branching.
To validate deployment without privileges, we deploy the frozen coordinator without additional training on a real-world AgileX PiPER dual-arm System for a long-horizon compositional task: first reorient two beverage bottles lying on the table with both hands, then place them neatly and simultaneously into a plate. RoboAware exhibits execution-coordination behavior: modular executors handle approaching the bottles and the final placement onto one side of the plate, while E2E executors handle fine-grained grasping and in-air pose rotation to upright the bottle. This real-robot study supports that a coordinator trained under privileged simulation can operate without those privileges at deployment. Please refer to Section 3.4 in the revision for this real-world evaluation.
W3. Counterfactual pairing
Please see Counterfactual pairing for the matched-budget comparison of RoboAware and RoboAware-UO. As shown there, although Eq. (4) learns from individual transitions, paired collection exposes unsuccessful and nonpreferred branches that unpaired executed-only data miss, and the gains isolate the benefit of same-state counterfactual pairing.
W4. Hybrid interfaces, disabled memory, embodiment compatibility, and coordinator quality
The P5 stage decision boundary and hybrid modular and end-to-end interface are RoboAware's core agent-system contributions. As the pre-trained π0.5 model is not directly compatible with dual-arm setups, existing baselines cannot be used for RoboTwin. Benefiting from its flexible modular composition, RoboAware can adapt to more embodiment settings and a broader range of task scenarios.
We have conducted various ablation studies under shared infrastructure to isolate the contributions of available execution tools, hierarchical decomposition, and learned family selection, respectively. (Abbreviations: MO = modular only; EO = end-to-end only; TF = training free; UO = unpaired outcomes; FH = fixed handoff; ES = episode selection.)
First, MO and EO keep only one execution family and fall well below the hybrid interface, so both tool families are needed for broad coverage. Second, ES chooses a family once per episode and FH follows a fixed perceive/pre-manipulate to E2E handoff; both trail per-stage coordination, so fine-grained hierarchical decomposition at P⁵ boundaries matters. Third, TF skips coordinator training and UO trains without same-state pairing; both trail the full model, isolating the gain from learned family selection with paired counterfactual supervision.
To address the issue of cross-episode memory, please see Cross-episode memory for the Harness-VLA protocol and results. As reported there, RoboAware retains its advantage when the baseline can use accumulated task-specific experience, supporting the benefit of counterfactual coordination beyond execution memory.