acceptodds
Desk rejected at ICLR 2027

Agency at the Right Scale: Bimanual Manipulation through Active Observation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Robot agents must adapt their reasoning as they act on and acquire knowledge about the physical world. In robotic manipulation, this raises a central question: which decisions should remain open to the model as execution unfolds? We introduce (ARS), a bimanual robot-agent architecture that makes the scope of reasoning an online decision. ARS connects task-level reasoning with sustained local interaction, preserving established goals and arm roles during local correction while allowing physical feedback to trigger broader replanning. Active perception makes this coordination embodied: the agent moves a wrist camera to resolve uncertainty in an ongoing manipulation, then uses the acquired evidence to decide whether to continue correcting, execute an established operation, or revise the task arrangement. Reusable procedures retain established action structure across executions, leaving scene-dependent interpretation and recovery to online reasoning. We evaluate ARS on 18 RoboDojo tasks and use execution studies and physical-robot demonstrations to examine how observation, correction, and procedural reuse interact during manipulation. ARS connects what the robot has established through physical interaction to what it can commit to execution and what must remain open to model judgment.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Artificial Intelligence, Embodied Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2tjpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Agency at the Right Scale (ARS), a bimanual robot-agent architecture combining task-level planning, contract-bounded local visual correction, and offline skill consolidation. Physical wrist-camera movements acquire evidence, while delegation and return determine which goals and arm roles remain fixed. On a selected RoboDojo subset, ARS reports good performance.

Critical strengths

Separating task commitments from local corrections provides a clear design for preserving goals and arm roles while allowing escalation when the arrangement must change. Integrating physical view acquisition into this interface makes the architecture practically meaningful for occluded bimanual manipulation.

ARS improves both reported metrics in every task group. The insertion example also makes the observation-to-correction mechanism concrete, while explicitly distinguishing single-tube seating from full-task success.

Critical weaknesses

W1. The promised online-ARS observation and scale-selection ablations lack quantitative success-rate/decision-step results. Table 1 compares complete configurations. It seems that the benefits of adaptive delegation and active observation are not isolated from other interface differences. This leaves the central architectural claims insufficiently validated.

W2. Table 1 omits episode counts, per-task outcomes, and uncertainty estimates. Appendix B names priors, authored procedures, environment patches, and grasp-attachment settings without specifying their exact configuration or clearly tying them to the baseline.

W3. Examples from two tasks, without trial counts or full-task success rates, do not establish cross-embodiment generalization. Section 4.4 also says an Opus 5 controller was retained from simulation, whereas Table 1 identifies GPT-6. The relationship between these configurations is not explained.

Questions for authors

Where are the observation/scale ablation results referenced in Section 4.1? Please provide success-rate and decision-step comparisons against no-active-observation and fixed-delegation variants, with APIs, skills, physical-action budgets, and seeds matched.

Which priors, procedures, patches, and grasp-attachment settings were shared with Robocurve?

Which skills were generated from traces, and were evaluation layouts held out from skill construction? For Figure 4, what are the task set, trial counts, paired success rates, and offline construction/validation cost?

Which controller, prompts, and skills were used for each simulation and physical condition? Please reconcile the GPT-6/Opus 5 descriptions and report physical trial counts, completion criteria, and full-task outcomes for both tasks.

Reviewer_PrWPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The agentic robot uses an LLM as the system brain to control the robot is a timely topic this year. In this paper, the authors present Agency at the Right Scale (ARS), a hierarchical agent framework for bimanual robotic manipulation. It also tests 18 RoboDojo tasks (including generalization, memory, precision, long-horizon, and open-task) with GPT-6 as the backbone. However, the novelty and in-depth thinking and consideration of how the agentic robot is missing.

Critical strengths

1) Test how the latest LLM models (GPT -6 and Opus 5) can help in an agentic robot design. 2) Extensive tests (including on the real robot) is very solid and time-consuming. As a pioneer in testing SOTA coding LLMs, it is valuable.

Critical weaknesses
  1. The overall nights why and how the robot-agent architecture is very superficial, e.g., how to consider and build the robot harness from first principles (especially when it relates to the closed-source API calling). See this https://research.nvidia.com/labs/srl/projects/astranomy/ to consider the organization and in-depth thinking

  2. Can not find the GitHub link for this robot-agent architecture

  3. It is unclear why it is only bimanual manipulation. There should be a few works that comprehensively test the GPT -6 Astra's performance in locomotion, dexterous hands, whole-body control, etc.

  4. The tested model is very limited, more like a preliminary technical report but not a research paper.

Questions for authors
  1. Hierarchical planning, local closed-loop control, active perception, and reusable skills have been explored in previous robotic agent systems. Please clarify the exact insights and contributions in this work
  2. The reusable skills (behaviors) is very likely a language-instructed summarization of the skills, which in practice is impractical. Examples in Figure 5, what if the rotation is only 45 degrees. Also, if the skills are summarized in this way, how to maintain such a large skill space. It is neither adaptive nor robust.
  3. As frontier research, only testing with the closed-loop API LLM model does not impact the whole community. For example, tons of questions can not be answered; e.g., if the task fails, it is because of a reasoning error (LLM text reasoning) or a perception error (like vision encoder damage). Without understanding those questions, it is hard to convince people to borrow your agentic harness work.
Reviewer_sAe6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ARS, a framework for bimanual robot manipulation that combines task-level planning, local visual correction, active observation, and reusable skills. It reports improved performance over its implemented baseline on 18 tasks and provides physical-robot demonstrations.

Critical strengths
  • The difference between task-level planning and local correction is clear and makes sense in practice.
  • The results are promising, and the examples show how extra camera views can help.
Critical weaknesses
  • The main comparison looks at complete systems, so it is hard to tell whether the gains come from adaptive delegation, active observation, or other implementation differences. The mentioned ablation results are not clearly shown.
  • Key evaluation details, such as the number of trials and the results for each task, are missing. This makes it hard to judge how reliable the reported gains are.
  • The physical demonstrations are promising, but without repeated trials and results for complete tasks, they offer limited evidence that the approach works in new settings.
Questions for authors
  1. Could you provide controlled comparisons showing the separate benefits of adaptive delegation and active observation?
  2. Could you report trial counts, per-task outcomes, and clarify the GPT-6 versus Opus 5 configurations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.