acceptodds
Under review as a conference paper at ICLR 2027

RoboARCade: Benchmarking Agentic Robot Control

Abstract

Improvements in multi-modal capabilities have enabled general agents to act directly as robot policies, a setting we refer to as agentic robot control (ARC). We introduce RoboARCade, a benchmark of long-horizon manipulation tasks across multiple embodiments. It is designed to test in-context adaptation, memory, and reasoning for precise control, extending beyond the capabilities emphasized by prior robotic policy benchmarks. The benchmark differentiates leading agents: GPT-6 Astra achieves 49.2% compared with 41.5% for Claude Fable 5.1, leaving substantial room for improvement. We equip agents with different interfaces to study their effects on performance and behavior. Our benchmark identifies several bottlenecks that prevent agents from zero-shot completion of robotic tasks: identifying an approach, executing it precisely, and recognizing subtask completion. Additionally, we find that performance can be improved without additional human input through retries, successful demonstrations, and prompt optimization.

Primary area:
datasets and benchmarks
Keywords:
Agentic Robotics, Autonomous Improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Er47OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RoboARCade, a benchmark for Agentic Robot Control (ARC) comprising 22 long-horizon manipulation tasks involving single-arm and bimanual manipulation, assembly, physical puzzles, and laboratory-inspired procedures. The benchmark integrates MuJoCo simulation, high-fidelity Blender rendering, and unified evaluation protocols, supporting both VIA and multi-view RGBD interfaces. The authors evaluate several frontier multimodal agents and investigate their limitations in visual understanding, action planning, precise execution, and task completion verification. Furthermore, the paper studies autonomous improvement through in-context learning and feedback-based prompt optimization without updating model parameters. GPT-6 Astra achieves a 49.2% subtask completion score with VIA, while the autonomous improvement experiments increase the mean rubric score from 54.0% to 72.7%.

Critical strengths
  1. A practical and unified benchmark for agentic robot control. RoboARCade integrates 22 diverse manipulation tasks with multiple robot configurations and standardized evaluation interfaces. Beyond conventional short-horizon manipulation benchmarks, it offers a useful setting for evaluating long-horizon planning, closed-loop control, and the use of execution feedback.

  2. Insightful and systematic analysis of agent failures. The paper goes beyond aggregate performance by investigating visual understanding, action planning, execution precision, retry-based recovery, and incorrect success claims. These analyses provide valuable empirical evidence about the current limitations of multimodal agents in robotic control.

Critical weaknesses
  1. The primary evaluation metric does not fully capture long-horizon task completion. The Success (%) metric in Table 1 measures the proportion of completed rubric subtasks rather than full-episode success. The paper does not provide complete task-level success rates and per-task statistics across all 22 tasks, making the overall performance difficult to interpret. Additionally, the comparison of trajectory lengths across benchmarks is based on simulation steps without sufficiently accounting for differences in control frequency and action granularity, weakening the claim regarding long-horizon complexity.

  2. Limited coverage and incomplete cost analysis of autonomous improvement. The main autonomous improvement evaluation focuses on only five tasks selected to have at least one successful rollout, which limits the representativeness of the results. Furthermore, T2 uses multiple rounds of graded rollout feedback, while the reported costs mainly reflect individual evaluation trials rather than the cumulative cost of obtaining demonstrations and optimizing prompts. The lack of cross-task generalization experiments also makes it difficult to establish the broader effectiveness of the proposed autonomous improvement setting.

  3. Limited Novelty and Insufficient Experimental Validation. The paper primarily integrates existing robotic manipulation tasks into a unified benchmark, with limited methodological novelty beyond the evaluation framework. Moreover, the experiments focus on a narrow set of proprietary agent models from only two model families, without evaluating open-source alternatives or a broader range of agent architectures. This limited model coverage makes it difficult to establish whether RoboARCade can comprehensively and reliably assess the general capabilities of agentic robot control systems. Additional evaluations across more diverse models and control paradigms would strengthen the benchmark's validity and significance.

Questions for authors
  1. Limited Model Coverage. The evaluation only includes proprietary models from two model families. Could the authors extend the experiments to additional open-source and closed-source models to demonstrate that RoboARCade can reliably differentiate the capabilities of diverse agentic robot control systems?

  2. Benchmark Novelty and Evaluation Scope. Most tasks are adapted from existing robotic benchmarks. What unique capabilities does RoboARCade evaluate beyond existing benchmarks? In particular, how can the authors demonstrate that it effectively assesses agentic capabilities such as reasoning, planning, and memory, rather than primarily measuring action execution and manipulation precision?

  3. Evaluation Metrics. The reported Success (%) mainly reflects subtask completion rather than full-episode success. Could the authors provide complete task success rates and per-task results across all 22 tasks to better demonstrate the benchmark's effectiveness in evaluating long-horizon manipulation?

  4. Autonomous Improvement and Generalization. The autonomous improvement experiments are limited to five selected tasks. Could the authors extend these evaluations to more diverse tasks, particularly those without successful demonstrations, and report the total computational and inference costs of the multi-round optimization process?

4 comments
AuthorsOpenReview ↗

[1/n] Reviewer Er47

Thank you so much for your feedback and thorough questions. We address all four questions and additional points below.

Q1: Limited Model Coverage. The evaluation only includes proprietary models from two model families. Could the authors extend the experiments to additional open-source and closed-source models to demonstrate that RoboARCade can reliably differentiate the capabilities of diverse agentic robot control systems?

Moreover, the experiments focus on a narrow set of proprietary agent models from only two model families, without evaluating open-source alternatives or a broader range of agent architectures. This limited model coverage makes it difficult to establish whether RoboARCade can comprehensively and reliably assess the general capabilities of agentic robot control systems. Additional evaluations across more diverse models and control paradigms would strengthen the benchmark's validity and significance.

Absolutely. We expand our models in Table 1 to include another frontier closed-source model, Gemini 4 Argon, and a frontier open-source model, Deepseek V4.1 Flash 1017. We also evaluate Gemini 4 Argon at high and medium reasoning levels (Appendix Table 12 and Table 13). High reasoning achieves 46.6% subtask success and a 34.5% full-episode solve rate, compared with 37.9% and 27.3% for medium reasoning. These differences show that the benchmark distinguishes performance across reasoning levels within the same model. Note that the numbers from the original manuscript are very slightly different as we discovered some errors in a few rollouts and reran them.

Main Paper Table 1:

ModelInterfaceSubtask (%)Full-episode solve (%)Cost ($)Turns/EEFCalls/EEF
GPT-5.6-SolVIA14.08.225.053.63.3
Claude Opus 5VIA30.320.960.004.95.5
Claude Fable 5.1Multi-view RGBD36.826.433.161.01.7
Claude Fable 5.1VIA43.634.555.502.27.0
GPT-6-AstraMulti-view RGBD48.337.313.891.31.3
GPT-6-AstraVIA49.836.431.842.33.0
DeepSeek V4.1 Flash 1017Multi-view RGBD13.18.20.651.81.9
DeepSeek V4.1 Flash 1017VIA2.51.90.293.44.1
Gemini 4 ArgonMulti-view RGBD51.440.98.261.92.6
Gemini 4 ArgonVIA46.634.512.685.45.4
AuthorsOpenReview ↗

[2/n] Reviewer Er47

Q2: Benchmark Novelty and Evaluation Scope. Most tasks are adapted from existing robotic benchmarks. What unique capabilities does RoboARCade evaluate beyond existing benchmarks? In particular, how can the authors demonstrate that it effectively assesses agentic capabilities such as reasoning, planning, and memory, rather than primarily measuring action execution and manipulation precision?

For all 22 tasks, we had to introduce large task-specific initial-position ranges because these benchmarks were designed for VLA evaluation with narrow randomization distributions. Since we made these ranges large, they had to be tested to ensure the task was feasible with the agents. In addition, we substantially modified the visual and collision meshes and tuned physics parameters to address issues in the existing assets and scenes. The contributions therefore extend beyond asset reuse to task construction, scene configuration, and physics tuning. Below, we breakdown all 22 tasks. The Appendix Table 3 has citations to all named benchmarks included below and below are just some of the differences for all 22 tasks:

  • Beam placement, Cooling manifold, Gamepad, Circular stool: The task is original. The assets are from Fabrica. Visual meshes were fine. Collision meshes and friction had to be tuned to all for feasible insertion under sub-mm tolerances.
  • Square nut assembly: The task is not original. The assets are from Robomimic. The visual and collision meshes had issues.
  • Shell game: Completely original
  • Lights Out, 6×7: The task is not original. Expand the original 4×6 board to 6×7 for much more difficult task. The assets are not unique.
  • Lock coins in strongbox: Completely original
  • Pack tools into toolboxes: Completely original besides tool assets from RoboDojo where visual and collision meshes had to be tuned.
  • Organize table and plug in charger: The task is original. The charger and pen assets from RoboDojo. We modified the charger’s visual and collision meshes and tuned friction and restitution for the assets.
  • Sort LEGO bricks: The task is original. The assets are from Robodojo.
  • Set Up PCR Run: The task is original. The asset is from AutoBio. The plate friction and meshes were tuned.
  • Start incubation: The task is original. The asset is from AutoBio but buttons were added. The test tube visual and collision meshes were tuned.
  • Vortex sample: The task is original. The asset is from AutoBio but switches were added and it was made articulated. The test tubes visual and collision meshes were tuned.
  • Serve warm drinks: The task is original. The assets are from LIBERO but heavily tuned meshes and physics parameters. A button was added to the microwave as part of the task.
  • Tidy counter: The task is not original. The assets are from LIBERO but heavily tuned meshes and physics parameters.
  • Start making breakfast: The task is original. The assets are from LIBERO but heavily tuned meshes and physics parameters. A toaster asset with bread was added.
  • Handover dominos: The task is not original from Pix2Dex.
  • Cup pyramid: The task is not original. The assets are from Pix2Dex but heavily tuned. The cup friction had to be tuned and the meshes aligned.
  • Jenga tower: The task is not original from Pix2Dex.
  • Hang mugs and rack plates: The task is not original. The assets are from Pix2Dex but heavily tuned. Center of mass of mug stand, plate thickness, mug handle friction.
  • Bottles in bin: The task is not original. The assets are from Pix2Dex but heavily tuned. The bottles friction and restitution were heavily tuned.
AuthorsOpenReview ↗

[3/n] Reviewer Er47

Q3: Evaluation Metrics. The reported Success (%) mainly reflects subtask completion rather than full-episode success. Could the authors provide complete task success rates and per-task results across all 22 tasks to better demonstrate the benchmark's effectiveness in evaluating long-horizon manipulation?

Absolutely. We include in Table 1 above the full-episode solve percentage. In Appendix Table 12, we include per-task results across all 22 tasks.

Appendix Table 12: Each cell reports subtask success (%) / full-episode solve (%), with five seeds per task.

TaskGPT-5.6-Sol (VIA)Claude Opus 5 (VIA)Claude Fable 5.1 (VIA)Claude Fable 5.1 (Multi-view RGBD)GPT-6-Astra (VIA)GPT-6-Astra (Multi-view RGBD)Gemini 4 Argon Medium (VIA)Gemini 4 Argon High (VIA)Gemini 4 Argon High (Multi-view RGBD)DeepSeek V4.1 Flash 1017 Xhigh (Multi-view RGBD)DeepSeek V4.1 Flash 1017 Xhigh (VIA)
Beam placement0.0 / 00.0 / 060.0 / 6020.0 / 2040.0 / 4080.0 / 8040.0 / 4040.0 / 4060.0 / 600.0 / 00.0 / 0
Puzzle 6×7100.0 / 100100.0 / 100100.0 / 100100.0 / 100100.0 / 100100.0 / 100100.0 / 100100.0 / 100100.0 / 100100.0 / 10040.0 / 40
Shell game93.8 / 8098.5 / 8080.0 / 8086.2 / 80100.0 / 10087.7 / 8081.5 / 80100.0 / 100100.0 / 10026.2 / 200.0 / 0
Sort LEGO bricks46.7 / 088.9 / 40100.0 / 10095.6 / 8095.6 / 6097.8 / 80100.0 / 10097.8 / 80100.0 / 10013.3 / 00.0 / 0
Tidy counter6.7 / 026.7 / 0100.0 / 10066.7 / 6060.0 / 6020.0 / 040.0 / 2053.3 / 4073.3 / 606.7 / 00.0 / 0
Square0.0 / 0100.0 / 10060.0 / 60100.0 / 100100.0 / 100100.0 / 100100.0 / 100100.0 / 100100.0 / 10020.0 / 200.0 / 0
Bottles in Bin5.0 / 075.0 / 60100.0 / 10065.0 / 4095.0 / 8095.0 / 8080.0 / 6095.0 / 8090.0 / 805.0 / 00.0 / 0
Handover Dominos13.3 / 030.0 / 063.3 / 2053.3 / 2070.0 / 2080.0 / 6026.7 / 056.7 / 2066.7 / 406.7 / 00.0 / 0
Cup Pyramid6.7 / 026.7 / 046.7 / 2046.7 / 046.7 / 4026.7 / 2026.7 / 040.0 / 2066.7 / 400.0 / 00.0 / 0
Set Up PCR Run0.0 / 00.0 / 020.0 / 2015.0 / 055.0 / 4040.0 / 4055.0 / 4015.0 / 080.0 / 8020.0 / 00.0 / 0
Jenga Tower2.2 / 06.7 / 020.0 / 08.9 / 024.4 / 017.8 / 020.0 / 08.9 / 013.3 / 00.0 / 00.0 / 0
Cooling manifold0.0 / 00.0 / 00.0 / 00.0 / 020.0 / 200.0 / 00.0 / 020.0 / 2020.0 / 200.0 / 00.0 / 0
Gamepad0.0 / 020.0 / 200.0 / 00.0 / 00.0 / 020.0 / 200.0 / 020.0 / 2040.0 / 400.0 / 00.0 / 0
Hang mugs / rack plates0.0 / 010.0 / 025.0 / 05.0 / 010.0 / 010.0 / 020.0 / 025.0 / 025.0 / 05.0 / 00.0 / 0
Organize table / charger0.0 / 00.0 / 020.0 / 023.3 / 056.7 / 046.7 / 06.7 / 026.7 / 010.0 / 03.3 / 00.0 / 0
Pack tools0.0 / 00.0 / 00.0 / 02.5 / 02.5 / 05.0 / 05.0 / 00.0 / 015.0 / 00.0 / 00.0 / 0
Start incubation4.0 / 00.0 / 08.0 / 024.0 / 016.0 / 048.0 / 400.0 / 00.0 / 08.0 / 08.0 / 04.0 / 0
Start making breakfast30.0 / 080.0 / 60100.0 / 10065.0 / 60100.0 / 10080.0 / 6040.0 / 080.0 / 6085.0 / 6075.0 / 400.0 / 0
Circular stool0.0 / 00.0 / 00.0 / 020.0 / 2040.0 / 4040.0 / 4060.0 / 6080.0 / 8020.0 / 200.0 / 00.0 / 0
Vortex sample0.0 / 04.0 / 020.0 / 012.0 / 040.0 / 036.0 / 024.0 / 020.0 / 028.0 / 00.0 / 08.0 / 0
Serve warm drinks0.0 / 00.0 / 036.0 / 00.0 / 024.0 / 032.0 / 208.0 / 044.0 / 024.0 / 00.0 / 00.0 / 0
Lock coins in strongbox0.0 / 00.0 / 00.0 / 00.0 / 00.0 / 00.0 / 00.0 / 02.5 / 05.0 / 00.0 / 00.0 / 0
Overall14.0 / 8.230.3 / 20.943.6 / 34.536.8 / 26.449.8 / 36.448.3 / 37.337.9 / 27.346.6 / 34.551.4 / 40.913.1 / 8.22.5 / 1.9
AuthorsOpenReview ↗

[4/n] Reviewer Er47

Q4: Autonomous Improvement and Generalization. The autonomous improvement experiments are limited to five selected tasks. Could the authors extend these evaluations to more diverse tasks, particularly those without successful demonstrations, and report the total computational and inference costs of the multi-round optimization process?

Absolutely. We extend the task-specific autonomous improvement experiments for tasks without successful demonstrations to a set of 5 tasks from just 1. Table 2 has the results and also includes the one-time computational cost of the optimization process. The inference cost of the optimization process for each individual episode is already included in the per-episode cost as the optimization incorporates a prompt that is prepended when the agent starts the task. We see that multiple rounds of optimization provide increased subtask success and full-episode success, keeping with the earlier trends. We see the optimization costs are extremely high compared to the per-episode inference cost, however, that cost is a one-time cost.

Main paper Table 2:

MethodSubtask (%)Full-episode solve (%)Per-episode cost ($)Optimization cost ($)Turns/EEFCalls/EEF
GPT-6-Astra baseline53.538.027.60—2.53.4
In-context success demo (ICL)73.064.030.55—3.14.2
Text prompt, round 1 (T1)62.552.030.7657.413.03.3
Text prompt, round 2 (T2)72.064.022.75914.272.33.2
No successful baseline episode
GPT-6-Astra baseline25.30.042.00—2.43.1
Text prompt, round 1 (T1)40.016.036.18539.931.83.2
Text prompt, round 2 (T2)53.320.035.401,149.602.13.1

Here we address all of the Critical Weaknesses:

The paper does not provide complete task-level success rates and per-task statistics across all 22 tasks, making the overall performance difficult to interpret.

We included in Table 1 the complete task-level success rates and in Appendix Table 12 the per-task statistics.

Additionally, the comparison of trajectory lengths across benchmarks is based on simulation steps without sufficiently accounting for differences in control frequency and action granularity, weakening the claim regarding long-horizon complexity.

We agree that raw step counts alone do not establish long-horizon. We have therefore revised the main paper to report approximate simulated time durations to incorporate control frequency alongside trajectory lengths, using duration = number of steps / control frequency. Across our 22 tasks, rollouts average 3,311 control steps, corresponding to approximately 166 s at 20 Hz. For comparison, we use average trajectories and control frequencies from that benchmark. For example, trajectories contain 276 steps for LIBERO-Long [1] (14 s at 20 Hz), 372 for RoboCasa [2] (19 s at 20 Hz), 502 for VLABench [3] (approximately 50 s using its nominal 10 Hz), and 2,972 for RoboCerebra [4] (149 s at 20 Hz). RoboDojo’s long-horizon demonstrations average 630 steps [5] (25 s at 25 Hz). These conversions account for different control frequencies of different benchmarks. We note the closest is the RoboCerebra benchmark [4], however, it primarily tests the single Franka Panda embodiment. Our tasks test the single Franka Panda, bimanual Panda and bimanual YAM embodiments.

Furthermore, T2 uses multiple rounds of graded rollout feedback, while the reported costs mainly reflect individual evaluation trials rather than the cumulative cost of obtaining demonstrations and optimizing prompts.

We now provide the per-episode individual evaluation cost and the optimization cost for T1 and T2. As expected, the optimization cost for T2 is high but it is a one-time cost.

The lack of cross-task generalization experiments also makes it difficult to establish the broader effectiveness of the proposed autonomous improvement setting.

We agree with the reviewer and made changes to the manuscript to limit the scope of the autonomous improvement section to task-specific autonomous improvement.

[1] Liu et al. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. 2023.

[2] Nasiriany et al. RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots. 2024.

[3] Zhang et al. VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks. 2025.

[4] Han et al. RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation. 2025.

[5] Chen et al. RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies. 2026.

Reviewer_Ck8XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new benchmark for agentic robot control, evaluating foundation models (FM) as closed-loop policies on simulated robotics tasks. The paper constructed a benchmark of 22 tasks, which are claimed to be long-horizon and require visual understanding, precise control, memory, and multi-step planning. The authors evaluate frontier models' performance on their benchmark and analyze the failure modes of these models. Additionally, the authors investigate the in-context learning as well as self-improvement capabilities of these models.

Overall, I find this benchmark to be a valuable addition to the robot learning community, for its complexity of tasks as well as its ready-to-use infrasturcture for foundation models.

Critical strengths

This paper proposes a useful and challenging benchmark for evaluating foundation models as closed-loop policies. The authors explicitly distinguish this benchmark from existing robotic benchmarks and highlight its focus on planning and reasoning.

The analysis from the experiments shed light on the current capabilities of the frontier models as well as their limitations. The authors conduct experiments on the failure modes of the models, the types of tools that are given to the models, as well as the self-improvement capabilities of the models.

Critical weaknesses

The experiment scope is quite limited. Only a few closed-source models are used. There are many open-source models that can be evaluated on this benchmark but are skipped. Only 5 seeds (trials) are used per task. There is no report of statistical significance of the results.

The failure analysis lacks details to validate the authors' claim. The authors classifie failures using geometric thresholds followed by verification from either a VLM or a human annotator. However, this process is prone to error. No additional details are provided for how the verification is done to ensure that the failure labels are correct.

Questions for authors

What is a retry, and how is it identified? Does a retry refer to another attempt at the same subtask within a continuous rollout without resetting the environment?

The pre-compaction prompt provides the scene state to the agent. When is this being used? And is this considered privileged information that should not be accessed by the agent?

What are the challenges of incorporating robot hands and dexterous manipulation tasks into this benchmark?

4 comments
AuthorsOpenReview ↗

[1/n] Reviewer Ck8X

Thank you for your questions Reviewer Ck8X. We address the concerns and all three questions below.

Q1. What is a retry, and how is it identified? Does a retry refer to another attempt at the same subtask within a continuous rollout without resetting the environment?

Yes: a retry is another attempt at the same subtask within a continuous rollout, without reset. Displaced or dropped objects persist and simulation steps and wall clock time continue increasing. We determine retries with simulation state then check with human verification with a custom visualization tool that will be open-source.

Q2: The pre-compaction prompt provides the scene state to the agent. When is this being used? And is this considered privileged information that should not be accessed by the agent?

Compaction is a common practice with agents that run for multiple hours as their context windows are limited. Before compacting 245,000 tokens from the full transcript, a pre-compaction prompt is a prompt that is added when the agent needs to compact and it indicates what the agent should focus on. For example, the agent should focus on the state of the observed scene, progress, failures, and calibration.

By scene state the prompt does not refer to simulator state. The scene state in the prompt refers to the estimated state from the transcript of image observations and tool calls the agent has done in the session. This information is not privileged information as it is from the agent's inputs and outputs during the episode so far.

AuthorsOpenReview ↗

[2/n] Reviewer Ck8X

Q3: What are the challenges of incorporating robot hands and dexterous manipulation tasks into this benchmark?

Although current embodiments use two-finger parallel-jaw grippers, there are no challenges in incorporating robot hands into this benchmark. With a single simulation backend, given a file of the bimanual arm + hand embodiment, it can be used to test all of the existing tasks. Tasks can also be easily added with agentic real-to-sim scenes to simulate dexterous manipulation such as hammering a nail or using a screwdriver.

Here we address all of the Critical Weaknesses:

The experiment scope is quite limited. Only a few closed-source models are used. There are many open-source models that can be evaluated on this benchmark but are skipped.

We expand our models in Table 1 to include another frontier closed-source model, Gemini 4 Argon, and a frontier open-source model, Deepseek V4.1 Flash 1017. We also evaluate Gemini 4 Argon at high and medium reasoning levels (Appendix Table 13). High reasoning achieves 46.6% subtask success and a 34.5% full-episode solve rate, compared with 37.9% and 27.3% for medium reasoning. These differences show that the benchmark distinguishes performance across reasoning levels within the same model. Note that the numbers from the original manuscript are very slightly different as we discovered some errors in a few rollouts and reran them.

Main Paper Table 1:

ModelInterfaceSubtask (%)Full-episode solve (%)Cost ($)Turns/EEFCalls/EEF
GPT-5.6-SolVIA14.08.225.053.63.3
Claude Opus 5VIA30.320.960.004.95.5
Claude Fable 5.1Multi-view RGBD36.826.433.161.01.7
Claude Fable 5.1VIA43.634.555.502.27.0
GPT-6-AstraMulti-view RGBD48.337.313.891.31.3
GPT-6-AstraVIA49.836.431.842.33.0
DeepSeek V4.1 Flash 1017Multi-view RGBD13.18.20.651.81.9
DeepSeek V4.1 Flash 1017VIA2.51.90.293.44.1
Gemini 4 ArgonMulti-view RGBD51.440.98.261.92.6
Gemini 4 ArgonVIA46.634.512.685.45.4

Only 5 seeds (trials) are used per task. There is no report of statistical significance of the results.

We now report 95% confidence intervals in the Appendix Table 7.

The failure analysis lacks details to validate the authors' claim. The authors classifie failures using geometric thresholds followed by verification from either a VLM or a human annotator. However, this process is prone to error. No additional details are provided for how the verification is done to ensure that the failure labels are correct.

We now provide full details in Appendix Section A.9. The annotation protocol covers every attempted but unsuccessful subtask across all rollouts. We classify these subtasks according to where they fail in the sequence of (1) identifying the correct object (visual understanding), (2) manipulating the object to the correct pose (approach), and (3) precisely completing the subtask (precision). Two decision boundaries separate these stages. The first uses simulator state to determine whether the robot manipulated the intended object: manipulating the wrong object is labeled as a visual understanding failure; otherwise, the attempt is evaluated at the approach stage. The second uses a diagnostic geometric boundary T, specified by a human annotator for each subtask, to distinguish approach from precision failures: attempts outside T are labeled as approach failures, while those within T that remain unsuccessful are labeled as precision failures. A script assigns preliminary labels from recorded simulator states and actions. A human annotator then reviews synchronized rollouts in a custom annotation tool, checks the manipulated object and the attempt’s approach, and corrects subtask labels as needed. These deliberately narrow definitions make the labeling criteria explicit and reduce ambiguity for human verification.

I thank the authors for addressing my concerns. My review remains positive.

Thank you so much for the feedback and reply!

Reviewer_7a6NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RoboARCade, a benchmark for agentic robot control in which foundation-model agents directly observe simulated scenes and issue robot controller targets. It comprises 22 manipulation tasks spanning assembly, physical puzzles, laboratory-inspired procedures, household activities, and single- and dual-arm embodiments. The infrastructure combines randomized MuJoCo scenes, Blender-rendered observations, sandboxed agent access, and simulator-based subtask grading.

The authors evaluate four agents and compare a multi-view RGBD interface with the Visual Interface Agent (VIA) interface for two of them. GPT-6 Astra achieves the highest mean subtask rubric score, reaching 49.2% with VIA and 47.3% with RGBD. The paper additionally analyzes retries, failure categories, and incorrect completion claims. On five selected tasks, successful demonstrations and iterative task-specific prompt optimization improve Astra’s mean rubric score from 54.0% to approximately 72%.

Critical strengths
  1. A useful benchmark setting with diverse tasks and explicit isolation of privileged information. RoboARCade evaluates direct, closed-loop control by general agents rather than requiring a separately trained robot policy. The 22 tasks cover multiple embodiments and combine manipulation with planning, memory, and precision requirements. The sandbox explicitly prevents access to object poses, task assets, success predicates, and grading results during execution. Together with shared simulation and controllers across interfaces, this provides a useful foundation for studying agentic robot control. The contribution should be judged primarily by the quality of this evaluation resource rather than by whether it introduces a new learning algorithm.

  2. Analysis beyond aggregate scores. The paper examines recovery after failed attempts, incorrect completion claims, interaction overhead, and adaptation from previous rollouts. These analyses are practically relevant: an agent’s ability to recognize failure and verify completion matters substantially for long-horizon control. Reporting both interfaces’ monetary costs also reveals a useful trade-off: VIA improves the reported rubric scores but incurs considerably higher inference expenditure. The successful-demonstration and text-prompt experiments provide an additional perspective on improving control without updating model weights.

Critical weaknesses
  1. The benchmark’s measurement and difficulty calibration are insufficiently specified and validated. The principal metric is the proportion of completed rubric subtasks, not the fraction of fully solved episodes. Partial credit is appropriate, but the paper does not fully specify how subtasks are weighted, how ordering and dependencies are handled, or whether earlier achievements must remain satisfied at termination. Different decompositions can change aggregate scores without changing actual task competence. Full-episode success rates and per-task zero-shot results are also needed to interpret the headline numbers. Furthermore, the benchmark combines tight success tolerances with interfaces whose information and control limitations require calibration. Appendix A.2.6 quantizes depth over 0.05–3.0 m into 255 valid levels, corresponding to approximately 11.6 mm between levels, while Section 4.2 gives an example requiring 0.6 mm lateral placement accuracy. This does not establish that the task is impossible—RGB feedback and iterative correction may support finer positioning—but it makes a reference-controller or expert feasibility evaluation important. Without such calibration and complete task specifications, it is difficult to determine how much measured difficulty reflects agent capability versus the observation/control pipeline.

  2. The diagnostic and improvement claims need stronger controls and statistical support. The failure analysis relies on a VLM or human annotator to distinguish visual understanding, approach, and precision failures, but the manuscript does not report a sufficiently detailed annotation protocol, annotator identities or models, agreement, or validation against an independently labeled subset. Spatial perception errors can manifest as placement errors, so the conclusion that visual understanding is rarely the bottleneck depends materially on how these categories are operationalized. The interface comparison also changes several factors simultaneously, including spatial representation, target preview, interaction structure, and default observation resolution. It supports a comparison of complete interfaces, but does not isolate why their performance differs. With five seeds per task and no uncertainty estimates, the significance of Astra’s 1.9-point interface gap is unclear. Finally, autonomous improvement is evaluated on five tasks selected for having a successful baseline episode, plus a separate five-trial Gamepad experiment. The reported gains are encouraging, but the demonstration/optimization/evaluation seed separation and total adaptation costs need explicit reporting. These results currently support task-specific adaptation more strongly than a broadly validated autonomous-improvement capability.

Questions for authors
  1. Could you fully specify the scoring protocol and report complete-task performance?
  2. How was task feasibility validated under the actual observation and control interfaces?
  3. How reliable is the failure taxonomy?
  4. What exactly was held out during autonomous improvement?
  5. Could you clarify the reproducible execution protocol?
5 comments
AuthorsOpenReview ↗

[1/n] Reviewer 7a6N

We greatly appreciate Reviewer 7a6N for their questions and clarifications to better our manuscript. They describe the paper as “clear and easy to follow.” They highlight “a useful benchmark setting with diverse tasks and explicit isolation of privileged information" with tasks that “combine manipulation with planning, memory, and precision requirements.”

Q1: Could you fully specify the scoring protocol and report complete-task performance?

Partial credit is appropriate, but the paper does not fully specify how subtasks are weighted, how ordering and dependencies are handled, or whether earlier achievements must remain satisfied at termination.

We agree that more details are needed and all additions to the manuscript are in red. In Appendix Section A.8, we added the full scoring protocol and the exact subtasks and rubrics for all tasks in the Appendix Table 11. Because rubric decomposition is a design choice, we also report full task success in Table 1.

To clarify, subtasks are equally weighted within each task. Full success requires completing all subtasks, maintaining the task’s invariants throughout the episode, and satisfying a 0.5-second hold, which cannot be credited on timeout. Invariants make the tasks more realistic, and if one is violated, partial credit freezes at the subtask where the violation occurred. For example, in Start Incubation, the test tube cannot tilt more than 15° from upright at any point because its contents could spill. Scores are computed from saved simulator states outside the agent-accessible environment, and the agent indicates when the episode is ready to be graded. In the table below, comma-separated subtasks can be completed in any order (e.g., storing the headphones and inserting the charger in Organize Table), while each arrow (→) indicates a required sequence (e.g., in Set Up PCR Run, the robot must close the lid before latching the green lever). We show a condensed version below, all tasks are included in the manuscript.

Example rows from Appendix Table 11:

TaskNo. of subtasksSubtask orderInvariants
Beam placement1Seat and release part.—
Puzzle 6×71Turn all cells red.—
Set Up PCR Run4Seat PCR plate → Close machine lid → Latch green lever → Tighten force knob.Plate tilt ≤30° (rubric); 15° (full sequence).
Organize table / charger6Store headphones, insert charger, store 1–4 pens.—

Full-episode success rates and per-task zero-shot results are also needed to interpret the headline numbers.

We also report full-episode task success rates in Table 1, along with per-task subtask completion in Appendix Table 12. Note we have expanded our model evaluations as Reviewer Er47 requested. Note that the numbers from the original manuscript are very slightly different as we discovered some errors in a few rollouts and reran them.

Main Paper Table 1:

ModelInterfaceSubtask (%)Full-episode solve (%)Cost ($)Turns/EEFCalls/EEF
GPT-5.6-SolVIA14.08.225.053.63.3
Claude Opus 5VIA30.320.960.004.95.5
Claude Fable 5.1Multi-view RGBD36.826.433.161.01.7
Claude Fable 5.1VIA43.634.555.502.27.0
GPT-6-AstraMulti-view RGBD48.337.313.891.31.3
GPT-6-AstraVIA49.836.431.842.33.0
DeepSeek V4.1 Flash 1017Multi-view RGBD13.18.20.651.81.9
DeepSeek V4.1 Flash 1017VIA2.51.90.293.44.1
Gemini 4 ArgonMulti-view RGBD51.440.98.261.92.6
Gemini 4 ArgonVIA46.634.512.685.45.4
AuthorsOpenReview ↗

[2/n] Reviewer 7a6N

Q2: How was task feasibility validated under the actual observation and control interfaces?

Without such calibration and complete task specifications, it is difficult to determine how much measured difficulty reflects agent capability versus the observation/control pipeline.

For every task and seed, a human verified that the task could be completed using the same observation and control interface as the agent (either VIA or Multi-view RGBD). The human made iterative improvements to the tasks to ensure feasibility. As a secondary check, we used agents to generate scripted motions, which we will release with the code.

Appendix A.2.6 quantizes depth over 0.05–3.0 m into 255 valid levels, corresponding to approximately 11.6 mm between levels, while Section 4.2 gives an example requiring 0.6 mm lateral placement accuracy.

The reviewer’s example of approximately 11.6 mm between control levels and a required lateral placement accuracy of 0.6 mm refers to the Beam placement task. Humans verified that the task was solvable for these initializations, and the per-task success rates in Appendix Table 12 show that Astra with Multi-view RGBD, Astra with VIA, and Fable each fully solved the task in at least one episode. To address the reviewer’s concern about distinguishing agent capability from limitations of the observation and control pipeline, we verified feasibility under the same interfaces used for evaluation and incorporated tool calls in both interfaces that allow for multiple high-resolution viewpoints to ensure occlusions and resolution were not an issue.

AuthorsOpenReview ↗

[3/n] Reviewer 7a6N

Q3: How reliable is the failure taxonomy?

The failure analysis relies on a VLM or human annotator to distinguish visual understanding, approach, and precision failures, but the manuscript does not report a sufficiently detailed annotation protocol, annotator identities or models, agreement, or validation against an independently labeled subset.

We now add Appendix Section A.9 to specify the annotation protocol. The failure taxonomy classifies every attempted but unsuccessful subtask across all rollouts according to where it fails in the sequence of (1) identifying the correct object (visual understanding), (2) manipulating the object to the correct pose (approach), and (3) precisely completing the subtask (precision). Two decision boundaries separate these stages. The first uses simulator state to determine whether the robot manipulated the intended object: manipulating the wrong object is labeled as a visual understanding failure; otherwise, the attempt is evaluated at the approach stage. The second uses the simulator and a diagnostic boundary TT that a human annotator specifies for each subtask. For pose-based subtasks, TT specifies a position tolerance and, where applicable, an orientation tolerance. Appendix Section A.9 describes each boundary type, and Appendix Table 14 lists the boundaries for all tasks.

For each rollout, we first run the annotation script on the recorded states and actions. The script assigns each attempted but unsuccessful subtask the label of its last eligible attempt. A human annotator then reviews the synchronized rollout in a custom annotation tool, verifies the script’s labels against the recording, and corrects attempt or subtask labels as needed. We will release the annotation tool with the code.

Few Rows of the Appendix Table 14. Subtask order and diagnostic boundaries. Arrows indicate dependent stages. Subtasks within a stage may be completed in any order.

TaskSubtask order and diagnostic boundariesInvariants
Beam placementSeat and release part (Pose, 25 mm and 6°).—
Tidy counterPlace upright bottle, store released bowl (Region, 20 mm each) → Close loaded drawer (Non-grasp, joint displacement > 0.02 m).—
Handover DominosStand aligned blocks, counts 1–5 (Line, 40 mm, Gap, ≤125 mm, and Standing checks) → Trigger toppling cascade (Non-grasp).—
Serve warm drinksOpen microwave door (Non-grasp, joint displacement > 0.02 rad) → Insert upright mug (Region, 20 mm) → Close microwave door (Non-grasp, joint displacement > 0.02 rad) → Press start button → Place mug on plate (Region, 20 mm).Mug upright (≤30°).

Spatial perception errors can manifest as placement errors

We use a coarse definition of visual understanding based on whether the agent manipulates the correct object given the language command for the subtask. We ensure that each manipulation can be completed using the observations and control stack available to the agent. Thus, since the agent is able to perceive the scene, if it cannot act spatially precisely that would be classified under precision. We use this failure taxonomy because finer-grained categories would require denser annotations and take longer to apply. Future work could subdivide these categories to capture more specific failure modes.

The interface comparison also changes several factors simultaneously

Since we wanted to compare agent capability, we wanted to provide a state-of-the-art interface, VIA, so they could better accomplish the tasks. The intention wasn't to ablate different design decisions of the interface, instead we selected a state-of-the-art interface to probe agent capability on difficult tasks. We also evaluated Multiview RGBD as a minimal baseline that provided agents with only raw sensor readings. The stronger agents performed well even with this minimal interface.

With five seeds per task and no uncertainty estimates, the significance of Astra’s 1.9-point interface gap is unclear.

We do not make the claim that the VIA interface is better than the minimal Multiview RGBD interface. The intention is to provide an extensible benchmark to better compare agents.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comments from the Authors

As the discussion period comes to an end, we thank all reviewers for their thorough feedback as it helped us make the manuscript better. Any last-minute responses would be very appreciated. All changes to the manuscript are in red. We would be very appreciative of the area chair if our rebuttal can be taken into account as some reviewers may not respond before the discussion ends.

Reviewer Ck8X indicated by engaging with us that they have a positive review of the paper.

We believe we addressed all points made by Reviewer 7a6N. Since Reviewer 7a6N indicates they have minor or fixable concerns about the paper’s soundness, we hope that the addressed points was able to satisfy the reviewer and fix their concerns. Specifically, we included all details about the scoring protocol and task completion with Appendix Table 11 and added full-episode solve percentage to the main paper Table 1. We also explain that for every task and seed, a human verified that the task could be completed using the same observation and control interface as the agent (either VIA or Multi-view RGBD). We also describe how the failure taxonomy is designed to be unambiguous and we now add Appendix Section A.9 and Appendix Table 14 to specify the annotation protocol. The failure taxonomy classifies every attempted but unsuccessful subtask across all rollouts according to where it fails in the sequence of (1) identifying the correct object (visual understanding), (2) manipulating the object to the correct pose (approach), and (3) precisely completing the subtask (precision). As aptly identified by the reviewer, we also limited the scope of autonomous improvement to task-specific autonomous improvement. The reviewer wanted the held out seeds for the experiment for task-specific autonomous improvement, which we added in Appendix Table 10 for the expanded task set we conducted as asked by reviewer Er47. Lastly, for reproduction, we indicated all configs, assets, and code would be open-sourced, including the full traces for all agents.

We believe we addressed all points made by Reviewer Er47. Since Reviewer Er47 also indicates they have minor or fixable concerns about the paper’s soundness, we hope that the addressed points below were able to satisfy the reviewer. The reviewer asked for additional open-source and closed-source models and we expanded Table 1 with two additional models: a frontier open-source agent, DeepSeek V4.1 1017, and another frontier agent, Gemini 4 Argon. We also tested Gemini 4 Argon with both medium and high thinking runs and show that the benchmark is able to differentiate performance with different reasoning effort levels (Appendix Table 12 and 13). We also report full-episode solve percentage as asked by the reviewer in Table 1 and per-task success rates in Appendix Table 12. Reviewer Er47 mentions that there might be limited novelty due to reusing aspects of existing benchmarks, however, we emphasized that for all 22 tasks, we had to introduce large task-specific initial-position ranges because these benchmarks were designed for VLA evaluation with narrow randomization distributions. Since we made these ranges large, they had to be tested to ensure the task was feasible with agents. In addition, we substantially modified the visual and collision meshes and tuned physics parameters to address issues in the existing assets and scenes. The contributions therefore extend beyond asset reuse to task construction, scene configuration, and physics tuning. Reusing entire tasks from existing benchmarks would not have allowed us to test frontier agent performance as many tasks had issues and were designed for VLAs. In our response to Reviewer Er47, we note for every task what aspects were used from existing benchmarks and what was original. Lastly, the reviewer mentioned extending the task-specific autonomous evaluations to more diverse tasks, particularly those without successful demonstrations. We extended the set of tasks without successful demonstration from one to five, in addition to the already five tasks that have at least one successful demonstration, and reported per-episode and full optimization costs as requested. We see a similar trend holds were text-only optimization can better performance but has a high optimization cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.