acceptodds
Under review as a conference paper at ICLR 2027

RoboQuest: Generalist Physical Agents that Search, Inspect and Test

21 views

Abstract

Recent advances in multimodal foundation models have them capable generalist physical agents for a range of manipulation tasks. However, successful operation in an unfamiliar environment may require an agent to seek task-relevant information through interaction should it be absent in the observations. It may need to determine where a relevant object is, inspect an unobserved property, discover the effect of an unfamiliar tool, or manipulate the environment to make the required information accessible. We thus introduce a benchmark for goal-directed embodied exploration, where agents must actively acquire task-relevant information through physical interaction, use the resulting evidence to adapt subsequent actions, and autonomously decide when to commit to task completion. The benchmark centers on three forms of uncertainty—search, manipulation-based inspection, and interactive testing—while allowing viewpoint changes and preparatory manipulation to emerge as task-dependent strategies rather than prescribed procedures. Our evaluation mainly focuses on three frontier multimodal agents that operate through a common visuomotor interface and are evaluated by physical task outcomes, interaction efficiency, and explicit task submission. We additionally provide full-episode trajectories to support learning-based methods and evaluate a trained policy , allowing zero-shot evaluation of out-of-the-box frontier robotic agents and trained vision-language-action policies under the same task framework. Our benchmark aims to study not only whether a robot can execute an action, but whether it can determine what it needs to learn from the environment, physically obtain that information, and use it to complete a goal. This also allows teasing out the failure points in the trajectory of task execution.

Primary area:
datasets and benchmarks
Keywords:
VLA, Planning, Robotics, Simulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JGYYOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

RoboQuest is a simulation benchmark for goal-directed embodied exploration, comprising ten mobile manipulation tasks. The tasks are grouped into Search, Inspect, and Test according to whether the missing information concerns object locations, properties revealed through manipulation, or outcomes of physical interventions. Several tasks make information acquisition consequential, and episodes require an explicit physical submission. The paper provides 3,211 successful scripted demonstrations, approximately 405 hours of interaction, and evaluates three zero-shot multimodal agents and a fine-tuned π0.5 policy. It reports task success, partial progress, interaction costs, and failure analyses.

Critical strengths
  1. The benchmark addresses a meaningful information-acquisition problem through concrete physical tasks. Search, Inspect, and Test cover different sources of task uncertainty. Hidden cube faces, vessel labels, stamp patterns, and access dependencies connect missing information to subsequent actions. Several tasks also make exploration consequential: accessing storage changes what can be searched, inspecting vessels can spill contents, and final stamp impressions are irreversible. Together with explicit submission, these designs provide a useful setting for studying how agents acquire and use information to complete physical goals.

  2. The evaluation establishes substantial headroom and useful differentiation between agents. Overall success rates are 19.4%, 10.0%, and 2.0%; excluding Puzzle Box, they remain 10.9%, 3.3%, and 0.9%, preserving the ranking (Tab.3). The suite is not uniformly at a floor: the strongest agent succeeds on seven tasks, and task-level progress supplies graded outcome measures. The matched-chain-length Puzzle Box comparison is particularly informative: covering the mechanism leaves Astra's success approximately unchanged while increasing its decisions, but reduces Fable's success from 100% to 34.6% (Tab.20). Configurable task difficulty and successful demonstration trajectories further support the benchmark's potential as a resource for measuring future improvements.

Critical weaknesses
  1. Some intended capability requirements are not sufficiently established by task success, particularly in Test. Stamp Composition specifies unmarked housings and hidden patterns, but does not establish whether turning a stamp toward a camera could reveal its die and avoid trial impressions. Such a solution would still involve active inspection, but would not establish testing capability. For monotonic Puzzle Box chains, repeatedly commanding every bolt toward its open position could potentially succeed without interpreting outcomes or recovering dependencies. The stated trap moves may prevent this, but their coverage and the effectiveness of this alternative are not established. Low scores from current models do not establish that such routes are unavailable, limiting the precision of the capability interpretation. Fixed exhaustive search, face-by-face inspection, and individual weighing remain legitimate uses of the corresponding basic capabilities; the concern is which capabilities successful behavior actually demonstrates.

  2. Task specifications and score interpretation need further reconciliation and calibration. Painted Cubes is described as collecting one or all qualifying objects in Appendix D, whereas Appendix K and the progress definition describe sorting every cube into two bins. Unfamiliar Containers similarly shifts between a single hidden target and multiple items to collect. Most concretely, Appendix J acknowledges Odd Parcel episodes that selected exactly the correct odd set but failed because parcels remained on the balance, although the instruction does not require empty pans. These ambiguities affect the interpretation of benchmark results. Partial progress also needs an information-free reference: Tab.24 contains 201 negative and 101 positive cubes, so assigning every cube to the negative bin would attain 66.6% object-weighted progress without reading its marks. This calculation uses the diagnostic cohort, which differs from Tab.3, and is not directly comparable to its episode-averaged progress. It does not establish a high episode success rate, but shows why progress alone cannot demonstrate effective inspection.

Questions for authors
  1. Can a stamp's pattern be recovered by directly viewing its die after rotation? For Puzzle Box, can a fixed cyclic sequence of opening commands succeed without using observed outcomes, and how do traps or reversals prevent this in the evaluated configurations? Please distinguish basic physical testing from the stronger claims of adaptive experimental design or dependency inference.

  2. Please reconcile the Painted Cubes and Unfamiliar Containers descriptions with their evaluated goals and success predicates. For Blackout Search, does the released lamp continue illuminating the target, are all three camera streams available, and how do wall-switch and timed-light variants change the required behavior? These details determine which capability requirements are universal versus variant-specific.

  3. How would Odd Parcel results change when scoring follows the stated instruction without an empty-pan condition? Please report the effect on the Tab.3 cohort and distinguish it from the larger diagnostic cohort containing the acknowledged cases. For Painted Cubes, does every instance contain a positive object, and how do initial-view-only or no-inspection assignments perform under the actual episode-level metrics?

  4. For one representative near-zero-success task, is there a simple systematic policy or an existing verified execution under the agents' allowed observations, tool interface, and 200-decision budget? The successful scripts have privileged information and use native control, so they establish physical feasibility rather than a directly comparable reference. Please also report the π0.5\pi_{0.5} total training steps, evaluation sample sizes and horizons, and any available training-distribution performance, so that its near-zero results can be interpreted.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer JGYY (1/2)

Dear Reviewer JGYY,

Thank you for the detailed and constructive review. We answer each separate point under your questions, which also cover both weaknesses. We additionally evaluate three new models (Gemini 4 Argon, Claude Opus 5.5 and GPT-6.1 Sol), and some answers below use their results. Our general comment lists all changes.

Q1 & W1. Capability requirements not sufficiently established by task success, particularly in Test

  • Stamp, reading the die by rotating it. No. The die dots are enclosed inside an opaque stamp face, so the face shows no marks from any viewpoint. The pattern appears only through ink.
  • Puzzle Box, fixed cyclic sequence and traps. Yes. Trying every bolt in a fixed order, over and over, can open the box, since each pass frees at least one more bolt. The designed trap is a decoy slider that pushes the lid's lock bolt back in, so an order that pulls the decoy last never succeeds. Reversing the order on alternate passes avoids this. We consider this strategy legitimate, like exhaustive search.
  • Basic testing versus stronger claims.
    • We agree these should be separated. Success shows basic physical testing. It doesn't always show adaptive experimental design or dependency inference. We changed Appendix F, which now states that the capability marks describe what an efficient solution uses, not what success proves.
    • Our additional analysis shows that efficiency matters under the decision budget. With the mechanism hidden, Opus 5.5 drops from 94.4% to 36.8% success, as Fable 5.1 does, and its mean number of decisions rises from 94 to 176. Importantly, 12 of its 26 failed requirements are from three or more pulls on a blocked bolt without a change of plan.

Q2 & W2. Reconciliation of task specifications in Appendix E

We thank you for pointing this out. And the revision corrects three Appendix E descriptions from the original submission (table below). The evaluation itself, the instructions given to the models, and all results were unaffected. The paragraphs in the original submission described some internal earlier task versions by mistake.

TaskOutdated descriptionCorrected description (as evaluated)
Painted CubesCollect one or all qualifying objectsSort every cube: those that satisfy the rule in the blue bin, all others in the yellow bin
Unfamiliar ContainersOne target inside one box2–4 items, each in its own box, and all must go in the bowl
Blackout SearchA light aimed at the target, with wall-switch and timed-light variantsA free lantern that lights about 1 m around itself, with no switch or timer variants

Appendix E now also prints each task's goal text beside its success rule. On your Blackout questions:

  • Lamp after it is put down. Yes, it keeps lighting. The lantern lights about 1 m around wherever it stands. The robot carries it and sets it down beside whatever it works on.
  • Camera streams. All three camera streams are delivered at every decision, as in every task.
AuthorsOpenReview ↗

Author Response to Reviewer JGYY (2/2)

Q3 & W2. Score interpretation needs calibration

  • Odd Parcel without the empty-pan condition. The main table now scores Odd Parcel by the stated instruction. After this change, GPT-6 Astra goes from 0 to 5 successes, Claude Fable 5.1 from 0 to 1, and Gemini 3.8 Flash stays at 1. And our revised failure analysis uses exactly the main-table episodes.
  • Painted Cubes, positive objects and initial views. Every instance is designed to have both kinds of cube. The deciding marks are designed to be invisible from the initial 3 camera views, so no cube can be classified without manipulating it.
  • No-inspection assignments, and does progress show inspection? Blindly putting every cube in the yellow bin indeed gives 67.9% progress and 0% success, so in Painted Cubes progress alone does not show inspection. Therefore, for this task we propose a second score, Score=12(Cs−WsNs+Co−WoNo)\text{Score} = \frac{1}{2}\left(\frac{C_s - W_s}{N_s} + \frac{C_o - W_o}{N_o}\right). Here NsN_s is the number of cubes that satisfy the rule, CsC_s of them are put in the correct bin and WsW_s in the wrong bin. NoN_o CoC_o and WoW_o are the same counts for the other cubes. The score ranges from −100% to 100%. A perfect sort scores 100%, while strategies that do not inspect, such as one bin for everything or random guessing, score 0 on average. We also find this score follows the success rate more closely than progress does:
Gemini 4 ArgonGPT-6 AstraGPT-6.1 SolClaude Opus 5.5Claude Fable 5.1Gemini 3.8 FlashAll cubes in yellow
Success rate363018141420
Progress74.674.869.259.261.511.367.9
New Score65.958.754.440.437.77.40

Q4. Feasibility under the agents' interface, and π0.5 details

  • A systematic solution under the agents' interface. With the added models, Gemini 4 Argon solves 3 Search Room episodes, and Blackout Search is now the only task that no model solves. To show that the tasks are solvable through the agents' tools and budget, we converted the scripted demonstrations into tool calls, and verified in simulation that the scripts succeeded in the tasks. The revision adds this conversion to Appendix I. Like an agent, these scripts do not use the hidden information to skip exploration. They act only on what their own exploration has revealed. Search Room needs a median of 66 calls (all episodes within 200), and Blackout Search a median of 122 (96% within 200). These counts are upper bounds, not what a good agent needs. The scripts move the robot in many short steps, and the conversion keeps their path. An agent that plans its own moves can cover the same path with fewer calls.
  • π0.5 details. We fine-tune for 10,000 steps per task at batch size 32 (now in Table 6, Appendix D). The policy predicts 20-step action chunks (1.0 s at 20 Hz) and replans after each chunk, and an episode ends at SUBMIT or after 12,000 control steps (10 minutes). The revision evaluates π0.5 on all 10 tasks (5 in the original submission), on the same 50 instances per task as the frontier models (§3.4). It solves only one Puzzle Box episode (2.0% success, 43.3% progress), against 12–96% success for the frontier models on that task. On its training distribution, the policy fits the demonstrations closely: its predicted action chunks reach R² = 0.97 against always predicting the mean action, both with its own predicted subtask and with the correct one. Appendix D gives the full training-distribution details.

Thank you again for your detailed and constructive review and please let us know if you have any other questions. We are more than happy to discuss.

Thanks for the effort made by the authors for clarification about the uncertain settings and I'm quite impressed by the authors who do fix some of the main concerns I care about in the initial reviews, especifically about whether the tasks can test what the authors care, while it also fit new models into the benchmark for testing. Therefore I think the contribution of this paper can be serve as a useful benchmark to test new robotic policies capabilities. Since I hope I myself can test on it, I have change my recommendation to accept.

Some small advice for the author:

  • Since the rebuttal is still not over, I'd like to have some glance of the videos of the existing tasks (maybe in the supplementary) to check the tasks further. Some annotations that can help understanding may be even better.
  • An advice related to ICLR specific rebuttal is that, since it can change paper, you may make the important modifications of your current paper with a different color (other than the default black), so people can easily find the main modification you have made to strengthen the paper.
Reviewer_dRFQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RoboQuest, a benchmark with ten robotic tasks that test whether models can gather the information needed to complete a task by searching, inspecting objects, and trying things out, then decide when to submit their answer. The authors provide 3,211 demonstration trajectories and evaluate three frontier multimodal models and a fine-tuned π₀.₅ policy. The three multimodal models achieve average success rates ranging from 2% to 19.4%, while π₀.₅ also struggles. These results suggest that the tasks remain challenging for the evaluated models. The authors also analyze failures in information gathering, planning, and action execution. (§3–5, Table 3, Figure 3)

Critical strengths
  1. The tasks test how robots gather information through physical actions. For example, a robot needs to make test impressions to learn the patterns on the stamps, while mistakes on the final surface cannot be undone. In the search tasks, finding a token determines which compartments the robot can open next. These designs require the robot to adjust its actions based on new findings, making the benchmark useful for studying how exploration and manipulation work together. (§3.1, Appendix D)

  2. The experiments compare zero-shot models with a VLA policy under the benchmark settings. This offers an interesting insight: generalization in embodied tasks may be difficult to obtain through policy fine-tuning, while using frontier models directly could be a promising alternative.

Critical weaknesses
  1. Some scoring requirements are not communicated to the models. In Odd Parcel, a model can select the correct odd parcels and still fail because it has not cleared the balance pans. Appendix J explicitly acknowledges that the instruction does not include this requirement and reports six such cases. This counts failure to meet an unstated requirement as task failure, which affects how the results should be interpreted. It is still unclear how much these cases affect the success rates in Table 3. (Appendices G and J, pp. 20 and 22)

  2. The explanations for why models fail need more care. The paper uses indirect rules to assign failure causes. For example, a gripper that never approaches a target may be classified as insufficient information gathering, while enough placements on the balance are taken to indicate that sufficient information has been collected. However, not approaching an object does not mean the model has not seen it, and more weighing actions do not necessarily mean the comparisons were useful. The authors acknowledge these limitations in the appendix, so the conclusion that information gathering is the main bottleneck still needs more direct validation. (§5, Appendix J, Table 15)

Questions for authors
  1. How would the success rates change after correcting the unstated scoring requirement in Odd Parcel? Do any other tasks have similar issues? If this affects only a few cases, its impact on the overall assessment would be limited. If similar issues occur across multiple tasks, the reliability of the benchmark's scoring would need to be reconsidered.

  2. Could you manually review a sample of full trajectories to check whether the cases labeled as insufficient information gathering actually lacked the necessary information? If this remains the most common failure type after the check, the paper's main analysis would be more convincing. Otherwise, the failure categories and the corresponding conclusions would need to be adjusted.

  3. One concern about the fairness of the comparison is the difference in model size. The VLA is a small model with only a few billion parameters, while the models used for zero-shot evaluation are on the scale of several trillion parameters or more. Have you tried open-source backbones such as Qwen? How would models ranging from a few billion to tens of billions of parameters perform?

7 comments

Dear Reviewer dRFQ,

We sincerely thank you for the careful review and the helpful questions! Thanks to your feedback, we added two experiments and two analyses to the paper. Below, we address your concerns point by point.

W1 & Q1. Unstated scoring requirements. Thank you for pointing this out. Odd Parcel indeed scored an unstated requirement, and we removed it in the revised paper. Following your question about other tasks, we re-checked every scorer and also found the Marked Mugs placement rule too strict to show the real differences between models. The old scorer required the whole vessel footprint on the pad, so a vessel on the correct pad that slightly overhung its edge did not count. We relaxed it to 3/4 of the footprint. Success rates on the 50 evaluation instances:

TaskRule changeGPT-6 AstraFable 5.1Gemini 3.8 Flash
Odd ParcelParcels may remain on the pans0% → 10%0% → 2%2% → 2%
Marked Mugs≥ 3/4 of the vessel footprint on its pad (was the whole footprint)10% → 38%0% → 12%0% → 0%

No result of any other task changed, and we added each task's goal text beside its success rule in Appendix E.

W2 & Q2. Failure attribution. We appreciate this insightful point, and we agree that the conclusion needed more direct validation. Following your comment, we re-implemented both indirect rules as direct measurements and added a human check of the attribution, where 46 of 50 sampled units agree with the automatic labels (§4.1, Appendix K).

  • Not approaching vs. not seeing. As you pointed out, not approaching an object does not mean the model has not seen it. We added a direct measure of whether the model saw an object on the exact images it received: we restore the recorded state at every decision and render segmentation masks for those images. An object counts as seen if it appears clearly in any of them. With this measure, the unopened hiding places had usually appeared in the model's images, typically five or more times, and hiding places that never appeared are only 1–3% of failures (§4.2). Therefore, the evaluated models often see a hiding place and still leave it unopened.
  • More weighings vs. useful comparisons. In Odd Parcel, we rewrote the attribution rule so that a parcel now counts as determined only when the settled weighings that appeared in the model's images logically identify it.
  • Human review on “missing evidence vs. failed execution”. When a hidden object was never seen, the robot either never tried to open its hiding place (missing evidence) or tried but failed (execution failure). We distinguish them with a threshold: an attempt counts when the robot touches the hiding place or commands its gripper to within 10–12 cm of its handle. Since this threshold is a heuristic, we added a manual review of 50 randomly sampled units, blind to the automatic label, and 46 match. Of the four mismatches, three met the threshold unintentionally while the robot was manipulating other objects, and in the fourth the robot tried to open a box but its gripper never came within 12 cm of it.

With the newly revised attribution, missing evidence remains the largest failure family for every model (share of failures, §4.1):

Failure familyGPT-6 AstraClaude Opus 5.5GPT-6.1 Sol
Missing evidence39%38%41%
Wrong decision25%34%29%
Execution failure23%21%19%
Side effect (e.g. spilled balls)13%7%12%

Besides exploration that stops too early being the largest failure mode, our improved analysis (§4.2) also finds:

  • Execution is imperfect but not the bottleneck. It accounts for 19–23% of failures.
  • Exploration disturbs the scene, and the damage is rarely repaired, for example balls spilled while tipping a mug to read its label.
  • Learning by trial and error remains hard. In Unfamiliar Containers, when not told how a box opens, Opus 5.5 opens only 45% of the boxes it tries, against 90% when told how. Similarly, covering the Puzzle Box mechanism lowers Opus 5.5 from 94% to 37%.

Q3. Model size. We are grateful for this valuable suggestion. Our π0.5 baseline was meant to test a different route, a policy fine-tuned on demonstrations, rather than to provide a size-matched comparison. As you suggested, we added two open-source Qwen models, run through the same interface as the frontier models: the open Qwen3.8-27B and the larger Qwen3.8-Flash (125B parameters, 6B active). Neither open model solves any of the 60 episodes we ran. The results are in our follow-up below, and we added them to the revised PDF (§3.2, §3.3, Appendix H).

We would deeply appreciate it if you could let us know whether your concerns have been addressed.

AuthorsOpenReview ↗

Follow-up on Q3 and Additional Results

Dear Reviewer dRFQ,

Q3 (follow-up). Smaller open models. Thank you again for suggesting open-source models. As promised, here are the results for the two Qwen models we added, run through the same interface as the frontier models: the open Qwen3.8-27B (27B parameters, dense) and Qwen3.8-Flash (125B parameters, 6B active per token).

Full tasks. We picked the easiest task of each family: Locked Storage (Search), Painted Cubes (Inspect) and Puzzle Box (Test). Within each task, we took the 20 of its 50 evaluation instances that frontier models solved most often. Neither Qwen model solves any of these 60 episodes, and both make almost no progress. All models below are scored on these 20 instances per task.

GPT-6 AstraClaude Opus 5.5GPT-6.1 SolQwen3.8-FlashQwen3.8-27B
Puzzle Box (Test): SR95%80%75%0%0%
Puzzle Box (Test): Progress99.7%93.6%91.6%0.0%2.7%
Painted Cubes (Inspect): SR75%25%35%0%0%
Painted Cubes (Inspect): Progress90.4%61.2%73.8%0.0%0.0%
Locked Storage (Search): SR70%5%15%0%0%
Locked Storage (Search): Progress89.5%46.5%45.1%1.2%0.0%

Isolated skill tests. We also added isolated skill tests to diagnose the low capacity of the Qwen models. Each short test episode checks one basic skill in a kitchen scene, with 20 scenes per skill and at most 50 turns per episode. The instruction names exactly what to do, for example which object to pick up and where to put it, so no search or exploration is needed (§3.3). On skills the frontier models handle reliably, the Qwen models mostly fail:

Skill (success rate)AstraOpus 5.5SolQwen3.8-FlashQwen3.8-27B
Pick an object and place it on a target95%90%95%15%0%
Drive to an object and pick it up100%100%90%5%0%
Turn a cube onto another face100%100%100%70%15%
Take an object out of an open container90%90%70%5%0%

The Qwen models cannot yet act as robot-control agents, and how to enable smaller models to control robots remains an active research question in the community too. Thanks to your suggestion, we added these results to the revised PDF (§3.2, Table 3 in §3.3, and Appendix H).

New experiments, results and analysis. Beyond your questions, we add the same skill tests on three frontier models (§3.3), expanded the demonstration set to 5,000 verified episodes (§2.2), and added three more frontier models for full-task evaluation, so the revision evaluates six instead of three (§3.2, Table 2; 500 episodes each, SR and mean progress in %). Our general comment lists all changes in the revision.

ModelSRProgressSR w/o Puzzle Box
Gemini 4 Argon (new)23.849.316.4
GPT-6 Astra23.248.015.1
Claude Opus 5.5 (new)13.837.37.8
GPT-6.1 Sol (new)12.234.05.6
Claude Fable 5.111.431.04.9
Gemini 3.8 Flash2.09.20.9

Again, thank you very much for your suggestions for improvement! We'd deeply appreciate it if you could let us know whether these results and our response have addressed your concerns.

AuthorsOpenReview ↗

Seeking your response to our rebuttal

Dear Reviewer dRFQ,

Thank you again for your time and your thoughtful review! Following your suggestions, we added two experiments and two analyses, and all the new results are now in our responses above and in the revised PDF.

We understand how busy this period is. We would deeply appreciate it if you could let us know whether your concerns have been addressed.

--

Authors

Reviewer_WYJFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ethics concern

Critical strengths

ethics concern

Critical weaknesses

ethics concern

Questions for authors

ethics concern

1 comment
AuthorsOpenReview ↗

Clarification on anonymity: the citation names a third-party developer, not the authors

Dear Reviewer WYJF,

Thank you for checking the anonymity of our submission so carefully. Anonymity is a serious matter, and we appreciate that you raised it. We would like to clarify the sentence in question.

Inspect Robots is third-party open-source software. It is a MIT-licensed framework developed by Robocurve (github.com/robocurve/inspect-robots). We cite it in the same way we cite MuJoCo or RoboCasa365, as software our work builds on. None of the authors is affiliated with Robocurve, and the citation names the developer of the framework, not the authors of this paper.

“Our custom harness” refers to our own adaptation of the framework. We reuse its model clients and adapted it for RoboQuest, including the simulator, prompt caching, the scoring etc. Appendix B lists these changes in details. We have also revised the sentence in the updated PDF so that it can no longer be misread. It now reads “built on the Inspect Robots agent framework [cite] and adapted for RoboQuest”.

Since the review rests only on this concern, we would be grateful if you could reassess our paper on its technical content. We have also updated the paper and the experiments since submission. The revision evaluates six frontier models, adds isolated tests of the execution skills the tasks require, builds the failure analysis on the images each model received, and releases 5,000 demonstration episodes. We hope you will find the benchmark and its findings interesting and useful, and we are happy to answer any questions during the discussion.

Comments on the paper

2 comments

Dear Reviewer WYJF,

Thank you for your response, and for being open to reconsidering once the flag is resolved. We have also asked the Area Chair and Program Chairs to help resolve it.

We would also like to clarify that the originally submitted PDF already presents Inspect Robots as third-party software throughout:

  • Line 212 refers to Appendix B for details.
  • Lines 315–317: “built on the Inspect Robots agent framework (Robocurve, 2026) and adapted for RoboQuest”.
  • Appendix B (lines 623–630, 666–668) gives its version and MIT license, and lists what we reused and the changes we made to “the published … configuration of Inspect Robots”.
  • References (lines 564–565): “Inspect robots: The open-source evaluation framework for physical ai”, with its public repository.

We agree that line 212 was ambiguous on its own and have reworded it.

We would be glad to discuss the technical content of the paper with you and hope you will find the benchmark and its findings interesting and useful.

AuthorsOpenReview ↗

Summary of revisions

Dear Area Chair and Reviewers,

Thank you for the reviews and the helpful suggestions. Following the feedback, we conducted 5 additional experiments and 2 new analyses, and revised the paper accordingly. For your reference, we have uploaded a revised PDF with the following major improvements:

  1. Added three more frontier models. We added Gemini 4 Argon, Claude Opus 5.5 and GPT-6.1 Sol (§3.2, Table 2). The best two reach 23.8% and 23.2% success. Without Puzzle Box, the easiest task, no model exceeds 16.4%.

  2. Added tests of 8 control skills in isolation (§3.3, Appendix H). We test the control skills that the RoboQuest tasks require, each in isolation, with the hidden information supplied and a budget of 50 decisions. The frontier models reliably carry out most of them, such as pick-and-place (90–95%), driving to an object and picking it up (90–100%) and turning a cube (100%). Some motor skills remain harder, for example opening a drawer or cabinet.

  3. Added a comparison of skills in isolation and in the full tasks (§4.2). Although the isolated tests are a much easier setup, the models do better on the hard skills inside the full tasks. For example, GPT-6 Astra opens 90% of the compartments it tries, against 50% in isolation, and needs 10 decisions on average instead of 22. We hypothesize that the models adapt to the environment and tasks in context during an episode.

  4. Added two smaller open models (§3.2, §3.3, Appendix H). We evaluated Qwen3.8-27B and Qwen3.8-Flash (125B parameters, 6B active) through the same interface. Neither solves any of the 60 full-task episodes we ran. In the isolated skill tests they fail most of the skills too, for example pick-and-place (15% and 0%). Smaller open models therefore cannot yet carry out physical tasks through this interface.

  5. Expanded the demonstration set and added a tool-call version (§2.2, Appendix I). We release 5,000 verified demonstrations, 500 per task, up from 3,211, and every frame is annotated with its stage and subtask. We also converted the demonstrations into the agents' tool-call format, which can be used to train VLM agents.

  6. Improved failure attribution with a human check (§4, Appendix K). Whether a model saw an object is now measured directly on the exact images it received. A blind manual check agrees with the automatic labels on 46 of 50 sampled units.

  7. Rewrote task descriptions and appendices. Among other fixes, we corrected a few task descriptions to match the evaluated goals, and Appendix E now prints each task's goal text beside its success rule. Appendix F now states that the capability marks describe what an efficient solution uses, not what success proves, and Appendix D adds the π0.5 training details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.