- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
RoboQuest is a simulation benchmark for goal-directed embodied exploration, comprising ten mobile manipulation tasks. The tasks are grouped into Search, Inspect, and Test according to whether the missing information concerns object locations, properties revealed through manipulation, or outcomes of physical interventions. Several tasks make information acquisition consequential, and episodes require an explicit physical submission. The paper provides 3,211 successful scripted demonstrations, approximately 405 hours of interaction, and evaluates three zero-shot multimodal agents and a fine-tuned π0.5 policy. It reports task success, partial progress, interaction costs, and failure analyses.
Critical strengths
-
The benchmark addresses a meaningful information-acquisition problem through concrete physical tasks. Search, Inspect, and Test cover different sources of task uncertainty. Hidden cube faces, vessel labels, stamp patterns, and access dependencies connect missing information to subsequent actions. Several tasks also make exploration consequential: accessing storage changes what can be searched, inspecting vessels can spill contents, and final stamp impressions are irreversible. Together with explicit submission, these designs provide a useful setting for studying how agents acquire and use information to complete physical goals.
-
The evaluation establishes substantial headroom and useful differentiation between agents. Overall success rates are 19.4%, 10.0%, and 2.0%; excluding Puzzle Box, they remain 10.9%, 3.3%, and 0.9%, preserving the ranking (Tab.3). The suite is not uniformly at a floor: the strongest agent succeeds on seven tasks, and task-level progress supplies graded outcome measures. The matched-chain-length Puzzle Box comparison is particularly informative: covering the mechanism leaves Astra's success approximately unchanged while increasing its decisions, but reduces Fable's success from 100% to 34.6% (Tab.20). Configurable task difficulty and successful demonstration trajectories further support the benchmark's potential as a resource for measuring future improvements.
Critical weaknesses
-
Some intended capability requirements are not sufficiently established by task success, particularly in Test. Stamp Composition specifies unmarked housings and hidden patterns, but does not establish whether turning a stamp toward a camera could reveal its die and avoid trial impressions. Such a solution would still involve active inspection, but would not establish testing capability. For monotonic Puzzle Box chains, repeatedly commanding every bolt toward its open position could potentially succeed without interpreting outcomes or recovering dependencies. The stated trap moves may prevent this, but their coverage and the effectiveness of this alternative are not established. Low scores from current models do not establish that such routes are unavailable, limiting the precision of the capability interpretation. Fixed exhaustive search, face-by-face inspection, and individual weighing remain legitimate uses of the corresponding basic capabilities; the concern is which capabilities successful behavior actually demonstrates.
-
Task specifications and score interpretation need further reconciliation and calibration. Painted Cubes is described as collecting one or all qualifying objects in Appendix D, whereas Appendix K and the progress definition describe sorting every cube into two bins. Unfamiliar Containers similarly shifts between a single hidden target and multiple items to collect. Most concretely, Appendix J acknowledges Odd Parcel episodes that selected exactly the correct odd set but failed because parcels remained on the balance, although the instruction does not require empty pans. These ambiguities affect the interpretation of benchmark results. Partial progress also needs an information-free reference: Tab.24 contains 201 negative and 101 positive cubes, so assigning every cube to the negative bin would attain 66.6% object-weighted progress without reading its marks. This calculation uses the diagnostic cohort, which differs from Tab.3, and is not directly comparable to its episode-averaged progress. It does not establish a high episode success rate, but shows why progress alone cannot demonstrate effective inspection.
Questions for authors
-
Can a stamp's pattern be recovered by directly viewing its die after rotation? For Puzzle Box, can a fixed cyclic sequence of opening commands succeed without using observed outcomes, and how do traps or reversals prevent this in the evaluated configurations? Please distinguish basic physical testing from the stronger claims of adaptive experimental design or dependency inference.
-
Please reconcile the Painted Cubes and Unfamiliar Containers descriptions with their evaluated goals and success predicates. For Blackout Search, does the released lamp continue illuminating the target, are all three camera streams available, and how do wall-switch and timed-light variants change the required behavior? These details determine which capability requirements are universal versus variant-specific.
-
How would Odd Parcel results change when scoring follows the stated instruction without an empty-pan condition? Please report the effect on the Tab.3 cohort and distinguish it from the larger diagnostic cohort containing the acknowledged cases. For Painted Cubes, does every instance contain a positive object, and how do initial-view-only or no-inspection assignments perform under the actual episode-level metrics?
-
For one representative near-zero-success task, is there a simple systematic policy or an existing verified execution under the agents' allowed observations, tool interface, and 200-decision budget? The successful scripts have privileged information and use native control, so they establish physical feasibility rather than a directly comparable reference. Please also report the total training steps, evaluation sample sizes and horizons, and any available training-distribution performance, so that its near-zero results can be interpreted.
3 comments
Author Response to Reviewer JGYY (2/2)
Q3 & W2. Score interpretation needs calibration
- Odd Parcel without the empty-pan condition. The main table now scores Odd Parcel by the stated instruction. After this change, GPT-6 Astra goes from 0 to 5 successes, Claude Fable 5.1 from 0 to 1, and Gemini 3.8 Flash stays at 1. And our revised failure analysis uses exactly the main-table episodes.
- Painted Cubes, positive objects and initial views. Every instance is designed to have both kinds of cube. The deciding marks are designed to be invisible from the initial 3 camera views, so no cube can be classified without manipulating it.
- No-inspection assignments, and does progress show inspection? Blindly putting every cube in the yellow bin indeed gives 67.9% progress and 0% success, so in Painted Cubes progress alone does not show inspection. Therefore, for this task we propose a second score, . Here is the number of cubes that satisfy the rule, of them are put in the correct bin and in the wrong bin. and are the same counts for the other cubes. The score ranges from −100% to 100%. A perfect sort scores 100%, while strategies that do not inspect, such as one bin for everything or random guessing, score 0 on average. We also find this score follows the success rate more closely than progress does:
| Gemini 4 Argon | GPT-6 Astra | GPT-6.1 Sol | Claude Opus 5.5 | Claude Fable 5.1 | Gemini 3.8 Flash | All cubes in yellow | |
|---|---|---|---|---|---|---|---|
| Success rate | 36 | 30 | 18 | 14 | 14 | 2 | 0 |
| Progress | 74.6 | 74.8 | 69.2 | 59.2 | 61.5 | 11.3 | 67.9 |
| New Score | 65.9 | 58.7 | 54.4 | 40.4 | 37.7 | 7.4 | 0 |
Q4. Feasibility under the agents' interface, and π0.5 details
- A systematic solution under the agents' interface. With the added models, Gemini 4 Argon solves 3 Search Room episodes, and Blackout Search is now the only task that no model solves. To show that the tasks are solvable through the agents' tools and budget, we converted the scripted demonstrations into tool calls, and verified in simulation that the scripts succeeded in the tasks. The revision adds this conversion to Appendix I. Like an agent, these scripts do not use the hidden information to skip exploration. They act only on what their own exploration has revealed. Search Room needs a median of 66 calls (all episodes within 200), and Blackout Search a median of 122 (96% within 200). These counts are upper bounds, not what a good agent needs. The scripts move the robot in many short steps, and the conversion keeps their path. An agent that plans its own moves can cover the same path with fewer calls.
- π0.5 details. We fine-tune for 10,000 steps per task at batch size 32 (now in Table 6, Appendix D). The policy predicts 20-step action chunks (1.0 s at 20 Hz) and replans after each chunk, and an episode ends at SUBMIT or after 12,000 control steps (10 minutes). The revision evaluates π0.5 on all 10 tasks (5 in the original submission), on the same 50 instances per task as the frontier models (§3.4). It solves only one Puzzle Box episode (2.0% success, 43.3% progress), against 12–96% success for the frontier models on that task. On its training distribution, the policy fits the demonstrations closely: its predicted action chunks reach R² = 0.97 against always predicting the mean action, both with its own predicted subtask and with the correct one. Appendix D gives the full training-distribution details.
Thank you again for your detailed and constructive review and please let us know if you have any other questions. We are more than happy to discuss.
Thanks for the effort made by the authors for clarification about the uncertain settings and I'm quite impressed by the authors who do fix some of the main concerns I care about in the initial reviews, especifically about whether the tasks can test what the authors care, while it also fit new models into the benchmark for testing. Therefore I think the contribution of this paper can be serve as a useful benchmark to test new robotic policies capabilities. Since I hope I myself can test on it, I have change my recommendation to accept.
Some small advice for the author:
- Since the rebuttal is still not over, I'd like to have some glance of the videos of the existing tasks (maybe in the supplementary) to check the tasks further. Some annotations that can help understanding may be even better.
- An advice related to ICLR specific rebuttal is that, since it can change paper, you may make the important modifications of your current paper with a different color (other than the default black), so people can easily find the main modification you have made to strengthen the paper.
Author Response to Reviewer JGYY (1/2)
Dear Reviewer JGYY,
Thank you for the detailed and constructive review. We answer each separate point under your questions, which also cover both weaknesses. We additionally evaluate three new models (Gemini 4 Argon, Claude Opus 5.5 and GPT-6.1 Sol), and some answers below use their results. Our general comment lists all changes.
Q1 & W1. Capability requirements not sufficiently established by task success, particularly in Test
Q2 & W2. Reconciliation of task specifications in Appendix E
We thank you for pointing this out. And the revision corrects three Appendix E descriptions from the original submission (table below). The evaluation itself, the instructions given to the models, and all results were unaffected. The paragraphs in the original submission described some internal earlier task versions by mistake.
Appendix E now also prints each task's goal text beside its success rule. On your Blackout questions: