- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents an interesting benchmark targeting experience-driven self-improving agents in Minecraft. It has a fixed catalog of 19 registered evolution chains. The benchmark not only measures whether an agent reaches the goal, but also measures progress through the task graph, knowledge transfer, different source conditioning consists of no prior experience, related experience, unrelated experience, and shuffled experience. Comprehensive experiments reach several interesting findings:
- Low-level execution is the first barrier to higher performance.
- Experience does not reliably acquire skills for agents.
- Self-improvement is gone under controlled comparisons (i.e., no statistical significance).
- More experience is not helping either.
- Early failures make evaluation of later advanced capabilities hard to measure.
Overall, this is an interesting and timely work for embodied agents in an open world.
Critical strengths
-
The 5 findings make the paper's conclusions explicit and good writing, as well as nice figures. Readers don't need to search for the key conclusions, and the paper's key ideas are easy to grasp. Very good presentation. All findings are supported by strong experiments and statistical significance measures. This greatly improves the paper's soundness and scientific validity.
-
This should be a very useful benchmark / test suite for researchers who study open-world embodied agents in the Minecraft environment. The failure patterns identified in the paper clearly show the key weaknesses of current VLM embodied agents even in a simple Minecraft task, let alone robotic tasks. This calls for more attention from the research community on questions like how to better connect high-level plans and low-level execution.
Critical weaknesses
- No proprietary frontier model evaluation. I only see smaller open-sourced models. It is important and recommended to add models like GPT-6, Opus 5.5, Gemini 4 to understand how these frontier multimodal vision language model work on these tasks because the gap between open source models and those proprietary models is still large so omitting such evaluation makes readers question whether the findings and claims still hold for these stronger models. Ideally show a scaling law from gpt-4o to gpt-5 to gpt-6 to see if keep scaling VLMs can solve the problem directly.
Questions for authors
No questions.
7 comments
Response to Reviewer c3yP (2/3)
3. Why stronger planners do not help: alignment between plan form and controller.
Stage-level records reveal where this gap comes from. Under the same world, the same scene profiles (wooden-tool and stone-tool families), the same STEVE-1 controller and the same step budget, for the same instruction "Collect at least one oak log.":
| Planner and plan form | Oak-log stage passed |
|---|---|
| Minecraft-finetuned 7B planner: passes the stage instruction through verbatim | 27/32 (84%) |
| Minecraft-finetuned 7B planner: rewrites it into subgoals | 0/34 (0%) |
Across the Table 3 stage records, verbatim instructions pass 37/151 while rewritten subgoals pass 1/118 (Fisher p ≈ 1.5×10⁻⁹). These observational contrasts suggest that plan form and compatibility with the controller's learned instruction distribution deserve attention. Stronger general planning does not automatically provide that compatibility. MC-EvoBench's stage records let us inspect the planner output, the instruction supplied to the controller, and the stage outcome together.
All 45 runs generated and dispatched a plan, with no empty plan; failures concentrate where plans must become multi-step crafting dependencies:
| Statistic | Value |
|---|---|
| Runs with a plan generated and dispatched | 45/45 |
| Runs ending with an empty inventory | 31/45 |
| Runs passing the first collection stage | 9/45 |
A representative high-progress run collected a log in 279 steps and crafted planks in 103 steps, then exhausted the 300-step cap at crafting sticks. These records show that generating a plan and collecting the initial resource need not lead to reliable execution of the remaining crafting steps, consistent with Findings 1 and 5 and with your emphasis on better connecting high-level plans and low-level execution.
4. Comparison on 12 task chains
We compared gpt-5.6-sol with gpt-6-astra on all 12 primary task chains. The experiment covers both controller branches. It contains 144 runs across the full task chains. The Optimus-1 planner receives first-person images. This branch covers the setting with visual input.
Metric (gpt-6-astra − gpt-5.6-sol) | Cluster mean difference | 95% confidence interval | Test result |
|---|---|---|---|
| TSR | −0.80 percentage points | [−2.05, +0.45] | Statistically equivalent within ±3 percentage points |
| WGP | −0.65 percentage points | [−1.66, +0.35] | Statistically equivalent within ±2 percentage points |
| VMP | −3.47 percentage points | [−9.57, +2.63] | Interval includes 0 |
The newer model shows no improvement on the three metrics in this comparison. The TSR difference falls within the specified equivalence range. The WGP difference meets its equivalence criterion. The plan texts differ within all 72 pairs. Mean text similarity is 0.400. Execution outcomes remain identical in 66 pairs.
The TSR difference between generations is 0.6 percentage points for JARVIS-1. It is 1.0 point for Optimus-1. The spread across the five planner families in Table 2 is 1.3 points for JARVIS-1. It is 1.5 points for Optimus-1. The controllers alone remain above every evaluated planner combination. Replacing the planner has not resolved the main execution difficulties in this setting.
Response to Reviewer c3yP (3/3)
5. Evaluation setting
MC-EvoBench supports planner replacement under three fixed settings: the world, controller, stage budget. We will provide the launcher, scorer, individual run records. These materials support evaluation through model services compatible with the OpenAI interface. The JARVIS-1 planner receives structured visible state. The Optimus-1 planner receives first-person images. The 144-run comparison covers these two input formats.
These experiments evaluate frontier models in the role of a planner. Released controllers execute the actions. This follows JARVIS-1's original design. Optimus-1 uses the same layered approach. The results concern the tested models under the specified interfaces. Better general planning has not automatically resolved the compatibility issue with the controller. End-to-end embodied models require evaluation in their own setting. Different control interfaces require separate tests.
The revision will identify model versions in Table 2. It will include the comparison across three generations, individual run records, further analysis of instruction form. Thank you for your suggestions. They help us clarify the evidence behind our conclusions. They help us state the scope of these findings more clearly.
Response to Reviewer c3yP (1/3)
Thank you for your positive assessment of our five findings. We appreciate your questions about stronger proprietary models. You ask about the potential of more capable VLMs to solve these tasks. We added comparisons across model generations. We examined execution records to understand the remaining gap. The results below clarify the scope of our findings.
1. The existing evaluation includes proprietary frontier models
Table 2 covers five planner families: GPT, Gemini, Qwen, GLM, DeepSeek. The GPT configuration uses a proprietary model. The Gemini configuration does the same. Appendix Tables F8–F9 give the configurations. Both controller branches were evaluated with
gpt-5.6-sol. The model uses the Responses interface with high reasoning effort. The frontier model generates subgoals. The underlying controller produces the actions.gpt-5.6-solplannerThe controller is unchanged in this comparison. Adding the planner reduces JARVIS-1's mean TSR from 7.5% to 1.5%. This matches the no-op baseline. Optimus-1 falls from 8.1% to 3.4%. The Gemini planners show a similar pattern. Their mean TSR is 1.8% for JARVIS-1, 3.9% for Optimus-1. Both values remain below the corresponding controller alone.
These results are consistent with Finding 1 within the tested architecture. A stronger planner has not improved execution in this setting. The revised Table 2 will replace icons with explicit model names. It will give the relevant model versions for each planner.
2. Three model generations in repeated evaluations
We used the target configuration from Table 3 to compare three successive GPT generations. Only the planner was replaced. The world, controller weights, memory interface, stage budgets stayed fixed. The design covers three task families, five agent seeds, three model generations. All 45 runs were scored.
gpt-5.6-solgpt-6-astragpt-6.1-solMean TSR ranges from 4.8% to 6.0% across the frontier models. The gap from the domain planner remains 23.0–24.2 percentage points. The newest generation gains 1.2 points over the earliest generation. The gap remains large. Success at the first stage totals 9/45 for the frontier models. The domain planner passes 13/15.
We repeated the frontier evaluation in a separate batch. The table below summarizes the two batches. Every row uses the same domain planner reference from Table 3.
Both batches show a substantial gap in pass rates. The pattern remains visible across repeated runs. For the tested models, progress in general planning has not fully translated into better embodied execution. Agent performance has room for improvement on this benchmark.