acceptodds
Under review as a conference paper at ICLR 2027

MC-EvoBench: Auditing Experience-Driven Self-Improvement in Minecraft Agents

Abstract

A central challenge in embodied recursive self-improvement is enabling agents to convert interaction history into capabilities that persist beyond one world and transfer to later tasks. A saved memory or successful final action does not by itself establish that a reusable capability was acquired; evaluation must separate capability change from carried-over resources, unequal source exposure, and early execution failures. We construct MC-EvoBench as a controlled Minecraft benchmark with a fixed catalog of 19 registered evolution chains, including 12 primary evaluation chains and seven calibration or probe chains, together with three graph-aligned paired probe families for transfer. The benchmark represents each task as a task-state graph, defines stage-level environment verifiers, assigns matched source and target roles, resets the target world, and permits only a declared agent artifact to cross the source–target boundary. For each probe, no-prior, related, unrelated, and shuffled source conditions are evaluated under matched environment-step budgets and five seeds; we report TSR, WGP, and VMP and retain provenance records. The benchmark reveals a consistent direct-capability gap between controller and planner-backed variants, while related experience yields no stable transfer: it does not improve either tool probe, its small wood-species gain hinges on one seed, and all intervals include zero. Stage audits further show that early wood-collection and crafting bottlenecks often leave later capabilities unobserved. MC-EvoBench thus offers a controlled testbed for durable, transferable embodied capabilities. Code and data are available at https://anonymous.4open.science/r/MC-EvoBench.

Primary area:
datasets and benchmarks
Keywords:
self-improving agents, embodied AI, agent evaluation, memory-augmented agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_c3yPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an interesting benchmark targeting experience-driven self-improving agents in Minecraft. It has a fixed catalog of 19 registered evolution chains. The benchmark not only measures whether an agent reaches the goal, but also measures progress through the task graph, knowledge transfer, different source conditioning consists of no prior experience, related experience, unrelated experience, and shuffled experience. Comprehensive experiments reach several interesting findings:

  1. Low-level execution is the first barrier to higher performance.
  2. Experience does not reliably acquire skills for agents.
  3. Self-improvement is gone under controlled comparisons (i.e., no statistical significance).
  4. More experience is not helping either.
  5. Early failures make evaluation of later advanced capabilities hard to measure.

Overall, this is an interesting and timely work for embodied agents in an open world.

Critical strengths
  1. The 5 findings make the paper's conclusions explicit and good writing, as well as nice figures. Readers don't need to search for the key conclusions, and the paper's key ideas are easy to grasp. Very good presentation. All findings are supported by strong experiments and statistical significance measures. This greatly improves the paper's soundness and scientific validity.

  2. This should be a very useful benchmark / test suite for researchers who study open-world embodied agents in the Minecraft environment. The failure patterns identified in the paper clearly show the key weaknesses of current VLM embodied agents even in a simple Minecraft task, let alone robotic tasks. This calls for more attention from the research community on questions like how to better connect high-level plans and low-level execution.

Critical weaknesses
  1. No proprietary frontier model evaluation. I only see smaller open-sourced models. It is important and recommended to add models like GPT-6, Opus 5.5, Gemini 4 to understand how these frontier multimodal vision language model work on these tasks because the gap between open source models and those proprietary models is still large so omitting such evaluation makes readers question whether the findings and claims still hold for these stronger models. Ideally show a scaling law from gpt-4o to gpt-5 to gpt-6 to see if keep scaling VLMs can solve the problem directly.
Questions for authors

No questions.

7 comments
AuthorsOpenReview ↗

Response to Reviewer c3yP (1/3)

Thank you for your positive assessment of our five findings. We appreciate your questions about stronger proprietary models. You ask about the potential of more capable VLMs to solve these tasks. We added comparisons across model generations. We examined execution records to understand the remaining gap. The results below clarify the scope of our findings.

1. The existing evaluation includes proprietary frontier models

Table 2 covers five planner families: GPT, Gemini, Qwen, GLM, DeepSeek. The GPT configuration uses a proprietary model. The Gemini configuration does the same. Appendix Tables F8–F9 give the configurations. Both controller branches were evaluated with gpt-5.6-sol. The model uses the Responses interface with high reasoning effort. The frontier model generates subgoals. The underlying controller produces the actions.

ConfigurationJARVIS-1 mean TSR / WGP / VMP (%)Optimus-1 mean TSR / WGP / VMP (%)
Controller alone7.5 / 5.5 / 25.08.1 / 6.1 / 25.4
With the gpt-5.6-sol planner1.5 / 0.6 / 0.03.4 / 2.1 / 6.7

The controller is unchanged in this comparison. Adding the planner reduces JARVIS-1's mean TSR from 7.5% to 1.5%. This matches the no-op baseline. Optimus-1 falls from 8.1% to 3.4%. The Gemini planners show a similar pattern. Their mean TSR is 1.8% for JARVIS-1, 3.9% for Optimus-1. Both values remain below the corresponding controller alone.

These results are consistent with Finding 1 within the tested architecture. A stronger planner has not improved execution in this setting. The revised Table 2 will replace icons with explicit model names. It will give the relevant model versions for each planner.

2. Three model generations in repeated evaluations

We used the target configuration from Table 3 to compare three successive GPT generations. Only the planner was replaced. The world, controller weights, memory interface, stage budgets stayed fixed. The design covers three task families, five agent seeds, three model generations. All 45 runs were scored.

PlannerFirst stage passedTSR (%)WGP (%)VMP (%)
7B planner fine-tuned for Minecraft (Table 3)13/1529.022.249.8
gpt-5.6-sol3/154.83.111.1
gpt-6-astra3/155.23.912.2
gpt-6.1-sol3/156.03.810.0

Mean TSR ranges from 4.8% to 6.0% across the frontier models. The gap from the domain planner remains 23.0–24.2 percentage points. The newest generation gains 1.2 points over the earliest generation. The gap remains large. Success at the first stage totals 9/45 for the frontier models. The domain planner passes 13/15.

We repeated the frontier evaluation in a separate batch. The table below summarizes the two batches. Every row uses the same domain planner reference from Table 3.

BatchFrontier planners: first stage passed7B planner fine-tuned for MinecraftFisher's exact test, two-sided p
Batch 1 (2026-11-15)1/2813/153.8×10⁻⁸
Batch 2 (2026-11-16, table above)9/4513/156.8×10⁻⁶
Combined10/7313/157.7×10⁻⁸

Both batches show a substantial gap in pass rates. The pattern remains visible across repeated runs. For the tested models, progress in general planning has not fully translated into better embodied execution. Agent performance has room for improvement on this benchmark.

AuthorsOpenReview ↗

Response to Reviewer c3yP (2/3)

3. Why stronger planners do not help: alignment between plan form and controller.

Stage-level records reveal where this gap comes from. Under the same world, the same scene profiles (wooden-tool and stone-tool families), the same STEVE-1 controller and the same step budget, for the same instruction "Collect at least one oak log.":

Planner and plan formOak-log stage passed
Minecraft-finetuned 7B planner: passes the stage instruction through verbatim27/32 (84%)
Minecraft-finetuned 7B planner: rewrites it into subgoals0/34 (0%)

Across the Table 3 stage records, verbatim instructions pass 37/151 while rewritten subgoals pass 1/118 (Fisher p ≈ 1.5×10⁻⁹). These observational contrasts suggest that plan form and compatibility with the controller's learned instruction distribution deserve attention. Stronger general planning does not automatically provide that compatibility. MC-EvoBench's stage records let us inspect the planner output, the instruction supplied to the controller, and the stage outcome together.

All 45 runs generated and dispatched a plan, with no empty plan; failures concentrate where plans must become multi-step crafting dependencies:

StatisticValue
Runs with a plan generated and dispatched45/45
Runs ending with an empty inventory31/45
Runs passing the first collection stage9/45

A representative high-progress run collected a log in 279 steps and crafted planks in 103 steps, then exhausted the 300-step cap at crafting sticks. These records show that generating a plan and collecting the initial resource need not lead to reliable execution of the remaining crafting steps, consistent with Findings 1 and 5 and with your emphasis on better connecting high-level plans and low-level execution.

4. Comparison on 12 task chains

We compared gpt-5.6-sol with gpt-6-astra on all 12 primary task chains. The experiment covers both controller branches. It contains 144 runs across the full task chains. The Optimus-1 planner receives first-person images. This branch covers the setting with visual input.

Metric (gpt-6-astra − gpt-5.6-sol)Cluster mean difference95% confidence intervalTest result
TSR−0.80 percentage points[−2.05, +0.45]Statistically equivalent within ±3 percentage points
WGP−0.65 percentage points[−1.66, +0.35]Statistically equivalent within ±2 percentage points
VMP−3.47 percentage points[−9.57, +2.63]Interval includes 0

The newer model shows no improvement on the three metrics in this comparison. The TSR difference falls within the specified equivalence range. The WGP difference meets its equivalence criterion. The plan texts differ within all 72 pairs. Mean text similarity is 0.400. Execution outcomes remain identical in 66 pairs.

The TSR difference between generations is 0.6 percentage points for JARVIS-1. It is 1.0 point for Optimus-1. The spread across the five planner families in Table 2 is 1.3 points for JARVIS-1. It is 1.5 points for Optimus-1. The controllers alone remain above every evaluated planner combination. Replacing the planner has not resolved the main execution difficulties in this setting.

AuthorsOpenReview ↗

Response to Reviewer c3yP (3/3)

5. Evaluation setting

MC-EvoBench supports planner replacement under three fixed settings: the world, controller, stage budget. We will provide the launcher, scorer, individual run records. These materials support evaluation through model services compatible with the OpenAI interface. The JARVIS-1 planner receives structured visible state. The Optimus-1 planner receives first-person images. The 144-run comparison covers these two input formats.

These experiments evaluate frontier models in the role of a planner. Released controllers execute the actions. This follows JARVIS-1's original design. Optimus-1 uses the same layered approach. The results concern the tested models under the specified interfaces. Better general planning has not automatically resolved the compatibility issue with the controller. End-to-end embodied models require evaluation in their own setting. Different control interfaces require separate tests.

The revision will identify model versions in Table 2. It will include the comparison across three generations, individual run records, further analysis of instruction form. Thank you for your suggestions. They help us clarify the evidence behind our conclusions. They help us state the scope of these findings more clearly.

Reviewer_8qVrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MC-EvoBench tests whether experience acquired in Minecraft transfers to later tasks. Only a declared agent artifact crosses the reset between source and target episodes. The benchmark contains 80 tasks and 19 chains, including 12 primary evaluation chains. Three paired probes compare different sources of experience using five agent seeds. The reference memory method shows no stable positive transfer, and the stage analysis identifies bottlenecks that prevent later capabilities from being observed.

Critical strengths
  1. Resetting the world and inventory and creating a fresh target agent is a sensible way to separate transferred experience from carried resources (Section 3.1 and Appendix D.1). The stage verifiers also make partial progress inspectable. The current release includes the task catalog, runner, verifiers, metrics, and tests.

  2. The negative result is reported carefully. Table 4 reports paired contrasts and confidence intervals, Table 5 reports source costs, and Appendix Table D3 gives the per-seed results. The paper also acknowledges that equal interaction budgets do not imply equal computation. Distinguishing failure at a stage from never reaching that stage is useful when interpreting low success rates.

Critical weaknesses
  1. The released benchmark code does not yet provide a clear route to reproducing the reference experiments. In the repository available on 20 October, I could not find the JARVIS online-memory adapter named in Appendix G.6, the paired source/target evaluation launcher, or the frozen artifacts and run records for Tables 2-5. The generic external-policy interface and the appendix pseudocode are helpful, but the missing components include the experience mechanism being evaluated. This is my main reservation about accepting the benchmark in its current form.

  2. It is unclear how sensitive the paired probes are to useful experience. The related-source TSR values are only 0.08–0.15 (Table 5), and the stage audit rarely reaches later crafting steps (Table F10). The authors acknowledge this limitation. The null result is still informative, but a small check with a verified successful source artifact would help distinguish limited acquisition from a target probe that cannot expose the benefit.

Questions for authors
  1. Can you point to the reference agent adapter, paired evaluation launcher, and artifacts or run records used for the reported experiments? If they are not in the release, please clarify what is currently reproducible from it.

  2. Is there a probe where the source demonstrably acquired the relevant skill or memory, and does that artifact help after the reset? An existing successful case, or a small positive control on one tool probe, would address the sensitivity concern.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 8qVr (1/2)

Thank you for your detailed suggestions. We have organized the materials needed to reproduce the reference study. We checked the delivery of source experience to the target agent. We traced its use during execution. We address your questions below.

Q1. Reference implementation The updated reproduction package is now available in the repository.

Offline recomputation reads the original stage records. It uses the planned number of stages in each denominator. The procedure rebuilds every metric in the 105 role reports. It rebuilds the 45 paired summaries, individual seed contrasts, source costs. The results match Tables 3–5. All 45 files match. Outputs remain identical at the byte level after independent extraction of the package.

AuthorsOpenReview ↗

Response to Reviewer 8qVr (2/2)

Q2. Successful source experience

Verified source success

The wooden tool tasks share their first 4 stages. The stone tool tasks share their first 7 stages. Sources leave verified success records at these stages.

We selected target stages meeting three requirements. The target must reach the stage. Memory must contain a source success for the identical instruction. The original retrieval rule must rank this record first. Twelve instances qualify. Comparisons use the same seed, world, stage.

StageInstancesNo prior: passedRelated: passed
Collect log1010/105/10
Craft planks21/21/2
Total1211/126/12

Source success is verified here. The target retrieves the corresponding record. Related memory shows no advantage within these selected instances. This supports Finding 2.

Use of retrieved experience

Prompts without prior experience contain 294, 306, 307 tokens across the three families. Prompts with memory contain 396–796 tokens. Retrieved content enters the planning context.

Source episodes reset the world between attempts. Memory is retained. We examined cases with a verified success for the identical instruction ranked first.

Controller instructionStages passedLog collection passed
Original instruction retained18/2617/20
Rewritten into subgoals1/410/27

Log collection rates are 0.85 versus 0.00. The verifier can record high success rates. These are observational contrasts.

The planner rewrites instructions in 8/45 cases with empty memory. The count is 110/224 with retrieved memory. Retrieved text appears directly in 62 rewrites. Of 272 reflections, 201 contain Grounding/Motion/Action: strings, such as Grounding: Explore. These entries lack reusable procedural guidance.

Across all 269 execution records, retained instructions pass 37/151. Rewritten subgoals pass 1/118. In 67 attempts reusing verified experience after reset, the planner rewrites 41 instructions. Experience reaches the planner. Converting it into effective controller input remains difficult. These records help explain Finding 2.

The final stage is executable

We agree the limited coverage of later stages needs clarification. A manual policy starts with an empty inventory. It completes all five original wooden shovel predicates in one episode. It uses 370 ordinary actions. The final public inventory contains 1 wooden shovel, 1 plank, 2 sticks.

StageActions usedFirst pass index
Collect log139138
Craft planks102240
Craft sticks29269
Craft table34303
Craft wooden shovel66366
Total370All five stages passed

Independent checks cover the raw actions, frame images, verifier recomputation, journal with a hash chain. The diagnostic series contains 11 runs with separate resets. The runs returned 2,874 ordinary actions. The complete episode confirms task feasibility under the existing interface. The verifier detects the success.

The released Voyager skill bridge provides another check under controlled resources. It uses the same task graph. The unchanged verifier records ten consecutive stages. Examples include craft_stone_pickaxe, craft_furnace, collect_coal.

Learned agents rarely reach later stages. This limits coverage of their capabilities (Findings 1, 5). The revision will report reach counts separately from pass counts.

Information controls

We conducted two controls with a fixed controller. E1 provides correct stage information written by hand. E1b provides a verified manual demonstration of all five predicates from an empty inventory. The reset target receives it through the original read-only retrieval interface.

Each experiment uses 5 matched seed pairs. Runs record the delivery path, retrieval events, initial inventory. Each arm scores 0/25 planned stages in both experiments. Conditional rates for later stages are NA. No downstream gain was observed in these tests. The compatibility of planner output with the controller merits further study.

Dear Reviewer,

Thank you for your careful review and suggestions. If these additions have adequately addressed your main concerns, we would appreciate it if you could reassess the paper’s contribution in light of the clarifications and revisions, and consider updating your score accordingly. If any concerns remain that affect your assessment, we would welcome the opportunity to clarify them.

Thank you for your time and for helping us improve this work.

Best regards, The Authors

Reviewer_pCKbOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MC-EvoBench, a benchmark that tests whether agents truly gain reusable skills from past experience. It resets the target world, lets only a declared memory artifact carry over, and compares four source conditions: no prior, related, unrelated, and shuffled experience. A task-state graph with stage-level verifiers tracks partial progress. In the experiments, the authors find that related experience gives no stable transfer gain, which is another evidence that the current straightforward approaches cannot achieve the true RSI.

Critical strengths
  • Problem framing, the paper treats embodied RSI as an attribution problem and separates transferable agent state from resettable world state, which makes sense to me.

  • Fine-grained controls and metrics, four-condition paired protocol uses matched step budgets, world resets, and artifact-only transfer to rule out confounds. Stage-level verifiers and TSR/WGP/VMP measure partial progress and show where execution stalls.

  • Evaluation and honest reporting.

Critical weaknesses
  • My major concern might be the evaluation on this benchmark currently cannot demonstrate the transfer capability. To test transfer, the agent must first complete the source task, but related-source TSR might be too low, so the experiments cannot yet show whether transfer works. Thus, how often did source runs complete the full chain, and could transfer be analyzed only on seeds where the source task succeeded.

  • Followed by the question above, I doubt the probes may not reach the target step: Most target runs stop after the first one or two stages, so the final tool step where source and target differ is rarely observed.

  • My minor concern is that Minecraft's suitability for embodied evaluation is not fully justified. The paper argues that Minecraft is suitable mainly because it is easy to control, dependencies are visible, recipes form a graph, and the world can be reset. However, these are properties of a crafting game rather than of physical embodiment, and the main bottleneck (plank, stick, and crafting-table steps) may come from the game-specific crafting interface rather than from embodied capability. What's more, for controlling Minecraft, there is no powerful baseline models, which is the exact bottlenecks for the evaluation as discussed above. So, could the authors discuss whether the findings and the reset-based protocol would apply to other embodied environments, such as robotic simulators?

Questions for authors

See weakness

11 comments
AuthorsOpenReview ↗

Response to Reviewer pCKb (1/4)

Thank you for your suggestions. We first identify the experience acquired by the source agent. We trace the stages reached by the target agent. These checks help us interpret the transfer results. Our response covers four topics: successful source experience, its use at the target, stage coverage, relevance to robotics.

1. Transfer after a successful source stage

In each tool probe, the source task shares a prefix with the target task. The wooden tool pair contains 5 stages. The first 4 are identical. The source ends with a wooden axe. The target ends with a wooden shovel. The stone tool pair contains 8 stages. The first 7 are identical. The source ends with a stone pickaxe. The target ends with a stone shovel. Tables D2, F11 give these sequences.

The fixed settings are world seed 31, the environment profile, the agent seed. We checked all 7 roles within each of the 15 evaluation units. These settings match throughout.

We can retain seeds with a verified source pass at a given stage. Target performance is compared at the same stage. The source passes log collection on 5/5 seeds in both tool families. Source plank success occurs on 2/5 seeds for wooden tools. It occurs on 3/5 seeds for stone tools. The table includes only these successful source seeds.

ProbeStage comparedSeeds with a source passNo priorRelatedUnrelatedShuffled
Wooden toolCollect log; identical source/target task55/53/53/54/5
Wooden toolCraft planks; identical source/target task21/21/21/21/2
Stone toolCollect log; identical source/target task55/52/51/53/5
Stone toolCraft planks; identical source/target task31/30/30/30/3
Wood speciesBirch log at source; acacia log at target11/10/10/10/1

The table contains 16 units. Each unit specifies a probe, stage, seed. The source has passed the relevant stage in every selected unit. Related experience does not exceed no prior within this subset. A subset of 12 instances retrieves the successful source record first. The retrieval evidence is given below.

Requiring success on the full source chain yields no eligible samples. This analysis concerns successful intermediate stages. The paired probes use short sequences of 2, 5, 8 stages. Tool tasks retain a shared prefix. This design supports evaluation of transfer at intermediate stages (Appendix D.3). The revision will distinguish stage success from success on the full chain. This will clarify the scope of the analysis.

AuthorsOpenReview ↗

Response to Reviewer pCKb (2/4)

2. Records trace the experience received by the target

We checked the planning records to verify the delivery of experience. The evidence covers several steps.

Retrieved content appears in the planner input. Within each probe, all five targets without prior experience have the same prompt length for the first stage. The prompt for the wooden tool probe contains 294 tokens. The prompt for the stone tool probe contains 306 tokens. The prompt for the wood species probe contains 307 tokens. Targets with a memory file receive prompts of 396–796 tokens. These lengths agree with the retrieval records.

Successful source records rank first in retrieval. We replayed the retrieval rule used for Tables 3–5 on the 45 saved memory files. The first retrieved record matches the original log at all 24 reached target stages in the tool probes. Twelve instances retrieve a successful record for the identical instruction. Its status is passed. The comparison below uses matched seeds.

StageInstancesNo prior: passedRelated: passedOnly no prior passed / Only related passed
Collect log1010/105/105 / 0
Craft planks21/21/20 / 0
Total1211/126/125 / 0

The same verifier captures differences across instruction forms. Source episodes reset the world between attempts. Memory is retained. These episodes provide repeated observations of experience reuse after reset. We examined attempts with a successful record for the identical instruction ranked first. Outcomes differ across the instruction forms received by the controller.

Instruction received by the controllerStages passed / recordsLog collection passed
Original instruction from the successful record retained18/2617/20
Instruction rewritten into subgoals by the planner1/410/27

Both comparisons use Fisher's exact test with a two-sided alternative. The p value for all stages is approximately 2.7×10⁻⁹. The value for log collection is approximately 4.2×10⁻¹⁰. Retaining the original instruction gives 17/20 success after reset at log collection. This confirms the verifier's ability to record high success rates. The rewritten instructions have lower pass rates. This observation motivates closer study of instruction compatibility with the controller.

Memory content is associated with instruction rewriting. With empty memory, the planner rewrites instructions in 8/45 cases. With retrieved memory, the count is 110/224 (Fisher's exact test, two-sided p ≈ 1.1×10⁻⁴). At least one step directly copies retrieved text in 62 cases. Of the 272 reflections, 201 use the Grounding/Motion/Action: format. An example is Grounding: Explore. These entries lack reusable procedural guidance.

The source leaves successful experience. The target retrieves the experience. It reaches the planner. During subsequent use, the planner rewrites the original instruction into subgoals in 41 cases. These records connect Finding 2 to a specific part of experience use. The record includes retrieved content, plan text, controller input, stage outcomes. Readers can inspect the step associated with the low transfer benefit.

AuthorsOpenReview ↗

Response to Reviewer pCKb (3/4)

3. Stage reach counts

We agree the limited coverage of later stages needs explicit reporting. The table gives passed/reached counts at each stage. NA denotes an unreached stage.

ProbeStageNo priorRelatedUnrelatedShuffled
Wood species1: collect acacia log; the tasks diverge here3/52/51/53/5
Wood species2: craft acacia planks1/32/21/12/3
Wooden tool2: craft planks; a shared stage1/52/31/32/4
Wooden tool3: craft sticks1/10/20/10/2
Wooden tool5: craft wooden shovel; the tasks diverge here0/1NANANA

The wood species probe changes the target tree species from the first step. All 20 target runs reach this step. The benchmark directly observes performance after this change. GATG is positive for all three metrics: TSR +0.05, WGP +0.06, VMP +0.05. At plank crafting, related experience passes 2/2. Unrelated experience passes 1/1. No prior passes 1/3. Shuffled experience passes 2/3. These small samples give a concrete example of positive gains recorded by the benchmark.

The shared stages leave room for improvement. In the wooden tool probe, all 5 runs without prior experience reach plank crafting. Only 1 passes. Related experience reaches this stage in 3 runs. It passes in 2 runs. The benchmark records reach rates separately from conditional pass rates. This distinction separates earlier execution barriers from failure at the current stage.

The final stage of the wooden shovel task is executable under the existing interface. We added a manual policy using ordinary actions. It completes all five original predicates in one episode. The policy uses the same world setting. Each stage remains within its original budget. Action counts follow the stage order: 139, 102, 29, 34, 66. These stages cover log collection, planks, sticks, the table, the shovel. The total is 370 actions. The final public inventory contains the shovel.

This complete record establishes the reachability of the final stage. The verifier records the success. Learned agents rarely reach this point. This limits our assessment of their later capabilities. The revision will separately report stage reach counts, pass counts, conditional pass rates. These details will make the limitation clearer.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Rebuttal Summary

Dear Area Chair and Reviewers,

Thank you for your careful reviews and helpful discussion. Your comments helped us improve the experiments, release materials, and explanation of MC-EvoBench. We added tests, checked the saved records, and gave more detail on the evaluation design. We summarize the main additions and the latest reviewer feedback below.

First, we updated the reproduction materials in response to Reviewer 8qVr. We released the updated code and reproduction package. We recomputed the metrics for 105 reports and 45 paired summaries from the original stage records. The results match Tables 3, 4, and 5. All 45 summary files match the original outputs. These materials give readers a clear way to check the reported results.

Second, we tested stronger models as planners in response to Reviewers c3yP and pCKb. We clarified that the original evaluation already included GPT and Gemini planners. We added 45 runs comparing three GPT generations across three task families and five agent seeds. Another comparison covered 12 primary task chains, two controllers, and 144 runs. The world, controller, and stage budgets stayed fixed within each comparison. These tests show that execution problems remain when stronger models serve as planners. We will state the model names and versions clearly in the revised Table 2.

Third, we checked successful source experience in response to Reviewers 8qVr and pCKb. Across the two tool tests, all ten planned comparisons had successful source records for collecting logs. Five also had successful records for making planks. All ten targets with related memory retrieved a successful log record first. These records let us test experience reuse at shared stages, even when the full source task is unfinished. We also traced the experience through memory retrieval, planner input, and controller instructions. This shows what experience was available and how the agent received it.

Fourth, we checked later task stages in response to Reviewers 8qVr and pCKb. A manual policy completed all five stages of the wooden shovel task in 370 ordinary actions. It started with an empty inventory. Every stage stayed within its original budget, and the existing checks recorded success. We also tested correct stage instructions and a complete demonstration through the original memory interface. These tests showed no gain with the fixed controller. In the revision, we will report how often each stage is reached and passed separately. This will make the results at later stages easier to inspect.

Fifth, we explained our choice of Minecraft in response to Reviewer pCKb. Minecraft recipes let us change the target goal while keeping earlier steps shared. Resetting the world and inventory lets us test saved experience without resources left by earlier runs. We expanded the discussion of previous work in Minecraft and robotics. We also mapped key parts of the protocol to robot simulators. These include scene resets, saved agent state, and checks for each task stage. The reported performance results concern the Minecraft agents we tested.

The latest reviewer comments show the progress made during the discussion. Reviewer c3yP wrote, “I maintain my original positive assessment of the paper.” Reviewer 8qVr stated, “The updated release addresses my main concern about reproducing the reference experiments.” This reviewer raised their rating. Reviewer pCKb maintained their overall assessment. They also noted that they may have missed some related work, especially studies on Minecraft. After reading our additional response, they lowered their confidence in their negative assessment.

We will include the new results and clearer explanations in the revised paper. We believe these additions make MC-EvoBench easier to check and use. We hope the added evidence and the benchmark’s contribution support a positive final assessment. Thank you again for your time and careful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.