- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
PRIME is a Minecraft agent that separates scheduling failures from capability-grounding failures. It combines a priority agenda that is rescored after every bounded skill call with a gated skill-evolution path, where generated Mineflayer programs are admitted only after passing at least 7 of 10 independent validation trials.
Critical strengths
-
The distinction between scheduling failures and capability-grounding failures is a clear and reasonable way to frame open-world execution. It gives a concrete criterion for when an agent should reschedule and when it should extend its skill library.
-
The evaluation uses external hard checkers shared across methods, and the paper reports its limitations openly. The skill admission protocol with independent validation trials is a careful design choice.
Critical weaknesses
-
The work is incomplete and the amount of experimental work is insufficient. The paper states that the matched no-seed ablation and component ablations are not measured, and the execution example in Figure 3 is a schematic, so no experiment tests the central claim about separating the two failure modes.
-
The difference from prior Minecraft agents is unclear. Voyager and Odyssey already provide skill libraries with verification and DEPS provides feedback-driven replanning, and the paper does not report whether online skill evolution was triggered in the formal episodes.
-
The evaluation only contains seven tasks, one world seed, and five trials per cell, reported only as thresholded pass/fail. With Gemini, PRIME_skill ties three baselines at 3/7.
Questions for authors
-
In the 735 formal episodes, how often was online skill evolution triggered, and how many new skills were admitted?
-
Please report the raw five-trial success rate for every cell. The majority threshold hides differences between 3/5 and 5/5.
-
How do the baselines perform when given the same 20-skill seed?
4 comments
Response 2 to Reviewer svpk
W3 & Q2: Multiple seeds and success rates without majority thresholding
1) Expanded replication
GPT uses five world-generation seeds and ten trials per seed per round. Gemini and Grok use three world seeds and five trials per seed per round. Each independent experiment runs two seven-task curricula. Task worlds reset, while validated skills accumulate only within the corresponding experiment and carry into its second round. Separate libraries prevent cross-experiment injection.
2) Raw success counts underlying the revised comparison
Table 1 reports trial-level success percentages, and Appendix Table 4 gives aggregate success counts. For the strongest baseline and PRIME conditions:
| Condition | GPT | Gemini | Grok |
|---|---|---|---|
| Highest-SR baseline | 451/700 | 102/210 | 118/210 |
| 249/350 | 58/105 | 69/105 | |
| 307/350 | 82/105 | 96/105 |
Baseline rows aggregate both rounds; each PRIME row reports one round. The counts above aggregate across tasks. The majority threshold no longer obscures differences in the main comparison. For Gemini, the strongest baseline achieves 48.6%, compared with 55.2% for PRIME₁ and 78.1% for PRIME₂.
Across all three backbones, exceeds the strongest baseline by 23.3–35.2 percentage points and achieves the lowest penalized active and wall times. The abstract and main-result analysis now reflect this expanded evaluation.
Q3: Matched initial skill availability
The revised evaluation addresses this concern by removing PRIME's additional pre-populated skills. All baselines, PRIME_sch, and PRIME₁ now start from the same frozen base skill library, with PRIME's learned-skill store initially empty. The comparison therefore aligns initial skill availability rather than retaining the previous 20-skill advantage.
Table 3 shows a consistent ordering in task success across all three backbones: best baseline < PRIME_sch < PRIME₁ < PRIME₂. Scheduling alone improves success over the strongest baseline by 3.6–6.7 percentage points; enabling online skill acquisition adds 1.9–3.1 points in Round 1; inheriting the skills validated during that experiment's Round 1 adds a further 16.6–25.7 points in Round 2. Overall, PRIME₂ exceeds the strongest baseline by 23.3–35.2 points. These results demonstrate gains from scheduling under matched initial skills, followed by additional benefits from online acquisition and persistent reuse.
Official Comment by Authors
Dear Reviewer svpk,
Thank you again for your time and effort in reviewing our submission.
As the discussion period will end in 24 hours, we hope you can review our rebuttal and share your feedback on whether our responses have addressed your concerns. We would be happy to clarify any remaining questions or issues.
Thank you for your time and consideration.
Dear Reviewer svpk,
Thank you again for your time and effort in reviewing our submission.
As the discussion period will end in 2 hours, we hope you can review our rebuttal and share your feedback on whether our responses have addressed your concerns. We would be happy to clarify any remaining questions or issues.
Thank you for your time and consideration.
Response 1 to Reviewer svpk
Thank you for your constructive feedback. We have expanded the evaluation across seven tasks, six baselines, and three backbones with additional world seeds and repeated trials over two rounds. GPT uses five seeds and ten trials per seed per round; Gemini and Grok each use three seeds and five trials per seed per round. We also added component ablations and analyses of skill evolution, reuse, and execution success. These additions are incorporated into the revised paper, with changes highlighted in blue.
W1: Component evidence and recorded examples of the two recovery paths
1) Scheduling, online acquisition, and persistence comparisons
Section 3.4 and Table 3 compare the strongest baseline with scheduling-only PRIMEsch, first-round PRIME1, and inherited-library PRIME2. Values are task success rates (%).
Scheduling alone improves success by 3.6–6.7 percentage points over the strongest baseline. Adding the skill module yields another 1.9–3.1 points in the first round. With the controller and evaluation setting unchanged, retaining validated skills improves second-round SR by 16.6–25.7 points. The scheduling-only condition uses five seeds for GPT and three for Gemini/Grok, with five trials per seed; the full conditions follow the main protocol.
2) Recorded executions beyond the schematic
Figure 3 remains explicitly labeled as a schematic. Section 3.5 adds two recorded GPT executions, with step-level evidence in Appendix E:
The first case changes execution priority; the second introduces a new execution behavior. In Task 1, the recovery capability already exists, and the scheduler preserves inventory progress while bringing it forward. In Task 3, PRIME generates a candidate that excludes the failed mining target and searches other deposits. It passes 10/10 independent validation trials and then obtains coal in the live task. These traces illustrate the two recovery paths; the component comparisons evaluate their contribution to task performance.
W2 & Q1: Online evolution and the distinction from existing agents
1) Evolution is measured in the revised evaluation
Table 2 reports episode-level triggers and subsequent reuse/call statistics. All values are percentages.
Evolution becomes less frequent as the validated library accumulates, while task success increases. Later-episode reuse and live call success show that acquired capabilities remain useful beyond their creation episodes. These results concern the expanded evaluation, not the original 735-episode cohort. Trigger rates measure episodes invoking evolution; they are not counts of distinct admitted skills.
2) Priority-conditioned acquisition and independent admission
DEPS combines feedback-driven planning with subgoal ranking by estimated completion steps [1]. Voyager develops reusable programs through iterative prompting and self-verification [2], while Odyssey provides primitive and compositional skills [3]. PRIME connects these concerns through an explicit runtime decision: after each bounded call, it reprioritizes feasible subgoals; synthesis becomes eligible only when an evidence-revised agenda retains a necessary subgoal whose prerequisites are met but which lacks suitable executable grounding. Persistent admission requires at least seven successes in ten independent, environment-checked validation trials. The contribution is the connection between runtime priority selection, capability acquisition, and independently validated reuse, clarified in Section 5.
References
[1] Wang, Z., et al. Describe, Explain, Plan and Select: Interactive Planning with LLMs Enables Open-World Multi-Task Agents. NeurIPS, 2023.
[2] Wang, G., et al. Voyager: An Open-Ended Embodied Agent with Large Language Models. TMLR, 2024.
[3] Liu, S., et al. Odyssey: Empowering Minecraft Agents with Open-World Skills. IJCAI, 2025.