- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MineOdyssey, a Minecraft-based benchmark for evaluating long-horizon spatial agentic intelligence in large, reconstructed real-world environments. It contains 180 ordered multi-waypoint tasks across 30 indoor and outdoor maps, together with a programmable agent interface and automated progress verification. Experiments across several frontier multimodal models reveal substantial performance differences, while qualitative examples illustrate several recurring failure patterns. Overall, this paper appears carefully implemented and potentially useful but can be improved in some aspects. My main concern is that the current analysis does not establish which underlying capabilities are responsible for these failures or for differences in aggregate performance. The benchmark could become a useful testbed for studying general-purpose embodied agents, although its construct validity and difficulty calibration require stronger evidence.
Critical strengths
- Diverse environments and task design. The benchmark covers diverse indoor and outdoor environments with varied spatial structures, while its multi-waypoint tasks introduce meaningful long-horizon planning and sequential constraints.
- Comprehensive evaluation framework and qualitative analysis. Besides the benchmark performance, the work provides sandboxed execution, automatic checkpoint verification, broad model evaluation, ablation studies, failure analysis, and clear qualitative visualizations.
Critical weaknesses
- Unclear interpretation of the aggregate score. Success rate jointly depends on perception, map reading, localization, planning, tool use, memory, low-level control, and error recovery. Because agents can access a global map, textual state information, and programmable tools, the current experiments do not establish which capabilities the benchmark score actually measures. Existing ablations mainly vary environmental or interface conditions rather than isolating these underlying abilities. Without further capability decomposition, it remains unclear to what extent aggregate performance can be interpreted specifically as agentic spatial intelligence rather than broader end-to-end agent competence.
- Limited characterization of benchmark quality and difficulty calibration. Although the paper describes LLM-assisted task generation, human refinement, reachability testing, and manual traversal, the resulting task set is not quantitatively characterized in sufficient detail. In particular, the paper provides limited information about the validation process and task-difficulty distribution. Task counts also vary substantially across maps, but the allocation criteria and the sensitivity of aggregate results to this imbalance are not reported. Finally, a human or oracle baseline would help calibrate benchmark difficulty and remaining headroom, particularly given GPT-6 Astra’s 85.56% success rate.
Questions for authors
- Could the authors provide controlled ablations for some of the major information and action channels, e.g. the global map, coordinate access, programmable tools, or interaction history, to clarify what capabilities primarily determine benchmark performance?
- How are task counts assigned to different maps? Would the main conclusions remain unchanged when performance is macro-averaged across maps rather than averaged across individual tasks?
- Could the authors provide additional information about the task validation process and report task-difficulty statistics, such as optimal path length, number of waypoints, elevation or floor changes, and required interactions?
- What performance do human experts or an oracle navigation system achieve? Such a baseline would help calibrate task difficulty and remaining benchmark headroom.
4 comments
Response to Reviewer rwMu(2/2)
Q3. Validation and difficulty statistics
During benchmark construction, all 180 tasks across the 20 outdoor and 10 indoor worlds were validated through a three-step process: (1) waypoint and boundary checks to ensure valid, unobstructed target locations on the correct floor; (2) automated offline route verification on the approximate movement graph; and (3) full manual in-game traversal confirming reachability and required structural interactions (locating entrances, opening doors, and traversing multilevel stairs and ladders).
To quantify task difficulty, we score each task along three structural dimensions: reference path length (≤300 / >300–600 / >600 blocks), ordered destination count excluding the start (≤3 / 4 / ≥5), and maximum consecutive-stop elevation difference (≤3 / >3–10 / >10 blocks), each contributing 0, 1, or 2 points. Total scores of 0–2, 3–4, and 5–6 define low, medium, and high difficulty, respectively:
| Difficulty | Tasks (%) | Mean Path Length (blocks) | Median Path Length (blocks) | Mean |Δy| (blocks) |
|---|---|---|---|---|
| Low | 79 (43.89%) | 280.34 | 227.54 | 2.07 |
| Medium | 82 (45.56%) | 788.22 | 612.61 | 4.88 |
| High | 19 (10.56%) | 930.11 | 854.39 | 6.15 |
Across all 180 tasks, ordered destinations excluding the start average 3.71 (median 4; range 1–11). Detailed breakdowns of destination counts, elevation transitions, and structural interactions observed across the 180 baseline trajectories are summarized below:
| Dimension / Interaction Category | Tasks | Share (%) |
|---|---|---|
| 1–3 ordered destinations | 84/180 | 46.67 |
| 4 ordered destinations | 51/180 | 28.33 |
| ≥5 ordered destinations | 45/180 | 25.00 |
| Nonzero consecutive-stop |Δy| (>0 blocks) | 165/180 | 91.67 |
| Consecutive-stop |Δy| >3 blocks | 103/180 | 57.22 |
| Door / gate interactions | 41/180 | 22.78 |
| Button interactions | 14/180 | 7.78 |
| Trapdoor interactions | 16/180 | 8.89 |
Together, these statistics show that Mine Odyssey spans a broad spectrum of spatial scales, multi-stop horizons, vertical transitions, and structural interactions. We will include this validation breakdown and the full per-dimension difficulty distributions in the revised manuscript.
Q4. Human/oracle calibration
To calibrate task difficulty and benchmark headroom, we evaluated two participants with regular Minecraft experience across the benchmark:
| Metric | SR (%) | CC (%) | SPL (%) |
|---|---|---|---|
| Human | 68.00 | 82.67 | 50.24 |
Human participants achieve 68.00% SR, 82.67% CC, and 50.24% SPL, outperforming six of the eight evaluated models and trailing only GPT-6 Astra and Claude Opus 5.5. Meanwhile, even the strongest models reach only 50.35%–53.70% SPL (on par with the human 50.24% SPL), showing that while top proprietary models surpass human completion rates, most models still lag significantly behind humans and substantial headroom remains in route efficiency across all models.
Thank you for your efforts in responding to my comments. My concerns have been partially addressed. However, answering Q1 does not fully address Weakness 1. A post hoc explanation of the benchmark scores does not directly show that performance is mainly driven by spatial layout understanding, map-guided visual navigation, and long-horizon progress tracking. Both “Map unavailable, waypoints kept” and “No summary” have negative effects, suggesting that navigation metrics alone cannot adequately measure Spatial Agentic Intelligence. In addition, the human calibration lacks details. Why do humans perform significantly worse than GPT-6 Astra and Claude Opus 5.5? Therefore, I maintain my original rating.
Response to Reviewer rwMu
Dear reviewer,
We thank the reviewer for raising this concern and providing an opportunity to make further discussions.
Regarding the interpretation of benchmark scores
We define agentic spatial intelligence as agents' ability to proactively acquire and continually update spatial information through interaction, use it to guide observation and action, and adapt those actions as new evidence emerges. This requires agents to integrate spatial understanding, planning, memory, and tool use to accomplish long-horizon spatial goals. In this process, map access and cross-window summarization serve important supporting roles. The former provides spatial information for navigation planning, while the latter helps retain spatial observations and task progress over extended interactions. Both mechanisms are closely related to the capabilities being evaluated, rather than merely external factors affecting performance. Consistent with these roles, our ablations also show that removing either mechanism leads to relative larger performance drops. In comparison, restricting direct coordinate readouts or programming tools has smaller impacts, suggesting that these additional affordances are less critical to task completion.
Similar evaluation practices are also adopted in established domain-specific agent benchmarks. For example, the Terminus 2 harness for Terminal-Bench incorporates automatic context summarization (Merrill et al., 2026); and OSWorld supports Python-based actions and structured observations such as accessibility trees (Xie et al., 2024). These benchmarks also assess integrated agentic capabilities take task-level success rate as evaluation metrics. So we think our evaluation can also provide a reasonable measure in spaital environments.
We will add this discussion in our revised manuscript.
[1]. Merrill, M. A., Shaw, A. G., et al. (2026). Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. arXiv preprint arXiv:2601.11868.
[2]. Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., & Yu, T. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Advances in Neural Information Processing Systems, 37.
Regarding Human evals.
Human evals are under the same task settings and scoring rules as the models. Our evaluation results show the main failures come from recurring out-of-bounds failures following falls and unsuccessful recovery, as well as premature completion claims. Multilingual instructions introduce an additional language-comprehension requirement. We will invite more participants for a more rigorous human evaluation.
Response to Reviewer rwMu(1/2)
We thank the reviewer for the helpful suggestions on controlled ablations, benchmark analysis, and task-difficulty calibration.
W1: Regarding the interpretation of benchmark scores
Regarding the interpretation of benchmark scores, please see our response to Q1, where we report controlled information- and action-channel ablations and discuss what they reveal about benchmark performance.
W2: Regarding the difficulty calibration
Regarding benchmark quality and difficulty calibration, please see Q2 for task allocation and map-macro evaluation, Q3 for validation and task-difficulty statistics, and Q4 for human-baseline calibration.
Q1. Information and action channel ablations
We evaluated four information and action-channel interventions with Astra and GLM as follows:
No direct position readouts: remove player-position fields from API responses and coordinate-bearing debug/HUD displays.
Fixed-command execution: reject arbitrary Python/shell programs, loops, substitutions, and general-purpose computation.
Map unavailable, waypoints kept: disable the global map and minimap while keeping first-person waypoint markers visible without the default distance limit.
No interaction-history summary (No summary): retain within-window interaction history, but reset at the default thresholds without generating or retaining a summary.
The following table shows the results:
Across interventions, removing the map or cross-window summaries causes the largest drops for both Astra and GLM. By contrast, removing direct position readouts, disabling general-purpose programming, or restricting control to raw keyboard/mouse events has a much smaller impact.
This indicates that benchmark performance is primarily driven by spatial layout comprehension, map-guided visual navigation, and long-horizon progress tracking rather than coordinate arithmetic or arbitrary scripting. We will include these ablation results in the revised manuscript.
Q2. Task allocation and map-macro averaging
Tasks are manually selected and refined for semantic coherence, route plausibility, and diversity, while limiting repeated ordered waypoint pairs within each map (Sec. 2.1). Consequently, maps contain 1–12 tasks, with both the mean and median equal to 6.
We recomputed metrics by first averaging within each map and then giving all 30 maps equal weight. The comparison below shows that the success-rate ranking of the main models remains unchanged, confirming that model ordering is robust to equal map weighting: