acceptodds
Under review as a conference paper at ICLR 2027

Mine Odyssey: Benchmarking Spatial Agentic Intelligence in the Wild

Abstract

Advances in foundation models are driving efforts to introduce agents to assist people in the physical world. Such agents require agentic spatial intelligence: exploring unfamiliar environments, updating spatial understanding through interaction, and adapting actions based on feedback to sustain progress toward a sequence of goals. Evaluating these capabilities requires environments that reflect the diversity and structural complexity of real-world places, yet existing benchmarks typically cover only a limited range of spatial layouts, scales, and traversal requirements. To address this gap, we introduce Mine Odyssey, a benchmark for evaluating agentic spatial intelligence using Minecraft reconstructions of real-world locations. It comprises 180 tasks covering 30 such locations across 20 countries and regions on five continents, including 20 outdoor and 10 indoor settings. These settings are selected to provide broad coverage across spatial scales, layouts, terrains, and connectivity patterns, ranging from the urban streets of Midtown Manhattan to the rural Entrup, and from the steep terrain of Santa Luc´ıa Hill to Buckingham Palace. Within each reconstruction, we select meaningful locations as waypoints (e.g., historic landmarks, buildings, and rooms) and manually verify their accessibility. Each task provides a natural-language instruction specifying which waypoints to visit and in what order. Completing these tasks requires agents to find accessible routes and entrances, open doors, and move between levels using stairs and ladders, while monitoring their progress and recovering from navigation errors. Across eight evaluated state-of-the-art models, GPT-6 Astra achieves the highest success rate of 85.6%. However, the second-best model, Claude Opus 5, completes only 50.6% of tasks, while the strongest evaluated open-weight model, DeepSeek-V4.1-Flash, reaches 23.9%, highlighting substantial room for improvement in the agentic spatial intelligence of current models. Comprehensive analyses and ablation studies on Mine Odyssey reveal current models’ limitations and provide insights for advancing agentic spatial intelligence.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Spatial Intellgence, Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rwMuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MineOdyssey, a Minecraft-based benchmark for evaluating long-horizon spatial agentic intelligence in large, reconstructed real-world environments. It contains 180 ordered multi-waypoint tasks across 30 indoor and outdoor maps, together with a programmable agent interface and automated progress verification. Experiments across several frontier multimodal models reveal substantial performance differences, while qualitative examples illustrate several recurring failure patterns. Overall, this paper appears carefully implemented and potentially useful but can be improved in some aspects. My main concern is that the current analysis does not establish which underlying capabilities are responsible for these failures or for differences in aggregate performance. The benchmark could become a useful testbed for studying general-purpose embodied agents, although its construct validity and difficulty calibration require stronger evidence.

Critical strengths
  1. Diverse environments and task design. The benchmark covers diverse indoor and outdoor environments with varied spatial structures, while its multi-waypoint tasks introduce meaningful long-horizon planning and sequential constraints.
  2. Comprehensive evaluation framework and qualitative analysis. Besides the benchmark performance, the work provides sandboxed execution, automatic checkpoint verification, broad model evaluation, ablation studies, failure analysis, and clear qualitative visualizations.
Critical weaknesses
  1. Unclear interpretation of the aggregate score. Success rate jointly depends on perception, map reading, localization, planning, tool use, memory, low-level control, and error recovery. Because agents can access a global map, textual state information, and programmable tools, the current experiments do not establish which capabilities the benchmark score actually measures. Existing ablations mainly vary environmental or interface conditions rather than isolating these underlying abilities. Without further capability decomposition, it remains unclear to what extent aggregate performance can be interpreted specifically as agentic spatial intelligence rather than broader end-to-end agent competence.
  2. Limited characterization of benchmark quality and difficulty calibration. Although the paper describes LLM-assisted task generation, human refinement, reachability testing, and manual traversal, the resulting task set is not quantitatively characterized in sufficient detail. In particular, the paper provides limited information about the validation process and task-difficulty distribution. Task counts also vary substantially across maps, but the allocation criteria and the sensitivity of aggregate results to this imbalance are not reported. Finally, a human or oracle baseline would help calibrate benchmark difficulty and remaining headroom, particularly given GPT-6 Astra’s 85.56% success rate.
Questions for authors
  1. Could the authors provide controlled ablations for some of the major information and action channels, e.g. the global map, coordinate access, programmable tools, or interaction history, to clarify what capabilities primarily determine benchmark performance?
  2. How are task counts assigned to different maps? Would the main conclusions remain unchanged when performance is macro-averaged across maps rather than averaged across individual tasks?
  3. Could the authors provide additional information about the task validation process and report task-difficulty statistics, such as optimal path length, number of waypoints, elevation or floor changes, and required interactions?
  4. What performance do human experts or an oracle navigation system achieve? Such a baseline would help calibrate task difficulty and remaining benchmark headroom.
4 comments
AuthorsOpenReview ↗

Response to Reviewer rwMu(1/2)

We thank the reviewer for the helpful suggestions on controlled ablations, benchmark analysis, and task-difficulty calibration.

W1: Regarding the interpretation of benchmark scores

Regarding the interpretation of benchmark scores, please see our response to Q1, where we report controlled information- and action-channel ablations and discuss what they reveal about benchmark performance.

W2: Regarding the difficulty calibration

Regarding benchmark quality and difficulty calibration, please see Q2 for task allocation and map-macro evaluation, Q3 for validation and task-difficulty statistics, and Q4 for human-baseline calibration.

Q1. Information and action channel ablations

We evaluated four information and action-channel interventions with Astra and GLM as follows:

  • No direct position readouts: remove player-position fields from API responses and coordinate-bearing debug/HUD displays.

  • Fixed-command execution: reject arbitrary Python/shell programs, loops, substitutions, and general-purpose computation.

  • Map unavailable, waypoints kept: disable the global map and minimap while keeping first-person waypoint markers visible without the default distance limit.

  • No interaction-history summary (No summary): retain within-window interaction history, but reset at the default thresholds without generating or retaining a summary.

The following table shows the results:

ModelConditionSR (%)CC (%)SPL (%)
AstraNo direct position readouts84.12 (-1.44)89.32 (+0.28)52.35 (-1.35)
AstraFixed-command execution82.46 (-3.10)88.66 (-0.38)50.88 (-2.82)
AstraMap unavailable, waypoints kept79.66 (-5.90)85.50 (-3.54)43.82 (-9.88)
AstraNo summary77.33 (-8.23)85.05 (-3.99)47.85 (-5.85)
GLMNo direct position readouts15.08 (-0.48)34.16 (-1.93)6.45 (+0.73)
GLMFixed-command execution18.44 (+2.88)39.93 (+3.84)9.12 (+3.40)
GLMMap unavailable, waypoints kept9.50 (-6.06)32.63 (-3.46)3.79 (-1.93)
GLMNo summary10.23 (-5.33)31.45 (-4.64)3.64 (-2.08)

Across interventions, removing the map or cross-window summaries causes the largest drops for both Astra and GLM. By contrast, removing direct position readouts, disabling general-purpose programming, or restricting control to raw keyboard/mouse events has a much smaller impact.

This indicates that benchmark performance is primarily driven by spatial layout comprehension, map-guided visual navigation, and long-horizon progress tracking rather than coordinate arithmetic or arbitrary scripting. We will include these ablation results in the revised manuscript.

Q2. Task allocation and map-macro averaging

Tasks are manually selected and refined for semantic coherence, route plausibility, and diversity, while limiting repeated ordered waypoint pairs within each map (Sec. 2.1). Consequently, maps contain 1–12 tasks, with both the mean and median equal to 6.

We recomputed metrics by first averaging within each map and then giving all 30 maps equal weight. The comparison below shows that the success-rate ranking of the main models remains unchanged, confirming that model ordering is robust to equal map weighting:

ModelTask SRMap SRTask CCMap CCTask SPLMap SPL
Astra85.5688.0089.0491.0653.7054.67
Opus 5.573.8975.2286.1187.3450.3549.41
Opus 550.5649.2667.5366.3424.2923.25
Sol43.8943.7161.3659.9022.8723.25
Gemini26.6726.8656.3256.0516.9116.77
DeepSeek23.8922.4644.3744.1111.6210.95
Grok17.2221.5440.3642.857.849.91
GLM15.5614.6336.0936.855.725.39
AuthorsOpenReview ↗

Response to Reviewer rwMu(2/2)

Q3. Validation and difficulty statistics

During benchmark construction, all 180 tasks across the 20 outdoor and 10 indoor worlds were validated through a three-step process: (1) waypoint and boundary checks to ensure valid, unobstructed target locations on the correct floor; (2) automated offline route verification on the approximate movement graph; and (3) full manual in-game traversal confirming reachability and required structural interactions (locating entrances, opening doors, and traversing multilevel stairs and ladders).

To quantify task difficulty, we score each task along three structural dimensions: reference path length (≤300 / >300–600 / >600 blocks), ordered destination count excluding the start (≤3 / 4 / ≥5), and maximum consecutive-stop elevation difference (≤3 / >3–10 / >10 blocks), each contributing 0, 1, or 2 points. Total scores of 0–2, 3–4, and 5–6 define low, medium, and high difficulty, respectively:

DifficultyTasks (%)Mean Path Length (blocks)Median Path Length (blocks)Mean |Δy| (blocks)
Low79 (43.89%)280.34227.542.07
Medium82 (45.56%)788.22612.614.88
High19 (10.56%)930.11854.396.15

Across all 180 tasks, ordered destinations excluding the start average 3.71 (median 4; range 1–11). Detailed breakdowns of destination counts, elevation transitions, and structural interactions observed across the 180 baseline trajectories are summarized below:

Dimension / Interaction CategoryTasksShare (%)
1–3 ordered destinations84/18046.67
4 ordered destinations51/18028.33
≥5 ordered destinations45/18025.00
Nonzero consecutive-stop |Δy| (>0 blocks)165/18091.67
Consecutive-stop |Δy| >3 blocks103/18057.22
Door / gate interactions41/18022.78
Button interactions14/1807.78
Trapdoor interactions16/1808.89

Together, these statistics show that Mine Odyssey spans a broad spectrum of spatial scales, multi-stop horizons, vertical transitions, and structural interactions. We will include this validation breakdown and the full per-dimension difficulty distributions in the revised manuscript.

Q4. Human/oracle calibration

To calibrate task difficulty and benchmark headroom, we evaluated two participants with regular Minecraft experience across the benchmark:

MetricSR (%)CC (%)SPL (%)
Human68.0082.6750.24

Human participants achieve 68.00% SR, 82.67% CC, and 50.24% SPL, outperforming six of the eight evaluated models and trailing only GPT-6 Astra and Claude Opus 5.5. Meanwhile, even the strongest models reach only 50.35%–53.70% SPL (on par with the human 50.24% SPL), showing that while top proprietary models surpass human completion rates, most models still lag significantly behind humans and substantial headroom remains in route efficiency across all models.

Thank you for your efforts in responding to my comments. My concerns have been partially addressed. However, answering Q1 does not fully address Weakness 1. A post hoc explanation of the benchmark scores does not directly show that performance is mainly driven by spatial layout understanding, map-guided visual navigation, and long-horizon progress tracking. Both “Map unavailable, waypoints kept” and “No summary” have negative effects, suggesting that navigation metrics alone cannot adequately measure Spatial Agentic Intelligence. In addition, the human calibration lacks details. Why do humans perform significantly worse than GPT-6 Astra and Claude Opus 5.5? Therefore, I maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer rwMu

Dear reviewer,

We thank the reviewer for raising this concern and providing an opportunity to make further discussions.

Regarding the interpretation of benchmark scores

We define agentic spatial intelligence as agents' ability to proactively acquire and continually update spatial information through interaction, use it to guide observation and action, and adapt those actions as new evidence emerges. This requires agents to integrate spatial understanding, planning, memory, and tool use to accomplish long-horizon spatial goals. In this process, map access and cross-window summarization serve important supporting roles. The former provides spatial information for navigation planning, while the latter helps retain spatial observations and task progress over extended interactions. Both mechanisms are closely related to the capabilities being evaluated, rather than merely external factors affecting performance. Consistent with these roles, our ablations also show that removing either mechanism leads to relative larger performance drops. In comparison, restricting direct coordinate readouts or programming tools has smaller impacts, suggesting that these additional affordances are less critical to task completion.

Similar evaluation practices are also adopted in established domain-specific agent benchmarks. For example, the Terminus 2 harness for Terminal-Bench incorporates automatic context summarization (Merrill et al., 2026); and OSWorld supports Python-based actions and structured observations such as accessibility trees (Xie et al., 2024). These benchmarks also assess integrated agentic capabilities take task-level success rate as evaluation metrics. So we think our evaluation can also provide a reasonable measure in spaital environments.

We will add this discussion in our revised manuscript.

[1]. Merrill, M. A., Shaw, A. G., et al. (2026). Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. arXiv preprint arXiv:2601.11868.

[2]. Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., & Yu, T. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Advances in Neural Information Processing Systems, 37.

Regarding Human evals.

Human evals are under the same task settings and scoring rules as the models. Our evaluation results show the main failures come from recurring out-of-bounds failures following falls and unsuccessful recovery, as well as premature completion claims. Multilingual instructions introduce an additional language-comprehension requirement. We will invite more participants for a more rigorous human evaluation.

Reviewer_zw3dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Mine Odyssey, a benchmark for long-horizon spatial navigation in unfamiliar 3D environments built from Minecraft reconstructions of real places. It includes 180 tasks across 30 locations, requiring agents to visit multiple semantic waypoints in order, gather spatial information, and adapt to obstacles and navigation errors. GPT-6 Astra achieves an 85.6% success rate, compared with 50.6% for Claude Opus 5 and 23.9% for DeepSeek-V4.1-Flash, the strongest open-weight model. Most failures occur before reaching the first waypoint, highlighting difficulties in initial orientation and recovery from unproductive local exploration.

Critical strengths
  1. The main contribution is one repeatable embodied benchmark that combines diverse real-world structures, long-horizon closed-loop decisions, tool use, and automatic position checks. Scale, terrain, connectivity, and vertical movement are broader than home or single-city tests.
  2. The paper provides an interesting insight: Route guidance does not necessarily raise success. A HUD, third-person view, or multi-view helps only a little. The bottleneck is not only missing information, but turning information into stable actions and recovering from local failure.
  3. Tasks are long-horizon and compositional: several places must be visited in order, not a single goal, with a closed loop evaluation, the agent moves, observes, acts, and recovers, and an independent position sampler scores the result rather than an LLM judge. The sandbox blocks hidden coordinates, teleport, Baritone auto-navigation, and external network access.
Critical weaknesses
  1. “In the wild” is a bit overclaiming. These are hand-built Minecraft reconstructions. Vision, physics, interaction, and maps are much cleaner than in a real setting.
  2. The best model is already at 85.6%, so the current tasks are near saturation for the strongest system; meanwhile, the results depend heavily on one proprietary model.
Questions for authors
  1. Limit the claim to open-ended Minecraft navigation in real-place reconstructions, and state the gap to real-world embodiment.
  2. Should add more baselines, such as skilled humans, Baritone, a classical hierarchical navigator, and different planners.
  3. Should hold out some private new maps as a server-side test set, and freeze world saves, mods, and configs by version, to avoid leakage in model training.
1 comment
AuthorsOpenReview ↗

Response to Reviewer zw3d (2/2)

Q2. Additional experiments

Our existing route-guidance experiment in Sec. 4.1 already uses Baritone to compute and visualize routes while leaving movement and interaction under agent control. For reference, the Table 2 comparison is reproduced below (values in parentheses indicate changes relative to the Table 1 main baselines):

ModelSR (%)CC (%)SPL (%)
DeepSeek21.11 (-2.78)46.32 (+1.95)9.72 (-1.90)
GLM18.33 (+2.77)41.77 (+5.68)7.99 (+2.27)

Following the reviewer’s suggestion, we added an HPA*-guided agent to examine hierarchical navigation and an FMM-guided agent as an alternative planning approach. HPA* (Hierarchical Path-Finding A*) [1, 3] plans through a hierarchy of local regions, whereas FMM (Fast Marching Method) [2, 4, 5] derives routes through arrival-time propagation. In both settings, a classical planner automatically computes and visualizes routes toward the ordered waypoints, while the LLM observes the guidance in screenshots and retains responsibility for movement and interaction. This follows the same guidance-and-execution setup as our existing Baritone ablation; no additional planner tool is exposed to the LLM.

HPA* and FMM share the same movement graph, a ±48-block horizontal search window, a 30,000-node limit, and a 20-second budget per query. Both support partial routes and replanning. These budgets were not imposed on the original Baritone experiment.

The results are as follows:

ModelConditionSR (%)CC (%)SPL (%)
GLMHPA*11.93 (-3.63)31.63 (-4.46)5.26 (-0.46)
GLMFMM12.50 (-3.06)37.25 (+1.16)5.22 (-0.50)
AstraHPA*87.01 (+1.45)91.28 (+2.24)53.87 (+0.17)
AstraFMM84.00 (-1.56)86.58 (-2.46)50.21 (-3.49)

Across all three classical planners, adding external route guidance yields at most marginal gains and otherwise slightly lowers end-to-end SR and SPL, even when intermediate checkpoint coverage increases modestly. Consistent with Sec. 4.1, this shows that local path planning over known terrain is not the sole bottleneck; successful navigation still depends on high-level spatial grounding, translating visual guidance into reliable movement and interactions (such as opening doors and traversing stairs or ladders), and recovering from local execution failures.

We also evaluated two human participants with regular Minecraft experience across the benchmark:

MetricSR (%)CC (%)SPL (%)
Human68.0082.6750.24

Human participants achieve 68.00% SR, 82.67% CC, and 50.24% SPL, outperforming six of the eight evaluated models and trailing only GPT-6 Astra and Claude Opus 5.5. Meanwhile, even the strongest models reach only 50.35%–53.70% SPL (on par with the human 50.24% SPL), showing that while top proprietary models surpass human completion rates, most models still lag significantly behind humans and substantial headroom remains in route efficiency across all models.

Q3. Private maps and version freezing

Reproducibility controls are already built into our setup: Sec. 2.1 describes the versioned world saves and specifies the exact Minecraft and mod versions, while Appendix C.1 (Table 4) documents the fixed evaluation settings. Together, these controls ensure consistent initial environments across evaluations.

We agree that reproducibility and training contamination are separate issues. To address potential contamination, we will expand the benchmark with additional held-out private maps and tasks.

Reference:

[1] Botea et al. Near Optimal Hierarchical Path-Finding, 2004.

[2] Sethian. A Fast Marching Level Set Method for Monotonically Advancing Fronts, 1996.

[3] Wibawa et al. Evaluating Hierarchical Pathfinding A* (HPA*) for Multi-Order Routing in a Small Warehouse Layout. Cauchy, 2026.

[4] Gao et al. DRIVE-Nav: Directional Reasoning, Inspection, and Verification for Efficient Open-Vocabulary Navigation. arXiv:2603.28691, 2026.

[5] Pan et al. One MLLM, One Call: Efficient Zero-Shot Vision-and-Language Navigation via Spatial-Aware Waypoints. arXiv:2609.06476, 2026.

Reviewer_y3thOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduce an novel benchmark called Mine Odyssey, which comprises 180 navigation tasks from 30 Minecraft reconstructions of real-world locations. The results reveal that GPT-6-Astra is the best model, while the performance of other models highlight substantial room for improvement in such evaluation setting.

Critical strengths

The works presents an very interesting evaluation setting, and demonstrates great engineering practice to make a feasible environment, which provides a useful reference for future research in the community.

Critical weaknesses

The evaluation focuses predominantly on navigation tasks. Expanding the benchmark to cover a broader range of tasks would make the contribution more valuable.

Questions for authors
  • the control flow based on mcapi largely simpify the control of the Minecraft environment, has the author trys to evaluate agents exceptionally with xdo,to comprehensively evaluates the capability of agents on visual intelligence and subject manipulation?

  • During the evaluation, how many Minecraft instances are initilialized? and how many agent rollouts are simutaneously sampled? What is the hardware requirements for running such evaluation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer y3th

We thank the reviewer for the helpful suggestions on the evaluation and experimental setup.

W1 Regarding more tasks

We agree that a broader range of tasks would further increase the benchmark’s value. While navigation is our primary focus, completing these tasks can also involve environmental interactions, such as opening doors and gates, activating buttons, and negotiating trapdoors, stairs, and ladders. These interactions connect spatial understanding with action during multi-goal navigation. We will extend the benchmark to manipulation, crafting, and construction as a valuable direction for future work.

Q1. Regarding xdo-only evaluation

Following your suggestion, we added an xdo-only ablation in which agents observe screenshots and control movement, camera orientation, and interactions through keyboard/mouse events, with mcapi controls and game-state APIs disabled.

ModelConditionSR (%)CC (%)SPL (%)
Astraxdo-only82.76 (-2.80)86.34 (-2.70)52.37 (-1.33)
GLMxdo-only7.26 (-8.30)25.02 (-11.07)2.99 (-2.73)

Under xdo-only control, Astra retains an 82.76% SR (down 2.80 points from 85.56%), whereas GLM's SR drops from 15.56% to 7.26%. This indicates that strong models can navigate and interact directly via low-level keyboard/mouse actions, while weaker models rely heavily on mcapi for low-level control.

Q2. Regarding Instances, concurrency, and hardware

Concurrency is jointly constrained by CPU/GPU rendering capacity, system memory, and the model provider's request/token limits and response latency. Each model is evaluated across 180 task instances, with one agent rollout per isolated Minecraft environment; rather than running all 180 instances simultaneously, we configure each batch to run up to 100 concurrent rollouts. Our server is equipped with two 128-core AMD CPUs (256 physical cores in total), approximately 1 TiB of RAM, and eight NVIDIA A800 (80 GB) GPUs, with the concurrent rollouts distributed across the eight GPUs. Each instance is allocated a 2-vCPU quota and an 8 GiB memory limit (including 2 GiB of shared memory), with GPU-backed OpenGL rendering at 800×600.

I thank the authors for their response, and I will maintain my positive rating of the work.

AuthorsOpenReview ↗

Response to Reviewer y3th

Dear Reviewer y3th,

We sincerely thank your thoughtful suggestions and support of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.