- Rating
- 2
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces NetHack+, a configurable engine with checkpointing, flexible agent interfaces, and faster simulation. It also proposes BALmin to measure balanced progression and Prime-Agent-Explore for orchestrator-guided checkpoint search. Experiments show improved progression in NetHack and three additional game environments.
Critical strengths
-
Configurable mechanics and checkpointing enable controlled studies of agent failures and search strategies.
-
Removing checkpoint selection or guidance reduces performance, supporting the value of both components.
Critical weaknesses
-
The experiments do not sufficiently distinguish long-term planning from local survival, game knowledge, and interface improvements. BALmin’s relationship to eventual success is also not established.
-
The headline gains compare multi-attempt checkpoint search with single-life baselines, making it difficult to separate methodological benefits from additional computation and retries.
Questions for authors
-
What evidence shows that improved NetHack performance reflects better long-horizon planning rather than better local decisions or game knowledge?
-
Compared with more open-ended environments such as Minecraft, what distinctive advantages does NetHack offer for evaluating long-horizon agents? This asks for clarification of benchmark scope, rather than a new experiment.
-
Does Prime-Agent-Explore outperform independent restarts or simpler checkpoint strategies under matched computational budgets?
-
Is BALmin more predictive of eventual ascension than BALmax, beyond encouraging balanced depth and experience?
3 comments
Author Review (cont. 2/2)
Compared with more open-ended environments such as Minecraft, what distinctive advantages does NetHack offer for evaluating long-horizon agents? (Q2)
NetHack differs from Minecraft in three ways that matter for evaluating long-horizon agents. First, information is restricted: potions, scrolls and wands have randomized appearances in every game, so the agent must identify them anew, which is not always possible and often relies on indirect evidence such as shop prices [1]. Second, later encounters depend greatly on earlier decisions, because the monsters that are generated depend on the hero's experience level and depth [2]. Third, NetHack is a roguelike, so a single mistake can end the game. In Minecraft, recipes and progression are fixed across worlds and mistakes are recoverable.
As a result, LLM agents in Minecraft can accumulate skills into reusable libraries of executable code [3] and have completed the game [4]. NetHack remains far from solved even when retries are allowed. When we provide 30 checkpointed attempts, PAE only reaches 11.70% BALmin with GLM-5.2, and single-attempt agents reach only about 2%, leaving substantial headroom as a benchmark.
References
[1] NetHack Wiki. "Identification." nethackwiki.com/wiki/Identification
[2] NetHack Wiki. "Monster creation." nethackwiki.com/wiki/Monster_creation
[3] Wang, G. et al., 2023. "Voyager: An Open-Ended Embodied Agent with Large Language Models." arXiv:2305.16291
[4] Faldor, M., Singh, S., Raileanu, R., 2026. "A team of Gemini 4 Argon agents beats Minecraft." arXiv:2610.22319
I have lowered my confidence from 2 to 1 to reflect my unce rtainty about parts of my assessment. I hope this helps the AC weigh my review.
Author Review
Dear Reviewer gE2H,
Thank you for the time and effort put into the review. We appreciate your recognition that configurable mechanics and checkpointing enable controlled studies of agent failures and search strategies, and that our ablations support the value of both checkpoint selection and guidance.
We understand the main concerns to be (1) whether the gains reflect long-horizon planning rather than local decisions or game knowledge, (2) BALmin's relationship to eventual success, and (3) separating PAE's benefit from additional computation and retries. We have run new experiments and revised the paper (changes highlighted in yellow). Together with the discussion below, does this fully address your concerns? We look forward to continuing the discussion!
Our harness adds navigation skills and a structured text encoding of observations, so that low-level control and perception do not become the bottleneck and higher-level decisions can be isolated. We then measured the remaining low-level confounders with the single-life probes of Section 5.1, for example simplifying interactions such as unlocking doors (control) and providing streamed observations (perception). None changes BALmin significantly (unlocked doors p = 0.31, ASCII map p = 0.16, JSON map p = 0.78).
In contrast, PAE, a much stronger search and exploration mechanism, produces the largest improvement of any intervention: with GLM-5.2 it reaches 11.70% BALmin at 30 attempts, against 4.15% for the best of 30 independent lives (combined response). Within PAE, letting the model choose which earlier state to continue from, a decision about the whole trajectory, raises BALmin from 2.45% to 4.21% over a fixed rule (table below).
PAE also benefits from additional search and from chances to recover from mistakes. The small effects of control and perception, against the much larger gains from PAE, therefore indicate that higher-level planning and exploration remain the major bottleneck for current agents in NetHack.
We address this in the combined response above, in the section "BALmin vs BALmax".
For independent restarts, we address this in the combined response above, in the section "Evaluating NetHack", which compares PAE@k with Best@k. For simpler checkpoint strategies, Table 6c compares Best@15 with PAE@15 and its ablations, all with GLM-5.2 at 15 attempts:
The strategies with no selection resume the most recent checkpoint, and the orchestrator models in evaluations with no guidance do not hand over a plan to the subagent. We find that Best@15 exceeds both ablations with no selection. Selection without guidance is on par with Best@15 (4.21% versus 4.15%). We thus find that simpler checkpoint strategies at best match Best@15, whereas PAE@15, which combines selection with guidance, exceeds Best@15 by 2.3×.