- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a framework for measuring exploration and exploitation errors in language model agents using partially observable grids with initially unknown symbolic task dependencies. It detects unproductive actions through distance-based progress and structural redundancy, then assigns errors according to the opportunities available in the current state. Experiments cover 35 models and examine relationships with task success, prompting, memory, semantic task labels, and an extension to Minecraft Clone. The intended contribution is a behavioral diagnostic that reveals information beyond binary task success.
Critical strengths
-
The controlled environment is well suited to studying sequential agent behavior. Partial observability and symbolic task dependencies require agents to acquire information, maintain state, and revisit actionable tasks while reducing dependence on prior semantic knowledge. This makes the setting useful for investigating failures that are difficult to isolate in less controlled benchmarks.
-
The broad model coverage and trajectory analyses provide useful descriptive evidence. The examples show how agents with similar task outcomes can behave differently, and the reported correlations are numerically consistent with Table 10. The framework could therefore become a valuable diagnostic tool if the interpretation of its two error measures is validated.
Critical weaknesses
-
The evidence does not yet establish that the metric distinguishes exploration failures from exploitation failures. The method first detects an error using a shared rule and then assigns it according to the opportunities available; in Case 4, the same error contributes to both metrics. This describes the circumstances in which an error occurs, but the connection to distinct failure mechanisms needs validation. Correlation with success is insufficient on its own because repetition and lack of progress can also predict failure under a fixed step budget. The paper does not establish incremental diagnostic value over these simpler measures or validate the attribution on policies with known failure modes. Since measurement is the central contribution, this is my main reason for weak rejection and the primary basis for my soundness concern.
-
The relationship between the written metric, the supplied implementation, and the reported results is unresolved. Section 4 describes entering an unobserved cell as a progress event. However, src/symbolic_benchmark/metrics_upd.py uses membership in the current target set, and when the goal is pending that set contains only the goal. Its default partially observed branch also computes distances over visited cells and their frontier. These details address part of the implementation question, but differ from the broad written progress rule. Separately, Table 16 uses an exploration denominator of 188 even though the final 110 steps are described as Case 2, while Section 5 restricts that denominator to Cases 1 and 4. The supplied files alone do not establish which implementation produced the reported numbers. Appendix I also leaves the mapping from heterogeneous Minecraft actions to the metric's targets, progress, and redundancy quantities unclear. These gaps limit reproducibility and interpretation across environments.
Questions for authors
-
Which implementation and configuration produced the main results and Table 16? Please reconcile the paper's progress definition with the target-based rule in metrics_upd.py, and clarify the exploration denominator, including episodes with no eligible opportunities. For Minecraft, please provide the action-to-metric definitions and the scored opportunity counts underlying Figure 6. These clarifications would establish whether the discrepancies are confined to exposition or affect the reported calculations.
-
Using existing trajectories where possible, can you compare the proposed measures with simple repetition/no-progress baselines and report Case 4 errors separately? A small diagnostic using policies with a known exploration or exploitation failure would also help assess the attribution. Evidence that the two measures identify distinct failures beyond general inefficiency would materially improve my assessment; a comprehensive new benchmark study is not required.
8 comments
(W2 & Q1)
Exploration denominator
Thank you for identifying this discrepancy. We audited the scoring code and confirmed that the exploration denominator incorrectly included Case 2 steps, where the goal is pending and only exploitation is eligible. We corrected the denominator to count only Cases 1 and 4, as specified in Section 5. Our audit found no other error in our code.
We recomputed the affected exploration-error rates and the corresponding values for their relationship with task success. The strong negative relationship between log exploration error and task success persists across all three settings.
| Setting | Original | Corrected |
|---|---|---|
| Default | 0.897 | 0.893 |
| Reasoning | 0.802 | 0.796 |
| Combined | 0.885 | 0.881 |
For the example in Table 16, the correction excludes the final 110 Case 2 steps, giving 78 eligible exploration steps for GPT-OSS-120B, and the final 6 Case 2 steps are removed for Hy3.
| Model | Original (errors / eligible steps) | Corrected (errors / eligible steps) |
|---|---|---|
| Hy3 | 12/148 (8.1%) | 12/142 (8.5%) |
| GPT-OSS-120B | 0/188 (0%) | 0/78 (0%) |
We also updated the exploration-error rates in the harness and semantic ablations and in the sensitivity analyses for and , and the corresponding figures.
Minecraft Clone setting
Appendix H of the revised manuscript clarifies how we score the actions (Figure 7; Figure 6 in the submitted version). Since Minecraft Clone allows 3D continuous actions, we discretize the actions the agents can take into 21 predefined actions: movement in four directions, camera adjustment in four directions, jumping, mining, block placement, selection of one of nine inventory slots, and waiting. Movements last for 1 second, jumping lasts 0.4 seconds and mining holds the left mouse button for 2 seconds. Camera adjustment applies a fixed mouse displacement with a 0.2 second action budget. The simulation advances only within these action budgets and remains frozen between actions, preventing model inference time from affecting execution.
The goal of the agent is to find two iron-ores. An iron-ore block becomes a known target once the agent aims at it. Exploration remains active when the known number of ores is insufficient to meet the goal, and exploitation is active whenever known ore remains to be mined. Moving to new positions, changing the viewpoint, and discovering ores are gainful exploration. Approaching known ore, productively reorienting toward it, mining, or collecting iron contribute to gainful exploitation. Discovering an ore or collecting it resets the no-progress thread. Within a no-progress thread, staleness tracks cycles and node/edge reuse beyond two visits/traversals, with stationary actions counting towards node visits (since we do not have a self-edge). Actions without active gain are errors; gainful actions with increase in staleness score are also errors.
As requested by the reviewer, the scored opportunity counts underlying Figure 6 of the submitted version are provided below.
| Model / visual memory | Exploration errors / opportunities | Exploitation errors / opportunities |
|---|---|---|
| Gemini 3.1 Flash Lite / with | 390 / 483 | 0 / 8 |
| Gemini 3.1 Flash Lite / without | 845 / 891 | 0 / 1 |
| GPT-4.1 / with | 438 / 592 | 10 / 18 |
| GPT-4.1 / without | 733 / 836 | 0 / 0 |
We hope the response addresses your concerns. We would be happy to answer any further questions.
Thank you for the detailed rebuttal. I appreciate the correction to the exploration denominator, the reference-policy experiments, and the clearer Minecraft definitions. I also understand that the two rates describe errors under available opportunities rather than infer an agent’s intention.
My central concern is whether the rates identify distinct failures in LM trajectories. The scripted policies show that the rates can differ in constructed cases, but 81.59% of exploitation-error events in the main analysis are Case 4 events counted in both metrics. In the Terminal-Bench extension, all 29 identified faults contribute to both numerators. The repetition and no-progress baseline also predicts success at least as well, so success correlation does not independently validate the attribution.
I would also appreciate clarification of a possible inconsistency: Equation (1) assigns zero error to entering an unobserved cell, while the new example counts moves into unseen side-branch cells after the goal becomes actionable as exploitation errors. An action-level scoring trace and the scorer version used for the main results would help resolve this. For now, I am keeping my rating at 2.
We thank the reviewer for recognizing the value of our controlled environment for studying sequential agent behavior and for highlighting how our trajectory analyses reveal differences in behavior among agents with similar task outcomes. We address the reviewer’s concerns below.
(W1 & Q2)
Distinguishing exploration and exploitation failures
We agree with the reviewer that correlation with task success is not a direct validation of error attribution. First, we kindly remind the reviewer that Appendix B (Lines 923–1016) includes challenging edge cases where we validate the error metrics against human annotation of the errors.
To further test our metric, we generate four scripted policies with known stalling behaviors. We first fix a small map consisting of a five-cell corridor and a two-cell side branch. We design four policies: "Efficient return" returns to the activated goal node directly. "Stall before discovering prerequisite" and "Stall after goal becomes actionable" add 12 alternating moves before discovering the prerequisite or after the goal becomes actionable, respectively. "Enter new side branch after goal becomes actionable" makes the agent perform unnecessary exploration after goal is pending. The results are reported below:
While all four policies succeed, "Efficient return" incurs no errors, while the two stalling policies receive different exploration and exploitation errors despite having the same counts for repetition and no-progress threads. The metric also flags two moves into unseen side-branch cells after the goal becomes actionable as an exploitation error as desired.
Comparison with simpler measures
We would like to emphasize that the purpose of our metric is to quantify exploration and exploitation errors, and not to use it as a predictor of success. The reported R2 values simply are empirical findings that add to the validity and soundness of our metric. We show below that repetition and lack of progress are both strong predictors of success. However, repetition and lack of progress are entities defined in our paper towards the error metric, and these alone cannot measure exploration and exploitation errors.
We perform leave-one-model-out evaluation with repetition and no-progress rates alongside our error rates. We trained logistic regression on episodes from 34 models and tested on the held out model, repeating 35 times. We report AUROC and Brier scores, and the results are shown below.
The results show that both repetition and no-progress threads are also strong predictors of episode success, showing comparable AUROC and Brier scores to our exploration and exploitation errors. However, repetition and no-progress threads cannot be used to attribute actions as errors.
Case 4 attribution
We report the breakdown of the errors into four cases:
Case 4 contributes 4.71% of exploration-error events and 81.59% of exploitation-error events.