acceptodds
Under review as a conference paper at ICLR 2027

Representation and Motivation Are Superadditive: Unlocking Exploration for LLM Agents in Structured Search

Abstract

LLM agents are increasingly expected to discover solutions beyond what is already known, yet prior work shows that they search locally, optimizing among known options rather than pursuing unexplored alternatives. We identify an upstream representational bottleneck in structured search: when alternatives must be recovered from relational history, unexplored regions must first be represented as candidate destinations before motivation can redirect choice. In a task adapted from Wu et al. (2018), baseline agents across DeepSeek V4.1 Flash, Qwen3.8-flash, and GPT-4.1-mini select new positions with mean uncertainty percentiles of only 1.6–3.6, compared with 22.8 for matched humans; the deficit persists when provider-native reasoning is enabled and in a GPT-5.6 terra diagnostic. Factorially combining a coverage representation with topology-free motivation produces superadditive gains in search extent: the joint effect exceeds the sum of the individual effects by 4.2–12.6 uncertainty-percentile points. The combination raises the uncertainty percentile to 21.9–23.2, closing the roughly order-of-magnitude gap to the human benchmark. Same-information controls isolate coverage status from generic history summarization, while rendering controls show that computing the same coverage fields does not guarantee their use in choice. In a restless bandit with named arms and 857 human participants, the same motivation increases switching without a positive interaction. Together, the task contrast identifies a structural boundary: motivation acts directly among named alternatives but gains leverage from representation when destinations must be recovered from history.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, exploration-exploitation, sequential decision-making, human behavior benchmarks, prompt design, structured search, super-additivity, spatial representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mCiCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies exploration in LLM agents during structured search. It finds that explicitly describing search coverage and providing motivational prompts have superadditive effects on search extent, and uses several control experiments to examine the role of information presentation.

Critical strengths
  1. The paper is clearly written and well structured. The experimental design and main results are easy to follow.

  2. It offers an interesting perspective, particularly the distinction between information being available in context and actually being used in decisions.

Critical weaknesses
  1. I find the novelty and contribution somewhat limited. The main intervention combines a coverage summary with a motivational prompt. I would appreciate a clearer explanation of what these findings add to existing work on state representation and exploration.

  2. The results may reflect a general decision-making effect rather than a limitation specific to LLMs. This makes the work read more as a study of cognition and decision-making.

  3. The practical relevance to tool-using agents remains unclear. The task is well suited to Bayesian optimization, and GP-UCB is already included as a reference. Comparing against an LLM agent that can call such an optimizer would help establish whether the proposed bottleneck remains important in a more complete agent system.

Questions for authors

Please refer to weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer mCiC

Thank you for this careful and constructive review. Your two suggestions, to say more clearly what the work adds and to test an agent that can call an optimizer, shaped much of the revision, and the second turned out to be one of the most informative new experiments.

On what the findings add to work on state representation and exploration (W1). We agree that the submission did not make this clear, and we have rewritten the introduction and related work around it (Sections 1 and 4). Work on state representation has shown that summaries of past experience can help LLM agents explore, and count-based methods in reinforcement learning already use coverage to direct exploration. What remained open is which part of such a remedy does the work. The evidence comes mostly from bandits, where every option is already named, and from systems evaluated as a whole; Krishnamurthy et al. (2024) note that their prompt modifications appeared to interact in ways that were hard to attribute. Our design pulls the two parts apart in a task where the agent must infer from its history where it has not yet looked. There they turn out to be complements. Motivation alone leaves search local (in two models it makes agents try more new positions, but not farther ones), and it widens search only once the untouched regions are named. A direct instruction to try unvisited positions shows the same pattern: it adds new choices right next to known ones without widening search (Section 3.3, Figure 3). Among named bandit arms, by contrast, motivation alone is enough. So the contribution is less that coverage helps than when an LLM agent actually uses coverage information that is already in its context.

On whether this is a general decision-making effect (W2). We think this is a fair point, and we cannot rule it out, since we did not give the interventions to people. What we can say is that the starting point differs. In the same task, humans search far more widely without any help and find better options than random search with the same number of tries, while baseline agents stay next to what they have already tried and do worse than that benchmark (Sections 3.1 and 3.2). Because the human data come from a visual interface, we treat this comparison as descriptive. We have taken your comment about the framing to heart: the revision presents the work as a question about what to put in an agent's context, and keeps the human data as a reference rather than as a claim about cognition.

On tool-using agents (W3). We added the comparison you proposed. In two new conditions, the agent may ask for a GP-UCB recommendation, computed from its own observations, before every choice and at no cost in points (Section 3.3, Table A10). Having the optimizer available did not make the problem go away, because agents rarely asked for it. With the original instructions, DeepSeek, Qwen and GPT-4.1-mini requested advice on about 19%, 13% and 1% of choices. With coverage and motivation, requests rose to about 58%, 38% and 3%, and adding the tool raised the reward of that condition in all three models. In this more complete agent, then, whether useful information gets used still depends on what the context gives the agent. GPT-4.1-mini almost never called the tool, so we do not read its reward change as an effect of the optimizer.

Thank you again for these suggestions. If the revised introduction still leaves the contribution unclear, we would be grateful to know where it falls short.

Reviewer_zWtLOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates why LLM agents exhibit predominantly local exploration in structured search tasks. The authors hypothesize that effective exploration requires both an explicit representation of unexplored alternatives and sufficient motivation to pursue them. They introduce a training-free intervention combining a coverage-based representation of interaction history with a topology-free motivation prompt, and evaluate their interaction through a 2x2 factorial design across three LLMs, structured search environments, and a restless bandit task. The main results show positive representation–motivation interactions in uncertainty-percentile-based exploration, with additional analyses examining information rendering, self-generated representations, and the distinction between leaving known options and selecting unexplored destinations.

Critical strengths
  1. The paper evaluates "representation" and "motivation" separately and jointly across three model families, reporting positive interactions of +10.2, +12.6, and +4.2 uncertainty-percentile points. Additional experiments examine two-dimensional search, rough landscapes, provider-native reasoning, and alternative metric formulations. This provides reasonably broad empirical evidence for a positive interaction in the measured exploration behavior. The factorial approach is more informative than simply reporting improvements from a combined prompt intervention.

  2. The distinction between departure and destination selection is supported by the contrast between structured search and the restless bandit (Sections 3.3-3.4). Moreover, the availability-versus-use analysis shows a substantial effect of sentence rendering over a structured list. These findings provide potentially useful insights into how LLM agents utilize externally organized information, independently of whether the proposed mechanism is fully established.

Critical weaknesses
  1. Inconsistent experimental controls in the main factorial design.

The released structured-exploration-supp-min/run.py disables the comprehension quiz for the baseline (wu_bare) but enables it for the other three factorial conditions. The quiz answers are subsequently included in decision contexts (structured-exploration-supp-min/wu1d.py). This introduces an uncontrolled prompt difference into the central R×M comparison (Section 2.2), weakening the causal attribution of the reported interaction. Matched experimental conditions are needed to establish the claimed effect reliably.

  1. Mismatch between the primary metric definition and implementation.

Equation (1) defines uncertainty percentile as qt=100(rt(at)−1)/(∣Ut∣−1)q_t=100(r_t(a_t)-1)/(|U_t|-1), using average ranks for ties. However, structured-exploration-supp-min/uncertainty_percentile.py (lines 18–20) uses np.searchsorted(v, sig[c-1])/len(v), implementing neither convention. Independent recalculation produces different numerical results, although the positive interaction remains. The reported estimates and confidence intervals should be reconciled with the stated metric definition.

  1. Inconsistent action handling between code and saved trajectories.

The saved trajectories, including structured-exploration-supp-min/out/wu_bare_deepseek.jsonl, contain invalid position-zero actions (16 occurrences in the DeepSeek baseline), despite the released fallback implementation enforcing valid positions from 1 to 30 (agents.py; wu1d.py). Such actions can also enter subsequent posterior calculations through obs[c] = r (uncertainty_percentile.py). This raises concerns about implementation-version consistency and the reliability of the evaluation pipeline.

  1. Inconsistencies in statistical analysis and reproducibility.

Figure 2 reports seed-paired bootstrap confidence intervals, whereas structured-exploration-supp-min/figs/main_data.py independently resamples factorial cells. A separate script, structured-exploration-supp-min/reviewer2_checks.py, implements paired resampling, leaving the authoritative analysis pipeline unclear. Some intermediate data required by the supplementary analysis scripts are also unavailable, preventing complete independent verification.

  1. Superadditive exploration does not establish superadditive task performance.

The central positive interaction is demonstrated primarily through uncertainty percentile rather than task utility. The best-found-value interaction is approximately −1.1 percentage points for DeepSeek (Table A11), while the two-dimensional interaction is not statistically distinguishable from zero (Table A13). The GPT-5.6 diagnostic also shows reduced per-step reward (Table A18). Thus, the evidence supports increased exploration extent, but not necessarily improved exploration efficiency or task performance.

  1. The proposed representational bottleneck is not sufficiently isolated.

All 30 candidate positions are already available in the primary task. The coverage representation primarily reorganizes and emphasizes unexplored locations rather than introducing previously unavailable alternatives. Moreover, sentence rendering alone produces a substantial improvement over a structured list (+12.9 points, Table A17). These findings leave prompt salience and linguistic framing as plausible alternative explanations for the claimed mechanism.

  1. Limited novelty and generalizability beyond structured search.

Externally organized histories and explicit exploration targets have already been investigated, notably by Krishnamurthy et al. (2024) [1] and Lu et al. (2025) [2]. The present contribution is primarily a factorial behavioral analysis rather than a new exploration algorithm. Furthermore, the experiments rely on finite, enumerable action spaces, leaving the broader implications for open-ended LLM agents insufficiently established.

[1] Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? Advances in Neural Information Processing Systems, 37, 2024.

[2] Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing on the shoulders of giant foundation models. In International Conference on Learning Representations (ICLR), 2025.

Questions for authors
  1. Why is the comprehension quiz disabled only for wu_bare in structured-exploration-supp-min/run.py? Can the authors provide results with identical quiz configurations across all four conditions, or evidence that this discrepancy does not affect the reported interaction?

  2. Why do trajectories such as structured-exploration-supp-min/out/wu_bare_deepseek.jsonl contain position-zero actions that the released agents.py and wu1d.py should prevent? Which implementation generated these trajectories, and how were invalid actions handled in subsequent reward and uncertainty calculations?

  3. Which metric definition and bootstrap implementation generated the reported results? Can the authors reconcile Equation (1) with structured-exploration-supp-min/uncertainty_percentile.py, and the paired-bootstrap claim in Figure 2 with structured-exploration-supp-min/figs/main_data.py? Corrected estimates and confidence intervals would help assess the reliability of the findings.

  4. Given the negative or statistically inconclusive best-found-value interactions (Tables A11 and A13), what evidence supports a superadditive improvement in effective exploration rather than merely increased uncertainty-seeking behavior? How should the observed reductions in per-step reward be interpreted?

  5. What evidence distinguishes the proposed representational bottleneck from prompt salience or linguistic framing? In particular, how does the interpretation account for the large rendering effect in Table A17 and the fact that all candidate positions are already explicitly available?

  6. What is the substantive conceptual distinction from prior work using externally structured exploration histories or explicit exploration targets, particularly Krishnamurthy et al. (2024) [1] and Lu et al. (2025) [2]? Which findings support extending the conclusions beyond the evaluated finite-action search tasks?

[1] Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? Advances in Neural Information Processing Systems, 37, 2024.

[2] Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing on the shoulders of giant foundation models. In International Conference on Learning Representations (ICLR), 2025.

3 comments
AuthorsOpenReview ↗

Response to Reviewer zWtL (1/2): implementation

Thank you for auditing the released code so carefully. All four implementation problems you found were real, and we found two more while checking. We answer your questions in order: this first part covers the implementation (Q1-Q3), and the second covers interpretation (Q4-Q6).

Q1 and W1: the quiz. This was an error in our run configuration, not a design choice, and we did not catch it before submission. We have rerun the whole factorial on all three models with identical settings in every cell, meaning the same quiz, display and final action parser, interleaved within provider on seeds 1000-1049 (Sections 2.2 and 3.2). The interaction remains positive: 12.26 [10.28, 14.23] points for DeepSeek, 14.24 [12.91, 15.61] for Qwen and 4.91 [3.67, 6.17] for GPT-4.1-mini (Table 2). In an earlier batch we also crossed the quiz as a factor, and the interaction was positive and of similar size with the quiz on and off (Table A5).

Q2 and W3: position-zero actions. You were right, and the problem was larger than the 16 cases you found. Across the four cells of the submitted factorial, 99 of 72,000 choices were recorded as position 0: 16, 3, 8 and 0 in DeepSeek's B, R, M and RM cells, 20, 17, 0 and 0 in Qwen's, and 35, 0, 0 and 0 in GPT-4.1-mini's. They came from an earlier fallback that is not in the released code, and we could not recover the exact commit. Replaying the saved trajectories shows what happened: position 0 read the reward of position 30 through negative indexing, while 0 stayed in the history and, as you noted, entered later posterior updates. The submission understated this, and Appendix A now reports it in full. None of the new estimates uses these trajectories. In all new runs the parser accepts only an unambiguous action and otherwise repeats a legal one, and every recorded reward replays exactly from the stored history.

Q3, W2 and W4: the metric, the bootstrap and reproducibility. The released code used an insertion rank and divided by the number of positions, so it did not implement Eq. (1). It now uses average ranks and divides by |U|-1; as a check, uniform random choice scores 49.7, against 46.2 under the old code (Section 2.3, Appendix A). Because some posterior SDs are nearly tied, we also report the interaction after rounding SDs to 1e-8 before ranking, which changes it by less than 0.4 points (Table A2). On the bootstrap, the intervals printed in the submission came from a paired analysis, but the released figure script resampled cells independently, and the separate reviewer2_checks.py added to the confusion. All intervals now come from one seed-block paired bootstrap. The supplement now contains every choice and reward, so no intermediate data are missing, and two scripts replace the old ones: python reproduce.py recomputes Table 2, the validation in Table A1 and the controls and checks them against stored values, and python reproduce_novelty.py recomputes Table A9.

Two problems you did not raise. A full audit showed that the earlier parser could also pick up a number from unmarked reasoning text. This affected 1,213 of 2,700 earlier trajectories, and we reran all of them (Appendix A). The submitted rendering experiment also mixed quiz settings across its eight cells; in the repaired version, all eight cells omit the quiz.

AuthorsOpenReview ↗

Response to Reviewer zWtL (2/2): interpretation and scope

Q4 and W5: exploration versus performance. We agree: the evidence supports wider search, not better task performance, and the revision now says so. In the new factorial, the reward interaction includes zero in all three models, and the best-found-value interaction is positive only in Qwen (Table 2). The submitted diagnostics you cite, the two-dimensional task and GPT-5.6, point the same way; we kept them in the appendix as submitted and did not rerun them. To separate finding many positions from finding good ones, we compare each agent's best discovery with random search over the same number of new positions. Baseline agents do worse than this benchmark, coverage plus motivation brings them to about chance level, and humans do better (Section 3.2, Table A4). So the intervention widens search but does not make it more efficient. The drop in per-step reward has a simple behavioral source: in DeepSeek and Qwen, agents with coverage and motivation pick new positions on 90-94% of choices, compared with 56-59% at baseline, and rarely return to the good positions they have found (Table A9). We then tested a feedback-guided policy that tells agents when to return. On new seeds it raised both reward and best-found value in Qwen and GPT-4.1-mini, but not reward in DeepSeek (Section 3.6, Table A1).

Q5 and W6: representation versus salience. We agree that the coverage summary adds no alternatives the agent could not already choose; it reorganizes what is in the history. That is the comparison we care about, because the information is available but often not used. You are right, however, that the submission's "representational bottleneck" went beyond the evidence, and we have dropped it. The new controls narrow the explanation. An instruction that mentions distance widens search without any summary, so coverage is not necessary. An instruction to try unvisited positions, without naming them, leaves search extent unchanged even though agents pick more new positions (Section 3.3, Figure 3). Content matters: naming unvisited positions beats a summary of similar length that only reorganizes past readings by about 15-18 points (Appendix Figure A2). Presentation matters too: with the same fields, sentences beat a list by about 16 points after the repair, compared with 12.9 in the submission (Table A11). We cannot separate this from salience or framing, so we treat the rendering as part of the intervention and say so in the Limitations.

Q6 and W7: relation to prior work and scope. Krishnamurthy et al. (2024) study bandits, where every alternative is named. Their robust configuration pairs chain-of-thought with a per-arm summary; they report that prompt modifications appeared to interact, which made individual effects hard to attribute, and they leave open how to organize history beyond arm-wise statistics. Lu et al. (2025) keep an archive of discovered states, let a foundation model choose where to return and explore, and evaluate the system on task performance. In both, a reason to explore and a record of candidates come together. Our design separates them in a task where candidates have to be inferred from history. There, motivation alone leaves search local and becomes effective only with coverage, whereas among named arms it raises switching on its own. We agree that this is a behavioral analysis rather than a new algorithm. Its use is to show which part of a context-based remedy does what, and that results obtained with named alternatives need not carry over. We also agree that nothing here establishes effects in open-ended agents, and the Limitations now say so.

Because you found parts of the paper hard to follow, we also rewrote the introduction and the related work. If running the scripts turns up any remaining discrepancy, we will correct it before the discussion closes. Thanks again for the detailed comments!

I appreciate the authors' detailed rebuttal and substantial revisions. The corrected factorial experiments, metric implementation, action parsing, and bootstrap analysis have addressed most of my original soundness concerns. I also appreciate the additional experiments and the more careful interpretation of the findings.

Some limitations remain, particularly regarding the distinction between exploration extent and task performance, the influence of prompt salience, and generalizability beyond finite-action settings. Nevertheless, the revised manuscript is substantially more convincing.

I therefore raise my Soundness score from 1 to 2 and my Overall Rating from 1 to 2 (Weak Rejection), acknowledging the significant improvements while retaining some reservations about the novelty and broader significance of the contribution.

Reviewer_z7xgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies how representation and motivation affect the exploration behavior of LLM agents. The authors find that combining explicit coverage information with exploration-oriented instructions leads to significantly more exploration than using either alone.

Critical strengths
  1. The experimental design is clear, and the interaction effect is consistent across multiple models.
Critical weaknesses
  1. The improvement in exploration is clear, but the gains in actual reward are much smaller. In some experiments, the combined method even performs worse than representation alone in terms of reward.

  2. The generalization is relatively limited, and the experiments are relatively toy. Most experiments focus on search tasks. The bandit experiments are quite different. It cannot show general improvements about exploration in LLM agents.

  3. The explanation is not fully convincing. Explicitly showing unexplored locations may simply encourage the LLMM to visit them. It is unclear whether this really shows a representation bottleneck.

Questions for authors
  1. Can the authors provide more results on cumulative reward rather than just exploration metrics?

  2. How much of the improvement comes from better representation versus simply highlighting unexplored locations?

  3. How does the method compare with simply telling the model to explore distant or unvisited locations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer z7xg

Thank you for these questions. They shaped most of the new experiments in the revision. We also fixed the implementation problems described in our general response, and the interaction in search extent is still positive in all three models after the fixes (Table 2).

Reward (W1, Q1). You are right that wider search did not translate into reward. The revision now reports cumulative reward for every condition (Tables A4 and A9), and we no longer describe the joint intervention as improving task performance. The reward interaction is indistinguishable from zero in all three models, and, as you point out, coverage plus motivation earns less than coverage alone in DeepSeek and Qwen. Agents in that condition keep moving to new positions and rarely go back to the good ones they have found.

That led us to test whether telling agents when to return would help. We picked one policy out of 13 candidates and then ran it on fresh seeds, with the success criterion fixed in advance. It improved both reward and best-found value over baseline in Qwen and GPT-4.1-mini; in DeepSeek it found better positions but earned fewer points (Section 3.6, Table A1). So it helps in two of three models, and we do not present it as a general fix.

Simply telling the model to explore (Q3). We ran this with four wordings (Section 3.3, Figure 3), and the answer depends on what the instruction says. When it mentions distance, agents search far without any coverage summary, in two models farther than with our intervention, although how far varies a lot with the wording and the model. When it only says to try unvisited positions, agents pick more new positions, but almost all of them sit right next to positions they have already tried, so search extent does not change and reward goes down. Given the first result, we have dropped the claim that coverage is necessary.

Representation versus highlighting (W3, Q2). We agree that the evidence does not show a representation bottleneck, and we no longer frame the paper that way. What we can say is that both content and presentation matter. In the submitted experiments, naming the unvisited positions raised search extent by about 15-18 points more than a summary of similar length that only reorganized past readings (Appendix Figure A2). In DeepSeek, the same information also worked better as sentences than as a list, by about 16 points (Table A11). We cannot cleanly separate this from salience, so we treat the wording of the summary as part of the intervention.

Toy tasks and generality (W2). We agree. These are controlled tasks with a finite set of actions, and they cannot show a general improvement in LLM exploration; the revised paper does not claim one. We chose them because the factorial needs identical landscapes across conditions, and because human data exist for the same task. The bandit is meant as a contrast rather than a test of generalization: when every option already has a name, motivation alone is enough to make agents switch. The Limitations section now says this directly.

Thank you again for all the constructive insights. We hope that those changes help us improve the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and summary of the revision

We thank all three reviewers for their careful reading. We have corrected the implementation errors that Reviewer zWtL identified, and the main result holds: the interaction in search extent is positive in all three models (12.26, 14.24 and 4.91 points; Table 2). We have also clarified the paper's novelty and contribution. Below, we go through each comment, say what we changed, and point to where the change appears in the revised PDF, in which all changes are highlighted; our reply to each reviewer gives the details.

Reviewer zWtL

  • On the quiz (W1, Q1): you were right that the baseline ran without the quiz while the other three cells used it. We reran the factorial on all three models with the same quiz, display and final parser in every cell, interleaved within provider (Sections 2.2 and 3.2, Table 2). An earlier batch that crossed the quiz as a factor gives a positive interaction both with and without it (Table A5).
  • On the metric (W2, Q3): the code did not implement Eq. (1). It now uses average ranks and divides by |U|-1, and uniform random choice scores 49.7 rather than 46.2 (Section 2.3, Appendix A). Table A2 shows that rounding near-tied posterior SDs changes the interactions very little.
  • On position-zero actions (W3, Q2): the problem was larger than the submission reported. In total, 99 of 72,000 archived choices were recorded as position 0 and read the reward of position 30, and Appendix A now gives the counts by cell. None of the new estimates uses these trajectories, the new parser accepts only an unambiguous action, and every new reward replays exactly. While checking, we also found ambiguous parsing in 1,213 of 2,700 earlier trajectories and reran all of them (Appendix A).
  • On the bootstrap and reproducibility (W4, Q3): all intervals now come from one seed-block paired bootstrap. The supplement contains every choice and reward, together with two scripts that replace main_data.py and reviewer2_checks.py: reproduce.py recomputes Table 2, the validation and the reviewer controls, and reproduce_novelty.py recomputes Table A9.
  • On exploration versus performance (W5, Q4): we agree, and we have narrowed the claim. The reward interactions all include zero (Table 2). We now report cumulative reward for every cell and compare discovery with random search over the same number of new positions (Section 3.2, Tables A4 and A9). We also validated a feedback-guided policy on new seeds (Section 3.6, Table A1).
  • On salience (W6, Q5): we cannot fully separate representation from salience, and we have withdrawn the claim that coverage must be represented before motivation can redirect choice. Four direct instructions show that mentioning distance widens search, while asking only for unvisited positions does not (Section 3.3, Figure 3, Tables A7-A9). We treat sentence rendering as part of the intervention, and Section 4 states what this leaves open.
  • On novelty and scope (W7, Q6): the introduction and Section 4 now explain how the work differs from Krishnamurthy et al. (2024) and Lu et al. (2025), and the Limitations restrict our claims to finite-action tasks.

Reviewer z7xg

  • On reward (W1, Q1): we now report cumulative reward for every cell, including the comparison of RM with R that you raised (Section 3.2, Tables A4 and A9). We also tested a policy aimed at reward itself (Section 3.6, Table A1).
  • On highlighting and direct instructions (W3, Q2, Q3): we ran the control you suggested with four wordings and crossed the strongest one with coverage (Section 3.3, Figure 3). The results led us to withdraw the claim that coverage is necessary.
  • On generality (W2): we present the bandit as a boundary test rather than a test of generalization (Section 3.4), and the Limitations say plainly that our evidence comes from enumerable tasks and does not cover open-ended agents.

Reviewer mCiC

  • On the contribution (W1): we rewrote the introduction, the contributions and the related work to say what the paper adds to existing work on state representation and exploration (Sections 1 and 4).
  • On whether the effect is specific to LLMs (W2): we compare agents and humans on the same measures (Section 4, Table A9). We do not claim that the effect is specific to LLMs, and the human participants did not receive our interventions.
  • On tool-using agents (W3): we added the comparison you suggested, an optional GP-UCB tool that agents can call before each choice, and we report how often they call it and follow its advice (Section 3.3, Table A10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.