- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper constructs a sequential search task to evaluate LLMs' ability to trade off potential future gains against search costs. The paper investigates two cost regimes: one where search costs are explicitly deducted from the rewards and another where costs are determined by token consumption. The paper also extends its evaluation to a real-world shopping environment.
Critical strengths
The paper is clearly presented, and the experimental results are easy to understand.
Critical weaknesses
(i) The paper examines only a single sequential decision-making task (I would not call it a "sequential search game," as there is only one decision-maker) and does not make a substantial technical contribution. There have already been numerous studies evaluating the capabilities of LLMs in various decision-making tasks. The paper neither proposes a novel approach to improve the decision-making capabilities of LLMs nor provides a comprehensive evaluation across a diverse set of tasks. Therefore, a purely empirical study focusing primarily on a single decision-making task appears to offer limited contributions.
(ii) The motivation for the study is not sufficiently clear. For the sequential decision-making task considered in the paper, a relatively simple Bayes-optimal algorithmic benchmark already exists. In practice, one could directly employ such an algorithm to make decisions rather than delegate the task to LLMs. The use of LLMs for sequential decision-making would be more compelling in complex environments where theoretically optimal algorithms are unavailable or computationally impractical. For example, in more contextual decision-making environments, LLMs could potentially leverage their knowledge and reasoning capabilities to better assess the value of further exploration. Unfortunately, the paper provides limited investigation into such settings, making the practical motivation for evaluating LLMs on the proposed task less convincing.
Questions for authors
Can the authors further justify the technical contributions and practical motivation of the study, particularly given the existence of a Bayes-optimal algorithmic benchmark for the considered task?
6 comments
Author response to Reviewer ZPGs [2/2]
W1: We believe our work is novel since we build a costly information acquisition task environment that is challenging for LLMs to solve but that still admits a computable optimal policy. We are not aware of similar papers in the literature doing that. We describe this as a “sequential search” task in the sense that agents decide sequentially whether to search further and continue the task or to stop. Existing work evaluating agents at in-context sequential decision-making tasks either does not have a meaningful ground truth, limiting evaluations to be only relative between models, or formulates the task as a bandit problem with opportunity costs for exploration rather than explicit search costs, a less direct representation of how agents must moderate their exploration when making rational economic decisions. As noted in our answer to W2, our work instead directly compares agents to a ground truth and shows how agents adjust their behavior to directly changing search costs. We also contribute to the literature by, to our knowledge, being the first to examine how sequential search behavior changes when the LLM’s own token consumption is charged against its payoff, a natural and common form of search cost for LLMs.
In response to your concern about our experiments failing to be comprehensive, we extrapolate our main task to an additional medical diagnosis environment (further details in our response to Reviewer KGRf or in Section 4.4.2) beyond what we already studied in the product shopping WebShop environment. The accordance of our findings in these environments with that of our main experiments is evidence that our results are not purely an artifact of our test environment but a genuine insight about LLM agent behavior. We additionally provide further comparison to existing work on in-context exploration by LLM agents in a new paragraph in the related work section (Section 2), explicitly stated in our response to Reviewer iKvo.
We hope this clarifies the contribution and would be grateful if the reviewer would reconsider their assessment and score. We are happy to continue the discussion and to further clarify additional details.
I would like to thank the authors for their responses. The clarifications are helpful, especially the new experiments. I am afraid I cannot give a "weak acceptance", as my main concern remains.
The paper is essentially an empirical evaluation of LLMs' capabilities on a particular decision-making task, without a new methodological contribution. Therefore, I believe the significance and practical relevance of the task itself are particularly important.
Conceptually, balancing the exploration against search costs is certainly an important problem. However, the original task is highly artificial. As expressed in my original comments, I believe the strength of LLMs in decision-making lies partly in their ability to leverage contextual information and knowledge.
The new medical diagnosis experiment is more closely aligned with this motivation, and I find it interesting. However, it still has some impractical simplifications, e.g., the LLM is not allowed to actively choose which information to acquire about the patient.
Overall, I remain unconvinced that the experimental tasks are sufficiently representative of practical decision-making scenarios. I sincerely appreciate the authors' efforts in conducting the additional experiments and providing thoughtful responses.
Author response to Reviewer ZPGs [1/2]
Thank you for your careful review and insightful comments. We have taken them into consideration and address your two main concerns below, starting with Weakness 2 to contextualize our response to Weakness 1.
W2: In the abstract sequential decision-making environment we devised, our goal is not to test agents’ ability to design an algorithm to solve the task but instead to study their search behavior in complex costly information acquisition tasks which the task serves as a model of. In order to provide meaningful evaluations, we need to have some ground truth of what the correct search behavior is. Thus, we deliberately build an environment with a solvable optimal policy which acts as ground truth while still maintaining enough task complexity that it is challenging for the agent to perform optimally. We use this optimal policy to evaluate whether agents are exploring or adjusting their exploration to changes in search costs as much as they should be, which are insights we couldn’t make without access to such a policy.
We perform an additional experiment to ensure that this task is not easily solvable by an LLM during their decisions and that agents instead act under heuristics, similar to how they would behave in real-world environments without a computable optimal policy. If LLM agents were able to solve for the optimal policy, then they should explore an identical amount, so we test whether LLM agents’ number of states explored is significantly different from that of the optimal. At the time we ran our experiments, GPT-5.4 was the best model available, but we have run an additional 120 simulations of the abstract task with GPT-6 Astra (high) at our highest search cost to stress-test the environment. We report in the table below the paired t-statistic for each model’s deviation in states explored from the optimal policy for Astra as well as all models from our main text experiments. Every model’s mean states explored differs from that of the optimal policy with 95% significance.
In our environment, agents must compare cost, their best observed reward, and the value of continued search in order to perform optimally. Solving for this continuation value does not have an analytic solution and is thus challenging to compute in-context. The universal failure of agents to explore similarly to the optimal policy shows their inability to recover such a policy and that they instead operate off of heuristics.
By deploying AI agents in an environment where we directly control their prior beliefs and understanding of the task, we can isolate their search behavior from task-specific biases and context-relevant knowledge. We also compare the results from our abstract task with similarly defined tasks in more realistic environments (Section 4.4.1, p. 9 and a newly added Section 4.4.2, p. 9 in the main text) and find that our results from the abstract task generally hold in these environments when they are directly comparable, demonstrating that our task is a strong model of performance in more realistic environments. See our response to Reviewer KGRf for further explanation on the relevance of the WebShop cost configuration to our results and details of a newly added medical diagnosis task in which acquiring patient findings is costly. Results from this new task support the findings of our main experiments.