acceptodds
Under review as a conference paper at ICLR 2027

Look Before You Stop: How Language Models Respond to Changes in Search Costs

Abstract

Recent developments in the capabilities of large language models (LLMs) have allowed for their increasingly effective deployment in situations where they are delegated to make decisions. In many of these economically relevant settings, LLMs must make dynamic choices where information acquisition is costly, which requires them to understand the value of their own future decision-making behavior. We evaluate how LLMs trade off the value of continued search against its cost across various dynamic economic contexts and cost regimes. In a sequential search game where exploration is costly, we find that meaningful sensitivity of exploration to changes in search costs, an essential feature of payoff-optimal Bayesian decision-makers, is a capability exhibited by only the most recent reasoning LLMs. We additionally allow this exploration cost to be charged in terms of token consumption rather than reward observations to more closely reflect the costs of deploying LLMs in real-world settings and measure LLMs' ability to internalize search costs more directly. We find that LLMs use this cost regime to adapt their thinking in order to observe more states, but that the extent to which they do so varies across models. We extend our game to a real-world shopping environment and find that reasoning LLMs' cost sensitivity generalizes to this setting but with less consistent results, indicating that this reasoning is key to dynamic decision-making in deployment but that models must possess broader capabilities in order to fully succeed in these settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
sequential search, economic decision-making, LLM evaluations, exploration-exploitation tradeoff, LLM agents, AI behavioral science

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZPGsOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3
Read review
Paper summary

The paper constructs a sequential search task to evaluate LLMs' ability to trade off potential future gains against search costs. The paper investigates two cost regimes: one where search costs are explicitly deducted from the rewards and another where costs are determined by token consumption. The paper also extends its evaluation to a real-world shopping environment.

Critical strengths

The paper is clearly presented, and the experimental results are easy to understand.

Critical weaknesses

(i) The paper examines only a single sequential decision-making task (I would not call it a "sequential search game," as there is only one decision-maker) and does not make a substantial technical contribution. There have already been numerous studies evaluating the capabilities of LLMs in various decision-making tasks. The paper neither proposes a novel approach to improve the decision-making capabilities of LLMs nor provides a comprehensive evaluation across a diverse set of tasks. Therefore, a purely empirical study focusing primarily on a single decision-making task appears to offer limited contributions.

(ii) The motivation for the study is not sufficiently clear. For the sequential decision-making task considered in the paper, a relatively simple Bayes-optimal algorithmic benchmark already exists. In practice, one could directly employ such an algorithm to make decisions rather than delegate the task to LLMs. The use of LLMs for sequential decision-making would be more compelling in complex environments where theoretically optimal algorithms are unavailable or computationally impractical. For example, in more contextual decision-making environments, LLMs could potentially leverage their knowledge and reasoning capabilities to better assess the value of further exploration. Unfortunately, the paper provides limited investigation into such settings, making the practical motivation for evaluating LLMs on the proposed task less convincing.

Questions for authors

Can the authors further justify the technical contributions and practical motivation of the study, particularly given the existence of a Bayes-optimal algorithmic benchmark for the considered task?

6 comments
AuthorsOpenReview ↗

Author response to Reviewer ZPGs [1/2]

Thank you for your careful review and insightful comments. We have taken them into consideration and address your two main concerns below, starting with Weakness 2 to contextualize our response to Weakness 1.

W2: In the abstract sequential decision-making environment we devised, our goal is not to test agents’ ability to design an algorithm to solve the task but instead to study their search behavior in complex costly information acquisition tasks which the task serves as a model of. In order to provide meaningful evaluations, we need to have some ground truth of what the correct search behavior is. Thus, we deliberately build an environment with a solvable optimal policy which acts as ground truth while still maintaining enough task complexity that it is challenging for the agent to perform optimally. We use this optimal policy to evaluate whether agents are exploring or adjusting their exploration to changes in search costs as much as they should be, which are insights we couldn’t make without access to such a policy.

We perform an additional experiment to ensure that this task is not easily solvable by an LLM during their decisions and that agents instead act under heuristics, similar to how they would behave in real-world environments without a computable optimal policy. If LLM agents were able to solve for the optimal policy, then they should explore an identical amount, so we test whether LLM agents’ number of states explored is significantly different from that of the optimal. At the time we ran our experiments, GPT-5.4 was the best model available, but we have run an additional 120 simulations of the abstract task with GPT-6 Astra (high) at our highest search cost to stress-test the environment. We report in the table below the paired t-statistic for each model’s deviation in states explored from the optimal policy for Astra as well as all models from our main text experiments. Every model’s mean states explored differs from that of the optimal policy with 95% significance.

ModelReasoningMean states exploredPaired t-test statisticPaired 95% CI
Bayes-optimal policy–27.92––
OpenAI GPT-5.4none34.903.87[3.41, 10.55]
OpenAI GPT-5.4low17.88-9.72[-12.09, -8.00]
OpenAI GPT-5.4 nanonone8.84-12.98[-21.98, -16.17]
OpenAI GPT-5.4 nanolow10.47-12.14[-20.30, -14.60]
Alibaba Qwen3.6 Flashoff46.315.03[11.15, 25.63]
Alibaba Qwen3.6 Flashon20.82-4.77[-10.04, -4.15]
Thinking Machines Inkling Smalloff9.38-12.74[-21.42, -15.66]
Thinking Machines Inkling Smallon18.57-8.91[-11.43, -7.27]
NVIDIA Nemotron 3 Ultraoff90.8521.56[57.15, 68.71]
NVIDIA Nemotron 3 Ultramedium19.78-8.25[-10.08, -6.18]
OpenAI GPT-6 Astrahigh25.15-5.20[-3.82, -1.71]

In our environment, agents must compare cost, their best observed reward, and the value of continued search in order to perform optimally. Solving for this continuation value does not have an analytic solution and is thus challenging to compute in-context. The universal failure of agents to explore similarly to the optimal policy shows their inability to recover such a policy and that they instead operate off of heuristics.

By deploying AI agents in an environment where we directly control their prior beliefs and understanding of the task, we can isolate their search behavior from task-specific biases and context-relevant knowledge. We also compare the results from our abstract task with similarly defined tasks in more realistic environments (Section 4.4.1, p. 9 and a newly added Section 4.4.2, p. 9 in the main text) and find that our results from the abstract task generally hold in these environments when they are directly comparable, demonstrating that our task is a strong model of performance in more realistic environments. See our response to Reviewer KGRf for further explanation on the relevance of the WebShop cost configuration to our results and details of a newly added medical diagnosis task in which acquiring patient findings is costly. Results from this new task support the findings of our main experiments.

AuthorsOpenReview ↗

Author response to Reviewer ZPGs [2/2]

W1: We believe our work is novel since we build a costly information acquisition task environment that is challenging for LLMs to solve but that still admits a computable optimal policy. We are not aware of similar papers in the literature doing that. We describe this as a “sequential search” task in the sense that agents decide sequentially whether to search further and continue the task or to stop. Existing work evaluating agents at in-context sequential decision-making tasks either does not have a meaningful ground truth, limiting evaluations to be only relative between models, or formulates the task as a bandit problem with opportunity costs for exploration rather than explicit search costs, a less direct representation of how agents must moderate their exploration when making rational economic decisions. As noted in our answer to W2, our work instead directly compares agents to a ground truth and shows how agents adjust their behavior to directly changing search costs. We also contribute to the literature by, to our knowledge, being the first to examine how sequential search behavior changes when the LLM’s own token consumption is charged against its payoff, a natural and common form of search cost for LLMs.

In response to your concern about our experiments failing to be comprehensive, we extrapolate our main task to an additional medical diagnosis environment (further details in our response to Reviewer KGRf or in Section 4.4.2) beyond what we already studied in the product shopping WebShop environment. The accordance of our findings in these environments with that of our main experiments is evidence that our results are not purely an artifact of our test environment but a genuine insight about LLM agent behavior. We additionally provide further comparison to existing work on in-context exploration by LLM agents in a new paragraph in the related work section (Section 2), explicitly stated in our response to Reviewer iKvo.

We hope this clarifies the contribution and would be grateful if the reviewer would reconsider their assessment and score. We are happy to continue the discussion and to further clarify additional details.

I would like to thank the authors for their responses. The clarifications are helpful, especially the new experiments. I am afraid I cannot give a "weak acceptance", as my main concern remains.

The paper is essentially an empirical evaluation of LLMs' capabilities on a particular decision-making task, without a new methodological contribution. Therefore, I believe the significance and practical relevance of the task itself are particularly important.

Conceptually, balancing the exploration against search costs is certainly an important problem. However, the original task is highly artificial. As expressed in my original comments, I believe the strength of LLMs in decision-making lies partly in their ability to leverage contextual information and knowledge.

The new medical diagnosis experiment is more closely aligned with this motivation, and I find it interesting. However, it still has some impractical simplifications, e.g., the LLM is not allowed to actively choose which information to acquire about the patient.

Overall, I remain unconvinced that the experimental tasks are sufficiently representative of practical decision-making scenarios. I sincerely appreciate the authors' efforts in conducting the additional experiments and providing thoughtful responses.

Reviewer_KGRfOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The rapid advancement of Large Language Models (LLMs) has led to their increasing role in scenarios requiring information acquisition and decision-making. Since acquiring information incurs costs, a factor that must be carefully weighed in economically significant contexts, this paper investigates how LLMs balance the value of search against the associated search costs within dynamic economic environments. The paper finds that LLMs adapt their behavior based on the cost. This includes employing different reasoning and search strategies, as well as adjusting their reasoning processes to observe more states. Enabling reasoning capabilities enhances the models' sensitivity to costs. Furthermore, experiments conducted on WebShop reveal that while reasoning models remain sensitive to costs, their performance becomes less consistent.

Critical strengths

This paper constructs a test environment in which the Bayesian optimal policy can be solved. By evaluating the performance of various models within this environment, the paper identifies the gaps between the behaviors of these models and the optimal behavior, thereby offering a new perspective for research.

Critical weaknesses
  1. Although the WebShop environment itself is designed to resemble real-world settings, its implementation in this paper incorporates several features that deviate from reality. For instance, the absence of an evaluation regarding the quality of the final purchase decision, and the introduction of an explicit monetary cost for viewing each additional item (Lines 2329–2337). These aspects diminish the relevance of this part of the experiment to real-world scenarios.
  2. The real-world scenario in the experiment is limited to online shopping, a rather narrow scope that may not be representative of the broader range of scenarios in which LLMs are used for decision-making. Other typical real-world scenarios involving LLM-based decision-making where outcomes are sensitive to search costs include information retrieval and programming. In these contexts, one must weigh factors such as retrieval time against search quality, or the costs of program execution against the risk of errors.
Questions for authors

Q1: In the WebShop experiments, could you provide additional reporting on the quality of the final purchasing decisions (e.g., a matching score for the purchased items)? For instance, by evaluating the accuracy of the models’ decisions under various search cost configurations?

Decision quality is a critical factor that must be considered; if a model significantly compromises decision quality in an effort to control costs, such decisions cannot be considered reasonable.

Q2: Why does the WebShop experimental setup impose a monetary cost for retrieving each product? Could a more realistic configuration for search costs be considered to enhance the persuasiveness of the experiments?

The prompts in WebShop (lines 2329–2337) explicitly state: "Reviewing each additional product costs ${search_cost} in time and fees, and the customer bears that cost in addition to the price of the item." Why should search costs be factored directly into a customer’s budget for purchasing goods? Could the authors please explain the rationale behind this design choice, or perhaps consider a more realistic approach to modeling search costs?

Q3: Could the real-world experiments be expanded to include tasks related to information retrieval or programming?

Drawing inspiration from related work mentioned in the paper (Line 151) "Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents" (https://arxiv.org/abs/2602.16699), you could incorporate information retrieval tasks (such as Search-R1, PopQA) to evaluate the trade-off between search time costs and search quality. This would broaden the range of application scenarios covered by the experiments.

5 comments
AuthorsOpenReview ↗

Author response to Reviewer KGRf [1/2]

Thank you for your thoughtful comments and for believing that our contribution offers a new perspective for research on AI agents’ information acquisition and decision-making under uncertainty. We answer your concerns and questions below and we aggregate our answers to the weaknesses and questions according to the topic.

W1, Q1 & Q2. You raised a fair question about the WebShop cost configuration, and we are glad to explain the rationale behind this design choice and why we believe this doesn’t change the relevance of this experiment to real-world scenarios.

We charge a cost for viewing each new item to reflect the latent cost an economic agent incurs when evaluating a product. This can be thought of as a token spent by an LLM or an implied dollar cost of time spent searching by a human or LLM. We make these search costs monetary to make the trade-off from exploration explicit and salient to the agent. We factor the cost into a customer’s budget specifically so that this budget reflects the complete acquisition cost of search and the product’s price. Additionally, the only notion of cost in the WebShop environment is budget, so we prefer to implement our search costs in a minimally invasive manner to the environment rather than imposing a more contrived and less realistic setup. Even though this environment diverges from our main experiments in multiple ways, we expect a priori that agents placed in it should still respond to rising search costs with reduced exploration. Thus, we believe that this configuration is still meaningful for evaluating agents’ search cost elasticity in a similar manner to our main experiments while still maintaining a realistic task environment.

To address your concern about agents significantly sacrificing decision quality, we evaluate agents’ purchase decisions. We have the LLM behind each agent judge the quality of their own purchases based on the exact prompt the decision-making agent was provided with to as closely match the decision-making agent’s assessment of quality during search. Judges report a score from 0 to 100. The results are shown below and demonstrate that judged product quality is similar across costs for each agent, which leads us to conclude that agents are not unreasonably sacrificing quality in order to save on search costs.

ModelSettingJudge reasoning0.05%0.15%0.5%All costs
GPT-5.4Non-reasoningdisabled46.0 ± 2.946.4 ± 2.845.7 ± 2.846.1 ± 2.7
Reasoning, turn costenabled45.2 ± 2.845.8 ± 2.844.8 ± 2.845.2 ± 2.7
Reasoning, token costenabled45.0 ± 2.844.2 ± 2.743.8 ± 2.744.3 ± 2.7
GPT-5.4 NanoNon-reasoningdisabled41.3 ± 3.240.4 ± 3.339.7 ± 3.140.5 ± 2.8
Reasoning, turn costenabled37.0 ± 3.236.0 ± 3.237.0 ± 3.336.7 ± 3.0
Reasoning, token costenabled38.5 ± 3.238.1 ± 3.237.2 ± 3.338.0 ± 3.1
Qwen3.6 FlashNon-reasoningdisabled40.2 ± 2.939.0 ± 2.939.5 ± 3.139.6 ± 2.7
Reasoning, turn costenabled63.5 ± 4.965.6 ± 4.663.1 ± 4.764.1 ± 4.4
Reasoning, token costenabled65.6 ± 4.665.2 ± 4.765.5 ± 4.965.4 ± 4.5
Inkling SmallNon-reasoningdisabled46.3 ± 3.744.8 ± 3.844.7 ± 3.745.3 ± 3.5
Reasoning, turn costenabled36.5 ± 3.836.5 ± 3.934.4 ± 3.835.8 ± 3.1
Reasoning, token costenabled31.8 ± 3.834.7 ± 4.033.7 ± 3.733.4 ± 3.2
Nemotron 3 UltraNon-reasoningdisabled47.2 ± 3.646.3 ± 3.646.6 ± 3.746.7 ± 3.5
Reasoning, turn costenabled37.4 ± 3.438.0 ± 3.437.1 ± 3.437.5 ± 3.3
Reasoning, token costenabled37.4 ± 3.438.0 ± 3.438.1 ± 3.637.8 ± 3.3

We added these results to Appendix B.4 (p. 25), where Figure 14 (p. 29) plots the judged score of each agent’s purchase across search costs.

AuthorsOpenReview ↗

Author response to Reviewer KGRf [2/2]

W2 & Q3. To provide more real-world application scenarios, we add a sequential information-retrieval task in medical diagnosis built on DDXPlus (Fansi Tchango et al., 2022). DDXPlus is a dataset with synthetic patients, each recorded with age, sex, a chief complaint, a ground truth pathology, and further findings (symptoms and medical history). We adapt this environment by adding an explicit price on retrieving information from the patient. In DDXPlus, no single finding of the patient can fully determine their pathology, so the agent decides by finding whether more evidence would sharpen the diagnosis enough to be worth its cost, which allows the depth of search to respond to cost. For each patient, the agent sees the patient's age, sex and chief complaint for free. It can then request the patient's recorded findings one at a time, up to 20, paying a cost for each. When it stops, it reports a probability distribution over the pathologies it considers possible out of 49 candidate pathologies. The following table reports the cost elasticity of search in DDXPlus when cost rises from 0.15% to 0.5%, and the pattern largely follows the main experiment (Figure 3 of our paper on p. 7). Without reasoning, search is essentially cost-insensitive. No non-reasoning elasticity is distinguishable from zero. With reasoning on and an explicit per-finding price, every model has a positive elasticity and exceeds the non-reasoning elasticity. We also find that cost sensitivity is weaker under token pricing, with the exception of Qwen 3.6 Flash. In about 12% of its token-arm games, Qwen reached the provider’s output limit while still reasoning and never gave a valid diagnosis. This adds noise to that result. By comparing LLMs’ diagnosis with the ground-truth pathology in DDXPlus, we also confirm that there are only slight changes in decision quality for cost-sensitive agents when the price rises.

ModelSettingCost elasticity of searchChange in diagnosis accuracy (%)
GPT-5.4Non-reasoning0.031 ± 0.0330.0
Reasoning, turn cost0.055 ± 0.030−2.0
Reasoning, token cost0.049 ± 0.041+1.5
GPT-5.4 NanoNon-reasoning0.004 ± 0.028+3.5
Reasoning, turn cost0.035 ± 0.047−2.0
Reasoning, token cost0.030 ± 0.038−2.5
Nemotron-3 UltraNon-reasoning00.0
Reasoning, turn cost0.029 ± 0.021−2.0
Reasoning, token cost0.009 ± 0.023+1.5
Inkling SmallNon-reasoning−0.055 ± 0.086+5.0
Reasoning, turn cost0.046 ± 0.025−2.0
Reasoning, token cost0.004 ± 0.024−2.5
Qwen3.6 FlashNon-reasoning−0.021 ± 0.0260.0
Reasoning, turn cost0.015 ± 0.051−0.5
Reasoning, token cost0.080 ± 0.042−2.0

We added this experiment to the paper in Section 4.4.2 (p. 9), where Figure 5(b) plots these elasticities. Appendix C (p. 26) gives the full configuration, the mean number of findings requested, and diagnosis accuracy for every model and arm.

We really appreciate your questions that helped to improve our paper and the frame of our contribution. We hope we have addressed all the points raised but let us know if there are further questions that require more clarification. We would appreciate it if you could raise your score and recommend our paper for acceptance if you believe that we have answered all your concerns and we are happy to continue engaging with you in the discussion.

Reference:

Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. DDXPlus: A new dataset for automatic medical diagnosis. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2022.

Thanks for the response and the new experiments. Q2 and Q3 are addressed. The cost rationale makes sense, and the DDXPlus experiment is a reasonable and interesting second domain.

But I still have concerns with Q1. The new quality analysis has each model grade its own purchases. WebShop already provides a native score against the reference product (see Section 3.1, Task Formulation, in [1]), and I would have found that more convincing. Self-grading makes the scales incomparable across models (Qwen3.6 Flash with reasoning gives itself 64 while others give 33 to 47), and a high grade does not show the correct product was bought. Even taking the grades at face value, they are low: around 46/100 for most models, although a fully matching product is guaranteed in every list (B.1). This suggests agents are mostly buying partial matches.

Therefore, I keep my rating. My expertise here is very limited and I cannot judge the value of the core contribution, so my confidence is 1 and the AC could down-weight this review.

[1] Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022). Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35, 20744-20757.

Reviewer_iKvoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission studies LLMs’ abilities to judge when obtaining more information is worth the cost. This is done by deploying LLMs as agents in a sequential game where at each round, the agent observes a numerical reward and chooses to stop, or to pay some cost to observe another reward. When the agent stops, they receive the highest reward they have seen so far. Concretely, rewards are drawn from a normal distribution whose mean and variance are unknown to the agent. However the agent does know the prior from which these parameters are drawn. Given this prior, the authors derive a Bayes-optimal benchmark to evaluate the LLMs against.

The rules of the game and history of play are explained to the LLMs in-context. The authors find that reasoning models generally earn higher rewards, with GPT-5.4 and Inkling Small performing close to the Bayes-optimal benchmark. With reasoning disabled, GPT-5.4 still performs close to the baseline, but other models see a significant drop in performance.

In the next set of experiments, the authors charge the models according to the number of tokens they generate, instead of for each reward they observe. This allows models to adjust both how much they think per decision and how many observations they decide to see. Under this setup, most models choose to think less per decision and see more observations. However one model (Nemotron 3 Ultra) takes more decisions and does not reduce its token usage per decision, thus causing its total cost to rise.

Finally, the authors deploy LLMs as agents in a modified WebShop environment, in which agents shop based on product and price constraints. (The modification is to add a search cost, and to allow the agents to decide how many products to search through before selecting one to buy.) In this setting, the results roughly track with the previous results in the purely numerical experiments, which the authors interpret as evidence that the observed behavior at least partially carries over to economically realistic contexts.

Critical strengths

This is the first paper I have seen that studies this type of exploration over time in the context of LLM decision-makers. Benchmarking and diagnosing LLMs' behavior in these settings is interesting/important because, as the authors point out, such exploration-exploitation trade-offs naturally arise in real-world decision-making settings. I found the authors' experiments to be fairly comprehensive and carried out well.

Another strength of the paper is the writing, as I had a relatively easy time following along.

Critical weaknesses

The biggest weakness is that there are a bunch of glaring omissions from the related work section. In particular, there is an entire literature on evaluating the ability of LLMs to solve exploration-exploitation tasks. This omission is especially puzzling since the authors include sections on "Bandits and Bayesian learning" and "Sequential decision-making with LLM agents" in their related work. Two of the first papers along this direction are "Can large language models explore in-context?" by Krishnamurthy et al (NeurIPS 2024) and "Do LLM Agents Have Regret? A Case Study in Online Learning and Games" by Park et al (ICLR 2025), although there is a fairly large line of work that builds off of these papers which is also quite related. While the type of exploration studied in these papers is generally different than the type studied in this paper, it is hard to contextualize the results of this paper without a discussion on/comparison to this line of work.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer iKvo

Thank you for the careful review, and for catching a gap in our related work section. Leaving out the work on in-context exploration in this section was a clear oversight and we are fixing it for the camera-ready version. Here is the new paragraph exactly as it will appear in Section 2 (p. 4) of the revision. Please check the revised paper as well. Let us know if there are still references that we are still missing and we will be happy to discuss them in our related work.

In-context exploration by LLM agents. A growing literature asks whether LLMs can resolve the exploration–exploitation tradeoff in-context. Binz and Schulz (2023) find that GPT-3 achieves lower regret than humans in-context in a two-armed bandit task, but with no signature of directed exploration. Krishnamurthy et al. (2024) test GPT-3.5, GPT-4, and Llama2 in multi-armed bandits with environment description and interaction history in-context. They find that only GPT-4 with CoT and externally summarized history explores robustly. Building on prior work, Nie et al. (2025) introduce a benchmark spanning multi-armed bandit and contextual bandit and enhance LLM’s in-context exploration using classical bandit algorithms. Park et al. (2025) study LLM agents in online learning and repeated games, finding that even strong models can fail to be no-regret in simple cases and proposing an unsupervised regret-loss for training Transformers to be no-regret. Pan et al. (2025) find that most LLMs underperform compared to humans in exploration during an open-ended task and attribute this to models making premature decisions before weighing the future potential of exploring. Schmied et al. (2026) similarly identify greediness, frequency bias, and a knowing-doing gap in LLMs on multi-armed bandit tasks. In these studies, agents obtain information through interaction with the environment, without a separate information-acquisition cost. We differ from this literature by charging agents an explicit cost from the reward for each additional draw. This cost setting allows us to evaluate how agents adjust their exploring behavior in response to cost changes. With a computable optimal policy, we show that no agents are sufficiently sensitive to cost and that they often explore less than is optimal. We further examine how search behavior changes when the agent pays for its own reasoning tokens, which makes deciding how much to think part of the search problem.

To explain the distinction between our contribution and previous ones, we will also add one sentence to Section 4.2 (p. 8):

“Taking Figure 2 and Figure 3 together, although reasoning models like GPT-5.4 may achieve net rewards close to the Bayes-optimal policy at the tested cost levels, their exploration elasticities remain lower. Thus, near-optimal realized payoffs alone do not necessarily imply that agents adjust their search behavior optimally when exploration costs change. This distinction matters for comparison with prior evaluations of in-context exploration, which score agents by regret or reward in a fixed environment (Krishnamurthy et al., 2024). By such a metric, GPT-5.4 would appear near-optimal, whereas its elasticity shows that it under-adjusts when exploration costs more. Appendix D provides a set of additional tests of agents’ decision-making behavior, finding that reasoning models make decisions more consistent with the Bayes-optimal policy.”

Thank you again. This addition makes our contribution easier to place and let us know if there is anything else we could answer to improve the readability of the paper or any questions you might have.

References:

Marcel Binz and Eric Schulz. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120, 2023.

Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? In Advances in Neural Information Processing Systems, volume 37, 2024.

Allen Nie, Yi Su, Bo Chang, Jonathan N. Lee, Ed H. Chi, Quoc V. Le, and Minmin Chen. EVOLvE: Evaluating and optimizing LLMs for in-context exploration. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 2025.

Lan Pan, Hanbo Xie, and Robert C. Wilson. Large language models think too fast to explore effectively. In Advances in Neural Information Processing Systems, volume 38, 2025.

Chanwoo Park, Xiangyu Liu, Asuman Ozdaglar, and Kaiqing Zhang. Do LLM agents have regret? A case study in online learning and games. In The Thirteenth International Conference on Learning Representations, 2025.

Thomas Schmied, Jörg Bornschein, Jordi Grau-Moya, Markus Wulfmeier, and Razvan Pascanu. LLMs are greedy agents: Effects of RL fine-tuning on decision-making abilities. In The Fourteenth International Conference on Learning Representations, 2026.

Thanks for your reply, I will maintain my positive score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Author Response to All Reviewers [1/2]

Our goal in this paper is to study LLM agent behavior in costly information acquisition tasks that are relevant across many economic and decision-making settings. We develop a sequential search task where exploration is costly and agents must assess the value of continued search against the best reward they have observed and the marginal cost of an additional observation. This requires agents to adapt their behavior to the cost of search and reason over their beliefs about the distribution of rewards. Our work departs from the previous literature by (i) directly pricing the cost of search and evaluating agents' ability to adapt to changes in this cost, (ii) developing an optimal policy for our task that we can use as ground truth in our evaluations of agents, and (iii) evaluating agents' search behavior when they are charged for each hidden reasoning token instead of for each interaction with the environment. Then, we tested diverse families of models and we find that strong capability at this task is only exhibited by the newest reasoning models and that many models fail to perform strongly. We also find that agents adapt to token cost pricing by thinking less in order to explore more states in the environment. We further extend to real-world shopping and medical diagnosis scenarios, finding that reasoning LLMs' cost sensitivity largely generalizes to these settings.

What reviewers found valuable. Reviewers emphasized the novelty of our work. Reviewer iKvo describes it as "the first paper [they] have seen that studies this type of exploration" and that model capability at our tasks have strong real-world implications. They additionally note that the experiments were comprehensive and well-executed. Reviewer KGRf identifies that the work "offers a new perspective for research" by comparing models to an optimal policy. Reviewers iKvo and ZPGs also found that the paper was well-written and easy to understand.

The main requests from the reviewers were:

  1. Extend the real-world scenarios to an information-retrieval task;
  2. Place our work in the literature of in-context exploration by LLMs;
  3. Evaluate the quality of agents' final purchase decisions in Webshop and justify the monetary search cost design;
  4. Justify our contribution and motivation, given that a Bayes-optimal benchmark exists for our task.

We have addressed all of them below and in the revised paper.

AuthorsOpenReview ↗

Author Response to All Reviewers [2/2]

New Contributions to the Work

Medical diagnosis task. We added a sequential information-retrieval task in medical diagnosis (Section 4.4.2, Appendix C). For each patient, the agent sees age, sex and chief complaint for free. It can then request the patient's recorded findings one at a time, paying an explicit cost for each. When it stops, it reports its diagnosis as a probability distribution over possible pathologies. We find that the patterns of cost elasticity in our main experiment largely generalize to this setting. Without reasoning, search is insensitive to cost. With reasoning and a per-finding price, every model has positive cost elasticity and exceeds its non-reasoning elasticity. Cost sensitivity is also generally weaker under token pricing.

In-context exploration related work section. We reviewed the literature on in-context exploration by LLM agents and contrasted our work from it. In those settings, agents obtain information through interaction with the environment and generally incur an implicit opportunity cost. We instead charge an explicit cost for each additional information observation or for the agent's own reasoning tokens, which lets us measure how agents adjust exploration as costs change against a computable optimal policy. We also note in Section 4.2 that GPT-5.4 earns near-optimal rewards but under-adjusts when exploration costs more. A metric based on regret or reward would miss this.

WebShop performance analysis. To address the concern that agents might give up decision quality to save on search costs, we asked each agent's LLM to judge the quality of its own purchases on a 0–100 scale, given the exact prompt the agent saw. For every agent, the variance of judged quality across costs is not statistically significant, which indicates that agents do not unreasonably sacrifice quality to save on search costs (Appendix B.4, Figure 14). We also elaborate further in our response on how the per-item search charge is a strong model for the latent cost the agent incurs when evaluating a product, such as tokens costs or the implied dollar cost of time spent searching. We define the cost monetarily to the agent so that the exploration tradeoff with the products it is searching for is explicit and salient. Budget is the only direct notion of cost in WebShop, so charging search against it is the least invasive change to the environment and the most realistic for how an agent would incur the cost of search.

Assessment of agent performance compared to the optimal policy. An optimal policy is very useful in our work to measure how far away agents' decisions are from ground-truth optimal decisions and whether they sufficiently adjust their exploration when costs change. We deliberately build an environment such that we have access to this optimal policy but that it is still challenging for frontier LLM agents to recover during evaluations, which has also been shown in our answer to reviewer ZPGs, when we empirically verify this with an additional set of 120 evaluations on a GPT-6 Astra agent. We find that its mean states explored differ from that of the optimal policy with 95% significance, which is also true for all other agents in our work. Agents therefore need to act on heuristics, as they would in real-world settings with no available optimal policy. We are not aware of prior work that builds an environment that is both challenging for LLMs and equipped with a computable optimal policy. Existing evaluations either lack a meaningful ground truth or use bandit-style opportunity costs instead of explicit search costs.

Our work presents a novel evaluation of LLM agent capabilities at responding to changes in search costs. We develop a unique task environment with an optimal policy to compare agent decisions with ground truth and connect this environment to a series of additional task environments that generalize our stylized model to the real world. We hope that the real-world economic implications of agents who can effectively respond to search costs is of interest to reviewers, area chairs, and senior area chairs and that this work is impactful for readers' understanding of LLM agent capabilities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.