- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates why large language models (LLMs) tend to underexplore in sequential decision-making tasks and whether metacognitive guidance can improve their exploration–exploitation balance.
The authors evaluate ten LLMs across three human-referenced bandit tasks and use computational cognitive modeling to characterize their decision-making strategies. They find that native reasoning disproportionately amplifies sensitivity to estimated value relative to uncertainty, leading to overly greedy choices and premature exploitation. Through reasoning-trace analysis, the authors identify a "value-to-action" shortcut, which they attribute to insufficient metacognitive monitoring and control.
To address this issue, the authors introduce structured metacognitive prompting that encourages models to assess value, uncertainty, confidence, and exploration–exploitation demands before making decisions. Their results suggest that this intervention promotes early uncertainty-directed exploration and improves subsequent exploitation. Additional experiments with longer-horizon bandit tasks show improvements in optimal-arm selection and cumulative rewards.
Critical strengths
-
Systematic empirical evaluation. The paper evaluates ten LLMs across multiple human-referenced bandit tasks, with additional experiments in longer-horizon settings. This provides reasonably broad empirical support for the findings.
-
Interpretable computational analysis. The authors use established cognitive models to separate value and uncertainty sensitivity, rather than relying only on behavioral measures. The finding that native reasoning disproportionately increases value sensitivity is interesting.
-
Consistent intervention effects. The proposed prompting strategy improves exploration behavior and subsequent decision performance across several settings. The authors also examine trial-level dynamics, providing evidence that the improvements involve earlier information gathering followed by better exploitation.
Critical weaknesses
1. Limited novelty and substantive advancement over prior work.
My main concern is how much the paper advances our scientific understanding of LLM exploration–exploitation. Prior studies have already investigated exploration failures from multiple perspectives, including in-context exploration and prompting (Krishnamurthy et al., 2024), computational modeling, reasoning processes, and internal representations (Pan et al., 2025), as well as algorithm-guided interventions and RL fine-tuning (Nie et al., 2025; Schmied et al., 2026; Arumugam & Griffiths, 2026).
In particular, Pan et al. (2025), which is not discussed, studied premature decision-making in LLM exploration through computational modeling, systematic CoT analysis, and mechanistic interpretability. The authors should better clarify how their proposed explanation advances beyond these existing findings.
While the paper combines established cognitive models, reasoning-trace analysis, and metacognitive prompting, it remains unclear what substantially new scientific understanding emerges from this combination. The proposed metacognitive explanation is potentially interesting, but its additional explanatory value over existing accounts is not sufficiently established.
2. Insufficient justification for interpreting value sensitivity as excessive.
The authors find that LLMs exhibit stronger value sensitivity than humans and interpret this as excessive exploitation. However, human behavior is not necessarily optimal in bandit tasks, and stronger value sensitivity does not necessarily imply worse decision-making. Larger fitted coefficients may also reflect differences in choice stochasticity.
Since these tasks have well-established algorithmic solutions, comparisons with appropriate normative baselines, such as Bayes-optimal policies where tractable, Thompson sampling, or UCB, would help clarify whether the observed behavior reflects suboptimal exploration. Applying the same cognitive model to these algorithmic policies could also provide a more meaningful reference for interpreting the fitted coefficients.
3. Insufficient evidence for the proposed metacognitive mechanism.
The authors interpret the observed value-to-action reasoning pathways as evidence of insufficient metacognitive monitoring and control. However, these patterns do not uniquely imply a metacognitive deficit. They could also arise from post-training, decoding configurations, or simpler first-order decision policies.
For example, Pan et al. (2025) showed that changing sampling temperature significantly affects uncertainty-driven exploration without a corresponding significant effect on empowerment-driven exploration. This suggests that exploration strategies can be influenced by inference configurations without necessarily involving metacognitive mechanisms.
More importantly, although the authors collect confidence reports, they do not systematically examine whether confidence tracks the reliability of value judgments or functionally regulates subsequent decisions. Such analyses would provide more direct evidence for the proposed metacognitive account.
4. Limited ablation and mechanistic analysis of the intervention.
Although the authors compare monitoring-only and monitoring-plus-control prompts, the intervention combines several components, including value assessment, uncertainty assessment, confidence reporting, and explicit exploration–exploitation guidance.
It remains unclear which components actually drive the improvements. Simply reminding models to consider uncertainty, or requiring structured value–uncertainty–choice reasoning, might produce similar effects without engaging metacognition. More targeted ablations would help distinguish these explanations.
Since the paper primarily aims to explain why LLMs underexplore, the intervention should ideally provide evidence that distinguishes the proposed metacognitive mechanism from simpler alternatives, rather than merely demonstrating improved performance.
References
- Arumugam, D., & Griffiths, T. L. (2026). Toward efficient exploration by large language model agents. ICLR 2026. https://arxiv.org/abs/2504.20997
- Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., & Slivkins, A. (2024). Can large language models explore in-context? NeurIPS 2024. https://arxiv.org/abs/2403.15371
- Nie, A., Su, Y., Chang, B., Lee, J., Chi, E. H., Le, Q. V., & Chen, M. (2025). EVOLvE: Evaluating and optimizing LLMs for in-context exploration. ICML 2025. https://proceedings.mlr.press/v267/nie25b.html
- Pan, L., Xie, H., & Wilson, R. C. (2025). Large language models think too fast to explore effectively. In Advances in Neural Information Processing Systems (NeurIPS 2025). https://proceedings.neurips.cc/paper_files/paper/2025/file/945c4b5d6daf5294befbc5b7c275c300-Paper-Conference.pdf
- Schmied, T., Bornschein, J., Grau-Moya, J., Wulfmeier, M., & Pascanu, R. (2026). LLMs are greedy agents: Effects of RL fine-tuning on decision-making abilities. ICLR 2026. https://openreview.net/forum?id=cf2fcda40de8520027af2eadcafdecd6
Questions for authors
I currently do not have any questions.
11 comments
Response to Reviewer hBzC (2/3)
W2 — Is higher value sensitivity "excessive"? Human reference, stochasticity and algorithmic baselines.
We agree that humans are not optimal, and we do not use them as a target. The human comparison was diagnostic: people in these tasks show uncertainty-directed exploration and LLMs behave greedily, so we fitted the same model to both to locate the difference. All ten models exceed the human mean in every setting (1.99–8.60×), whereas 28 of 30 model-level means lie within the central 95% of human coefficients (Appendix C.1.1). This led us to ask what native reasoning lacks relative to human decision-making — monitoring and control of provisional value judgments — which we then tested.
-
Excessive exploitation is established from behavior, not from coefficients. We use the definition of exploitation in the exploration–exploitation dilemma: choosing the option currently judged most valuable. Because the models judge values accurately (their reported better arm matches the empirical-value winner in at least 99.7% of trials; Appendix D.4), following the value winner is exploitation, and a very high greediness rate means persistent exploitation. Behaviorally:
- In the two-arm tasks, once both arms are sampled, Native follows the empirical-value winner in 98.29% / 99.75% / 98.53% of choices (Settings 1/2/3; original Table C.1).
- In the longer four-arm task, models stop exploring early. For example, native GLM-5.1 samples the optimal arm once, receives a zero, never returns to it, and follows a stable 5-point arm for the rest of the task (revised Figure 7).
- In the five-arm task, native GPT-5.6 Luna behaves almost like pure Greedy: its non-greedy choice stays low throughout, nearly matching Greedy (Figure 8a).
Our metacognitive guidance leads the models to form a strategy of early directed exploration followed by better-informed exploitation, which gives an overall advantage, especially over longer horizons (table below). Native reasoning therefore exploits too early, giving up exploration before enough information is gathered, while exploring still pays off. As stated in the paper, we fitted the sensitivities after observing this behavior, to explain it.
| Longer-horizon task (Native → Meta-M+C) | Late optimal choice | Cumulative reward |
|---|---|---|
| Four-arm, ten models (final ten trials; Section 6, Figure 6) | 37.7% → 59.2% | 282.0 → 300.5 |
| Five-arm, GPT-5.6 Luna (T51–100; Section 7, Figure 8, Table F.5) | 35.4% → 53.2% | 46.5 → 50.4 |
- Thompson sampling under the same cognitive model (new Appendix B.4). We added Thompson sampling, simulated in the same two-arm tasks and fitted with the same pipeline as the 217,600 LLM decisions per condition. Its value sensitivity is lower than native reasoning's in all three settings (Settings 1/2/3: 0.60 / 2.78 / 5.45 vs. 3.16 / 6.69 / 7.26).
- UCB and Thompson as executed baselines. Beyond the fitted Thompson reference, the expanded five-arm benchmark (new Section 7) executes UCB and Thompson directly for GPT-5.6 Luna and, in Appendix F.2.4, for Qwen3-4B and Qwen3-14B, together with prompts asking each model to follow these algorithms. The UCB-guided prompt was also tested in the four-arm task (Appendix E.5). Behaviorally, the algorithms do not converge: they keep a high exploration rate late in the task, when exploring has little value left (Figure 8a). Metacognitive guidance instead produces directed exploration early and then shifts to efficient exploitation. The algorithmic baselines generally fall short of it in late optimal choice and reward (e.g., Luna late optimal choice 53.2% vs. 41.2% for UCB and 37.0% for Thompson).
- Choice stochasticity. Larger coefficients could in principle reflect less random choice. But if reasoning only made choices less random, all coefficients would scale up together, keeping their signs and ratios. This is not what we observe: reasoning raises value sensitivity about 2–3×, whereas uncertainty sensitivity () stays flat in Setting 1 (0.25→0.23) and changes sign, from negative to positive, in Settings 2–3 (−0.32→0.45; −0.42→0.07). A common rescaling cannot change a sign. Metacognitive guidance then lowers value sensitivity while raising uncertainty sensitivity, again the opposite of a common rescaling. Both patterns hold under boundary exclusion and L2 refits (Table B.3).
Response to Reviewer hBzC (3/3)
W3 — Alternative explanations and the function of confidence.
We agree that these patterns do not uniquely imply a metacognitive deficit, and we do not claim that they do. Our account is an interpretation built on observed behavior: comparing human decision processes with the models' reasoning traces, we found that native reasoning rarely checks the reliability of a value judgment before acting, and this investigation led us to metacognitive guidance. We do not exclude other contributing factors, such as post-training. What the work does provide is a simple, task-independent way to make LLM exploration–exploitation behavior more balanced, for anyone who wants to reduce LLM greediness without external algorithms or fine-tuning. We also examined the alternatives you raised:
- Decoding configuration (new Appendix E.6). We had run temperature sweeps during the study, and your comment showed how relevant they are, so the revision now reports them. We varied the sampling temperature of Native reasoning in our two-arm task over ten levels from 0 to 1.6 (Qwen3-14B) and four levels from 0 to 1.5 (Qwen3-8B, Qwen3-32B). Temperature barely changed exploratory choice: Qwen3-14B's non-greedy choice stayed below 1% at every level, and Qwen3-8B and Qwen3-32B also stayed essentially unchanged (0.25–0.47% and 1.97–2.19%). (Many closed-source reasoning models no longer allow the temperature to be set when thinking is enabled, so these sweeps use open-weight models.)
- Simpler first-order policies. We also tested simpler prompts (W4): an explore-more instruction and a value→uncertainty→choice prompt did not reproduce the shift, and an uncertainty question alone raised early departures but kept departing late, when further information had little value.
Does confidence track judgments and regulate choice? We extended the confidence analysis (new Appendix F.1.6; four-arm Meta-M+C, ten models). Controlling for block, trial and observed evidence, +1 within-model SD of confidence is associated with:
| Endpoint | Association (pp) | 95% CI | Models negative / positive |
|---|---|---|---|
| Departure from own reported value winner | −25.33 | [−30.17, −20.48] | 10 / 0 |
| Next-trial same-arm persistence | +9.55 | [+6.77, +12.32] | 0 / 10 |
| Next-trial empirical non-greedy choice | +0.22 | [−3.32, +3.76] | 5 / 5 |
Higher confidence goes with acting on the reported judgment and with subsequent commitment, not with a general reduction of exploration. Value reports are slightly more accurate at higher confidence (agreement with the empirical ranking 98.1% vs. 95.3%), but accuracy is high at both levels; confidence is more clearly associated with acting on the report than with its accuracy. These are observational associations, not evidence of calibration or causation.
W4 — Which components drive the effect?
We added targeted ablations (new Appendix E.5; Qwen3-32B, Setting 3, 13,800 decisions each). Non-greedy choice (%) early (T3–4) / late (T8–10), and final-trial optimal choice (%):
| Prompt | Early / late non-greedy | Optimal T10 |
|---|---|---|
| Native | 1.54 / 2.32 | 88.26 |
| Uncertainty only | 10.46 / 3.70 | 89.57 |
| Value, then uncertainty, then choice | 0.35 / 0.43 | 90.58 |
| Format-matched task report | 0.04 / 0.35 | 91.30 |
| Our assessment fields, no guidance | 5.20 / 1.01 | 91.52 |
| Meta-M guidance, choice only | 1.77 / 0.87 | 92.39 |
| Meta-M+C guidance, choice only | 10.03 / 0.88 | 93.91 |
| Meta-M | 10.18 / 0.54 | 92.03 |
| Meta-M+C | 29.89 / 1.15 | 92.83 |
An uncertainty question alone raises early departures but keeps departing late, when new information has little use left, and structured value–uncertainty–choice reasoning stays greedy. Every metacognitive ablation keeps the same trend, more early exploration and less late departure than Native, with reduced strength, so each component of the current prompt contributes. The same pattern appears among the five-arm metacognitive variants (Section 7). We stress again that our contribution is a direction of intervention, not a specific prompt. Grounded in human cognitive science and metacognitive theory, it shows that scaffolding metacognitive processes in LLM reasoning, without external algorithms or direct instructions on behavior, lets the models spontaneously develop an adaptive, balanced exploration–exploitation strategy.
Response to Reviewer hBzC (1/3)
Thank you for the thorough review and pointers to related work; because of length, we reply in three parts.
W1 — Scientific advance beyond prior work, especially Pan et al. (2025).
Our question starts from the exploration–exploitation dilemma: exploitation secures immediate reward, whereas directed exploration carries an opportunity cost but gathers information that supports future decisions. We ask why reasoning-capable LLMs, despite being able to analyze this trade-off, still make excessively greedy decisions, and whether guidance can lead them to systematically change their decision strategy through their own reasoning, without external algorithms or fine-tuning. Krishnamurthy et al. document insufficient exploration and identify one successful engineered prompt for GPT-4; Nie et al., Arumugam & Griffiths and Schmied et al. improve exploration through algorithmic knowledge or training (all cited in our original submission). Our contribution:
Diagnosis. We find that reasoning speeds coverage but amplifies value sensitivity far more than uncertainty sensitivity, so post-coverage choices follow the provisional value winner in 98–99.8% of cases. This helps reconcile conflicting reports on whether reasoning aids exploration.
Process. Reasoning traces show that native reasoning often maps a provisional value comparison directly to action.
Direction, not a prompt. We show that guiding reasoning toward monitoring and control systematically reshapes the strategy toward early exploration and later, better-informed exploitation, across ten models and longer horizons; several instantiations shift behavior in the same direction.
Simpler explanations do not account for this shift. We tested four alternatives directly:
By contrast, metacognitive guidance concentrated exploration early and then gradually shifted to exploitation as evidence accumulated.
On Pan et al.: we did not discuss it because it addresses a different question in a different task. Pan et al. explicitly ask LLMs to explore (in Little Alchemy 2, the objective is to create as many elements as possible) and study how LLMs explore when exploration is the goal. We study whether and when LLMs choose to explore in an exploration–exploitation dilemma, where exploring forgoes immediate reward but gathers information that supports future decisions. Findings from an exploration-as-objective task therefore do not transfer directly to this dilemma. Temperature illustrates this: Pan et al. report temperature-related effects on exploration in their task, whereas in our dilemma, where exploration carries an opportunity cost, sampling temperature barely changes LLM decisions (see W3).