acceptodds
Under review as a conference paper at ICLR 2027

Thinking with Metacognition: Rebalanced Exploration-Exploitation in LLMs

Abstract

Despite recent advances in reasoning capabilities, Large Language Models (LLMs) often exhibit strikingly greedy behavior, exploring far less than humans in sequential decision-making tasks. We investigate this discrepancy across three bandit settings with human reference data. To characterize the underlying choice dynamics, we adapt a computational model of human exploration-exploitation (EE) behavior. Across ten LLMs, we find that their native reasoning (i.e., Thinking mode) amplifies sensitivity to estimated value, causing value differences to dominate choice while limiting the strength and persistence of uncertainty-directed exploration. Analyses of reasoning traces reveal that LLMs tend to compare value estimates and translate provisional judgments directly into choices, with little assessment of their reliability or evidential support. Such assessment and regulation of one’s own judgments constitute Metacognitive monitoring and control, whose limited engagement gives rise to a “value-to-action” shortcut. Motivated by this diagnosis, we scaffold metacognitive reasoning in LLMs by prompting them to assess value, uncertainty, confidence, and whether the current state calls for exploration or exploitation, and to use these assessments to regulate choice. Across LLMs and experimental settings, metacognitive reasoning rebalances sensitivity to value and uncertainty, promoting early uncertainty-directed exploration and improving subsequent exploitation. These gains persist in more challenging long-horizon four- and five-arm tasks, where improved optimal-arm selection and cumulative reward show that metacognitive reasoning can support adaptive exploration-exploitation over extended decision horizons.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Sequential decision making, Exploration–exploitation, Metacognitive monitoring and control, Multi-armed bandits

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_hBzCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why large language models (LLMs) tend to underexplore in sequential decision-making tasks and whether metacognitive guidance can improve their exploration–exploitation balance.

The authors evaluate ten LLMs across three human-referenced bandit tasks and use computational cognitive modeling to characterize their decision-making strategies. They find that native reasoning disproportionately amplifies sensitivity to estimated value relative to uncertainty, leading to overly greedy choices and premature exploitation. Through reasoning-trace analysis, the authors identify a "value-to-action" shortcut, which they attribute to insufficient metacognitive monitoring and control.

To address this issue, the authors introduce structured metacognitive prompting that encourages models to assess value, uncertainty, confidence, and exploration–exploitation demands before making decisions. Their results suggest that this intervention promotes early uncertainty-directed exploration and improves subsequent exploitation. Additional experiments with longer-horizon bandit tasks show improvements in optimal-arm selection and cumulative rewards.

Critical strengths
  1. Systematic empirical evaluation. The paper evaluates ten LLMs across multiple human-referenced bandit tasks, with additional experiments in longer-horizon settings. This provides reasonably broad empirical support for the findings.

  2. Interpretable computational analysis. The authors use established cognitive models to separate value and uncertainty sensitivity, rather than relying only on behavioral measures. The finding that native reasoning disproportionately increases value sensitivity is interesting.

  3. Consistent intervention effects. The proposed prompting strategy improves exploration behavior and subsequent decision performance across several settings. The authors also examine trial-level dynamics, providing evidence that the improvements involve earlier information gathering followed by better exploitation.

Critical weaknesses

1. Limited novelty and substantive advancement over prior work.

My main concern is how much the paper advances our scientific understanding of LLM exploration–exploitation. Prior studies have already investigated exploration failures from multiple perspectives, including in-context exploration and prompting (Krishnamurthy et al., 2024), computational modeling, reasoning processes, and internal representations (Pan et al., 2025), as well as algorithm-guided interventions and RL fine-tuning (Nie et al., 2025; Schmied et al., 2026; Arumugam & Griffiths, 2026).

In particular, Pan et al. (2025), which is not discussed, studied premature decision-making in LLM exploration through computational modeling, systematic CoT analysis, and mechanistic interpretability. The authors should better clarify how their proposed explanation advances beyond these existing findings.

While the paper combines established cognitive models, reasoning-trace analysis, and metacognitive prompting, it remains unclear what substantially new scientific understanding emerges from this combination. The proposed metacognitive explanation is potentially interesting, but its additional explanatory value over existing accounts is not sufficiently established.

2. Insufficient justification for interpreting value sensitivity as excessive.

The authors find that LLMs exhibit stronger value sensitivity than humans and interpret this as excessive exploitation. However, human behavior is not necessarily optimal in bandit tasks, and stronger value sensitivity does not necessarily imply worse decision-making. Larger fitted coefficients may also reflect differences in choice stochasticity.

Since these tasks have well-established algorithmic solutions, comparisons with appropriate normative baselines, such as Bayes-optimal policies where tractable, Thompson sampling, or UCB, would help clarify whether the observed behavior reflects suboptimal exploration. Applying the same cognitive model to these algorithmic policies could also provide a more meaningful reference for interpreting the fitted coefficients.

3. Insufficient evidence for the proposed metacognitive mechanism.

The authors interpret the observed value-to-action reasoning pathways as evidence of insufficient metacognitive monitoring and control. However, these patterns do not uniquely imply a metacognitive deficit. They could also arise from post-training, decoding configurations, or simpler first-order decision policies.

For example, Pan et al. (2025) showed that changing sampling temperature significantly affects uncertainty-driven exploration without a corresponding significant effect on empowerment-driven exploration. This suggests that exploration strategies can be influenced by inference configurations without necessarily involving metacognitive mechanisms.

More importantly, although the authors collect confidence reports, they do not systematically examine whether confidence tracks the reliability of value judgments or functionally regulates subsequent decisions. Such analyses would provide more direct evidence for the proposed metacognitive account.

4. Limited ablation and mechanistic analysis of the intervention.

Although the authors compare monitoring-only and monitoring-plus-control prompts, the intervention combines several components, including value assessment, uncertainty assessment, confidence reporting, and explicit exploration–exploitation guidance.

It remains unclear which components actually drive the improvements. Simply reminding models to consider uncertainty, or requiring structured value–uncertainty–choice reasoning, might produce similar effects without engaging metacognition. More targeted ablations would help distinguish these explanations.

Since the paper primarily aims to explain why LLMs underexplore, the intervention should ideally provide evidence that distinguishes the proposed metacognitive mechanism from simpler alternatives, rather than merely demonstrating improved performance.

References

Questions for authors

I currently do not have any questions.

11 comments
AuthorsOpenReview ↗

Response to Reviewer hBzC (1/3)

Thank you for the thorough review and pointers to related work; because of length, we reply in three parts.

W1 — Scientific advance beyond prior work, especially Pan et al. (2025).

Our question starts from the exploration–exploitation dilemma: exploitation secures immediate reward, whereas directed exploration carries an opportunity cost but gathers information that supports future decisions. We ask why reasoning-capable LLMs, despite being able to analyze this trade-off, still make excessively greedy decisions, and whether guidance can lead them to systematically change their decision strategy through their own reasoning, without external algorithms or fine-tuning. Krishnamurthy et al. document insufficient exploration and identify one successful engineered prompt for GPT-4; Nie et al., Arumugam & Griffiths and Schmied et al. improve exploration through algorithmic knowledge or training (all cited in our original submission). Our contribution:

  1. Diagnosis. We find that reasoning speeds coverage but amplifies value sensitivity far more than uncertainty sensitivity, so post-coverage choices follow the provisional value winner in 98–99.8% of cases. This helps reconcile conflicting reports on whether reasoning aids exploration.

  2. Process. Reasoning traces show that native reasoning often maps a provisional value comparison directly to action.

  3. Direction, not a prompt. We show that guiding reasoning toward monitoring and control systematically reshapes the strategy toward early exploration and later, better-informed exploitation, across ten models and longer horizons; several instantiations shift behavior in the same direction.

  4. Simpler explanations do not account for this shift. We tested four alternatives directly:

    • Decoding stochasticity. If greediness came from decoding settings, raising the sampling temperature should increase exploration. In our temperature sweeps of Native reasoning (Qwen3-8B, Qwen3-14B and Qwen3-32B; Appendix E.6), the models' exploratory choices barely changed with temperature (details in W3).
    • Too little exploration. If the models simply explored too little, asking them to explore more should help. With the instruction "Please engage in more exploratory behavior," they kept exploring late into the task, when few trials remained to benefit from new information, which lowered late optimal choice and cumulative reward relative to Meta-M+C in the four- and five-arm tasks (Appendix E.5; Section 7).
    • Output format. If the structured reports alone drove the effect, a format-matched report without metacognitive content should reproduce it. In our control, such a report remained largely greedy (W4).
    • Missing algorithmic knowledge. If the models lacked an exploration rule, prompts asking them to follow UCB or Thompson sampling should help. These prompts did not produce early exploration followed by a gradual shift to exploitation (Section 7).

    By contrast, metacognitive guidance concentrated exploration early and then gradually shifted to exploitation as evidence accumulated.

On Pan et al.: we did not discuss it because it addresses a different question in a different task. Pan et al. explicitly ask LLMs to explore (in Little Alchemy 2, the objective is to create as many elements as possible) and study how LLMs explore when exploration is the goal. We study whether and when LLMs choose to explore in an exploration–exploitation dilemma, where exploring forgoes immediate reward but gathers information that supports future decisions. Findings from an exploration-as-objective task therefore do not transfer directly to this dilemma. Temperature illustrates this: Pan et al. report temperature-related effects on exploration in their task, whereas in our dilemma, where exploration carries an opportunity cost, sampling temperature barely changes LLM decisions (see W3).

AuthorsOpenReview ↗

Response to Reviewer hBzC (2/3)

W2 — Is higher value sensitivity "excessive"? Human reference, stochasticity and algorithmic baselines.

We agree that humans are not optimal, and we do not use them as a target. The human comparison was diagnostic: people in these tasks show uncertainty-directed exploration and LLMs behave greedily, so we fitted the same model to both to locate the difference. All ten models exceed the human mean βVTU\beta_{\mathrm{VTU}} in every setting (1.99–8.60×), whereas 28 of 30 model-level βRU\beta_{\mathrm{RU}} means lie within the central 95% of human coefficients (Appendix C.1.1). This led us to ask what native reasoning lacks relative to human decision-making — monitoring and control of provisional value judgments — which we then tested.

  • Excessive exploitation is established from behavior, not from coefficients. We use the definition of exploitation in the exploration–exploitation dilemma: choosing the option currently judged most valuable. Because the models judge values accurately (their reported better arm matches the empirical-value winner in at least 99.7% of trials; Appendix D.4), following the value winner is exploitation, and a very high greediness rate means persistent exploitation. Behaviorally:

    • In the two-arm tasks, once both arms are sampled, Native follows the empirical-value winner in 98.29% / 99.75% / 98.53% of choices (Settings 1/2/3; original Table C.1).
    • In the longer four-arm task, models stop exploring early. For example, native GLM-5.1 samples the optimal arm once, receives a zero, never returns to it, and follows a stable 5-point arm for the rest of the task (revised Figure 7).
    • In the five-arm task, native GPT-5.6 Luna behaves almost like pure Greedy: its non-greedy choice stays low throughout, nearly matching Greedy (Figure 8a).

    Our metacognitive guidance leads the models to form a strategy of early directed exploration followed by better-informed exploitation, which gives an overall advantage, especially over longer horizons (table below). Native reasoning therefore exploits too early, giving up exploration before enough information is gathered, while exploring still pays off. As stated in the paper, we fitted the sensitivities after observing this behavior, to explain it.

Longer-horizon task (Native → Meta-M+C)Late optimal choiceCumulative reward
Four-arm, ten models (final ten trials; Section 6, Figure 6)37.7% → 59.2%282.0 → 300.5
Five-arm, GPT-5.6 Luna (T51–100; Section 7, Figure 8, Table F.5)35.4% → 53.2%46.5 → 50.4
  • Thompson sampling under the same cognitive model (new Appendix B.4). We added Thompson sampling, simulated in the same two-arm tasks and fitted with the same pipeline as the 217,600 LLM decisions per condition. Its value sensitivity is lower than native reasoning's in all three settings (Settings 1/2/3: 0.60 / 2.78 / 5.45 vs. 3.16 / 6.69 / 7.26).
  • UCB and Thompson as executed baselines. Beyond the fitted Thompson reference, the expanded five-arm benchmark (new Section 7) executes UCB and Thompson directly for GPT-5.6 Luna and, in Appendix F.2.4, for Qwen3-4B and Qwen3-14B, together with prompts asking each model to follow these algorithms. The UCB-guided prompt was also tested in the four-arm task (Appendix E.5). Behaviorally, the algorithms do not converge: they keep a high exploration rate late in the task, when exploring has little value left (Figure 8a). Metacognitive guidance instead produces directed exploration early and then shifts to efficient exploitation. The algorithmic baselines generally fall short of it in late optimal choice and reward (e.g., Luna late optimal choice 53.2% vs. 41.2% for UCB and 37.0% for Thompson).
  • Choice stochasticity. Larger coefficients could in principle reflect less random choice. But if reasoning only made choices less random, all coefficients would scale up together, keeping their signs and ratios. This is not what we observe: reasoning raises value sensitivity about 2–3×, whereas uncertainty sensitivity (βRU\beta_{\mathrm{RU}}) stays flat in Setting 1 (0.25→0.23) and changes sign, from negative to positive, in Settings 2–3 (−0.32→0.45; −0.42→0.07). A common rescaling cannot change a sign. Metacognitive guidance then lowers value sensitivity while raising uncertainty sensitivity, again the opposite of a common rescaling. Both patterns hold under boundary exclusion and L2 refits (Table B.3).
AuthorsOpenReview ↗

Response to Reviewer hBzC (3/3)

W3 — Alternative explanations and the function of confidence.

We agree that these patterns do not uniquely imply a metacognitive deficit, and we do not claim that they do. Our account is an interpretation built on observed behavior: comparing human decision processes with the models' reasoning traces, we found that native reasoning rarely checks the reliability of a value judgment before acting, and this investigation led us to metacognitive guidance. We do not exclude other contributing factors, such as post-training. What the work does provide is a simple, task-independent way to make LLM exploration–exploitation behavior more balanced, for anyone who wants to reduce LLM greediness without external algorithms or fine-tuning. We also examined the alternatives you raised:

  • Decoding configuration (new Appendix E.6). We had run temperature sweeps during the study, and your comment showed how relevant they are, so the revision now reports them. We varied the sampling temperature of Native reasoning in our two-arm task over ten levels from 0 to 1.6 (Qwen3-14B) and four levels from 0 to 1.5 (Qwen3-8B, Qwen3-32B). Temperature barely changed exploratory choice: Qwen3-14B's non-greedy choice stayed below 1% at every level, and Qwen3-8B and Qwen3-32B also stayed essentially unchanged (0.25–0.47% and 1.97–2.19%). (Many closed-source reasoning models no longer allow the temperature to be set when thinking is enabled, so these sweeps use open-weight models.)
  • Simpler first-order policies. We also tested simpler prompts (W4): an explore-more instruction and a value→uncertainty→choice prompt did not reproduce the shift, and an uncertainty question alone raised early departures but kept departing late, when further information had little value.

Does confidence track judgments and regulate choice? We extended the confidence analysis (new Appendix F.1.6; four-arm Meta-M+C, ten models). Controlling for block, trial and observed evidence, +1 within-model SD of confidence is associated with:

EndpointAssociation (pp)95% CIModels negative / positive
Departure from own reported value winner−25.33[−30.17, −20.48]10 / 0
Next-trial same-arm persistence+9.55[+6.77, +12.32]0 / 10
Next-trial empirical non-greedy choice+0.22[−3.32, +3.76]5 / 5

Higher confidence goes with acting on the reported judgment and with subsequent commitment, not with a general reduction of exploration. Value reports are slightly more accurate at higher confidence (agreement with the empirical ranking 98.1% vs. 95.3%), but accuracy is high at both levels; confidence is more clearly associated with acting on the report than with its accuracy. These are observational associations, not evidence of calibration or causation.

W4 — Which components drive the effect?

We added targeted ablations (new Appendix E.5; Qwen3-32B, Setting 3, 13,800 decisions each). Non-greedy choice (%) early (T3–4) / late (T8–10), and final-trial optimal choice (%):

PromptEarly / late non-greedyOptimal T10
Native1.54 / 2.3288.26
Uncertainty only10.46 / 3.7089.57
Value, then uncertainty, then choice0.35 / 0.4390.58
Format-matched task report0.04 / 0.3591.30
Our assessment fields, no guidance5.20 / 1.0191.52
Meta-M guidance, choice only1.77 / 0.8792.39
Meta-M+C guidance, choice only10.03 / 0.8893.91
Meta-M10.18 / 0.5492.03
Meta-M+C29.89 / 1.1592.83

An uncertainty question alone raises early departures but keeps departing late, when new information has little use left, and structured value–uncertainty–choice reasoning stays greedy. Every metacognitive ablation keeps the same trend, more early exploration and less late departure than Native, with reduced strength, so each component of the current prompt contributes. The same pattern appears among the five-arm metacognitive variants (Section 7). We stress again that our contribution is a direction of intervention, not a specific prompt. Grounded in human cognitive science and metacognitive theory, it shows that scaffolding metacognitive processes in LLM reasoning, without external algorithms or direct instructions on behavior, lets the models spontaneously develop an adaptive, balanced exploration–exploitation strategy.

Reviewer_kGFWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+2)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper explored why LLMs tend to make greedy decisions even when they have reasoning capabilities. The authors test 10 LLMs on multi-armed bandit tasks and find that reasoning often makes models focus more on immediate rewards than uncertainty. So they introduce metacognitive prompting which encourages models to consider uncertainty, confidence and when to explore or exploit. They found that this approach helps models to explore earlier and make better decisions later(especially in longer tasks). But I would say this is an icremental work since LLM exploration through uncertainty-aware guidance have already been studied in prior work [1,2,3].

Critical strengths

I find the problem is pretty straightforward and worth exploring. We all know currently LLMs are heavily used for multistep decision making. So it is important to understand why LLMs sometimes settle on a good looking option too quickly. In particular, I liked that they do not rely solely on final rewards. Instead they also analyze value sensitivity, uncertainty sensitivity, exploration patterns and reasoning traces. This gives a better picture of why the models make certain decisions.

The proposed metacognitive guidance is easy to apply without retraining the models. Separating monitoring(Meta-M) from monitoring plus control(Meta-M+C) is also a useful design choice.

Critical weaknesses
  1. Nie et al. [1] already showed how inference-time guidance using reward estimates and uncertainty can improve LLM exploration. Sun et al [2] also studied this problem using classical bandit algorithms. Zhang et al. [3] already examined how thinking affects LLM exploration using human bandit experiments, Bayesian belief estimates and interpretable choice models(I think this [3] substantially overlaps with the current paper's experimental setup and analysis). The authors provide some useful insights into LLM reasoning but overall the proposed metacognitive prompting approach does not seem different from these existing methods.

  2. The authors argue that the improvement comes from better metacognitive monitoring and control. But I can see the guided prompts explicitly ask models to consider uncertainty and exploration(Appendix E.1, items 2&3). Then it is difficult to tell whether the gains come from genuine metacognitive regulation? or simply from giving models more detailed instructions about how to approach the task?

  3. I found that the released code includes tied cases for the two-arm tasks. Specifically, in analysis.py, the condition is_eligible = covered and (setting != FOUR_ARM or len(winners) == 1) allows tied sample means in two-arm tasks. But this contradicts the definition in Appendix B.5(empirical greediness should only be measured when one arm has a uniquely higher historical average reward). I am concerned that this could artificially increase the reported greediness and affect one of the papers main findings. The authors should clarify this inconsistency.

[1] https://proceedings.mlr.press/v267/nie25b.html

[2] https://aclanthology.org/2026.acl-long.368.pdf

[3] https://arxiv.org/pdf/2505.09901

Questions for authors
  1. Stronger comparisons with [1,2,3] methods are needed to justify the contribution.

  2. Is it possible to compare your method with standard exploration strategies such as UCB or Thompson sampling along with equally detailed, length-matched prompts to better isolate the benefits of metacognitive guidance?

  3. The results are not always consistent. In Appendix E.4, Meta-M+C slightly lowers late optimal-choice performance in Setting 1 (97.30% to 96.97%). In the five-arm task, Meta-M also does better than Meta-M+C (50.8% vs. 49.6%). The authors should explain why adding extra control instructions does not always help.

7 comments
AuthorsOpenReview ↗

Response to Reviewer kGFW (1/2)

Thank you for the careful review and for inspecting our released code. Because of length, we reply in two parts.

W1 & Q1 — Relation to Nie et al. [1], Sun et al. [2] and Zhang et al. [3].

Our work addresses a question these studies leave open: why LLMs with strong reasoning capabilities, which can gather and analyze the evidence for a decision, still behave greedily, and whether guiding their own reasoning can change this strategy without external algorithms.

  • Zhang et al. [3], the study you cite, find that reasoning can make exploration more human-like. Yet under controlled histories reasoning strengthens value-guided choice without increasing exploration (Xiong et al., 2026), and training can improve performance while leaving greedy, prematurely committed policies (Schmied et al., 2026; Chen et al., 2026). We explain this conflict: reasoning speeds coverage but amplifies value sensitivity far more than uncertainty sensitivity, so post-coverage choices follow the provisional value winner in 98–99.8% of cases, and traces show the corresponding value-to-action pathway. Both studies build on standard tools from human exploration research (Gershman, 2018); what differs is the question and the diagnosis. Zhang et al. vary whether and how much models reason (CoT prompting, thinking budgets) to compare agents; they do not develop an intervention that targets a diagnosed mechanism, which is the core of our work.
  • Nie et al. [1] improve exploration by supplying algorithmic knowledge, such as externally computed statistics, demonstrations or fine-tuning. Our guidance supplies no algorithm, and the model forms its own assessments. Our new Section 7 compares Meta-M+C with directly executed UCB and Thompson sampling and with prompts asking the model to follow these algorithms. None of them shows the strategy elicited by Meta-M+C, namely early directed exploration followed by a gradual shift to exploitation. The algorithms keep exploring late, when exploration has little value left, and all four obtain lower late optimal choice and reward than Meta-M+C (see W2 & Q2).
  • Sun et al. [2] replace direct LLM arm selection with classical bandit algorithms (Thompson sampling, regression-oracle methods) that use the LLM only as a reward predictor. The algorithm makes the decisions, so the LLM's own decision process is neither analyzed nor changed. Although this setting differs from ours, we consider it relevant and have added it to Related Work in the revision.

W2 & Q2 — Detailed instructions vs. metacognition; UCB / Thompson with comparable prompts.

The metacognitive prompts prescribe no behavior and encourage neither exploration nor exploitation. The two items you cite (Appendix E.1: which arm is more uncertain, and whether the trial calls for exploration or exploitation) are part of our structured output, which asks the model to report its own assessments. These reports serve two purposes: they provide the monitoring signal that the control instruction uses to regulate choice, and they support our analyses, such as how far choices deviate from the reports (Appendix D.4). Instructions that mention exploration or uncertainty more directly do not produce the same strategy: an explore-more instruction keeps exploring late, when exploration no longer pays off, and lowers performance; an uncertainty question alone raises departures early but keeps them late; and value→uncertainty reasoning stays greedy (see our replies to N5Bu Q1 and hBzC W4). Detailed algorithmic instructions do not produce the strategy shift either. LLM-UCB and LLM-Thompson ask the model to follow UCB or Thompson and to output only its choice, without our structured reports. For a matched comparison, we therefore use Meta-M+C with the same choice-only output (M+C-CO), whose guidance has comparable length (146 vs. 136 and 139 tokens; Qwen tokenizer, task description and history excluded). With the same output, M+C-CO reaches higher late optimal choice than both prompts in the five-arm task (53.8% vs. 31.8% and 44.4%; Table F.5). The table below also compares full Meta-M+C with these prompts and the directly executed algorithms:

Task (model)ComparisonResult
Four-arm (Qwen3-32B; Luna)Late optimal choiceLLM-UCB 40.0%; 57.3% vs. Meta-M+C 59.3%; 89.3%
Five-arm (Luna; Qwen3-4B; Qwen3-14B)Late optimal choiceUCB 41.2%, Thompson 37.0%, LLM-UCB/LLM-Thompson ≤ 45.8% vs. Meta-M+C 53.2%; 52.0%; 52.2%
AuthorsOpenReview ↗

Response to Reviewer kGFW (2/2)

W2 & Q2 (cont.) — Detailed instructions vs. metacognition; UCB / Thompson with comparable prompts.

In our experiments, algorithm-guided prompts did not reproduce their algorithms or a balanced strategy. We found their behavior unstable across tasks and models: LLM-UCB explored little early in the two-arm task, yet in the 100-trial five-arm task it kept exploring late without converging in all three models, while LLM-Thompson stayed close to greedy (Section 7; Appendices E.5, F.2.4). None elicited the early directed exploration and later exploitation induced by metacognitive guidance. Their reward was lower than that of Meta-M+C and even lower than that of the pure algorithms they were asked to follow, in all three models (e.g., Luna: LLM-UCB 47.1 and LLM-Thompson 47.6 vs. UCB 48.6, Thompson 48.2 and Meta-M+C 50.4; Tables F.5–F.6). These results suggest that an external rule written into a prompt is not executed reliably, whereas metacognitive guidance lets the model assess its own evidence and decide.

W3 — Tied sample means in the two-arm greediness metric.

The reported two-arm rates exclude ties, as defined in original Appendix B.5 (revised B.7); the line you found was never used for them. The anonymous repository initially contained decision records only for the four- and five-arm tasks, which show the effect of metacognitive guidance more clearly over longer horizons. The two-arm records are large and served mainly our cognitive-model analyses, so we did not include them at that time, nor the analysis script for them. The line you found exists because, in that version, the five-arm analysis followed the greedy measure of the benchmark's authors (Krishnamurthy et al., 2024): a choice counts as greedy whenever the chosen arm has the highest empirical mean, even when that maximum is tied. Our definition instead requires a unique empirical-value winner and excludes ties. After consideration, we decided to use one definition throughout the paper, so the five-arm analysis now also uses our unique-winner rule: the revision states it for all tasks (Appendices B.7, F.1, F.2.1), and the released analysis.py applies it everywhere.

For your verification, we have now added the two-arm analysis script and the complete Qwen3-14B two-arm records for all three settings (Native, Meta-M and Meta-M+C, Thinking ON/OFF; 810 files). You can run python tools/reproduce_two_arm_qwen14b.py to reproduce all 18 Qwen3-14B greediness values reported in the appendices, which are unchanged in the revision.

Q3 — Why adding control does not always help.

Control mainly adds early exploration, and whether this pays off depends on the setting. In Qwen3-32B Setting 3, control raises early departures from 10.18% (Meta-M) to 29.89% (Meta-M+C). This extra exploration helps when early evidence is misleading but can cost reward when monitoring alone already identifies the best arm. The Setting 1 change you note (97.30%→96.97%; original Table E.4) is −0.32 pp with a 95% CI of [−1.78, +1.56], i.e., no detectable change near ceiling. In the expanded ten-run five-arm evaluation, Meta-M has the higher late optimal choice (60.4% vs. 53.2%) while Meta-M+C has the higher reward (50.4 vs. 49.7; Appendix F.2.3), so neither dominates. Appendix F.2.5 also shows a case where extra sampling costs reward. Our finding therefore concerns a direction rather than a specific prompt: both Meta-M and Meta-M+C shift the strategy toward early exploration and later exploitation.

Thanks to the authors for their detailed reply! I will go through the rebuttal carefully and get back to you soon.

Reviewer_N5BuOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper learns the exploration-exploitation behavior of LLMs through a metacognition framework. In detail, the authors find that native reasoning increases sensitivity to estimated action values, leading to greedy choices despite remaining uncertainty. Reasoning trace analyses suggest a value-to-action shortcut, where models make decisions based on provisional value comparisons without sufficiently evaluating their reliability. To address this issue, they introduce metacognitive prompting that explicitly encourages monitoring and control of value estimates, uncertainty, and confidence. Experiments show improved uncertainty-directed exploration and subsequent exploitation, with additional validation on longer-horizon arm bandit tasks.

Critical strengths
  1. The paper provides a comprehensive empirical analysis across 10 LLMs and 3 human-referenced bandit settings. The computational modeling framework separates value-related and uncertainty-related choice sensitivity, offering a useful perspective beyond simply measuring greedy behavior. The comparison between Thinking OFF and ON with the temporal analysis also provides interesting evidence about how reasoning affects sequential decisions.

  2. The proposed metacognitive guidance is simple and efficient. The distinction between monitoring and monitoring-plus-control is meaningful, and the experiments demonstrate improvements beyond the initial short-horizon tasks.

Critical weaknesses
  1. The central argument attributes excessive exploitation to insufficient metacognitive monitoring and control. However, the computational analysis relies on an external Bayesian observer and fitted choice sensitivities, rather than measuring the LLM's internal value or uncertainty. Also, reasoning traces observation only provide descriptive evidence but do not establish the causal mechanism. In the paper, the proposed prompts explicitly ask models to consider exploration and exploitation and encourage flexibility under uncertainty. I guess the improvements could arise from direct exploration instructions or structured decision prompting rather than metacognitive regulation specifically.

  2. I think the characterization of value-dominated behavior as excessive exploitation requires stronger justification. High sensitivity to estimated value does not necessarily indicate a suboptimal decision strategy, particularly when evidence is sufficiently informative. In Appd. C.2, the author shows that enabling Thinking improves optimal-choice rates across all three short-horizon settings. Also the table 1 shows their Meta methods provides almost no additional improvement(setting 1). These observations may suggest that the usefulness of additional exploration is highly task-dependent. The strong four-arm results partially address this concern, but the five-arm validation involves only five runs of one model.

Questions for authors
  1. Could the authors clarify what evidence distinguishes the benefits of metacognitive monitoring/control from those of explicitly encouraging exploration? Are comparisons available against simpler exploration-oriented prompts with matched output formats or reasoning budgets in the two-arm and four-arm tasks?

  2. Given that Thinking ON improves optimal-choice rates over OFF in the three short-horizon settings, how should excessive exploitation be distinguished from appropriate value-based commitment? Could the authors clarify whether the observed behavior deviates from a Bayes-optimal or other principled bandit reference, particularly in settings where metacognitive guidance provides little benefit?

4 comments
AuthorsOpenReview ↗

Response to Reviewer N5Bu (1/2)

Thank you for the thoughtful review and concrete questions; because of length, we reply in two parts.

W1 — Internal judgments, causal evidence and what the prompts ask.

We did examine the models' own judgments. In Appendix D.4, the better arm the models report matches the empirical-value winner computed from the observed history in at least 99.7% of trials in all ten models. With their strong reasoning, the models are clearly able to evaluate the evidence. This is the core of our question: why do such capable models still decide greedily, and what gap causes it? We therefore started from the behavior itself and decomposed choices into value- and uncertainty-related components with a computational model (Gershman, 2018). Comparing these findings and the reasoning traces with human decision processes, we found that native reasoning rarely engages the metacognitive monitoring and control that shape human exploration. This led us to design guidance that targets these processes, and we validated its effect in large-scale experiments. With this guidance, the models' own valuation and exploration capacities form an adaptive exploration–exploitation strategy. This finding is our core contribution.

We agree, and we neither establish nor claim a causal mechanism. Our account builds on comparing LLM reasoning with human cognitive processes, informed by human cognitive science. It is a possible explanation of LLM greediness based on our experimental findings, and our experiments demonstrated the effectiveness of the guidance it motivated. We do not exclude other contributing factors. However, we observe many "causal-like" cases. As shown in Figures 3 and 5b: for the same model in the same decision state, Native and Meta-M+C-guided reasoning follow different reasoning paths and lead to different choices. Appendix D.3 presents further such pairs.

The prompts follow metacognitive theory and carry no exploration bias. The monitoring prompt asks the model only to report its judgments, including whether a trial calls for exploration or exploitation; it encourages neither. The control instruction, following theories of human metacognitive control, asks the model to decide based on its own monitored state, namely the reliability of its value judgment and its confidence, rather than telling it how to act (Appendix E.1). The reports support several analyses, including the accuracy check above (Appendices D.4 and F.1.6). Importantly, our extended experiments show that the effect persists without the reports (Appendix E.5, Table E.5): with Meta-M+C guidance but choice-only output (Qwen3-32B, Setting 3), early non-greedy choice rises from 1.54% (Native) to 10.03%, while late non-greedy choice falls from 2.32% to 0.88%.

Q1 — Comparisons with explicit exploration prompts and matched output formats.

We ran these comparisons (new Appendix E.5; Qwen3-32B two- and four-arm tasks, GPT-5.6 Luna four-arm task):

  • Explicit exploration instruction (More Exploration: "Please engage in more exploratory behavior."). The model keeps exploring and does not shift to exploitation at the right time: in Setting 3, its non-greedy choice is 66.76% early and still 51.53% late, close to chance between the two arms, whereas Meta-M+C moves from 29.89% to 1.15%. Exploration that continues when little can be gained from it mainly costs reward: final-trial optimal choice is 54.78% vs. 92.83%, and in the four-arm task Luna's late optimal choice is 55.3% vs. 89.3% under Meta-M+C (Table E.7).
  • UCB-guided prompting (LLM-UCB), which asks the model to explore by the UCB rule. When an algorithm is used to guide the LLM, its behavior is unstable across tasks and models. Specifically, in the two-arm task and in Qwen3-32B's four-arm task, it explores little early. In Luna's four-arm task and in the 100-trial five-arm task, it keeps departing from the value winner late, without converging (Figures E.7 and 8a). In neither case does it show early directed exploration followed by a gradual shift to exploitation, and its performance is generally below that of Meta-M+C, which explores in a directed way early and then exploits.
  • A format-matched report without metacognitive content stays greedy (0.04% early non-greedy choice; Table E.5).

None of these reproduces what metacognitive guidance produces: early uncertainty-directed exploration followed by efficient exploitation, with high late optimal choice.

AuthorsOpenReview ↗

Response to Reviewer N5Bu (2/2)

W2 & Q2 — Excessive exploitation vs. appropriate value-based commitment; principled references.

Exploitation and exploration are behavioral concepts, so our characterization of premature exploitation is derived from behavior. As described in the paper, we fitted choice sensitivities after observing LLMs' greedy behavior, to analyze the decision mechanism behind it. High value sensitivity explains this behavior rather than defines it. The behavioral evidence includes:

  • Committing under highly unresolved uncertainty. In Setting 3, once both arms are sampled, native reasoning follows the current empirical-value winner in 98.53% of choices (original Table C.1). Metacognitive guidance instead explores early, leaves less remaining uncertainty and achieves higher optimal choice afterwards (Figure 5a), showing that native commitment came before enough information was gathered.
  • Abandoning the optimal arm on weak evidence. A more direct example comes from the four-arm task (revised Figure 7). The prompt states that variable arms pay 0 or 20, so one zero is weak evidence. Yet native GLM-5.1 and Grok-4.3 drop the optimal arm after one and two zeros and lock onto a stable 5-point arm, while under Meta-M+C both identify it (cumulative reward 235→335 and 225→350). Across ten models, optimal choice over the final ten trials rises from 37.7% to 59.2% (p=.004p = .004) and cumulative reward from 282.0 to 300.5 (p=.010p = .010) (Section 6).

Why Thinking ON still outperforms OFF is a central insight of our analysis. Reasoning strengthens uncertainty responsiveness and early information acquisition. For example, in Setting 3, both-arm coverage rises from 64.00% to 78.99% and first coverage advances from trial 3.15 to 2.46 (original Table C.1). However, reasoning amplifies value sensitivity far more, so once a provisional value winner emerges, choices follow it. OFF, by contrast, acquires early information more slowly and departs from the value winner at a steady rate throughout the block, reflecting more random than directed exploration, and leaves the most uncertainty (Figure 5a). Earlier information acquisition followed by strongly greedy commitment can therefore outperform OFF, but outperforming OFF does not rule out premature commitment.

We emphasize a shift in strategy, which does not guarantee better performance in every setting. In Setting 1, Native already reaches 98.81% final-trial optimal choice, leaving little room for improvement (Meta-M+C: 98.89%; Table 1). Where uncertainty matters, guidance raises late optimal choice (Settings 2–3: +2.18 and +3.32 pp, with 95% CIs excluding zero; original Table E.4), and over longer horizons it improves optimal-arm selection and reward across models (Section 6). Although a balanced exploration–exploitation strategy pays off in most tasks, it is not a guarantee in every one. For example, when the optimal arm becomes the value winner very early, immediate greedy commitment can be advantageous.

Principled references. Thompson sampling, fitted with the same pipeline (new Appendix B.4), shows lower value sensitivity than native reasoning in all three settings (Settings 1/2/3: 0.60 / 2.78 / 5.45 vs. 3.16 / 6.69 / 7.26). In the five-arm benchmark, native GPT-5.6 Luna behaves almost like pure Greedy (Figure 8a), while UCB and Thompson keep exploring late without converging. Meta-M+C explores early and gradually shifts to exploitation, with higher late optimal choice than both (53.2% vs. 41.2% and 37.0%).

W2 (cont.) — Concern about the five-arm validation (five runs of one model).

We strengthened this evaluation (new Section 7, Appendix F.2). Matching the experimental scale of Krishnamurthy et al. (2024), who introduced this benchmark, GPT-5.6 Luna completes ten independent 100-trial runs per condition. Besides algorithm-guided prompts, we compare directly executed bandit algorithms, prompting baselines from prior work (incl. K24), a direct instruction to explore more, and metacognitive variants, 14 conditions in total (Table F.5). The pattern of early exploration followed by a gradual shift to exploitation appears clearly only under metacognitive guidance and its variants. Meta-M+C and Meta-M obtain the two highest rewards (50.4 and 49.7) and higher late optimal choice than every non-metacognitive condition (53.2% and 60.4% vs. at most 48.0%). We additionally tested two further models, Qwen3-4B and Qwen3-14B (Appendix F.2.4), for the algorithm comparison, with the same conclusion: Meta-M+C shows the same timing and the highest late optimal choice among those methods (52.0%, 52.2%).

Reviewer_N5BuOpenReview ↗

Reponse to authors

Thanks for your response and additional experiments, which have addressed my main concerns. In my opinion, the current Meta-M+C prompt still encodes explicit decision heuristics in natural language, making it more appropriate to characterize the effect as prompt-guided conditional policy rather than intrinsic metacognitive control.

Overall, I view this paper as borderline and think that the paper still needs calibrating the framing and scope of their conceptual contributions. I will raise my score to 3 but I will lower my confidence.

AuthorsOpenReview ↗

1. Follow-up Response to Reviewer N5Bu

Thank you for reconsidering our paper and raising your score, and for this thoughtful comment.

We agree that our results should not be read as evidence of intrinsic metacognitive control, and we would like to clarify the level at which we use the term. In human research, metacognition is studied at several levels: behaviorally, through people's reports of their own judgments, such as confidence, and how these reports relate to their decisions; and neurally, through the brain systems that support these processes (Fleming, 2024). Our work is analogous to the behavioral level. "Thinking with metacognition" refers to the reasoning process: we guide the model's reasoning to monitor its own value judgments, uncertainty and confidence, and to use these assessments when choosing, and we evaluate the resulting decisions and reasoning traces. We make no claim about mechanisms inside the model.

Regarding the explicit decision heuristics you mention, the control instruction does contain a conditional guideline: limited confidence calls for flexibility, and strong evidence allows commitment. In our framework, this is what metacognitive control means, namely regulating action by one's own monitoring. What the prompt does not specify is the condition itself: whether confidence is limited or evidence is strong is assessed by the model from its own history. Moreover, the shift does not depend on this guideline alone: Meta-M, which contains no such guideline, and SO Only, which contains no guidance at all, move the strategy in the same direction (Appendix E.5).

Whether LLMs have intrinsic metacognition, however, remains a very open question. In humans, behavioral studies of metacognition are complemented by neuroscience, which relates it to specific brain systems. For LLMs, the role of the brain would be played by the model's weights and the activations they produce. Early attempts adapt human paradigms to this level, for example a neurofeedback paradigm that tests whether models can report and control their own activations (Ji-An et al., 2025), but how intrinsic metacognitive control in decision making should be measured there, and with what methods, remains a worthwhile and challenging question that lies beyond our scope.

Our immediate aim is more practical. Greedy behavior currently limits LLMs in sequential decision making, and recent work either supplies algorithmic knowledge (Nie et al., 2025) or hands the decisions to external bandit algorithms (Sun et al., 2026). We ask whether LLMs, as currently trained, can form an adaptive exploration–exploitation strategy themselves, and we offer a simple, training-free way to elicit it. For this concrete question, we analyze metacognition at the behavioral level. We have now stated this scope in the Limitations of the revised paper: our guidance elicits metacognitive monitoring and control at the level of reasoning and behavior, and whether LLMs possess intrinsic metacognitive control, and how it could be measured, remains open.

Thank you again for your suggestions and for this discussion.

References

  • Fleming, S. M. (2024). Metacognition and confidence: A review and synthesis. Annual Review of Psychology.
  • Ji-An, L., et al. (2025). Language models are capable of metacognitive monitoring and control of their internal activations. arXiv:2505.13763.
  • Nie, A., et al. (2025). EVOLvE: Evaluating and optimizing LLMs for in-context exploration. ICML.
  • Sun, J., et al. (2026). Large language model-enhanced multi-armed bandits. ACL.
Reviewer_v6QnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why LLMs under-explore in sequential decision-making. Using a cognitive model of human exploration-exploitation fitted across ten LLMs and three bandit tasks with human reference data, the authors find that uncertainty sensitivity is roughly human-like but value sensitivity is inflated, and that native reasoning amplifies value far more than uncertainty. Reasoning traces show a matching "value-to-action" shortcut, which the authors attribute to limited metacognitive monitoring and control. A lightweight, task-agnostic prompt that elicits these assessments rebalances the two sensitivities, shifts exploration toward early trials, and improves later exploitation, with gains carried over to a longer-horizon four-arm task and an established five-arm benchmark.

Critical strengths

The intervention follows from a specific account of where greedy behaviour originates, supported by both a human-anchored behavioural decomposition and reasoning-trace analysis. This is the paper's main contribution and also helps reconcile conflicting reports on whether reasoning aids exploration.

Disabling reasoning also yields frequent non-greedy choices, but these are poorly timed and hurt performance, whereas the guided choices are concentrated early and directed at uncertainty. The effect is also model-dependent in the expected direction, which rules out the most obvious alternative explanation.

Results span open and closed models of varying scale, use a paired design with matched payoff sequences in the long-horizon task, and are checked against a published prompting baseline. The intervention is simple enough to be easily adopted.

Critical weaknesses

Identifiability of the fitted sensitivities. LLM choices are close to deterministic, and probit coefficients estimated on near-separable data tend to inflate regardless of the underlying computation. The bounded estimation and median aggregation mitigate but do not resolve this. Since inflated value sensitivity is the central claim motivating the intervention, the paper needs evidence that it is not partly an estimation artefact.

Confounded comparison between guided and baseline conditions. The guided prompts also require several structured output fields, including naming the more uncertain option, while the baseline outputs only the choice. Without a format-matched control in the main settings, the gains cannot be attributed specifically to metacognitive content rather than to longer, more structured responses.

Questions for authors
  1. What fraction of participant-level fits reach the estimation bounds, and does the value–uncertainty asymmetry hold when these are excluded or a regularised estimator is used? Does a synthetic near-greedy agent with known sensitivities recover its true coefficients through the same pipeline?

  2. Can you add a control that matches the guided prompts in output structure and length but omits the metacognitive content?

  3. Do the main conclusions hold when restricted to models evaluated at the full human sample size, given that all models are weighted equally?

2 comments
AuthorsOpenReview ↗

Response to Reviewer v6Qn

Thank you for the detailed review and the precise questions.

W1 & Q1 — Is inflated value sensitivity an estimation artefact?

No. The value–uncertainty asymmetry persists under boundary exclusion, regularization and known-parameter recovery (new Appendix B.3).

  • Boundary hits (any coefficient within 10−610^{-6} of ±10\pm 10; model-equal % of finite fits, Settings 1/2/3): Thinking OFF 0.00/21.91/16.96; Native 0.67/17.86/26.57; Meta-M 0.00/21.36/23.04; Meta-M+C 0.00/10.73/16.17 (Table B.2). With reasoning on, hits involve value coefficients; βRU\beta_{\mathrm{RU}} never reaches a bound.
  • Exclusion and regularization. Excluding boundary fits, or refitting humans and LLMs with the same L2-penalized probit (λ=0.1\lambda = 0.1, no bounds), preserves all three contrasts (Table B.3). Under L2, OFF→ON raises median βVTU\beta_{\mathrm{VTU}} 1.17→3.09, 2.54→5.81 and 1.39→3.06; Native stays above humans (0.81, 2.59, 1.10); both guided prompts raise βRU\beta_{\mathrm{RU}} and lower overall value sensitivity (Setting 3 combined value norm 4.54 → 4.31 / 3.53). Additional refits at λ=0.01\lambda = 0.01 and 11 keep the same directions.
  • Recovery. Agents with βVTU=8\beta_{\mathrm{VTU}} = 8, βRU=0.3\beta_{\mathrm{RU}} = 0.3 (βV=3\beta_V = 3 in Setting 3), simulated on the original schedules (100 replications of the human samples) and refit with the unchanged estimator, recover βVTU\beta_{\mathrm{VTU}} = 8.33 / 8.37 / 7.58 and βRU\beta_{\mathrm{RU}} = 0.33 / 0.34 / 0.34 (Table B.4). In an additional check, a balanced but near-deterministic agent (both sensitivities = 6) is recovered as balanced (Settings 1–2: 6.37/6.40 and 6.57/6.73), so near-determinism alone does not create the asymmetry.

W2 & Q2 — Format-matched control.

We added this control; a format-matched report without metacognitive content does not reproduce the guided behavior (new Appendix E.5; Section 5 now explains why the prompts request reports). Task Reports keeps a five-field JSON but asks only for available options, whether all were sampled, the estimated best option, the current trial and the choice. SO Only keeps our four assessment fields (which involve metacognitive assessment) and the choice, without the metacognitive guidance, so only the output format requires these assessments.

  • Two-arm Setting 3 (Qwen3-32B, 13,800 decisions per condition): early (T3–4) non-greedy choice is 0.04% (Task Reports), 5.20% (SO Only) and 29.89% (Meta-M+C) (Table E.5).
  • Longer horizons (ten matched runs per condition; Table E.6, trajectories in Figure E.6). Late optimal choice (%) / cumulative reward:
Model, taskTask ReportsSO OnlyMeta-M+C
Qwen3-32B, four-arm35.3 / 267.040.0 / 271.559.3 / 295.5
GPT-5.6 Luna, four-arm69.3 / 334.584.7 / 339.589.3 / 337.0
GPT-5.6 Luna, five-arm44.6 / 48.052.6 / 49.253.2 / 50.4

Meta-M+C has the highest late optimal choice in every comparison. The assessment fields alone retain part of the effect, approaching Meta-M+C for Luna, indicating that the metacognitive content of the reports, not their format, contributes. Conversely, Meta-M+C guidance with a choice-only output still raises early non-greedy choice from 1.54% (Native) to 10.03%. We matched output structure, not token length.

Q3 — Models evaluated at the full human sample size.

The main patterns also hold in the six full-sample models (five Qwen3 models and GLM-5.1; 45/44/46 participants, Table A.4). Results for Settings 1/2/3 (Appendices C.3.1 and E.3.1):

  • Thinking OFF→ON: median βVTU\beta_{\mathrm{VTU}} 1.17→3.76, 2.58→7.20, 3.71→6.39 (humans: 0.81, 2.69, 1.76).
  • Native→Meta-M+C:
    • median βVTU\beta_{\mathrm{VTU}} 3.76→2.23, 7.20→4.53, 6.39→5.19, and Setting 3 βV\beta_V 3.12→1.96.
    • median βRU\beta_{\mathrm{RU}} 0.25→0.39, 0.25→0.89, −0.08→0.50.
    • early (T1–5) non-greedy choice 2.44%→9.13%, 0.14%→14.11%, 0.52%→9.71%.
    • late (T6–10) optimal choice 97.24%→96.89% (near ceiling), 94.44%→97.26%, 86.06%→90.14%.

Dear Reviewer v6Qn,

Thank you again for your constructive review. As the discussion period is closing, we would like to briefly check whether our response has addressed your concerns. In particular, we added boundary-exclusion, regularized and parameter-recovery analyses for the fitted sensitivities (Appendix B.3), a format-matched report control (Appendix E.5), and results for the six models evaluated at the full human sample size (Appendix A.5). We would be happy to clarify anything further before the discussion closes.

Best regards, The authors

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank all reviewers for their careful and constructive feedback. All revisions in the updated PDF are marked in blue, and new appendix sections are highlighted in blue in the appendix contents. In each reply, W# and Q# follow the numbering of that reviewer's weaknesses and questions.

Summary of our contribution. We ask why reasoning-capable LLMs still make excessively greedy decisions in the exploration–exploitation dilemma, and whether guidance can change their strategy through their own reasoning. Our answer is a direction of intervention rather than a specific prompt. Scaffolding metacognitive monitoring and control in LLM reasoning, without external algorithms or direct instructions on behavior, systematically reshapes the exploration–exploitation strategy: models explore in a directed way early and exploit with better information later. Several variants of the guidance shift behavior in the same direction, whereas an explore-more instruction, a format-matched report without metacognitive content, temperature changes and algorithm-guided prompts do not. The resulting performance gains are clearest over longer horizons.

In response to the reviewers' comments, we revised the abstract and conclusion to state this more clearly and made the following main revisions:

Concern (reviewers)What we added or revisedWhere
Estimation artefact (v6Qn W1; hBzC W2)Added boundary-hit rates, refits without boundary fits, L2-regularized refits and known-parameter recoveryApp. B.3; Sec. 4
Excessive vs. appropriate commitment; principled reference (N5Bu W2; hBzC W2)Revised Fig. 7 with two matched four-arm examples; added Thompson sampling fitted with the same cognitive model; added directly executed UCB and Thompson samplingSec. 6, Fig. 7; App. B.4; Sec. 7
Metacognition vs. format, exploration instructions, simpler prompts (v6Qn W2; N5Bu W1, Q1; kGFW W2; hBzC W4)Added format-matched output controls, choice-only ablations, simpler assessment prompts, an explore-more instruction and algorithm-guided prompts (LLM-UCB, LLM-Thompson) in two-, four- and five-arm tasks; stated the purpose of the reportsApp. E.5; Sec. 7; App. F.2; Sec. 5
Decoding and confidence (hBzC W3)Added sampling-temperature sensitivity analyses under Native reasoning (Qwen3-8B/14B/32B); added confidence–choice analyses (four-arm, ten models)App. E.6; App. F.1.6
Scale of the five-arm validation (N5Bu W2)Added a five-arm benchmark section: GPT-5.6 Luna, ten independent 100-trial runs per condition, matching the scale of Krishnamurthy et al. (2024), 14 conditions incl. K24; repeated the algorithm comparison in Qwen3-4B and Qwen3-14BSec. 7, Fig. 8; App. F.2, F.2.4
Related work; tie rule (kGFW W1, W3)Added Sun et al. to Related Work; stated the strict tie exclusion for all tasks; released the two-arm scoring code and Qwen3-14B recordsSec. 2; App. B.7, F.1, F.2.1; anonymous code repository

Key results and detailed answers are given in our replies to the individual reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.