acceptodds
Under review as a conference paper at ICLR 2027

Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents

Abstract

LLM agents are deployed in environments where they must interact to acquire information. In these scenarios, the agent must reason about inherent cost-uncertainty tradeoffs in how to act, such as when to stop exploring and commit to an answer. For instance, on a programming task, an agent might run the code it generates, or it might generate tests for that code snippet; the cost of writing and running a test is nonzero, but typically lower than the cost of running buggy code. In this work, we show that we can induce LLM agents to explicitly reason about balancing these cost-uncertainty tradeoffs, then act more optimally in their environments. We formalize multiple tasks, including retrieval-augmented QA, a file reading coding task, and interactive software engineering, as sequential decision-making problems under uncertainty. Each problem has latent environment state that impacts the agent's performance. We introduce a framework called Calibrate-Then-Act (CTA), where we pass the agent an inferred prior about this environment state to enable it to act more optimally. This information qualitatively changes agent behavior, and adds environment sensitivity to the agent which is not learned via standard RL training. Our results on three tasks show that making cost-benefit tradeoffs explicit with CTA helps agents discover more optimal decision-making strategies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agents, cost-aware

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Bi2LOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Calibrate-Then-Act (CTA), a framework that gives agents explicit probability estimates and action costs to decide whether to gather more information or proceed with execution. CTA is implemented through both prompting and GRPO-based training. Its evaluation includes question answering, file reading, and software engineering, aiming to evaluate how agents adapt their behavior to interaction costs.

Critical strengths
  1. The paper focuses on a practical decision problem. Agents must evaluate the potential benefit of additional information against the cost of obtaining it. Studying this trade-off is valuable because unnecessary exploration can consume substantial resources without improving task outcomes.

  2. The experiments help characterize when explicit uncertainty estimates are useful. Evaluations under different cost settings and ablations on prior quality offer useful evidence about the conditions under which supplying such estimates can improve agent decisions.

Critical weaknesses
  1. Selective exploration before commitment and the incorporation of exploration costs have already been studied in prior works such as [1]. In particular, EAPO learns when and how to explore within an integrated policy-learning framework. The paper needs to distinguish CTA’s contribution more clearly and explain what it adds beyond these existing ideas.

  2. CTA uses confidence calibration for question answering, a filename classifier for file reading, and a combination of confidence estimation and user-profile inference for software engineering. These heterogeneous components require substantial adaptation across tasks. Compared with EAPO’s unified exploration-aware formulation [1], it is unclear how much additional design and supervision would be needed to support a new environment.

  3. CTA represents the exploration–execution trade-off through discounted task success, without explicitly accounting for actions that may cause irreversible damage before the task concludes. For example, executing uncertain code can lead to data loss or system failure. The suitability of the resulting “try-and-go” behavior therefore remains unclear in settings that require explicit safety constraints.

[1] Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization

Questions for authors
  1. What are the specific methodological advances of CTA over existing exploration-aware approaches, particularly EAPO? Which aspects go beyond providing explicit probability estimates and action costs to the agent?

  2. How are the accuracy and calibration of the probability estimates evaluated on unseen tasks? How sensitive are CTA’s exploration and commitment decisions to errors in these estimates?

  3. How would CTA handle actions with potentially harmful or irreversible consequences?

5 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for your comments and feedback! We address some of the questions and concerns below:

(W1, Q1, W2, Q2) Comparison of CTA and EAPO.

Our paper differs from EAPO fundamentally in core objective and task setup, as well as the cost of adapting to new environments.

  • Task setup (W1, Q1): EAPO does not adapt to varying cost constraints. Their reward modeling [EAPO, Sec 4.3] applies a discount factor of γ\gamma^2 to prevent overconservatism, but that gamma is a global parameter and is fixed within a training/inference run; moving to a different cost regime means re-training the policy. This hyperparameter is set to prevent overconservatism in general, but not training the agents to react adaptively to varying costs and uncertainties.
    In contrast, CTA's objective is precisely that adaptive behavior under varying constraints: our test sets are constructed so that no static strategy wins across different costs, and we report results both aggregated and per cost level to show that the same system moves its operating point as the cost moves (e.g., ask rate 0.55→1.00 across γ, Table 8), with zero retraining.

  • Adaptation cost (W2): EAPO needs RL retraining on each new task or new cost regime, which is an inevitable adaptation cost. CTA's decoupled design is much cheaper and effective: each estimator is fit on a small validation set only, and RL is only an optional component in the CTA-RL variant. As measured in our response to Reviewer y4vN (Q2), this prior estimation costs at most ≈6% of one episode's tokens – orders of magnitude below an RL run.

  • Effort to support a new environment (W2, Q2): We would like to clarify that we do provide a unified framework to formalize and study this cost-aware exploration problem in a controlled setting (see Table 1 in our paper).
    Our paper demonstrates how one could adapt the unified framework to varying environments/underlying state variables with representative tasks, including model internal confidence (QA), patterns in seen data/training data (FileReading), and user preferences and interaction history (SWE-Clarify). New environments are largely covered by this span of signal types.
    Additionally, the prior estimates do not need to be perfect. As shown in our experiments in App. B and F, we show that CTA benefits from stronger priors while remaining effective with imperfect but informative ones. This tolerance further lowers the burden of adapting to a new environment.

We have also updated the related work section to explain this.

AuthorsOpenReview ↗

Response (2/2)

W1.1 - What goes beyond providing the model with prob estimates...?

Beyond providing probabilities and costs to the agent, CTA contributes a decision-making framework that explicitly decouples uncertainty from action selection under cost tradeoffs (Figure 2), where we (1) estimate priors with a decoupled calibrated estimator and (2) condition the agent on the priors during training and/or inference.

Prompting is one way to expose the uncertainty signal; the same interface also improves RL training (CTA-RL, Sec. 4), and it lets CTA work with closed-weight models where fine-tuning is not possible (e.g., Claude Haiku 4.5 in SWE-Clarify). Therefore, the key contribution is the explicit interface between uncertainty estimation and action selection, which enables more rational, cost-aware behavior, rather than the prompt itself.

Q2.1 - How well do the prior estimates transfer to unseen tasks?

Within a domain, the estimates do transfer to unseen tasks: every estimator is fit only on a small validation set and evaluated on held-out test instances it has never seen. The estimator also does not have to be perfect: CTA benefits from stronger priors while remaining effective with imperfect but informative ones (App. B; App. F ladder), providing the possibility of transfer to datasets in near domains.

Across distant domains, transferring the estimates themselves is not the objective. Different tasks have different sources of uncertainty and different environment variables, so by nature a prior estimated for one environment is neither expected nor required to apply to another. However, the same calibrate-then-act decision layer is reused unchanged across all four tasks, with only the cheap per-task estimator needing to change.

Q2.2 - How sensitive is CTA to imperfect estimates?

We refer the reviewer to our prior-quality experiments in App. B and F, where we degrade the priors given to CTA. The results demonstrate two points:

  • CTA is not tied to a specific estimator and the estimator does not have to be perfect. The results show that CTA benefits from stronger priors while remaining effective with imperfect but informative ones, which stably outperform the no-prior baseline.
  • Deliberately corrupted priors (noisy or adversarial) are expected to underperform the baseline. The over-confidence cases hurt the general performance, which illustrates the same failure mode as the miscalibrated confidence of off-the-shelf LLMs [1,2]. And this is precisely why CTA decouples prior estimates from decision making.

[1] Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs, Xiong et al., 2024.
[2] Calibration of Pre-trained Transformers, Shrey Desai and Greg Durrett, 2020.

W3, Q3 -Handling irreversible or harmful actions

We would like to clarify that CTA does not assume that exploration actions are reversible and they do not need to be. Harmful or irreversible actions are orthogonal to the exploration-cost design and are captured by the reward already. Concretely, if an action risks task failure, that risk lowers its expected task utility, and CTA weighs that utility against the exploration cost. If an action risked harm, the reward function could accommodate the severity of that harm.

Thank you for the rebuttal. However, my main concerns are not fully addressed.

  1. What does "varying cost constraints" mean? Is it the cost mentioned in Table.1? However, the cost in it are mostly γ\gamma, which is the same as it in EAPO.

  2. The claim "CTA's decoupled design is much cheaper and effective" is inaccurate as EAPO aims to train the agent with the capability to explore in unseen environments, which cost nothing for adaptation.

  3. An irreversible action means one that causes harm once executed, then we should not allow the agent to perform it at all, rather than letting it execute first and then reducing its utility after discovering the problem — because the harm has already occurred at the moment of execution, and lowering the utility afterwards cannot undo the damage that has already been done. A safety constraint is a prior restriction on the action set, which is an orthogonal level from weighing utility against cost among the legal actions.

Reviewer_Hw1dOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks when an LLM agent should act directly and when it should search, run tests, or ask the user. CTA gives the agent probability estimates, such as its chance of answering correctly without searching, to help it weigh the benefit of checking against the cost. The authors test this approach on QA, file reading, and software tasks. It improves reward under the chosen cost settings, although task success sometimes decreases.

Critical strengths
  1. The paper addresses a practical problem: deciding whether another search, test, or question is worth the cost. Giving the agent explicit probability estimates is a reasonable way to help it make these decisions.

  2. The experiments report task success and cost-adjusted reward separately, so the trade-offs are clear. The authors also compare against simple decision rules and test what happens when the probability estimates are inaccurate. These comparisons help show where the method is useful.

Critical weaknesses
  1. It is unclear what LLM reasoning adds once the probabilities are available. On SWE-CLARIFY, a simple expected-value rule scores 0.398 versus CTA’s 0.392, with no significant difference.

  2. The cost measure leaves out the extra work needed to estimate probabilities and reason over them. It is unclear whether CTA saves time or money overall.

  3. The gains are modest and vary across cost settings. Inaccurate probability estimates can also make CTA worse than the baseline, which raises concerns about using it on new tasks.

Questions for authors

Does CTA outperform a simple expected-value controller using the same probabilities on QA and file reading?

Do the gains remain after including the cost of estimating probabilities and the additional LLM reasoning?

How well do the probability estimates transfer to new tasks without recalibration?

2 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for your comments and feedback! We address some of the questions and concerns below:

W1, Q1 - What LLM reasoning adds once the probabilities are available

We compare CTA with a scripted expected-value (EV) controller on all three tasks, using the same priors.

TaskEV controller (same priors)CTA (LLM)Difference
QA (PopQA)0.2800.293+0.013, p<0.05 (paired bootstrap)
FileReading (new, App. O.1)0.262CTA-RL 0.268+0.006
SWE-Clarify (App. F)0.3980.392Δ=−0.006, 95% CI [−0.013, +0.001], p=0.09

First, the EV controller cannot replace the LLM completely. The EV controller selects actions using the estimated priors and their expected values. It does not replace the LLM that answers questions, writes code, or produces a patch; it provides an action-selection baseline. For example, in QA, the controller decides whether to retrieve, but the LLM still produces the answer.

From the results, CTA which relies on LLM for expected-value comparison outperforms or is on par with an EV controller. In QA, CTA significantly outperforms the controller in reward and task accuracy (0.512 vs. 0.466). This shows that the gains are not explained by improved uncertainty estimates alone, nor could it be replaced by simple rule based EV-controller: the LLM also contributes contextual reasoning beyond a fixed decision rule, which is particularly useful when the priors are imperfect.

We also refer the reviewer to Appendices D, K, L, and M. These case studies show that, when given explicit priors, the model reasons about exploration costs and updates its decisions after each observation. This becomes especially important in FILEREADING, where the sequential state and action space is nontrivial to exhaustively encode in a one-shot scripted controller. Thus, CTA combines informative uncertainty estimates with the LLM’s flexible reasoning over evolving beliefs and costs.

W2, Q2 - Overhead cost of calibration (prior estimator)

We report the overhead cost by prior estimator and the tokens added to the agent prompt, relative to the total cost of each episode.

Results show that the cost of prior estimation is minimal compared to the overall costs:

TaskPrior estimatorEstimator token overhead, vs. one episodeTokens added to agent prompt, vs. one episodeΔ reward vs no-prior
QA1 short call for verbalized confidence + a free offline isotonic fit≈6%≈8%+0.010
FileReadingone forward pass of the 4.4M BERT-tiny — no LLM call0 LLM tokens (<0.01% of episode compute)≈2.3%+0.009 (CTA-RL vs RL)
SWE-Clarify2 short calls (confidence + user profile), amortized over all six γ≈0.4%<0.01%+0.012

The results show that the calibration step takes less than 6% for all tasks in general, and the additional input tokens added to the agent are also minimal, supporting the effectiveness of CTA.

W3 - The gains seem modest. And CTA with bad estimation makes it worse.

We want to clarify that while the absolute reward differences may appear small, the reward is bounded between 0 and 1 and is further compressed by multiplicative discounts for exploration actions.

To further support the stability of the gains, we refer the reviewer to App. C, F, and O.4, where we provide significance test results by comparing CTA against baselines on individual cost settings. Overall, CTA achieves significant gains over different baselines in several cost regimes and is never statistically worse. These results show that the gains are stable and that CTA remains competitive across cost regimes.

The prior-quality experiments in App. B and F are not meant to say that CTA fails with bad estimation, but to demonstrate two points:

  • CTA is not tied to a specific estimator and the estimator does not have to be perfect. The results show that CTA benefits from stronger priors while remaining effective with imperfect but informative ones, which stably outperform the no-prior baseline.
  • Deliberately corrupted priors (noisy or adversarial) are expected to underperform the baseline. The over-confidence cases hurt the general performance, which illustrates the same failure mode as the miscalibrated confidence of off-the-shelf LLMs [1,2]. And this indeed emphasizes the need to decouple prior estimation from the LLM itself.

[1] Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs, Xiong et al., 2024.
[2] Calibration of Pre-trained Transformers, Shrey Desai and Greg Durrett, 2020.

AuthorsOpenReview ↗

Response (2/2)

Q3 - How well do the probability estimates transfer to new tasks without recalibration?

Within a domain, the estimates do transfer to unseen tasks: every estimator is fit only on a small validation set and evaluated on held-out test instances it has never seen. The estimator also does not have to be perfect: CTA benefits from stronger priors while remaining effective with imperfect but informative ones (App. B; App. F ladder), providing the possibility of transfer to datasets in near domains.

Across distant domains, transferring the estimates themselves is not the objective. Different tasks have different sources of uncertainty and different environment variables, so by nature a prior estimated for one environment is neither expected nor required to apply to another. However, the same calibrate-then-act decision layer is reused unchanged across all four tasks, with only the cheap per-task estimator needing to change.

Reviewer_y4vNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how LLM agents should balance the cost of gathering information against the uncertainty of acting without it. The authors propose Calibrate-Then-Act (CTA), which estimates prior probabilities about hidden environment states and provides them explicitly to the agent when choosing whether to explore or commit. These priors come from calibrated model confidence, a trained file-format predictor, or user interaction history. CTA can be applied through prompting alone or combined with GRPO training. The paper evaluates the approach on a synthetic Pandora's Box problem, retrieval-augmented QA, file-reading tasks, and software engineering with optional user clarification. CTA improves discounted reward over standard prompting and RL, although the gains on the three practical tasks are relatively small.

Critical strengths
  1. The paper addresses a meaningful problem in LLM agents: deciding whether additional exploration is worth its cost, rather than simply maximizing task accuracy. The unified formulation is intuitive and applies to several types of interaction, including retrieval, code execution, and user clarification. The Pandora's Box experiment is particularly useful for isolating the decision-making problem. With explicit priors, Qwen3-8B reaches a 94% optimal-policy match rate, compared with 23% for the prompted baseline. This provides evidence that the model can reason about cost and uncertainty when the relevant probabilities are available.

  2. The evaluation considers both task success and exploration costs, and examines how agent behavior changes under different cost settings. This is more informative than reporting accuracy or tool-call counts alone. The File Reading experiments compare prompting and RL with their CTA counterparts, showing that ordinary RL tends to learn a fixed test-first strategy, while CTA produces more cost-sensitive behavior. The prior-quality ablations are also useful, as they show how performance changes when the estimated probabilities become less reliable.

Critical weaknesses
  1. Limited novelty, and the baselines do not establish the benefit of LLM-based decision-making. Estimating probabilities and selecting actions by expected utility is standard decision theory, and prior LLM frameworks such as DeLLMa (ICLR 2025) have already explored this approach. CTA mainly adds calibrated prior estimates to the agent's prompt, without a clear methodological advance beyond applying this idea to exploration decisions. More importantly, it is unclear whether an LLM is needed once these probabilities are available. Appendix F shows that a simple expected-value controller achieves 0.398 reward on SWE-Clarify, compared with 0.392 for CTA, although the difference is not significant. Similar comparisons are missing for QA and File Reading. In addition, CTA uses separately trained or calibrated estimators, including a supervised BERT predictor for File Reading. This makes it difficult to determine whether the gains come from better decision-making or simply from providing additional information.

  2. The improvements are modest and appear only at extreme cost settings. Reward rises from 0.283 to 0.293 on QA, 0.380 to 0.392 on SWE-Clarify, and 0.259 to 0.268 for File Reading RL, with no confidence intervals for QA. The gains are concentrated where exploration is penalized most heavily. On QA, CTA improves reward only in the lowest discount bin, from 0.18 to 0.24 for discounts below 0.2, while the other bins differ by at most 0.01. On SWE-Clarify, most of the gain comes from a discount of 0.15, where one clarifying question removes 85% of the reward. A rule based only on the discount factor, which never asks at 0.15 and always asks otherwise, scores 0.3925, the same as CTA. The paper does not relate these discounts to real latency, monetary cost, or user preferences, and it does not charge for the extra LLM calls used to estimate the priors.

Questions for authors
  1. How does CTA differ from prior decision-theoretic LLM frameworks such as DeLLMa (ICLR 2025)? Could you compare against an expected-value controller using the same priors across all three tasks, rather than only SWE-Clarify?

  2. How were the discount factors chosen? Could you report the token, latency or API cost of each method, including the calls used to estimate the priors, alongside the discounted reward?

  3. In File Reading, the BERT predictor is trained directly on the format labels, which the RL baseline only observes through its own unit tests. Could you add an RL baseline trained with an auxiliary format-prediction loss but without priors in the prompt, to separate this extra supervision from the effect of the prompt?

3 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for your comments and feedback! We address some of the questions and concerns below:

W1, Q1 - Limited Novelty (e.g., DeLLMa)

Reasoning about expected utility is a classic notion in decision theory and our formulation builds on prior work such as [1]. DeLLMa [2] shares this decision-theoretic background. However, there are two key differences that establish the novelty of CTA:

(1) How the agent obtains and uses its beliefs: DeLLMa asks the LLM to forecast state probabilities from the provided context on their own and then decide from those forecasts. However, CTA explicitly separates prior estimation from action selection: a decoupled estimator provides calibrated priors, then the LLM uses them to reason about the expected-utility given costs and uncertainty.

This separation addresses the known difficulty of obtaining reliable probabilities directly from LLMs [3, 4] – on our QA task, raw verbalized confidence has ECE 0.62 before calibration (§4.1). This separation also lets CTA cover general settings where self-elicited confidence is not even applicable because the priors live outside the model, such as priors learned from training data (FileReading), or priors inferred from a user’s interaction history (SWE-Clarify). This separation also means that CTA works even when we can’t fine-tune the model (e.g., Haiku 4.5 in SWE-Clarify).

(2) One-shot vs. interactive decisions in an explorable environment: DeLLMa addresses one-shot decisions from information provided upfront. CTA studies interactive POMDP settings [5], where exploration actions can reveal new information but incur costs. The agent must decide whether to gather information or commit, and adapt as new observations arrive. We have now included this discussion in related work.

[1] Optimal Search for the Best Alternative, Martin L. Weitzman, 1979.
[2] DeLLMa: Decision Making Under Uncertainty with Large Language Models, Liu et al., 2025.
[3] Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs, Xiong et al., 2024.
[4] Calibration of Pre-trained Transformers, Shrey Desai and Greg Durrett, 2020.
[5] Acting Optimally in Partially Observable Stochastic Domains, Cassandra et al., 1994.

W2.1 - CTA mainly adds calibrated prior estimates to the agent's prompt.

We would like to clarify that the key contribution is not about designing better prompts, but about introducing a decision-making framework that explicitly incorporates uncertainty into action selection under cost tradeoffs. Concretely, the framework follows a two-stage process (Figure 2), where we (1) estimate priors and (2) condition the model on the priors during training and/or inference.

Prompting is one way to expose the uncertainty signal to the model. More generally, the same principle can be applied beyond prompting. For example, by integrating uncertainty estimates into RL policies (CTA-RL experiments, Sec. 4), tool-use agents, or other decision systems. The key contribution is the explicit interface between uncertainty estimation and action selection, which enables more rational, cost-aware behavior, rather than the prompt itself.

W2.2, Q1 - Comparison with EV controller results.

We evaluated a scripted expected-value (EV) controller on all three tasks, using the same priors as CTA:

TaskEV controller (same priors)CTA (LLM)Difference
QA (PopQA)0.2800.293+0.013, p<0.05
FileReading (new, App. O.1)0.262CTA-RL 0.268+0.006
SWE-Clarify (App. F)0.3980.392Δ=−0.006, 95% CI [−0.013, +0.001], p=0.09

First, the EV controller cannot replace the LLM completely. The EV controller selects actions using the estimated priors and their expected values. It does not replace the LLM that answers questions, writes code, or produces a patch; it provides an action-selection baseline. For example, in QA, the controller decides whether to retrieve, but the LLM still produces the answer.

From the results, CTA which relies on LLM for expected-value comparison outperforms or is on par with an EV controller. In QA, CTA significantly outperforms the controller in reward and task accuracy (0.512 vs. 0.466). This shows that the gains are not explained by improved uncertainty estimates alone, nor could it be replaced by simple rule based EV-controller: the LLM also contributes contextual reasoning beyond a fixed decision rule, which is particularly useful when the priors are imperfect.

We also refer the reviewer to App. D, K, L, and M. These case studies show that, when given explicit priors, the model reasons about exploration costs and updates its decisions after each observation. This becomes especially important in FILEREADING, where the sequential state and action space is nontrivial to exhaustively encode in a one-shot scripted controller. Thus, CTA combines informative uncertainty estimates with the LLM’s flexible reasoning over evolving beliefs and costs.

AuthorsOpenReview ↗

Response (2/2)

W2.3 - The gains seem modest.

We want to clarify that while the absolute reward differences may appear small, the reward is bounded between 0 and 1 and is further compressed by multiplicative discounts for exploration actions.

To further support the stability of the gains, we refer the reviewer to App. C, F, and O.4, where we provide significance test results by comparing CTA against baselines on individual cost settings, e.g., CTA-Prompted outperforms baseline Prompted on SWE-Clarify by +0.012 (significant, p=0.007). Overall, CTA achieves significant gains over different baselines in several cost regimes and is never statistically worse. These results show that the gains are stable and that CTA remains competitive across cost regimes.

W2.4 - Confidence interval for QA task.

We provide paired-bootstrap confidence intervals for QA. Overall, CTA vs. Prompted is +0.010 (95% CI [−0.004, +0.023]); against the static baselines the gains are larger and significant (+0.067 vs. Never-Retrieve, +0.077 vs. Always-Retrieve, both p<0.001). It is true that the lowest-discount bin is the only significant one (+0.061, p=0.002), which reflects the design: high-discount bins saturate because every reasonable policy retrieves when retrieval is nearly free, so the value of cost-adaptive behavior concentrates where cost binds. No bin is significantly negative (|Δ| ≤ 0.007, p ≥ 0.67).

W2.5 - A rule based on the discount factor, which never asks at γ=0.15\gamma=0.15 and always asks otherwise, scores 0.3925 which is similar to CTA.

The arithmetic is correct, but this rule was selected after inspecting our evaluation results; essentially it is tuned on the test set.

Furthermore, the rule’s cost-dependent behavior is consistent with our point that agents should adapt to varying costs (and uncertainties). CTA achieves a comparable score without a hindsight-selected threshold and uses per-instance priors to decide whether to explore. Therefore, CTA adapts to both cost and instance-level uncertainty.

Q2 - How are the discount factors chosen? And how are the cost of LLM calls, token usage accounted for?

The discount factors are chosen so that no static strategy wins across all regimes (see the static baseline, where never-explore and always-explore each dominate only at one end of the sweep, Tables 3,4,8). The design goal is to study how well the agent adapts to varying cost and uncertainty. Note that these controlled cost settings are not directly using specific dollar or latency cost notions, but cover different regimes by design, and therefore the conclusions generalize.

Nevertheless, we measure the overhead cost added by CTA by analyzing the run logs. Results show that the cost of prior estimation is minimal compared to the overall costs:

TaskPrior estimatorEstimator token overhead, vs. one episodeTokens added to agent prompt, vs. one episodeΔ reward vs no-prior
QA1 short call for verbalized confidence + a free offline isotonic fit≈6%≈8%+0.010
FileReadingone forward pass of the 4.4M BERT-tiny — no LLM call0 LLM tokens (<0.01% of episode compute)≈2.3%+0.009 (CTA-RL vs RL)
SWE-Clarify2 short calls (confidence + user profile), amortized over all six γ≈0.4%<0.01%+0.012

The results show that the calibration step takes less than 6% for all tasks in general, and the additional input tokens added to the agent are also minimal, supporting the effectiveness of CTA.

Q3 - Comparison with a new RL baseline, trained with an auxiliary format-prediction loss but without priors in the prompt, to separate this extra supervision from the effect of the prompt

We ran this “aux-RL” baseline. Specifically, aux-RL adds an initial turn which asks the model to predict the file format (delimiter, quotechar, skiprows), and in the objective we add the additional term of format-prediction accuracy R_{file} = α·(correct attributes)/3, where α is a tunable hyperparameter and is set to 0.25 in the experiments. This turn is excluded from the turn cost calculation for fair comparison. Then the training reward becomes R_{aux} = R + R_{file}.

MethodUCρ=0.5ρ=1ρ=2ρ=4AccReward
RL2.131.39.359.303.227.148.9970.259
aux-RL (new)2.031.43.363.307.229.149.9960.262
CTA-RL1.981.46.382.315.230.147.9910.268

The prediction is generated and parseable in all episodes, with accuracy 0.55, which is short of the supervised estimator's 0.67. Yet reward is statistically indistinguishable from RL (Δ=+0.003, 95% CI [−0.011, +0.017]) and does not close the prior-injection gap (vs CTA-RL: Δ=−0.007, 95% CI [−0.020, +0.007]). A learned point estimate cannot express "comma 0.67 / semicolon 0.23 / tab 0.10" and it is the calibrated distribution that determines how many unit tests are worth paying the cost for. More details in revised App. O.2.

Reviewer_y4vNOpenReview ↗

Response to Authors’ Rebuttal

Thank you for the detailed response and the substantial effort in running the additional experiments. Many of my main concerns have been resolved, such as the clarification of the differences from related work. I also appreciate the expected-value controller results on all three tasks, the cost analysis of prior estimation, and the new aux-RL baseline.

However, a few points are still open. For example, the improvement over the prompted baseline on QA is not significant. Furthermore, on FileReading, CTA-RL does not significantly outperform aux-RL, and no significance test is given against the expected-value controller. It would be helpful to clarify how the expected-value controller estimates the success probability after retrieval on QA, since it scores below the prompted baseline without priors. I encourage the authors to include these results and this discussion in the revision.

Overall, the response resolves my main concerns, and I have raised my score to 3.

Reviewer_UhVNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors present work to induce LLM agents to explicitly reason about balancing these cost-uncertainty tradeoffs, then act more optimally in their environments. They formalize multiple tasks, including retrieval-augmented QA, a file reading coding task, and interactive software engineering, as sequential decision-making problems under uncertainty. Each problem has latent environment state that impacts the agent’s performance. The authors introduce a framework called Calibrate-Then-Act (CTA), where we pass the agent an inferred prior about this environment state to enable it to act more optimally. This information qualitatively changes agent behavior, and adds environment sensitivity to the agent which is not learned via standard RL training.

Critical strengths
  1. The paper is well-presented with critical findings and contributions clearly highlighted.
  2. Authors take great care to present background and settings for readers new to the area.
  3. The methodology and experimental settings are sound.
Critical weaknesses
  1. The paper lacks enough qualitative examples to show the impact of CTA on the tasks when compared to baselines.
  2. The authors are encouraged to discuss valid limitations of their methods and future work.
Questions for authors
  1. Why are priors estimated from the training data via MBert? Is this a strict restriction or a deliberate experimental setup? Did the authors try out other similar techniques to account for variance due to this choice?

  2. Did the authors try other datasets beyonf PopQA and SWE-verified to check for robustness in their method? SWE-Verified has been reported to have its own set of issue related to answer leakage via git commits. Were these issues accounted for during the evaluations?

1 comment

Thank you for your comments and feedback! We address some of the questions and concerns below:

W1 - lack of qualitative examples to show the impact of CTA

We refer the reviewers to Appendices D, K, and L, which show how CTA’s behavior differs from the baselines on Pandora’s Box, FileReading, and SWE-Clarify. We additionally add the trace analysis for QA task in the revised version (Appendix M).

W2 - The authors are encouraged to discuss valid limitations of their methods and future work.

Thank you for your suggestion. Please see Appendix A for a discussion of limitations of the work and directions for future work.

Q1 - Why are priors estimated from the training data via MBert? Did the authors try other similar techniques?

We chose a BERT-tiny model (4.4M) in the main experiments deliberately to show (1) how CTA outperforms baselines by decoupling prior estimation from action selection and (2) that even a very light weight estimator suffices.

Importantly, CTA is not tied to a specific estimator. We refer the reviewer to App. B and F, where we run CTA with prior estimators of varying quality. The results show that it benefits from stronger priors while remaining effective with imperfect but informative ones, which stably outperforms the no-prior baseline.

Q2 - Potential issues with SWE task; About task coverage.

To clarify, while the benchmark in our fourth task is based on SWE-bench Verified, it is a different benchmark: Stateful SWE-bench (Zhou et al., 2025b), which is based on SWE-bench Verified. In Stateful SWE-bench, the original issues are rewritten as under-specified instructions and paired with user personas and interaction histories. Therefore, the agent never sees the original issue text, but only sees the vague instruction and what the simulated user tells it.

Moreover, the git-commit channel is blocked by the environment itself; all episodes run in the official per-instance Docker images, with networking disabled and git history trimmed to the base commit (remotes, branches, and tags removed), so future-fix commits are simply not present in the container. We also audited all 699 episode transcripts to verify this: although the agent frequently probed git history, no command ever returned a post-base commit. We have added these details in Appendix N in the updated version to be clear.

Our experiments range from a proof-of-concept problem (Pandora’s Box) to establish the theoretical model, to knowledge QA with optional retrieval (PopQA), a coding setting with history information (FileReading), and coding with human interaction/preferences (Stateful SWE-bench). Together, these show the breadth, applicability, and effectiveness of CTA across various domains.

Comments on the paper

1 comment

The author-reviewer discussion period will end soon. Please review and acknowledge the authors’ responses. If your concerns have been addressed, please reconsider your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.