- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Calibrate-Then-Act (CTA), a framework that gives agents explicit probability estimates and action costs to decide whether to gather more information or proceed with execution. CTA is implemented through both prompting and GRPO-based training. Its evaluation includes question answering, file reading, and software engineering, aiming to evaluate how agents adapt their behavior to interaction costs.
Critical strengths
-
The paper focuses on a practical decision problem. Agents must evaluate the potential benefit of additional information against the cost of obtaining it. Studying this trade-off is valuable because unnecessary exploration can consume substantial resources without improving task outcomes.
-
The experiments help characterize when explicit uncertainty estimates are useful. Evaluations under different cost settings and ablations on prior quality offer useful evidence about the conditions under which supplying such estimates can improve agent decisions.
Critical weaknesses
-
Selective exploration before commitment and the incorporation of exploration costs have already been studied in prior works such as [1]. In particular, EAPO learns when and how to explore within an integrated policy-learning framework. The paper needs to distinguish CTA’s contribution more clearly and explain what it adds beyond these existing ideas.
-
CTA uses confidence calibration for question answering, a filename classifier for file reading, and a combination of confidence estimation and user-profile inference for software engineering. These heterogeneous components require substantial adaptation across tasks. Compared with EAPO’s unified exploration-aware formulation [1], it is unclear how much additional design and supervision would be needed to support a new environment.
-
CTA represents the exploration–execution trade-off through discounted task success, without explicitly accounting for actions that may cause irreversible damage before the task concludes. For example, executing uncertain code can lead to data loss or system failure. The suitability of the resulting “try-and-go” behavior therefore remains unclear in settings that require explicit safety constraints.
[1] Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
Questions for authors
-
What are the specific methodological advances of CTA over existing exploration-aware approaches, particularly EAPO? Which aspects go beyond providing explicit probability estimates and action costs to the agent?
-
How are the accuracy and calibration of the probability estimates evaluated on unseen tasks? How sensitive are CTA’s exploration and commitment decisions to errors in these estimates?
-
How would CTA handle actions with potentially harmful or irreversible consequences?
5 comments
Response (2/2)
W1.1 - What goes beyond providing the model with prob estimates...?
Beyond providing probabilities and costs to the agent, CTA contributes a decision-making framework that explicitly decouples uncertainty from action selection under cost tradeoffs (Figure 2), where we (1) estimate priors with a decoupled calibrated estimator and (2) condition the agent on the priors during training and/or inference.
Prompting is one way to expose the uncertainty signal; the same interface also improves RL training (CTA-RL, Sec. 4), and it lets CTA work with closed-weight models where fine-tuning is not possible (e.g., Claude Haiku 4.5 in SWE-Clarify). Therefore, the key contribution is the explicit interface between uncertainty estimation and action selection, which enables more rational, cost-aware behavior, rather than the prompt itself.
Q2.1 - How well do the prior estimates transfer to unseen tasks?
Within a domain, the estimates do transfer to unseen tasks: every estimator is fit only on a small validation set and evaluated on held-out test instances it has never seen. The estimator also does not have to be perfect: CTA benefits from stronger priors while remaining effective with imperfect but informative ones (App. B; App. F ladder), providing the possibility of transfer to datasets in near domains.
Across distant domains, transferring the estimates themselves is not the objective. Different tasks have different sources of uncertainty and different environment variables, so by nature a prior estimated for one environment is neither expected nor required to apply to another. However, the same calibrate-then-act decision layer is reused unchanged across all four tasks, with only the cheap per-task estimator needing to change.
Q2.2 - How sensitive is CTA to imperfect estimates?
We refer the reviewer to our prior-quality experiments in App. B and F, where we degrade the priors given to CTA. The results demonstrate two points:
- CTA is not tied to a specific estimator and the estimator does not have to be perfect. The results show that CTA benefits from stronger priors while remaining effective with imperfect but informative ones, which stably outperform the no-prior baseline.
- Deliberately corrupted priors (noisy or adversarial) are expected to underperform the baseline. The over-confidence cases hurt the general performance, which illustrates the same failure mode as the miscalibrated confidence of off-the-shelf LLMs [1,2]. And this is precisely why CTA decouples prior estimates from decision making.
[1] Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs, Xiong et al., 2024.
[2] Calibration of Pre-trained Transformers, Shrey Desai and Greg Durrett, 2020.
W3, Q3 -Handling irreversible or harmful actions
We would like to clarify that CTA does not assume that exploration actions are reversible and they do not need to be. Harmful or irreversible actions are orthogonal to the exploration-cost design and are captured by the reward already. Concretely, if an action risks task failure, that risk lowers its expected task utility, and CTA weighs that utility against the exploration cost. If an action risked harm, the reward function could accommodate the severity of that harm.
Thank you for the rebuttal. However, my main concerns are not fully addressed.
-
What does "varying cost constraints" mean? Is it the cost mentioned in Table.1? However, the cost in it are mostly , which is the same as it in EAPO.
-
The claim "CTA's decoupled design is much cheaper and effective" is inaccurate as EAPO aims to train the agent with the capability to explore in unseen environments, which cost nothing for adaptation.
-
An irreversible action means one that causes harm once executed, then we should not allow the agent to perform it at all, rather than letting it execute first and then reducing its utility after discovering the problem — because the harm has already occurred at the moment of execution, and lowering the utility afterwards cannot undo the damage that has already been done. A safety constraint is a prior restriction on the action set, which is an orthogonal level from weighing utility against cost among the legal actions.
Response (1/2)
Thank you for your comments and feedback! We address some of the questions and concerns below:
Our paper differs from EAPO fundamentally in core objective and task setup, as well as the cost of adapting to new environments.
Task setup (W1, Q1): EAPO does not adapt to varying cost constraints. Their reward modeling [EAPO, Sec 4.3] applies a discount factor of γ^2 to prevent overconservatism, but that gamma is a global parameter and is fixed within a training/inference run; moving to a different cost regime means re-training the policy. This hyperparameter is set to prevent overconservatism in general, but not training the agents to react adaptively to varying costs and uncertainties.
In contrast, CTA's objective is precisely that adaptive behavior under varying constraints: our test sets are constructed so that no static strategy wins across different costs, and we report results both aggregated and per cost level to show that the same system moves its operating point as the cost moves (e.g., ask rate 0.55→1.00 across γ, Table 8), with zero retraining.
Adaptation cost (W2): EAPO needs RL retraining on each new task or new cost regime, which is an inevitable adaptation cost. CTA's decoupled design is much cheaper and effective: each estimator is fit on a small validation set only, and RL is only an optional component in the CTA-RL variant. As measured in our response to Reviewer y4vN (Q2), this prior estimation costs at most ≈6% of one episode's tokens – orders of magnitude below an RL run.
Effort to support a new environment (W2, Q2): We would like to clarify that we do provide a unified framework to formalize and study this cost-aware exploration problem in a controlled setting (see Table 1 in our paper).
Our paper demonstrates how one could adapt the unified framework to varying environments/underlying state variables with representative tasks, including model internal confidence (QA), patterns in seen data/training data (FileReading), and user preferences and interaction history (SWE-Clarify). New environments are largely covered by this span of signal types.
Additionally, the prior estimates do not need to be perfect. As shown in our experiments in App. B and F, we show that CTA benefits from stronger priors while remaining effective with imperfect but informative ones. This tolerance further lowers the burden of adapting to a new environment.
We have also updated the related work section to explain this.