- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies confidence estimation for individual actions of LLM agents before execution. It proposes Critic Experience Bank (CEB), which stores records from completed trajectories, including the task context, action, feedback, a hindsight productivity pseudo-label, and the critic's original confidence. A fixed critic retrieves productive and unproductive records to assess new actions, and the bank is updated after each trajectory. The model parameters remain fixed, and bank construction does not use ground truth step labels. Experiments cover Mind2Web, Mind2Web-Live, AMEX, and InterCode-Bash with three model backbones. The paper reports improvements in calibration and ranking over six baselines, supported by component ablations, an analysis of accumulated experience, and evaluation under two labeling policies. Downstream validation includes selective execution diagnostics, simulated oracle review, and action regeneration at fixed states.
Critical strengths
-
A practical use of experience on the critic side. CEB gives a fixed critic a persistent record of what it previously predicted and what later evidence suggested about those predictions. Retrieving both productive and unproductive cases makes this more informative than simply supplying successful examples. Keeping the bank fixed within each trajectory also makes the intended information boundary explicit. This is a useful contribution for settings where the underlying models cannot be updated and reliable step annotations are unavailable during deployment.
-
The empirical support is not limited to one calibration metric. The experiments span four benchmarks and three backbones, and CEB achieves the best reported AUC under both evaluation policies across all twelve setups. The bank ablation offers especially insightful comparisons using identical actions: for Mind2Web with GPT-5.4, incorporating the bank cuts ECE from 0.318 to 0.176 and raises AUC from 0.672 to 0.704. Appendix I further tests real action regeneration under fixed states instead of assuming all interventions succeed. Collectively, these results provide meaningful evidence that the confidence scores can help identify actions worth reconsidering.
Critical weaknesses
-
The source of execution feedback is underspecified in the offline experiments. The central argument is that CEB uses evidence about what happened after an action, rather than only judging whether the action looks plausible. However, Mind2Web uses recorded page snapshots, and Appendix C does not explain what feedback is associated with a predicted action that differs from the reference. Figure 6 also presents a hindsight labeling prompt containing actions and reasoning, without an explicit field for environment observations. It is therefore unclear how the offline pseudo-labels are grounded in the consequences of the agent's own actions, rather than in the continuation of a reference replay or the model's reasoning. Since most component and memory analyses use offline Mind2Web, this ambiguity affects the central explanation of why the method works, not just reproducibility.
-
The calibration comparison needs stronger reference points and better matched baselines. For GPT-4.1 on Mind2Web, Table 8 reports a positive rate of 0.181 under the rule labels, while Table 1 reports CEB's ECE as 0.319 and Brier score as 0.294. On this same action stream, a predictor that always outputs would have both ECE and binary Brier score of approximately 0.181. Such a predictor cannot rank actions, so this does not negate CEB's AUC improvement. It does show that outperforming the selected baselines is not sufficient to establish strong probability calibration. Simple controls using only previously accumulated pseudo-labels would help put the gains in context. In addition, Verbalization and PRO are evaluated on their own actions in Tables 1 and 2, unlike CEB and the other scorers. Those particular comparisons mix confidence quality with differences in the actions being evaluated.
Questions for authors
-
What exactly is in offline Mind2Web when the agent selects an action different from the reference? Please provide a complete hindsight labeler input showing the observations it receives and explain how those observations are associated with the proposed action.
-
How does CEB compare with constant predictions and a simple confidence estimate based on pseudo-label frequencies from earlier trajectories? These controls should require no ground truth labels during deployment and would help distinguish contextual calibration gains from learning the overall positive rate.
-
Could CEB also score the actions generated by Verbalization and PRO, so that each comparison uses identical actions and labels? Reporting these matched results with uncertainty across task orders would make the main ranking and calibration claims easier to assess.
2 comments
Thank you again for your helpful suggestions. We added ECE and Brier to the matched-action experiment requested in Q3. All methods score identical actions and use the same earlier pseudo-labels and recalibration:
| Action set | Backbone | Baseline ECE | CEB ECE | Baseline Brier | CEB Brier | Baseline AUC | CEB AUC |
|---|---|---|---|---|---|---|---|
| Mind2Web Verbalization | GPT-4.1 | 0.299 | 0.268 | 0.221 | 0.196 | 0.553 | 0.727 |
| AMEX Verbalization | GPT-4.1 | 0.221 | 0.217 | 0.246 | 0.244 | 0.647 | 0.719 |
| AMEX PRO | GPT-5.4 | 0.145 | 0.133 | 0.231 | 0.199 | 0.588 | 0.762 |
CEB has better ECE, Brier, and AUC point estimates in all three settings. We sincerely appreciate your suggestion to strengthen these comparisons. We would be grateful if you could let us know whether our response has addressed your concerns.
Response to Reviewer jHHz
W1/Q1: Meaning of the Offline Observation
Thank you for identifying this ambiguity. As explained in our response to GDyG, if a proposed action differs from the reference action in offline Mind2Web, the next recorded state is not its result. The hindsight LLM instead uses the full trajectory. Later states, reasoning, and actions can reveal repeated actions, an unchanged page, or a stuck agent. These are indirect clues about whether an earlier proposal was useful, not its observed outcome. In the Ritz-Carlton example above, the five hindsight labels match all five Mind2Web rule labels. Mind2Web-Live and InterCode-Bash further confirm the method using direct environment feedback, and CEB outperforms the confidence baselines on both online datasets.
W2/Q2: Constant Baselines and Metric Interpretation
Thank you for identifying the need for constant baselines. In the submitted Mind2Web/GPT-4.1 setting, only 18.1% of rule labels are positive. Always predicting zero therefore gives ECE and Brier of 0.181, but its AUC is 0.5 because it cannot rank successful and unsuccessful steps. This is a known limitation of calibration error under class imbalance: prior LLM confidence work therefore reports discrimination metrics such as AUC together with ECE and Brier [1].
W2/Q3: Same Feedback and Same Actions
Thank you for requesting simple controls that use only previously accumulated pseudo-labels and evaluate the same actions. We use the controlled experiment reported to GDyG. Within each row, the native baseline and CEB score the same actions and use the same online calibration fitted only on earlier pseudo-labels. Only CEB retrieves past examples at prediction time.
CEB improves AUC in all three controlled comparisons. The gains therefore remain after matching both the action stream and access to historical pseudo-labels.
References
[1] Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. EMNLP 2023.