acceptodds
Under review as a conference paper at ICLR 2027

Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents

Abstract

LLM agents operate in stateful environments, where a single erroneous step can waste limited interaction budget or cause irreversible effects before task failure becomes apparent. Reliable deployment therefore requires *step-level confidence estimation*: estimating, *before execution*, the probability that a proposed action will advance the task. Existing LLM confidence estimators are typically designed for static question answering under a fixed task context and evaluation criterion. For an agent, however, its action productivity depends on an environment transition that is observed only after execution. To address this challenge, we introduce Critic Experience Bank (CEB), a training-free framework that turns feedback from completed trajectories into reusable evidence for future confidence judgments. After each trajectory, an LLM assigns hindsight productivity pseudo-labels to individual actions and stores them with the critic's original pre-execution confidence, task context, action, and observed feedback. For a new action, by retrieving related productive and unproductive experiences, CEB grounds pre-execution confidence in feedback from completed trajectories to condition a fixed LLM critic. CEB thereby adapts over a task stream without parameter updates or ground-truth step labels at deployment. Across four agent benchmarks spanning offline and live web navigation, mobile GUI and shell tasks, and three critic backbones, CEB achieves the best or tied-best ECE, Brier score, and AUC in all twelve benchmark–backbone settings under rule-based step labels, reducing ECE by up to 53.8% relative to the strongest training-free baseline. Its confidence scores also improve downstream utility in selective execution and simulated task success.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, Confidence Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jHHzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies confidence estimation for individual actions of LLM agents before execution. It proposes Critic Experience Bank (CEB), which stores records from completed trajectories, including the task context, action, feedback, a hindsight productivity pseudo-label, and the critic's original confidence. A fixed critic retrieves productive and unproductive records to assess new actions, and the bank is updated after each trajectory. The model parameters remain fixed, and bank construction does not use ground truth step labels. Experiments cover Mind2Web, Mind2Web-Live, AMEX, and InterCode-Bash with three model backbones. The paper reports improvements in calibration and ranking over six baselines, supported by component ablations, an analysis of accumulated experience, and evaluation under two labeling policies. Downstream validation includes selective execution diagnostics, simulated oracle review, and action regeneration at fixed states.

Critical strengths
  1. A practical use of experience on the critic side. CEB gives a fixed critic a persistent record of what it previously predicted and what later evidence suggested about those predictions. Retrieving both productive and unproductive cases makes this more informative than simply supplying successful examples. Keeping the bank fixed within each trajectory also makes the intended information boundary explicit. This is a useful contribution for settings where the underlying models cannot be updated and reliable step annotations are unavailable during deployment.

  2. The empirical support is not limited to one calibration metric. The experiments span four benchmarks and three backbones, and CEB achieves the best reported AUC under both evaluation policies across all twelve setups. The bank ablation offers especially insightful comparisons using identical actions: for Mind2Web with GPT-5.4, incorporating the bank cuts ECE from 0.318 to 0.176 and raises AUC from 0.672 to 0.704. Appendix I further tests real action regeneration under fixed states instead of assuming all interventions succeed. Collectively, these results provide meaningful evidence that the confidence scores can help identify actions worth reconsidering.

Critical weaknesses
  1. The source of execution feedback is underspecified in the offline experiments. The central argument is that CEB uses evidence about what happened after an action, rather than only judging whether the action looks plausible. However, Mind2Web uses recorded page snapshots, and Appendix C does not explain what feedback is associated with a predicted action that differs from the reference. Figure 6 also presents a hindsight labeling prompt containing actions and reasoning, without an explicit field for environment observations. It is therefore unclear how the offline pseudo-labels are grounded in the consequences of the agent's own actions, rather than in the continuation of a reference replay or the model's reasoning. Since most component and memory analyses use offline Mind2Web, this ambiguity affects the central explanation of why the method works, not just reproducibility.

  2. The calibration comparison needs stronger reference points and better matched baselines. For GPT-4.1 on Mind2Web, Table 8 reports a positive rate of 0.181 under the rule labels, while Table 1 reports CEB's ECE as 0.319 and Brier score as 0.294. On this same action stream, a predictor that always outputs z^t=0\hat z_t=0 would have both ECE and binary Brier score of approximately 0.181. Such a predictor cannot rank actions, so this does not negate CEB's AUC improvement. It does show that outperforming the selected baselines is not sufficient to establish strong probability calibration. Simple controls using only previously accumulated pseudo-labels would help put the gains in context. In addition, Verbalization and PRO are evaluated on their own actions in Tables 1 and 2, unlike CEB and the other scorers. Those particular comparisons mix confidence quality with differences in the actions being evaluated.

Questions for authors
  1. What exactly is oto_t in offline Mind2Web when the agent selects an action different from the reference? Please provide a complete hindsight labeler input showing the observations it receives and explain how those observations are associated with the proposed action.

  2. How does CEB compare with constant predictions and a simple confidence estimate based on pseudo-label frequencies from earlier trajectories? These controls should require no ground truth labels during deployment and would help distinguish contextual calibration gains from learning the overall positive rate.

  3. Could CEB also score the actions generated by Verbalization and PRO, so that each comparison uses identical actions and labels? Reporting these matched results with uncertainty across task orders would make the main ranking and calibration claims easier to assess.

2 comments
AuthorsOpenReview ↗

Response to Reviewer jHHz

W1/Q1: Meaning of the Offline Observation

Thank you for identifying this ambiguity. As explained in our response to GDyG, if a proposed action differs from the reference action in offline Mind2Web, the next recorded state is not its result. The hindsight LLM instead uses the full trajectory. Later states, reasoning, and actions can reveal repeated actions, an unchanged page, or a stuck agent. These are indirect clues about whether an earlier proposal was useful, not its observed outcome. In the Ritz-Carlton example above, the five hindsight labels match all five Mind2Web rule labels. Mind2Web-Live and InterCode-Bash further confirm the method using direct environment feedback, and CEB outperforms the confidence baselines on both online datasets.

W2/Q2: Constant Baselines and Metric Interpretation

Thank you for identifying the need for constant baselines. In the submitted Mind2Web/GPT-4.1 setting, only 18.1% of rule labels are positive. Always predicting zero therefore gives ECE and Brier of 0.181, but its AUC is 0.5 because it cannot rank successful and unsuccessful steps. This is a known limitation of calibration error under class imbalance: prior LLM confidence work therefore reports discrimination metrics such as AUC together with ECE and Brier [1].

W2/Q3: Same Feedback and Same Actions

Thank you for requesting simple controls that use only previously accumulated pseudo-labels and evaluate the same actions. We use the controlled experiment reported to GDyG. Within each row, the native baseline and CEB score the same actions and use the same online calibration fitted only on earlier pseudo-labels. Only CEB retrieves past examples at prediction time.

Action setBackboneBaseline AUCCEB AUC
Mind2Web VerbalizationGPT-4.10.5530.727
AMEX VerbalizationGPT-4.10.6470.719
AMEX PROGPT-5.40.5880.762

CEB improves AUC in all three controlled comparisons. The gains therefore remain after matching both the action stream and access to historical pseudo-labels.

References

[1] Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. EMNLP 2023.

Thank you again for your helpful suggestions. We added ECE and Brier to the matched-action experiment requested in Q3. All methods score identical actions and use the same earlier pseudo-labels and recalibration:

Action setBackboneBaseline ECECEB ECEBaseline BrierCEB BrierBaseline AUCCEB AUC
Mind2Web VerbalizationGPT-4.10.2990.2680.2210.1960.5530.727
AMEX VerbalizationGPT-4.10.2210.2170.2460.2440.6470.719
AMEX PROGPT-5.40.1450.1330.2310.1990.5880.762

CEB has better ECE, Brier, and AUC point estimates in all three settings. We sincerely appreciate your suggestion to strengthen these comparisons. We would be grateful if you could let us know whether our response has addressed your concerns.

Reviewer_ghCvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training-free critic that estimates, before execution, the probability that a proposed agent action will advance the task. After each trajectory, a hindsight LLM labels steps productive/unproductive via majority vote and stores them with task, state, action, feedback, and the critic's original confidence. For a new action, the critic retrieves similar records and is prompted to give a confidence score based on current track and past records. Evaluation covers Mind2Web, Mind2Web-Live, AMEX and InterCode-Bash, with GPT-4.1 / GPT-5.4 / Qwen3-Next. CEB is best or tied-best on ECE, Brier and AUC in all 12 settings against rule-based step labels. Further results include ablations, an oracle-label bank comparison, and downstream diagnostics (selective execution, oracle review, regeneration).

Critical strengths

Provide evidence that retrieved experience helps identify unreliable actions. The bank and retrieval ablations suggest that relevant past interactions help the critic distinguish productive from unproductive actions. The offline regeneration study provides some evidence that this improved ranking can guide where to spend reconsideration effort, although its value in closed-loop execution remains unverified.

Critical weaknesses
  1. The calibration comparison lacks a baseline that uses the same historical feedback. CEB benefits from hindsight pseudo-labels accumulated across completed trajectories, while the main confidence baselines do not use that history to adjust their scores. The ranking improvements and retrieval ablations support the usefulness of experience, but do not establish how much additional probability-calibration benefit retrieval provides over a simple score recalibrator. A baseline that adjusts a bank-free critic's scores using only earlier trajectories' scores and pseudo-labels would make this comparison more informative, while keeping the underlying LLM fixed.
  2. The evaluation protocol is non-standard, preventing comparison with published work, and downstream utility is only shown in offline simulations.
  • Evaluation. The four benchmarks (Mind2Web, Mind2Web-Live, AMEX, InterCode) are agent task benchmarks, not confidence benchmarks. The paper defines its own step labels. No results are reported on established step-level evaluations like WebRewardBench or using the benchmark's official metrics (Step success rate, task success rate). All baselines are reimplemented, so none of the numbers can be directly compared with published results.
  • Downstream. All downstream experiments (selective execution, oracle review, regeneration) are offline simulations on frozen trajectories. There is no closed-loop deployment showing that CEB improves task success, reduces cost, or enables selective intervention in practice. Related work has demonstrated such integration: WebShepherd (NeurIPS 2025) reports 34.55% success on WebArena-lite when using its step scores to guide execution (+10.9 over the baseline evaluator).

As a result, the claimed superiority is established only within the paper's own setup, with no anchor to community-standard evaluations or real deployment scenarios.

Questions for authors
  1. Please report results on at least one established step-level evaluation, such as WebRewardBench, or on the official Mind2Web test splits using the benchmark's standard step labels.
  2. Please conduct a closed-loop experiment on Mind2Web-Live or InterCode where low-confidence steps trigger regeneration or human review, and report task success rate, cost.
  3. The critic and hindsight labeler share the same backbone. Their errors are likely correlated: a step the critic wrongly finds plausible may also be wrongly labeled productive, then retrieved as confirming evidence. Could you run the hindsight labeler with a different model family and report how bank-label accuracy and ECE change?
  4. Could you compare CEB with a bank-free critic whose scores are recalibrated using the same hindsight pseudo-labels from previously completed trajectories? This would test whether retrieval improves probability calibration beyond a simpler use of the available feedback.
3 comments
AuthorsOpenReview ↗

Response to Reviewer ghCv

W1/Q4: Same Historical Feedback Without Retrieval

Thank you for this suggestion. We use the controlled experiment described in our response to GDyG W3/Q3. Within each row, Verbalization or PRO and CEB score the same actions. Both methods also use the same pseudo-labels from earlier trajectories through the same online calibration procedure; only CEB retrieves past examples when making a prediction.

Action setBackboneBaseline AUCCEB AUC
Mind2Web VerbalizationGPT-4.10.5530.727
AMEX VerbalizationGPT-4.10.6470.719
AMEX PROGPT-5.40.5880.762

CEB has higher AUC in all three comparisons. This control shows that its gain is not explained only by access to historical labels or by evaluating different actions.

W2/Q1: Standard Evaluation

Thank you for asking for an established evaluation. Mind2Web provides an official step label: a step succeeds only when both the target element and operation match. Our Mind2Web rule label uses this criterion.

We also evaluated all 220 WebRewardBench tasks: 776 decision steps with five candidate actions each. MRR measures how highly the preferred action is ranked. Step accuracy is how often it ranks first, while trajectory accuracy requires every step to be correct.

MethodMRRStep accuracyTrajectory accuracy
P(True)0.4940.2730.023
CoT0.5800.3300.036
CEB, online bank0.6300.4090.077

CEB obtains the best result on all three official metrics. It improves MRR over P(True) by 0.136 and over CoT by 0.050. It also improves step accuracy over CoT by 0.079 [+0.038, +0.118]. Thus, CEB remains effective under an established step-level evaluation and the benchmark's official metrics.

W2/Q2: Closed-Loop Deployment

Thank you for requesting a closed-loop test. We compare no intervention with P(True), CoT, and CEB as confidence-based triggers for replanning. A low-confidence step generates one alternative action from the same state, and the higher-confidence action is executed. We allow at most three replans per task.

Key-node rate is the fraction of required intermediate goals reached. Task success is the fraction of tasks that reach every goal. Replans/task is the average number of extra action-generation attempts per task; each adds one actor call and one critic call.

SettingKey-node rateTask successAvg. replans/task
No intervention0.4320.1170.00
P(True)-gated0.4130.1172.75
CoT-gated0.4300.1332.20
CEB-gated0.4800.1672.40

CEB raises task success from 11.7% without intervention to 16.7%, or from 7 to 10 solved tasks. It also achieves the highest key-node rate with a replan budget similar to CoT and lower than P(True). This provides direct closed-loop evidence that CEB identifies more useful points for intervention.

Q3: Correlated Critic and Labeler Errors

Thank you for raising the risk that the critic and labeler make the same errors. We initially used the same backbone to keep model capacity fixed and to represent settings with only one available model. However, this choice may reinforce the model's own mistakes. We therefore rebuilt the Mind2Web bank with Qwen3-32B hindsight labels while keeping the GPT-4.1 critic and the same evaluation data.

Hindsight labelerRule AUCLLM-Judge AUCLLM-Judge ECE
GPT-4.10.6350.7590.211
Qwen3-32B0.6310.7500.170

The AUC values are nearly unchanged, while judge-label ECE improves. This single model-dataset setting does not show that the result generalizes to every model family, but it shows that the critic and hindsight labeler do not need to use the same family. We will add more model pairs in the revision.

Thank you for the detailed response. I acknowledge the WebRewardBench evaluation, the closed-loop experiment, and the cross-family labeler test as good-faith efforts, and several of my concerns are partially addressed.

I nevertheless maintain my score:

  1. Calibration metrics are missing from every new controlled experiment. My W1/Q4 request concerned probability calibration — whether retrieval improves ECE/Brier over a bank-free critic recalibrated with the same pseudo-labels. All controlled comparisons in the rebuttal report only AUC, which measures ranking, not probability quality. For a paper whose headline claim is calibration ("best or tied-best on ECE, Brier and AUC in all 12 settings"), this omission leaves the central question open. I would like to see the retrieval-free recalibration baseline's ECE/Brier, across all settings.

  2. The closed-loop gain is not statistically meaningful. Task success moves from 7 to 10 out of 60 tasks, with no significance test, and the human-review branch of my request was not run.

  3. Calibration claims still have no community-standard anchor. WebRewardBench is a ranking benchmark; ECE/Brier remain evaluated only under the paper's own protocol and labels.

I would raise my score if the authors provide (a) the same-feedback, retrieval-free baseline on ECE/Brier across all settings, and (b) a closed-loop result with statistical support.

Thank you for the careful follow-up and for clarifying the additional evidence needed. Following your suggestions, we added ECE and Brier to the controlled comparisons and expanded the closed-loop experiment from 60 to all 232 tasks.

Same-feedback, retrieval-free control. In each controlled comparison, the baseline and CEB score exactly the same actions, receive the same historical pseudo-labels, and use the same recalibration procedure. The controlled difference is whether relevant historical experiences are retrieved.

Action setBackboneBaseline ECECEB ECEBaseline BrierCEB BrierBaseline AUCCEB AUC
Mind2Web VerbalizationGPT-4.10.2990.2680.2210.1960.5530.727
AMEX VerbalizationGPT-4.10.2210.2170.2460.2440.6470.719
AMEX PROGPT-5.40.1450.1330.2310.1990.5880.762

CEB improves ECE, Brier, and AUC in all three controlled settings. These results provide additional evidence that retrieval improves probability quality, rather than only ranking, when access to historical feedback and recalibration are held fixed.

Full closed-loop experiment. Following your request for a larger evaluation with statistical support, we reran the deployment experiment on all 232 Mind2Web-Live tasks. All methods use the same task order and intervention protocol.

SettingKey-node rateDifference from no intervention [95% CI]Task successAvg. replans/task
No intervention0.388—0.0730.00
P(True)-gated0.406+0.018 [−0.001, +0.038]0.0952.30
CoT-gated0.405+0.017 [−0.004, +0.038]0.0861.56
CEB-gated0.421+0.034 [+0.014, +0.053]0.1212.38

The confidence intervals are computed using paired task-level bootstrap. CEB is the only method whose key-node improvement over no intervention has a 95% confidence interval excluding zero. It also increases task success from 7.3% to 12.1%. Compared with P(True), CEB improves task success by 2.6 percentage points while requiring only 0.08 additional replans per task.

We sincerely appreciate these suggestions, which helped us strengthen both the calibration controls and the deployment evaluation.

Reviewer_GDyGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Critic Experience Bank (CEB), a training-free method for estimating whether an agent’s next action will advance its task. It stores hindsight-labeled experiences from completed trajectories and retrieves relevant positive and negative examples to inform future confidence estimates. Experiments cover four benchmarks and three model backbones.

Critical strengths
  • The problem is well motivated. Estimating confidence before an action changes the environment is useful, and the method works with fixed API models without requiring ground-truth step labels at deployment.

  • The main results are consistent across environments and backbones. CEB achieves the best or tied-best ECE, Brier score, and AUC across all twelve settings under the rule-based labels.

  • The evaluation goes beyond aggregate calibration metrics. Alternative labeling, task-stream experiments, and action-regeneration tests provide complementary evidence that the confidence scores contain useful information.

Critical weaknesses
  • The evaluation labels are imperfect proxies for task progress. Reference matching can reject valid alternatives, while the Mind2Web-Live rule credits every step before the last attained key node, potentially including mistakes. The alternative LLM judge helps, but independent validation is still missing.

  • The feedback construction in offline replay needs clarification. When the proposed action differs from the recorded reference, the next recorded state may not represent its consequence. This matters because learning from execution outcomes is central to the method’s motivation.

  • The main baseline comparison is not fully controlled: Verbalization and PRO generate their own actions, whereas CEB scores a shared frozen stream. A stronger comparison would also include a simple calibrator using the same historical pseudo-labels and a comparable inference budget.

  • The downstream evidence remains limited to fixed trajectories or fixed states. Oracle corrections assume perfect intervention, and offline regeneration does not propagate changes through subsequent actions. These experiments support error selection, but do not yet establish improved closed-loop task success.

Questions for authors
  1. Can a small human audit assess both the hindsight pseudo-labels and the evaluation labels, particularly for exploratory actions and mistakes that precede eventual progress?

  2. How is post-action feedback constructed for offline Mind2Web and AMEX when the proposed action differs from the reference? What execution observations are actually passed to the hindsight labeler in Figure 6?

  3. Do the gains persist when all baselines score identical actions and have access to the same historical pseudo-labels, with inference costs accounted for?

  4. Can CEB guide actual replanning or regeneration in a live environment, with task success measured after the resulting state changes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer GDyG

W1/Q1: Label Validity

Thank you for highlighting label limitations. Rule and LLM-as-a-judge labels are imperfect but common. Mind2Web and WebCanvas use rule-based progress signals, while AgentRM [1] and WebJudge [2] use LLM judges. We use both as complementary proxies.

We blindly audited 120 Mind2Web steps sampled from cases where the rule and hindsight labels agree or disagree. Two authors labeled each step independently and resolved disagreements to obtain human reference labels. Both machine-label policies agree strongly:

Machine labelCohen's κ\kappa with human labels
Rule0.835
LLM judge0.848

Against these human labels, CoT and P(True) use scores from their corresponding Table 1 runs, and all three methods score the same actions.

MethodECE$\downarrow$Brier$\downarrow$AUC$\uparrow$
CoT0.4280.3420.716
P(True)0.3740.3750.613
CEB0.2300.2020.757

The audit supports both label policies and preserves the conclusion: CEB is best on all three metrics.

W2/Q2: Feedback in Offline Replay

Thank you for identifying this ambiguity. In offline Mind2Web and AMEX, if a proposal differs from the reference action, the next recorded state is not its consequence. The hindsight LLM instead uses the full trajectory. Repeated actions, an unchanged page, a change of direction, or a stuck agent provide indirect evidence of usefulness, not an observed consequence. We will make this distinction clear.

A Mind2Web trajectory illustrates this: the original hindsight LLM and rule labels both mark five steps as unproductive.

Task: Find Ritz-Carlton hotels in Washington, DC, for a May 9--12 student event requiring seven rooms and event space for 50 people.
Steps 7-11: repeatedly CLICK Marriott Bonvoy; the proposal rationales state that the page is still loading, stuck, or showing the same header.
Hindsight labels / Mind2Web rule labels: [0, 0, 0, 0, 0] / [0, 0, 0, 0, 0].
Independent GPT-5.4 audit: Step 7 resets navigation; steps 8-11 repeat the same top-level link without finding hotel or event-space details.

We also test Mind2Web-Live and InterCode-Bash, which provide direct environment feedback; CEB outperforms the confidence baselines on both.

W3/Q3: Controlled Verbalization and PRO Comparisons

Thank you for pointing out this problem. Our initial comparison followed native Verbalization and PRO protocols [3,4]. Because Verbalization jointly emits action and confidence and PRO samples outputs, either may change the evaluated action. We therefore freeze each action set; the baseline and CEB score the same actions and use the same earlier pseudo-labels for online calibration. The table reports AUC.

Action setBackboneNative AUCCEB AUC
Mind2Web VerbalizationGPT-4.10.5530.727
AMEX VerbalizationGPT-4.10.6470.719
AMEX PROGPT-5.40.5880.762

CEB has higher AUC on all three streams, so its advantage remains with matched actions and historical labels.

W4/Q4: Closed-Loop Task Success

Thank you for requesting a closed-loop test. We compare no intervention with P(True), CoT, and CEB on the same random 60 Mind2Web-Live tasks. A low-confidence step generates one alternative from the same state, and the higher-confidence action is executed, with at most three replans per task.

Key-node rate measures intermediate goals reached; task success requires reaching every goal. Replans/task measures extra action-generation attempts, each adding one actor and one critic call.

SettingKey-node rateTask successAvg. replans/task
No intervention0.4320.1170.00
P(True)-gated0.4130.1172.75
CoT-gated0.4300.1332.20
CEB-gated0.4800.1672.40

CEB has the highest key-node rate and raises task success from 11.7% to 16.7% at 2.40 replans/task.

References

[1] AgentRM: Enhancing Agent Generalization with Reward Modeling. ACL 2025. [2] An Illusion of Progress? Assessing the Current State of Web Agents. COLM 2025. [3] Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. EMNLP 2023. [4] Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. ICLR 2024.

Thank you again for your helpful suggestions. We added the following results, also reported in our follow-up to Reviewer ghCv.

For Q3, all methods score identical actions and use the same earlier pseudo-labels and recalibration:

Action setBackboneBaseline ECECEB ECEBaseline BrierCEB BrierBaseline AUCCEB AUC
Mind2Web VerbalizationGPT-4.10.2990.2680.2210.1960.5530.727
AMEX VerbalizationGPT-4.10.2210.2170.2460.2440.6470.719
AMEX PROGPT-5.40.1450.1330.2310.1990.5880.762

For Q4, we expanded the closed-loop evaluation to all 232 Mind2Web-Live tasks:

SettingKey-node rateDifference [95% CI]Task successReplans/task
No intervention0.388—0.0730.00
P(True)0.406+0.018 [−0.001, 0.038]0.0952.30
CoT0.405+0.017 [−0.004, 0.038]0.0861.56
CEB0.421+0.034 [0.014, 0.053]0.1212.38

CEB is the only method whose key-node improvement has a 95% CI excluding zero. We sincerely appreciate your suggestions to strengthen these comparisons. We would be grateful if you could let us know whether our response has addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.