acceptodds
Under review as a conference paper at ICLR 2027

Bayesian Evidence Fusion for Uncertainty Quantification of AI Agents

Abstract

AI agents complete tasks through sequences of diverse actions: large language model (LLM) generations, tool calls, and revisions. Existing uncertainty quantification (UQ) techniques typically assess the reliability of individual LLM generations, leaving open how to aggregate and propagate uncertainty across different types of actions within an agent trajectory. We formulate UQ for LLM agents as Bayesian belief tracking over final task correctness, integrating generation-level uncertainty with tool outcomes through Bayesian updates. Our framework requires no changes to the agentic harness or LLM, and can be applied post hoc without additional agent executions, thus introducing negligible latency. We conduct an extensive evaluation across five LLMs, eight agent harnesses, and 15 agentic environments spanning coding, retrieval, and interactive tasks. The basic Bayesian fusion achieves the highest average outcome-ranking scores on coding and retrieval, while a variant that separately models trajectory length, history, and final-step evidence leads on interactive tasks. Bayesian fusion generalizes well across models, harnesses, and datasets within the same domain. For larger environment and domain shifts, adaptation with little or no labeled target data improves performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agents, LLM, Uncertainty Quantification, Bayesian Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MB2NOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates trajectory-level uncertainty quantification for LLM agents as Bayesian belief tracking over final task success. For each evidence channel, such as pass or fail outcomes from tools and critics and generation-level uncertainty like perplexity and verbalized confidence, the method estimates class-conditional likelihoods on successful and failed trajectories and adds the resulting log-likelihood ratios to the prior log-odds to compute the probability of success. An extension called Structured Bayesian Fusion models trajectory length, evidence from earlier steps, and evidence from the final step separately, and recalibrates the history contributions. The method is applied post hoc to recorded rollouts and requires no additional agent executions. Across 15 coding, retrieval, and interactive environments and five LLMs, the paper evaluates how well the scores identify failed trajectories using [email protected], and reports that the basic fusion achieves the highest average on retrieval and coding while Structured Bayesian Fusion leads on interactive tasks.

Critical strengths

S1. The paper addresses a practical and well-motivated problem. The premise that judging whether an agent's final outcome is correct requires combining generation uncertainty with external feedback such as tool results and tests is sound. The method is applied to recorded trajectories without changing the agent or the harness, and its training and inference costs are negligible. The tool-only variant, which works without token probabilities, is also practical.

S2. The evaluation is broad. The method is compared with baselines that include recent trajectory-level methods such as SAUP, UProp, and HTC across 15 environments, five LLMs, and multiple harnesses. Covering within-domain and cross-domain transfer as well as performance with little training data also reflects attention to practical use.

S3. The analysis of the roles of different evidence is useful. The observation that tool feedback matters more in retrieval and generation uncertainty matters more in coding, together with the analysis showing that adding uninformative critics on CRAG hurts the strongest signal, verbalized confidence, provides insight into when each type of evidence is useful.

Critical weaknesses

W1. The methodological novelty is limited, and the Bayesian belief tracking framing is overstated. The basic fusion is textbook naive Bayes that treats the evidence as independent and sums log-likelihood ratios. Although the update is written step by step, the method is evaluated only on completed trajectories, so its ability to track success over time is never tested. Furthermore, the authors show in Proposition 1 that naively summing correlated evidence overstates it, and the resulting expression itself indicates how to correct for trajectory length, yet this correction is not incorporated into the method.

W2. Evaluations matching the probabilistic framing are missing, and the comparison favors the proposed method. The method outputs success probabilities intended for decision making, but calibration is not evaluated, nor is failure prediction at intermediate points, which the tracking framing would require. In addition, the best variant differs by domain, so the best of several variants is compared against each baseline. The best variant on retrieval relies on verbalized confidence and therefore requires an extra LLM call at every step. Without verbalized confidence, the basic fusion variants fall below HTC and logistic regression, and only Structured Bayesian Fusion, which itself contains learned logistic-regression components, exceeds them slightly. On interactive tasks, all basic fusion variants fall below logistic regression, so it is unclear whether Bayesian fusion itself is the source of the gains.

W3. Performance depends heavily on critics designed for each benchmark, which limits generality to new environments. The core evidence, tool feedback, consists of boolean critics designed by hand for each benchmark, and when the critics are uninformative, as on CRAG, fusion can even hurt, as it does for the strongest signal, verbalized confidence. Much of the result therefore comes from critic design, and the need for such design in every new environment should be discussed alongside the claim that the method requires no changes to the harness. Indeed, performance approaches random when the domain changes. It is also unclear which accumulation rule for critic outcomes, such as summing per step or summarizing once per episode, is used in each experiment, and LLM-as-judge verifiers are not compared as an independent baseline.

Questions for authors

Q1. How well calibrated are the predicted success probabilities, for example in terms of ECE or Brier score? Does correcting the history contribution for trajectory length, as suggested by Proposition 1, improve ranking and calibration?

Q2. Does a single prespecified variant used across all domains still outperform the best baseline? Do the conclusions hold when terminated trajectories are included as failures?

Q3. How costly is it to design critics for a new environment, and how much does performance vary with critic quality? How well does the method predict failure at intermediate points of a trajectory?

5 comments
AuthorsOpenReview ↗

Response to Reviewer MB2N (1/4)

Thank you very much for your time in reviewing our paper and your insightful comments.

Q1. The methodological novelty is limited, and the Bayesian belief tracking framing is overstated…

Our contribution is not Bayes' rule but its formulation for LLM agents: Bayesian belief tracking over final task correctness, where tool outcomes and generation-level uncertainty both act as evidence.

The basic fusion rule is simple and standard, and the paper presents it as such (Eq. 1; lines 154–156), while lines 171–174 state the independence assumption. This simplicity is an advantage: the basic fusion improves every raw uncertainty signal in all three task families (Tables 7–9), it is already strong with 30–50 training episodes, where logistic regression is close to random (Figure 3), and it keeps its ranking quality under model and harness shifts, where logistic regression loses 0.191 PRR on average (Figure 4).

Additionally, prior trajectory-level methods (SAUP, UProp, HTC) rely on generation-level signals only, while we convert both generation signals and tool outcomes into log-likelihood ratios and add them on a common log-odds scale. SBF also goes beyond naive Bayes: it models trajectory length, history, and the final step separately, uses a joint final-step model (Eq. 16), and recalibrates the history evidence (Eq. 17).

The belief update is sequential, so the score exists for every trajectory prefix. We additionally evaluate it on prefixes, where SBF has the best average [email protected] (see our answer to Q5).


Q2. The correction from Proposition 1 is not incorporated into the method…Does correcting the history contribution for trajectory length, as suggested by Proposition 1, improve ranking and calibration?

The length correction suggested by Proposition 1 improves both ranking and calibration, and SBF performs on par with it.

SBF rescales the accumulated history evidence by a fitted β\beta (Eq. 17; lines 256–258). For comparison, we also tested the length correction directly: we estimate ρ\rho on the training folds and divide each accumulated history term by 1+(K−1)ρ1+(K-1)\rho.

History term[email protected] ALFWorld[email protected] DeepSWE[email protected] AvgECE ALFWorldECE DeepSWEECE Avg
Independent accumulation (Eq. 15).756 ± .023.157 ± .039.457 ± .023.178 ± .006.202 ± .015.190 ± .008
+ length correction of Proposition 1.804 ± .020.177 ± .029.490 ± .018.109 ± .008.123 ± .009.116 ± .006
+ affine recalibration (SBF, Tables 8–9).825 ± .010.173 ± .036.499 ± .019.097 ± .005.129 ± .020.113 ± .010

The length correction improves both ranking and calibration, as Proposition 1 suggests. SBF's recalibration reaches the same quality without estimating ρ\rho, so it already provides a similar correction.

We will add this analysis to the camera-ready version.


Q3. The best variant on retrieval relies on verbalized confidence and therefore requires an extra LLM call at every step...On interactive tasks, all basic fusion variants fall below logistic regression, so it is unclear whether Bayesian fusion itself is the source of the gains.

The gains come from combining tool feedback and generation uncertainty as evidence on a common log-odds scale: SBF outperforms both HTC and logistic regression on average in every domain, without verbalized confidence.

First, SBF does not require verbalized confidence, so the gains do not depend on the extra LLM call.

Second, fusion alone already helps: without any logistic-regression components, fitting only per-signal class-conditional likelihoods, Bayes Fused improves every raw signal in all three task families (15 of 15 cases in Tables 7–9; solid blue bars in Figure 2). Its simplicity is especially beneficial out of domain (Figures 4 and 5).

Third, SBF outperforms logistic regression given the same evidence (Appendix B.1: aggregates of every generation signal, tool outcomes, and per-step pass rates), so the gain does not come from additional inputs. SBF uses logistic regression only to project several uncertainty signals onto one scalar for the history and one for the final step. The evidence itself comes from class-conditional Gaussian and Bernoulli models.

AuthorsOpenReview ↗

Response to Reviewer MB2N (2/4)

Q4. Performance depends heavily on critics designed for each benchmark, which limits generality to new environments…How costly is it to design critics for a new environment, and how much does performance vary with critic quality?

Critics are cheap to define: they are boolean checks on information the harness already provides, with no tuning on outcomes, and SBF stays the best learned method even when they carry almost no information (e.g., on CRAG).

The critics are lightweight boolean checks based on information that the harness already provides: whether an action is well-formed, whether a tool call executes successfully, and whether it changes the environment state (Tables 2, 4, and 6 list all of them). They follow directly from the information available in each task, and defining them requires no tuning on outcomes and no changes to the agent or the harness.

The method also remains effective when the critics carry little information. For example, on CRAG, tool feedback is nearly uninformative (Tool-only: −.020), yet SBF still achieves the best result among learned methods (.391 vs. .358 for logistic regression), because it weighs critic and generation evidence according to their estimated informativeness. As we note in lines 321–323, the CRAG critics do not check whether the search retrieved new documents, which explains the drop for Bayes Fused (Verb.). Additionally, Tables 7-9 already show how performance varies with critic quality (Tool-only row): the gain from fusion is largest where the critics are informative on their own (Tool-only reaches .353 on MuSiQue and .445 on BrowseComp-Plus) and smaller where they are not (−.020 on CRAG).

Under domain shift, we use only the critics shared by both domains, so the drop is not caused by missing critics. The same check can still predict success differently across environments, so critic likelihoods may also shift. However, the most visible shift is in the distributions of uncertainty scores, caused by differences in answer length, trajectory length, and response informativeness. A few labeled target examples are enough to re-estimate the likelihoods, after which the method outperforms the baselines even when tool feedback is nearly uninformative.

We will expand this discussion in the camera-ready version of the paper.


Q5. Although the update is written step by step, the method is evaluated only on completed trajectories, so its ability to track success over time is never tested…How well does the method predict failure at intermediate points of a trajectory?

On trajectory prefixes, SBF has the best average [email protected] for predicting both the final outcome and the current task state.

Thank you for this suggestion! We have added this evaluation. Each estimator sees only the trajectory up to step kk, and we test two setups: (1) whether it predicts the final outcome of the episode, and (2) whether it predicts the state of the task at step kk.


(1) Predicting the final outcome from the first kk steps ([email protected]).

MethodWebArena, 8 stepsBrowseComp-Plus, 10 stepsDeepSWE, mid-trajectory stepAvg
SP.171 ± .000.471 ± .000−.135 ± .000.169 ± .000
Logistic regression.307 ± .016.431 ± .009.035 ± .027.258 ± .011
Bayes Fused (SP).196 ± .022.446 ± .001.025 ± .091.222 ± .031
SBF.294 ± .014.453 ± .006.095 ± .061.281 ± .021

(2) Predicting the state of the task at step kk ([email protected]).

MethodBrowseComp-Plus, 10 stepsDeepSWE, mid-trajectory stepAvg
SP.902 ± .000−.040 ± .000.431 ± .000
Logistic regression.934 ± .003.095 ± .091.515 ± .046
Bayes Fused (SP).908 ± .001.210 ± .034.559 ± .017
SBF.939 ± .002.310 ± .016.625 ± .008

To obtain state labels on BrowseComp-Plus, we stop the agent after 10 steps and force it to give a final answer. On DeepSWE, we restore the repository after a random sample of the commands that modified the code (up to 8 per episode) and re-run the hidden tests.

SBF has the best average in both setups, with the largest margin for the intermediate state (.625 vs. .559 for the next-best estimator), and the highest PRR on DeepSWE in both. Prediction is harder very early in a trajectory: on WebArena, all methods stay below .15 after 2 to 4 steps.

We will add these experiments to the camera-ready version of the paper.

AuthorsOpenReview ↗

Response to Reviewer MB2N (3/4)

Q6. In addition, the best variant differs by domain, so the best of several variants is compared against each baseline. Does a single prespecified variant used across all domains still outperform the best baseline?

Yes, SBF is the best-performing method overall, outperforming the strongest baseline on average across all three domains.

It uses a single specification in every domain: no variant is selected per benchmark, and the same evidence model and hyperparameters are used throughout. Aggregated over all 15 benchmarks, SBF outperforms the best baseline overall and in each domain:

MethodRetrievalCodingInteractiveOverall (15 datasets)
HTC.502 ± .009.173 ± .025.443 ± .011.369 ± .010
LR.498 ± .009.277 ± .010.496 ± .009.424 ± .005
Bayes Fused (SP).355 ± .006.338 ± .007.405 ± .005.369 ± .004
Bayes Fused (Verb.).555 ± .004.231 ± .010.382 ± .008.378 ± .005
SBF.515 ± .005.306 ± .011.526 ± .006.450 ± .004

Bayes Fused (SP) remains stronger on coding, while Bayes Fused (Verb.) performs best on retrieval tasks. As expected, a single specification involves some trade-offs compared with the best variant in each domain. However, no baseline outperforms SBF in any domain: SBF is either the best- or second-best-performing method, with only other Bayes Fused variants outperforming it. Overall, SBF leads the best baseline by .026. Thus, a prespecified configuration, used unchanged across all domains, still outperforms the best baseline, and the reported gains do not depend on selecting a variant.


Q7. The method outputs success probabilities intended for decision making, but calibration is not evaluated...How well calibrated are the predicted success probabilities, for example in terms of ECE or Brier score?

We have added a calibration evaluation of the held-out predictions, where SBF + Platt scaling achieves the lowest average ECE.

Our conclusions concern outcome ranking rather than calibrated probabilities: the paper states that SBF's probabilities are not guaranteed to be calibrated (lines 263–264) and that PRR measures ranking quality, not calibration (lines 298–300), and the conclusion claims only the strongest average outcome rankings (line 480). Following your suggestion, we added the calibration evaluation below as a complementary analysis.

We report the expected calibration error (ECE, 10 bins) of the held-out predictions, averaged across seeds and configurations. Raw SP scores are min-max normalized to [0, 1]. SBF + Platt applies Platt scaling fitted on out-of-fold SBF predictions within the training folds only. For DeepSWE, ECE is computed against the cohort-median binary labels.

MethodWebArenaOSWorldALFWorldBrowseComp+DeepSWEAverage
SP.467 ± .000.059 ± .000.262 ± .000.507 ± .000.219 ± .000.303 ± .000
LR.054 ± .006.058 ± .009.094 ± .016.065 ± .002.075 ± .017.069 ± .005
Bayes Fused (SP).099 ± .002.064 ± .013.080 ± .009.077 ± .005.237 ± .012.111 ± .004
SBF.062 ± .005.112 ± .006.097 ± .005.060 ± .002.129 ± .020.092 ± .004
SBF + Platt.042 ± .005.072 ± .020.064 ± .004.022 ± .003.079 ± .016.056 ± .005

SBF is close to logistic regression on 3 of the 5 benchmarks (WebArena, ALFWorld, and BrowseComp+), and both learned estimators are far better calibrated than raw SP. If absolute probabilities are needed, a monotone recalibration fitted on the training folds (e.g., Platt scaling) can be applied while preserving the ranking within each fold, so PRR is essentially unaffected. SBF + Platt achieves the lowest average ECE and outperforms LR on 3 benchmarks, with ranking unchanged within each fold.

We will include these results in the camera-ready version of the paper.


Q8. Evaluations matching the probabilistic framing are missing, and the comparison favors the proposed method.

Our evaluation covers a broad range of tasks, models, and harnesses, without any tuning in our favor. We clearly report all results, including the cases where our method is not the best (e.g., cases where tools do not provide much signal).

All methods, including the baselines, are evaluated on the same trajectories with the same five folds and three split seeds, and no variant or hyperparameter is selected per benchmark. The evaluation covers 15 environments and 80 model-harness configurations, which leaves no room to fit the method to a particular setting. Tables 7-9 report all Bayes Fused variants and SBF for every dataset. For example, on CRAG, tool feedback is nearly uninformative and does not help to improve the performance. Finally, a single prespecified variant used unchanged across all domains, SBF, still outperforms the best baseline overall (see the table in Q6).

Reviewer_pGjMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to predict whether an agent run will succeed from its recorded trajectory. It proposes Bayesian fusion of generation uncertainty and tool feedback, with a structured version that treats trajectory length, history, and the final step separately. A theoretical analysis explains how correlated steps affect accumulated evidence and identifies a limit of the proposed history correction. Experiments across 15 public benchmark environments show gains in average outcome-ranking performance, with different variants leading on retrieval, coding, and interactive tasks.

Critical strengths
  1. The approach is well motivated. Tool checks and generation confidence provide different signals. Learning their relation to final success, and treating history and the final step separately, is a useful improvement.

  2. The evaluation is broad. It covers retrieval, coding, and interactive tasks, with both in-domain and transfer tests. This gives useful evidence about where the method applies.

Critical weaknesses
  1. The value of SBF's added structure is unclear. HTC already predicts trajectory success, so outcome scoring itself is not new. The paper adds specific evidence models, but its ablations test Bayes Fused rather than SBF's length term, history rescaling, and final-step joint model. It is therefore hard to tell which added choices drive the gains. The theory explains a limit of the history correction; it does not establish the benefit of these components. [Lines 116–117, 204–227, 250–264, 1751–1769; Tables 19–21.]

  2. A key training-label definition is missing. The method uses binary success/failure labels, but DeepSWE has no fully successful runs and is evaluated using the fraction of passed tests. How this score is used to fit the two-class models is unclear. This matters: from the rounded Table 8 values, DeepSWE contributes about 85% of the average gain of Fused (SP) over raw SP across the five coding datasets. I cannot fully assess this result without the label definition. [Lines 135–141, 165–170, 1101–1104, 1242–1245; Table 8, lines 1379–1397.]

Questions for authors
  1. How are the training labels defined for DeepSWE and OSWorld? The method uses binary labels, but these benchmarks allow partial credit, and DeepSWE has no fully successful runs. How are these scores used to fit the success/failure models? Please provide a table with the label rule and class counts for each configuration and fold, separately for training and testing. If soft labels or another objective are used, report that rule and the label distribution instead. Please distinguish these fitting labels from the quality scores used for PRR. [Lines 135–141, 165–170, 1090–1104, 1242–1245.]

  2. Which added components explain SBF's gains on interactive tasks? Do the length term, history correction, and final-step joint model each help? Please test this by removing the length term, disabling history rescaling and centering, and replacing the final-step joint model with separate evidence contributions, one change at a time. Compare each variant with full SBF using the same input signals and data splits, refit each variant, and report per-dataset [email protected] means and standard deviations over the same three split seeds. [Lines 204–227, 1101–1107, 1751–1769; Tables 9 and 19–21.]

4 comments

Thank you very much for your time in reviewing our paper and your insightful comments.

Q1. The value of SBF's added structure is unclear…Which added components explain SBF's gains on interactive tasks? Do the length term, history correction, and final-step joint model each help?

SBF's gains come mainly from the final-step model and history normalization, and removing any single component lowers the average [email protected].

We do not claim that outcome scoring itself is novel. Rather, as stated in the abstract and contributions, we propose integrating generation-level uncertainty with tool outcomes, and demonstrate that this approach outperforms HTC and the other baselines on average in all three task families. To clarify the contribution of each SBF component, we additionally conducted an ablation study on interactive (τ³-retail, WebArena, OSWorld, and ALFWorld) and retrieval (CRAG) tasks, following the same input signals, data splits, and evaluation procedure as in the paper:

VariantWebArenaOSWorldALFWorldCRAGτ³-retailAverage
Full SBF.545 ± .007.400 ± .010.825 ± .010.391 ± .015.278 ± .019.488 ± .006
Without trajectory length.553 ± .005.402 ± .005.836 ± .006.366 ± .014.273 ± .021.486 ± .005
Without critic history.542 ± .003.388 ± .008.835 ± .011.388 ± .015.259 ± .018.482 ± .005
Without generation history.542 ± .003.400 ± .018.820 ± .007.368 ± .014.221 ± .012.470 ± .005
Without final step.370 ± .006.382 ± .001.349 ± .022.318 ± .009.257 ± .020.335 ± .006
Without history normalization.443 ± .010.408 ± .007.647 ± .041.292 ± .011.271 ± .021.412 ± .010

Removing any component reduces the average [email protected]. The largest drops are observed when removing the final-step model (−.153) or history normalization (−.076). This is expected, since the final step contains the agent's solution, while history normalization prevents evidence from accumulating excessively across correlated steps. The length term contributes less on average, but it helps on CRAG (.391 vs. .366).

We will include this ablation study in the camera-ready version of the paper.

Q2. A key training-label definition is missing…How are the training labels defined for DeepSWE and OSWorld?

Partial-credit scores are used only for evaluation of the PRR metric. For training and for the stratified folds we use a binary outcome obtained with a simple binarization rule that requires no manually tuned thresholds.

Specifically, for OSWorld we treat every episode with a score below 1 as incorrect, which gives 154 successes and 121 failures out of 275 episodes. Only 9 of the 275 episodes received a partial score. They are treated as failures for training and evaluated with their original scores at test time.

For DeepSWE, outcomes are graded by the fraction of tests passed, so the binarization has to come from the score distribution rather than from a predefined success criterion. A fixed threshold would produce highly imbalanced classes. We therefore use the median as the threshold. For fold stratification, we compute the median over the full cohort; this affects only fold composition, not the labels any estimator is fitted on. During training, we recompute the median using only the training portion of each split, ensuring balanced classes by construction.

The class counts per split are:

ConfigurationLabel ruleEpisodesTrain (success / failure)Test (success / failure)
OSWorld, Qwen3.6score = 1275123–124 / 96–9730–31 / 24–25
DeepSWE, GPT-OSS-20Bpartial > training median9939–40 / 409–11 / 9–10
DeepSWE, GPT-OSS-120Bpartial > training median8733–35 / 35–366–11 / 7–11
DeepSWE, Qwen3.6partial > training median6525–27 / 25–275–8 / 5–7

Thus, the training classes remain balanced across all folds and seeds. The threshold is not tuned: it is simply the median of the training data, recomputed independently for each split. For transparency, the test columns show how this same training median applies to the held-out fold. Importantly, PRR is always computed using the raw continuous test-pass fractions, not the binarized labels. This is one advantage of the metric: it accepts a continuous outcome measure. The labels used for fitting never depend on held-out outcomes. No soft labels or other objectives are used.

We will make the distinction between the fitting labels and the quality scores used for PRR clearer in the camera-ready version of the paper.

Thank you for the additional ablations and for clarifying the training labels. The label explanation answers my question, and the results support history normalization.

Could you clarify what "without final step" removes? My original request was to keep the same final-step generation and tool-feedback signals and model their evidence separately. If the ablation removes all final-step evidence, the drop in PRR shows that these signals are useful, but the benefit of joint modeling is still unclear. In that case, could you add the comparison with separate models, leaving the other SBF components unchanged? Please refit each variant on the same splits and report [email protected] means and standard deviations for each dataset over the same three split seeds.

Thank you for the follow-up and for confirming that the label clarification answers your question.

"Without final step" removes all final-step evidence. Modeling the same final-step signals separately performs on par with the joint model on every dataset.

The new variant uses the same final-step projection rNr_N and tool-feedback indicator zNz_N, but replaces the joint model with two separate log-likelihood ratios (Gaussian for rNr_N, Bernoulli for zNz_N). All other components are unchanged, and each variant is refit on the same splits and the same three seeds.

VariantWebArenaOSWorldALFWorldCRAGτ³-retailAverage
Full SBF (joint final step).545 ± .007.400 ± .010.825 ± .010.391 ± .015.278 ± .019.488 ± .006
Separate final-step evidence.545 ± .007.400 ± .010.834 ± .011.398 ± .014.278 ± .019.491 ± .006
Without final step.370 ± .006.382 ± .001.349 ± .022.318 ± .009.257 ± .020.335 ± .006

This isolates the effect of joint modeling: SBF is robust to the form of the final-step model, and the gain (.153 on average) comes from the final-step evidence itself.

The reason is structural: the joint model differs from the separate one only through the within-class correlation ρ\rho between the two signals, and equals it when ρ=0\rho=0. On WebArena and OSWorld the final step is the agent's answer rather than a tool call, so the indicator is constant and the two forms are equal. Where the indicator varies, as on ALFWorld, they differ by at most within one standard deviation.

The joint form adds one parameter and is designed for settings where the two final-step signals are correlated, a concern similar to the one behind history normalization. Together with the first ablation, these results show what drives SBF's gains: the final-step evidence and history normalization.

We will report both variants in the camera-ready version. We hope this addresses your remaining concern.

Thank you for running the requested comparison. The results show the value of final-step evidence, with no clear advantage from joint modeling over separate modeling in the reported settings. Together with the earlier ablations and the clarification of the training labels, this addresses my main concerns. Please include both variants in the revised paper and update the contribution discussion to reflect these findings. I have raised my rating to 3.

Reviewer_33AWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines tool feedback and generation uncertainty to rank agent trajectories by likely outcome. Bayes Fused adds class-conditional likelihood ratios; SBF also models trajectory length, history, and final-step evidence. Evaluation covers 15 environments and 80 model–harness configurations, including transfer and adaptation with limited target labels.

Critical strengths
  1. A practical estimator. The method uses recorded trajectories without retraining the agent or generating new rollouts. The tool-only version is useful when token probabilities are unavailable (Section 3).

  2. Broad comparisons. The evaluation includes retrieval, coding, and interactive tasks, with simple statistics, raw uncertainty scores, and learned predictors as baselines. The per-dataset tables make clear that fusion does not always help (Tables 7–9).

  3. Informative transfer results. The experiments distinguish model/harness shifts from larger domain shifts. The latter expose a real limitation: unlabeled adaptation alone is insufficient for ALFWorld–DeepSWE transfer (Section 5.2).

Critical weaknesses
  1. The training labels are unclear for partial-credit tasks. The estimator and stratified folds require binary outcomes, but Appendix B.5 says no DeepSWE task is fully solved and uses test-pass fractions instead. The conversion to training classes is unspecified, as it is for OSWorld partial credit. This matters particularly for DeepSWE, which accounts for most of the coding-average gain over raw SP in Table 8.

  2. The error bars do not establish the precision of the reported gains. They measure three fold splits of the same trajectories, not task-sampling uncertainty. Appendix B.3 also propagates deviations as though configurations were independent, although configurations reuse benchmarks. Paired task-level intervals are needed to judge margins such as SBF versus logistic regression on interactive tasks.

  3. The evidence supports ranking, not calibrated success probabilities. The paper correctly acknowledges this distinction, and Proposition 1 explicitly limits the history correction. However, the success-probability output has no empirical calibration check. Its use should remain limited to ranking unless that additional interpretation is validated (Sections 3 and 4).

Questions for authors
  1. What binary labels and class counts are used for DeepSWE and OSWorld? If a threshold is used, how is it selected without held-out outcomes?

  2. Do paired task-level intervals support the main gains? For transfer on a shared benchmark, are source and target folds aligned by task identity?

  3. Are calibration results available for the held-out predictions? If not, please keep the conclusions explicitly about outcome ranking rather than reliable absolute success probabilities.

3 comments

Thank you very much for your time in reviewing our paper and your insightful comments.

Q1. The training labels are unclear for partial-credit tasks…What binary labels and class counts are used for DeepSWE and OSWorld?

Partial-credit scores are used only for evaluation of the PRR metric. For training and for the stratified folds we use a binary outcome obtained with a simple binarization rule that requires no manually tuned thresholds.

For OSWorld, every episode with a score below 1 is a failure (154 successes and 121 failures out of 275; only 9 episodes have a partial score).

For DeepSWE, no task is fully solved, so we binarize the test-pass fraction at the median: of the whole cohort for fold stratification, and of the training folds of each split for training. This gives balanced training classes (e.g., GPT-OSS-20B, 99 trajectories: 39–40 per class in training, 9-11 in testing).

The held-out fold is therefore never used to set the threshold for training. At test time, PRR is computed on the raw continuous test-pass fraction. We will make the distinction between the fitting labels and the quality scores used for PRR clearer in the camera-ready version.


Q2. The error bars do not establish the precision of the reported gains…Do paired task-level intervals support the main gains?

Paired task-level bootstrap intervals support the main gains: five of the six exclude zero, and the sixth is positive with bootstrap probability .97.

Tasks are resampled jointly across all configurations of each benchmark, and both methods are compared on the same out-of-fold predictions (ALFWorld and DeepSWE: 20 split seeds, B=5000; WebArena and OSWorld: the 3 seeds of the main tables, B=2000):

BenchmarkComparisonΔ PRR95% CI
ALFWorldSBF vs. LR+.078[+.012, +.153]
ALFWorldBayes Fused (SP) vs. raw SP+.188[+.043, +.353]
WebArenaSBF vs. raw SP+.116[+.052, +.177]
OSWorldSBF vs. raw SP+.175[+.035, +.312]
DeepSWEBayes Fused (SP) vs. LR+.104[−.003, +.230]
DeepSWEBayes Fused (SP) vs. raw SP+.190[+.022, +.357]

Q3. Appendix B.3 also propagates deviations as though configurations were independent.

Across benchmarks, the task sets are essentially independent, so propagating SDs as independent is standard practice.

Within a benchmark, we treat configurations as independent only to keep the SD pipeline uniform. Even under perfect correlation, the SDs stay of the same order of magnitude and below the margins. The paired bootstrap in Q2, which resamples tasks jointly across configurations, supports the same gains.


Q4. For transfer on a shared benchmark, are source and target folds aligned by task identity?

Yes, folds are assigned per task and shared by all agents on a benchmark, so no task-level information leaks between training and testing.

Folds are stratified by the joint vector of the configurations' binary outcomes on each task. In transfer, the estimator is fitted on the source agent's training-fold episodes and evaluated on the target agent's held-out fold, which contains exactly the tasks excluded from source training.


Q5. The evidence supports ranking, not calibrated success probabilities…Are calibration results available for the held-out predictions?

All reported results and conclusions already concern only ranking-based failure detection; we additionally report ECE here, where SBF + Platt scaling achieves the lowest average ECE.

We do not claim that our estimators produce calibrated success probabilities. Rather, the paper consistently limits its conclusions to outcome ranking: it states that SBF's predicted probabilities are not guaranteed to be calibrated (lines 263–264) and that our metric evaluates ranking quality rather than calibration (lines 298–300), and the conclusion claims only the strongest average outcome ranking (line 480).

For completeness, we report the expected calibration error (ECE, 10 bins) of the held-out predictions. SBF + Platt applies Platt scaling fitted on the training folds only.

MethodWebArenaOSWorldALFWorldBrowseComp+DeepSWEAvg
SP (min-max norm.).467 ± .000.059 ± .000.262 ± .000.507 ± .000.219 ± .000.303 ± .000
LR.054 ± .006.058 ± .009.094 ± .016.065 ± .002.075 ± .017.069 ± .005
Bayes Fused (SP).099 ± .002.064 ± .013.080 ± .009.077 ± .005.237 ± .012.111 ± .004
SBF.062 ± .005.112 ± .006.097 ± .005.060 ± .002.129 ± .020.092 ± .004
SBF + Platt.042 ± .005.072 ± .020.064 ± .004.022 ± .003.079 ± .016.056 ± .005

SBF is close to LR on 3 of the 5 benchmarks, and both learned estimators are better calibrated than raw SP. SBF + Platt outperforms LR on three benchmarks, with ranking unchanged within each fold.

We will include these results in the camera-ready version.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. As the discussion period is approaching its end, we would greatly appreciate it if you could consider our responses. Our rebuttal addresses the concerns raised in your review. Should any questions remain, we would be happy to provide further clarification and engage in further discussion.

Dear Reviewer 33AW,

As the last day of the discussion period is approaching its end, we would greatly appreciate it if you could consider our responses. Our rebuttal addresses the concerns raised in your review, and we would be happy to answer any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions by Authors

Dear Reviewers and Area Chair,

Thank you for taking the time to review and evaluate our work. We greatly appreciate your efforts and valuable feedback. Below is a summary of the main changes we have made to the revised version of the paper (highlighted in blue) to address the reviewers’ feedback:

  1. Added an ablation study of SBF components, including separate modeling of the final-step evidence, in Section 5.3 and Table 22 in Appendix E (requested by pGjM).
  2. Added an analysis of history normalization and a comparison with the length correction suggested by Proposition 1 in Sections 3.4 and 5.3 and Tables 23-24 in Appendix E (requested by MB2N and pGjM).
  3. Added a calibration evaluation (ECE) in Section 5.3 and Table 27 in Appendix F (requested by 33AW and MB2N).
  4. Added failure prediction at intermediate steps of the trajectory in Section 5.3 and Tables 28-29 in Appendix F (requested by MB2N).
  5. Added a paired task-level bootstrap analysis, an LLM-as-judge baseline, and an evaluation including terminated trajectories in Section 5 and in Tables 30-32 in Appendix F (requested by 33AW and MB2N).
  6. Clarified the training labels for partial-credit tasks, fold alignment in transfer, and SD propagation in Section 4 and Appendix B.3 (requested by 33AW and pGjM).
  7. Clarified design of the critics and the accumulation rule for each benchmark in Section 4 and Appendices A.1 and B.4-B.6, and expanded the Limitations section (requested by MB2N).

All of these changes are already included in the revised version of the paper. We thank the reviewers again for their constructive feedback, which has helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.