acceptodds
Withdrawn from ICLR 2027

TURF: Outcome-Grounded Uncertainty Quantification for LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM agents execute multi-step workflows with consequences beyond their final responses, making uncertainty quantification important for identifying tasks that warrant verification or human oversight. We introduce TURF, a task-training-free method that ranks task-level failure risk by combining smoothed dispersion of observable outcomes with frozen-NLI spectral diversity of terminal responses through Gaussianized-rank fusion, without fitting correctness-supervised predictors or fusion weights. TURF-Fixed uses a prescribed trajectory budget, while TURF-Adaptive selects a prefix using Bayesian evidence of outcome and semantic recurrence relative to the first execution. We evaluate both variants against 15 baselines on six benchmarks, three backbones, and three seeds, across supported trajectory caps from 2 to 16. Equally averaging across datasets, backbones, seeds, and budgets of 4, 6, 8, 12, and 16 trajectories, TURF-Fixed improves AUROC/AUPR by 5.19/2.99 percentage points over SGD, the strongest baseline with complete results on this grid. TURF-Adaptive retains gains of 4.68/2.60 points while using 20.90% fewer trajectories for scoring than TURF-Fixed, with corresponding quality losses of 0.52/0.39 points. The results support outcome-grounded, two-view uncertainty estimation across multiple sampling budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, uncertainty estimation, failure prediction, task-training-free learning, stochastic rollouts, adaptive computation, semantic uncertainty, risk ranking, multi-step reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7vBtOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

The paper proposes TURF, a method for ranking the failure risk of LLM agents from repeated executions. It combines variation in observable execution outcomes with semantic variation in terminal responses, using Gaussianized ranks to fuse the two signals. The method does not train a predictor on task correctness labels. Its adaptive version uses Bayesian evidence of agreement with the first execution to decide how many trajectories to inspect. The theoretical analysis establishes properties of the scoring components and a Bayesian guarantee for the stopping rule. Experiments cover six benchmarks, three language models, three random seeds, and 15 baselines. Averaged over the five main trajectory budgets, TURF-Fixed improves AUROC and AUPR over SGD by 5.19 and 2.99 percentage points. TURF-Adaptive uses 20.90% fewer trajectories for scoring, with respective losses of 0.52 and 0.39 points relative to TURF-Fixed. The evaluation measures failure ranking for previously recorded task results, with correctness labels held fixed across budgets and policies.

Critical strengths
  1. Direct evidence for the value of execution outcomes. The paper provides a convincing test of whether execution outcomes add useful information beyond final responses. The comparison in Figure 5 and Appendix F.5 keeps the tasks, trajectory prefixes, count statistic, and semantic scores the same, and removes the additional WebShop severity signal from both versions. Using execution outcomes improves AUROC and AUPR by 1.59 and 1.24 percentage points, respectively, averaged over the five main budgets. The largest gains occur on the database and shopping tasks. This experiment directly supports the paper's central motivation and is an important reason for my positive assessment.

  2. Careful comparisons with the baselines. I also appreciate the effort to check whether the results depend on how the baselines are adapted. Appendix E evaluates 13 baseline families both on final responses and by aggregating uncertainty across steps, using the same task set for every method. In the averages reported in Table 2, TURF-Fixed still exceeds SGD-Final by 5.58 percentage points in AUROC and 3.42 points in AUPR. This comparison addresses two plausible concerns: that the main results favor a particular baseline adaptation, or that methods are being compared on different subsets of tasks. It makes the empirical evidence considerably more persuasive.

Critical weaknesses
  1. The benefit of several components of the full method remains modest. In the core scoring ablation at B=16B=16, adding the semantic score to the count score improves AUROC and AUPR by only 0.16 and 0.53 percentage points. The AUROC confidence interval includes zero, and the AUPR result does not survive the reported Holm correction. Stouffer fusion also performs very close to simply averaging the two ranks (Appendix F.2). The count score is therefore already close to the complete scorer. In addition, the adaptive scoring ablations retain the stopping decisions made by the full method, so they do not establish the value of semantic computation across the entire procedure. Given the additional NLI computation, the practical benefit of the complete design needs a clearer demonstration.

  2. The WebShop comparison does not fully separate the contribution of the general scorer from an additional severity signal. This signal uses purchase status, request coverage, and price. Removing it lowers TURF-Fixed's AUROC from 0.7020 to 0.6454 at B=16B=16 (Appendix H.3). The paper clearly discloses this adjustment, and the controlled experiment in Appendix F.5 excludes it. However, I could not find a comparison in which a strong semantic baseline receives the same severity score and the same rule for combining it with its uncertainty score. This leaves an important question about how much of the main WebShop advantage comes from the general TURF scorer and how much comes from the additional rules.

Questions for authors
  1. Could you compare the full method with a version using only execution outcomes for both scoring and Bayesian stopping, at the same average trajectory budget? The current scoring ablation retains the full method's stopping decisions. Reporting NLI overhead alongside this comparison would also help assess whether the semantic component is worthwhile.

  2. How does a strong semantic baseline perform on WebShop when given the same severity score and combination rule as TURF? Comparing them on the same tasks and trajectory budgets would help separate the contribution of the general scorer from the additional WebShop rules.

  3. Keeping the archived correctness labels fixed, how do the stopping decisions and ranking results change if semantic availability is determined only from the trajectories inspected so far? Please recompute the reference ranks as well. This would clarify the effect of using the full trajectory pool for availability checks, as described in Sections A.4 and B.3.

2 comments

Thank you for the careful assessment. We agree that semantics should be evaluated jointly with stopping and computational overhead, and that WebShop requires severity-matched controls.

All AUROC/AUPR differences below are absolute changes on the percentage scale, not relative percentage improvements.

Q1. Outcome-only scoring and stopping at matched trajectory cost

We compared full TURF-Adaptive with outcome-only scoring and independently recomputed outcome-only Bayesian stopping across six datasets, three backbones, and three seeds (54 cells). Entries below are full minus outcome-only; AUPR denotes average precision.

BudgetMatched mean trajectoriesΔAUROC (absolute)ΔAUPR (absolute)
44.00+0.03%+0.08%
86.83+0.85%+0.91%
1610.97+0.90%+1.03%

At B=16, paired 95% intervals are [−0.01%, 1.79%]/[0.36%, 1.76%], supporting a modest AUPR benefit while the AUROC difference remains inconclusive. At B=8, the intervals are [0.00%, 1.66%]/[0.35%, 1.54%]. B=4 tests scoring alone because early stopping begins at prefix five. Intervals use 5,000 paired query resamples conditional on frozen scores/ranks, with equal-cell macro averaging.

Contribution of semantics to stopping. Holding the fused scoring rule fixed, we separately compared two-view and outcome-only stopping using the same retrospective trajectory-matching procedure. At B=16, using semantics in stopping improves AUROC/AUPR by +0.91%/+0.92%, with paired 95% intervals of [0.38%, 1.43%]/[0.55%, 1.25%]. This isolates the contribution of semantics to sampling decisions rather than to the scoring rule.

Computational overhead. A single directed NLI comparison took approximately 12 ms on an A100 GPU when processed individually. This is component timing, not complete per-task runtime. Matching trajectory counts does not equalize total computation, and we do not claim an overall deployment-cost advantage over the outcome-only version.

 

Q2. Identical WebShop severity adjustment

We applied the same outer WebShop severity score and combination rule to TURF and Semantic Graph Density (SGD), a strong semantic baseline, on matched tasks across all nine backbone–seed cells. Entries below are TURF-Fixed minus severity-matched SGD.

BudgetΔAUROC (absolute)ΔAUPR (absolute)
4+2.67%+0.83%
8+2.40%+0.68%
16+1.39%+0.51%

B=4 provides descriptive estimates only. At B=8, the secondary, unadjusted 95% intervals are [−0.55%, 5.35%] for AUROC and [0.01%, 1.30%] for AUPR. At B=16, both intervals include zero: [−1.30%, 4.01%]/[−0.11%, 1.10%]. All point estimates favor TURF after matching the WebShop severity adjustment. The intervals accommodate substantial positive effects but also a small negative effect.

 

Q3. Prefix-local semantic availability and recomputed reference ranks

The historical availability check used all 16 trajectories, allowing an invalid later response to disable semantics at an earlier prefix. We reran the evaluation using only trajectories inspected so far to determine availability, retaining the original validity criteria and outcome-only fallback.

Across all 54 cells and B={2,3,4,6,8,12,16}, we kept correctness labels, outcomes, trajectory order, and stopping parameters fixed while recomputing stopping decisions, scores, and reference ranks, including WebShop’s outer ranks. Entries below are prefix-local minus historical results.

Adaptive budgetΔAUROC (absolute) [95% interval]ΔAUPR (absolute) [95% interval]Changed stopsExtra trajectories
8+0.0101% [−0.0067%, 0.0250%]+0.0346% [−0.0191%, 0.0908%]6/22,23015
16+0.0062% [−0.0120%, 0.0193%]+0.0155% [−0.0149%, 0.0340%]9/22,23034

All changed stops are later; counts refer to task–cell records. Fixed B=16 is exactly unchanged. The correction produced very small observed changes in this replay, with all displayed intervals crossing zero.

Thank you for the detailed rebuttal and additional experiments. While some concerns have been addressed, I remain unconvinced by the methodological novelty and the benefits of the key components.

I am also concerned about the extensive rewriting of the manuscript during rebuttal, which I believe goes beyond the scope of reasonable revisions at this stage.

Reviewer_e53DOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents TURF, a training-free method for estimating task-level failure risk in LLM agents. TURF executes the same task multiple times and measures disagreement in both environment outcomes and final textual responses. These two uncertainty signals are combined into a single risk score. The adaptive version can stop collecting trajectories early when repeated executions appear sufficiently consistent.

Critical strengths
  • The paper studies an important practical problem: detecting unreliable agent executions without requiring ground-truth answers.

  • It incorporates environment outcomes, which can provide more meaningful signals than relying only on the agent’s generated text.

  • The evaluation is relatively broad, covering six benchmarks, three backbone models, multiple sampling budgets, and numerous baselines.

  • The paper provides extensive ablations and includes an adaptive variant that reduces the number of trajectories compared with always using the full rollout budget.

Critical weaknesses
  • The novelty is limited, the method is basically doing multi sample from a question and evaluating risk based on the consistency of answer, it is very similar to the voting method in the llm. The author should compared TURF and self-consistency, majority voting to see which one is better.

  • Different from voting in LLM reasoning, such multiple sampling method in long-horizon tasks is expensive and the author acknowledge consistency may not equal to correct. So such high cost finally get a incorrect answer, the early stopping method still doesn't solve such limitations.

  • In experiments, this paper first generate the rollout pool. Paper didn't report the reductions in wall-clock time, token usage, tool calls, or total deployment cost. without these, this paper does not demonstrate actual online cost savings.

  • TURF can not give independent risk score. It relies on empirical rank mappings derived from a reference cohort at the same operating point. This introduces an additional deployment requirement and leaves its robustness to mismatched or distribution-shifted reference cohorts unclear.

Questions for authors

See weaknesses.

4 comments

Thank you. We agree that repeated sampling itself is not novel. TURF's contribution is outcome-grounded failure-risk estimation: finite-sample count-posterior risk and graded semantic dispersion are fused without task-specific training.

W1-Part-A: Novelty beyond repeated sampling and answer consistency

We agree that repeated sampling and answer consistency are established ideas. TURF’s contribution is a specific outcome-grounded framework for failure-risk ranking, rather than a new sampling or majority-voting procedure.

First, TURF represents observable execution consequences—not only agreement between final-answer strings. Depending on the task, these include normalized answers, database states, purchase configurations, and termination outcomes. This distinction matters for agents: identical terminal messages can accompany different execution outcomes, while different execution paths can produce the same outcome.

Second, TURF does not reduce the sampled outcomes to the winning vote share. Its count-posterior component incorporates the distribution of outcome counts, reserved unseen-outcome mass, and the mean and variance of entropy under the specified smoother. This produces a finite-sample dispersion statistic, not a posterior probability of correctness. TURF then combines this discrete execution-outcome view with graded terminal-semantic dispersion through unlabeled reference-rank mappings. We present this particular construction and its integration with two-view stopping as our methodological contribution; we do not claim to invent Dirichlet smoothing, semantic entropy, or Stouffer’s transformation.

We also directly test whether this construction adds value beyond voting. Across six datasets, three backbones, and three seeds, TURF-Fixed improves over final-answer vote confidence by 5.03 AUROC / 5.84 AUPR percentage points at B=4, and 4.16 / 4.62 points at B=8. At B=8, the paired 95% intervals are [2.99, 5.36] and [3.66, 5.35], respectively. Additional comparisons against outcome dominance, entropy, top-two gap, and anchor agreement use the same execution-outcome buckets, helping distinguish the estimator’s benefit from the benefit of changing the representation alone.

 

W1-Part-B: Comparison with Voting and self-consistency, and novelty.

We added final-answer vote confidence (SC-Final: one minus the largest answer count divided by B). We also tested outcome dominance, entropy, top-two gap, and anchor agreement on the same execution-outcome buckets. TURF-Fixed and controls share tasks, the first B trajectories, three backbones, and three seeds. The table gives TURF-minus-control AUROC/AUPR gains in percentage points.

ControlB=4B=8B=16
Final-answer voting+5.03/+5.84+4.16/+4.62+2.97/+2.93
Outcome entropy+2.76/+4.44+1.52/+2.91+0.93/+2.01

Against SC-Final at B=8, paired 95% intervals are [2.99,5.36]/[3.66,5.35]. All four outcome controls have positive macro point differences across B={2,3,4,6,8,12,16}.

In addition to explicit voting controls, we compared TURF with broader final-response UQ methods, including semantic-consistency approaches. Figures 11–12 in the submitted appendix compare TURF with 13 Final-adaptation baselines across all 18 dataset–backbone combinations. These adaptations apply existing LLM uncertainty methods—including SE, KLE, and SGD—to the final responses of sampled trajectories, without aggregating intermediate-step uncertainty. Averaged across six datasets, three backbones, three seeds, and budgets B∈{2,3,4,6,8,12}B\in\{2,3,4,6,8,12\}, TURF-Fixed improves AUROC/AUPR by 5.58/3.42 percentage points, and TURF-Adaptive by 5.31/3.24 points, over SGD-Final—the strongest Final baseline on both budget-averaged metrics.

 

W2: Agreement is not correctness.

We agree that repeated execution is costly and consistency does not imply correctness. TURF estimates failure risk; it does not repair the agent’s result. When users lack the expertise or resources to verify every result, a risk ranking can help prioritize verification or human review. An incorrect result flagged as high risk can therefore be useful, even though uncertainty estimation does not correct it. TURF provides a relative risk score, not a calibrated failure probability.

Consistently wrong outputs are a shared limitation of consistency-based uncertainty estimation. Nevertheless, disagreement remains informative: at B=16, our completed-rollout diagnostics show higher failure rates for nonunanimous than unanimous outcomes. TURF uses outcome and semantic dispersion to rank failure risk, rather than treating agreement as proof of correctness. Early stopping addresses sampling cost, not correctness: it reduces the sampling work used for risk estimation but does not eliminate systematic errors or certify successful execution. We therefore report predictive quality alongside measured resource use to characterize this tradeoff.

W3: Measured online savings and the quality–cost tradeoff.

We agree that truncating a pre-generated pool is insufficient evidence of online savings. We now report all 300 DBBench tasks and all 200 WebShop tasks per backbone (Qwen and Gemma), including the original ten-task pilots, at B=16 and seed 101. These are actual paired live Fixed/Adaptive executions, not savings simulated solely from cached prefixes. Both policies use the shared first execution's correctness as their prediction target.

All-task resource savings. Entries are percentage reductions [paired 95% CI], computed as 100 × [1 − sum(Adaptive)/sum(Fixed)], not averages of task or cohort percentages. Generation tokens include input and output. Each displayed resource uses all paired tasks in its row.

All-task mixed scopeTrajectoriesGeneration tokensTool callsNLI comparisons
DBBench / Qwen27.77 [24.42, 31.15]18.85 [15.01, 23.19]22.57 [18.94, 26.58]39.26 [34.65, 43.95]
DBBench / Gemma26.40 [22.94, 29.92]19.44 [15.68, 23.58]21.75 [18.21, 25.55]36.93 [32.18, 41.77]
WebShop / Qwen30.09 [25.72, 34.50]16.98 [13.42, 20.95]18.33 [14.63, 22.38]41.48 [35.49, 47.45]
WebShop / Gemma53.91 [50.25, 57.31]47.85 [41.98, 53.72]48.70 [43.18, 54.09]73.60 [68.70, 78.12]

All displayed resource-saving intervals are above zero. Thus, these all-task historical aggregates show approximately 17–48% fewer recorded generation tokens and 26–54% fewer sampled trajectories. This provides measured online-work evidence while preserving the heterogeneous acquisition histories.

All-task predictive quality. Metrics are recomputed globally from task-level scores, not averaged across cohort AUROCs/AUPRs. Differences are Adaptive − Fixed in percentage points; AUPR denotes average precision.

All-task mixed scopeFixed AUROC / AUPRAdaptive AUROC / AUPRΔAUROC, pp [95% CI]ΔAUPR, pp [95% CI]
DBBench / Qwen0.7834 / 0.77700.7667 / 0.7768-1.68 [-4.22, +0.75]-0.01 [-2.46, +2.82]
DBBench / Gemma0.7546 / 0.86160.7395 / 0.8617-1.51 [-4.06, +0.97]+0.01 [-1.38, +1.56]
WebShop / Qwen0.6764 / 0.89200.6920 / 0.8952+1.56 [-2.16, +5.42]+0.32 [-0.62, +1.33]
WebShop / Gemma0.7187 / 0.93730.7368 / 0.9342+1.81 [-3.86, +7.26]-0.30 [-2.00, +1.20]

DBBench AUROC point estimates are lower under Adaptive, whereas WebShop AUROC point estimates are higher; the AUPR differences are small in both directions.

Protocol-aligned comparison retained separately. It shows the same qualitative pattern of recorded savings and inconclusive quality differences.

Protocol-aligned scopeΔAUROC, pp [95% CI]ΔAUPR, pp [95% CI]Token savings, % [95% CI]Tool-call savings, % [95% CI]
DBBench / Qwen-1.91 [-4.57, +0.59]-0.16 [-2.66, +2.79]18.86 [15.04, 23.17]22.60 [18.92, 26.57]
DBBench / Gemma-1.64 [-4.28, +0.86]-0.13 [-1.65, +1.59]19.45 [15.63, 23.60]21.84 [18.25, 25.73]
WebShop / Qwen+1.79 [-2.15, +5.98]+0.41 [-0.54, +1.44]16.35 [12.73, 20.45]17.71 [13.93, 21.96]
WebShop / Gemma+1.55 [-4.39, +7.44]-0.42 [-2.21, +1.21]48.24 [42.55, 53.95]49.24 [44.01, 54.41]

Measured timing on explicitly scoped support. The table reports Adaptive reductions in recorded NLI time and instrumented logical policy-arm time. These are paired measurements, not estimates inferred from trajectory counts.

Measured timing scopeNLI-time reduction, % [95% CI]Instrumented arm-time reduction, % [95% CI]
DBBench / Qwen38.78 [32.96, 44.65]17.82 [13.65, 22.64]
DBBench / Gemma35.11 [29.28, 41.13]20.73 [16.11, 25.73]
WebShop / Qwen36.48 [30.79, 42.10]17.68 [14.09, 21.72]
WebShop / Gemma64.06 [59.71, 68.08]47.74 [42.68, 52.62]

These savings concern recorded work. Setup, reference-bank construction, and incompletely measured failed-attempt/recovery costs remain outside the accounting; the reported timings are not complete end-to-end deployment latency.

The all-task results—300 DBBench and 200 WebShop tasks per backbone—include the initial ten pilot tasks. The protocol-aligned comparisons exclude these pilots, leaving 290 and 190 tasks, respectively. Because the pilots used different acquisition settings, the all-task results are mixed-protocol descriptive aggregates; the protocol-aligned results provide a separate sensitivity check.

W4: reference-bank dependence and domain mismatch

We agree that TURF’s fused score depends on a reference cohort of unlabeled tasks. The cohort defines percentile mappings for the count and semantic statistics. Stouffer fusion then transforms these percentiles to a common normal-score scale and combines them symmetrically. Under the explicit Gaussian working model in our appendix, equally informative views and shared symmetric covariance yield a likelihood-ratio ranking based on their sum. This provides a conditional design rationale, but does not guarantee optimality or robustness when the reference cohort differs from the evaluation population.

To assess this dependence empirically, we added a bank-size and domain-mismatch analysis with evaluation tasks and correctness labels held fixed. Query identities are excluded from the reference banks. To isolate scoring performance from adaptive stopping, we evaluate TURF-Fixed at B=4,8,16B=4,8,16 and compare it with reference-free outcome-dominance voting using exactly the same first BB trajectories.

The primary mismatch experiment covers HotpotQA, StrategyQA, and GAIA, with three backbones and three seeds. We replace 0%, 25%, 50%, 75%, or 100% of each 100-task bank with donors from a prespecified alternative domain. We use the following fixed donor-domain mapping:

Evaluation domainSame-domain donorsShifted-domain donors
HotpotQAHotpotQAStrategyQA
StrategyQAStrategyQAGAIA
GAIAGAIAHotpotQA

Results average 200 bank draws; uncertainty uses 2,000 paired query resamples. The table summarizes matched, half-shifted, and fully shifted banks. Entries are TURF minus outcome voting in AUROC/AUPR percentage points; positive values favor TURF. Because voting does not use a reference bank, its score remains unchanged as the donor mixture varies.

BudgetShifted donorsVersus outcome voting: ΔAUROC / ΔAUPR
B=4B=40%+6.06 / +6.49
B=4B=450%+5.99 / +6.45
B=4B=4100%+5.87 / +5.42
B=8B=80%+3.19 / +3.95
B=8B=850%+3.08 / +3.82
B=8B=8100%+1.41 / +1.64
B=16B=160%+1.36 / +1.41
B=16B=1650%+1.40 / +1.48
B=16B=16100%−0.85 / −1.92

With matched or half-shifted banks, TURF improves both metrics over outcome voting at B=4B=4 and B=8B=8, with paired 95% intervals excluding zero. Even with fully shifted banks at B=4B=4, its gains remain +5.87 AUROC points [3.86, 8.07] and +5.42 AUPR points [2.78, 7.01]. However, complete domain mismatch substantially reduces the advantages at larger budgets. At B=16B=16, the point estimates favor voting, although both intervals include zero. Thus, these results support resilience to the tested partial shifts, particularly at small budgets.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors’ efforts to address my concerns.

However, some concerns remain unresolved.

  • The methodological is still limited, as the approach largely builds upon existing sampling and consistency-based uncertainty estimation techniques.

  • The method still struggles with consistently incorrect outcomes, which early stopping does not address.

  • TURF remains dependent on a reference cohort, and the new experiments show performance degradation under severe domain mismatch.

Importantly, I check the revision manuscript and it seems that the authors rewrite the whole main paper, in my mind, the revision in rebuttal mainly let author do some minor fix rather than rewrite the full paper, it means this paper is not ready for submission, so I think this paper should be submitted in next round and I maintain my clear rejection.

Reviewer_q1QXOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper proposes a task-training-free method for task-level UQ in LLM agents which combines observable outcome dispersion across multiple agent trajectories with NLI-based semantic diversity (similar to semantic entropy) of terminal responses through rank-based fusion. It also introduces an adaptive variant that uses Bayesian recurrence estimates. The authors evaluate their method through experiments across six benchmarks and three LLM backbones, claiming improvements over existing uncertainty estimation baselines, with reduced trajectory usage for the adaptive variant.

Critical strengths
  • The proposed method makes sense to some degree.
  • The proposed method shows improvements in UQ quality across extensive empirical validation.
Critical weaknesses
  • Very poor writing quality and readability. Sections 3 and 4, which explain the methodology and theory, are extremely difficult to follow. The text is filled with dense paragraphs, unnecessary caveats, and lengthy explanations of minor details, while the main arguments remain unclear. Even after spending hours reading these sections, I still struggle to understand the authors' core claims, key insights, and implications. This makes it very hard to properly evaluate the technical content of the paper. For context, I have read at least ~ 40 LLM/agent uncertainty quantification papers from last year and published a couple of relevant papers, so I believe this is not because of my lack of expertise in this field.
  • Concerns about experimental rigor and presentation. The main baseline comparison is largely presented through the aggregated results in Figure 2, while detailed per-dataset and per-backbone comparisons are mostly left to the appendix. Also, the main comparison involves baselines with different task coverage and sampling-cost assumptions, where I find it difficult to assess how consistent and statistically reliable the claimed improvements are from the main text alone. Reviewers should not be expected to go through the entire appendix to evaluate the central experimental claims.
  • Unclear benefits of the proposed components. Figure 4 suggests that the additional benefit of the semantic component is relatively small, while the proposed Stouffer fusion does not clearly outperform simple mean-rank fusion. This raises questions about whether the additional methodological complexity of your method is justified.
  • Missing citations to closely related work. The paper missed several important related works, reflecting its premature literature survey and loose contextualization. First, in the third paragraph of the Introduction, the authors emphasize limitations of existing uncertainty quantification frameworks that have already been discussed extensively in Oh et al., ACL 2026, without citing this work. Second, there is a method called Holistic Trajectory Calibration (Zhang et al., ICML 2026), which uses trajectory-level information to estimate agent success and failure. Despite its close relevance to the paper's objective, it is neither discussed in the related work section nor included in the experimental comparisons. Another relevant work, Agentic Uncertainty Quantification (Zhang et al., 2026), also studies uncertainty-aware agent control but is not cited. These omissions make it harder to assess the novelty of the problem statement and proposed approach as well as its relationship to existing methods.
  • (minor): the framework visualization (figure 1) is also hard to understand due to many unexplained symbols and an unintuitive flow diagram.

Reference

  1. Oh et al. ACL 2026, Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities
  2. Zhang et al. ICML 2026, Agentic Confidence Calibration
  3. Zhang et al. 2026, Agentic Uncertainty Quantification
Questions for authors

please see the above weaknesses section

4 comments

W1. Writing quality and readability; W5: the framework visualization.

Thank you for emphasizing this issue. We have substantially rewritten Sections 3 and 4 to make the intuition, construction, and implications easier to follow. Section 3 now explains the method through a running example and the three-stage workflow in the updated Figure 1: representing execution outcomes, estimating outcome and semantic dispersion, and combining their evidence. The adaptive stopping policy is explained separately, with a concrete stopping example. Section 4 summarizes the meaning of each theoretical result in short, plain-language paragraphs, while detailed statements, proofs, and implementation specifications are provided in the appendix. We also added a table of contents in the Appendix.

 

W2. Experimental rigor and presentation

Detailed comparisons are now prominent. We replaced the aggregated main-paper figure with per-dataset, per-backbone, and per-budget AUROC comparisons. The revised Figure 2 shows all 18 dataset–backbone combinations and all 15 baselines at their supported budgets. The corresponding AUPR comparisons appear on the first appendix page, followed by the macro AUROC/AUPR comparison on the second appendix page. Both are directly referenced from the main text and Figure 2’s caption.

Task coverage and matched-task comparisons. As different methods require different inputs, missing likelihoods, incomplete candidate records, or unusable generated outputs can prevent some methods from scoring tasks that others can still evaluate. These coverage differences reveal practical limitations of some methods under the available inputs, so we retain native-support results and report coverage separately.

However, we agree that performance on different task sets can conflate scoring quality with task difficulty. Our revised main-paper comparison of two TURF variants and 15 baselines therefore uses a fixed common cohort of 22,158 task–backbone–seed records, retaining 99.68% of the original population.

The main aggregate conclusions remain unchanged: Our results indicate that both TURF variants retain higher observed macro AUROC and AUPR than every supported baseline at each evaluated budget. Restricting TURF to the matched cohort changes its macro scores by at most 0.34 percentage points. Thus, the aggregate advantage persists on shared tasks.

Shared budgets and baseline aggregation. We evaluated all the methods over the same trajectory-budget grid, B={2,3,4,6,8,12,16}B=\{2,3,4,6,8,12,16\}. All 15 baselines have complete six-dataset comparisons through B=12B=12; We did not generate the results for Degree and UProp B=16B=16, as their generaton cost is high because of the requirement to sample multiple candidate responses at each step, and the empirical observations on based on budgets up to B=12B=12 are clear.

All baselines are evaluated at the same nominal trajectory budget BB, using the first BB rollouts from their respective input pools.

For single-step baselines such as PE, we evaluated two adaptations. Final computes uncertainty from the final-step response of each rollout, then averages the available scores across the first BB rollouts. StepAgg follows the two-level aggregation approach in UProp, Section 4.1: compute uncertainty at eligible decision steps, average these scores within each rollout, and then average the available rollout-level scores across the first BB rollouts.

Both adaptations therefore aggregate evidence across sampled executions rather than relying on a single rollout. This averaging has a Monte Carlo interpretation under the corresponding sampling and score-availability protocol. Matching trajectory budgets does not, however, equalize candidate-generation or scoring costs.

We distinguish this shared budget design from equal total computation. TURF-Adaptive can inspect fewer trajectories; Degree and UProp require four local candidates per step and retain candidate-enriched pools; semantic and likelihood scoring also incur method-specific costs. The curves therefore show trajectory usage, not equalized end-to-end computational cost.

Statistical reliability. Our revised figures include pointwise 95% paired task-bootstrap intervals from 50,000 resamples, preserving pairing across methods, backbones, and seeds. For example, at B=16B=16, TURF-Fixed improves over the strongest baseline averaged across budgets by +4.77% AUROC [3.40%, 6.14%] and +2.48% AUPR [1.61%, 3.39%] on the revised common cohort. These are absolute percentage-point differences, not relative improvements. As both intervals lie entirely above zero, the results support higher AUROC and AUPR for TURF-Fixed and TURF-Adaptive than SGD and other baselines on the matched-task cohort.

W3: Unclear benefits of the proposed components.

Does semantic information help?

We compared full TURF-Adaptive with a variant that uses execution outcomes alone, removing semantic information from both the uncertainty score and the stopping rule. We matched the total number of inspected trajectories within each dataset–backbone–seed cell using label-independent random additions. This tests whether semantics helps beyond simply inspecting more trajectories.

At B=16, full TURF-Adaptive gains +1.03% AUPR [95% interval: 0.36%, 1.76%] and +0.90% AUROC [−0.01%, 1.79%]. All point estimates favor TURF-Adaptive. The intervals accommodate substantial positive effects but also a small negative effect.

We would like to mention that semantic information has demonstrated its value in LLM uncertainty quantification through methods such as semantic entropy (Farquhar et al., 2024) and SGD (Li et al., 2025). Indeed, SGD is the strongest budget-averaged baseline among those with complete results on our primary evaluation grid. Our ablation asks a different question: how much additional value semantics provides when execution-outcome information is already available. Although the incremental gains on our benchmarks are modest, the matched-cost AUPR improvement supports combining these complementary sources. The semantic view can capture relationships between terminal responses that discrete outcome categories do not represent. This motivates us to include it in TURF. We will further investigate where this additional information is most beneficial in our future work.

Is Stouffer fusion better than simple averaging?

We agree that replacing Stouffer fusion with equal-rank averaging produces similar performance in our empirical results. In particular, as demonstrated in Figure 4 (a), Stouffer fusion outperforms slightly but consistently better than mean-rank fusion on AUPR within the range [0, 0.25] across budgets, but comparable on AUROC. We present mean-rank fusion as a simpler alternative and do not attribute TURF’s gains in our empirical comparisons specifically to the Stouffer transformation. We have clarified this in the empirical section.

We would like to clarfiy that we use Stouffer as a theoretically motivated, task-training-free default. It maps the two component ranks to a common normal-score scale and combines them symmetrically. Under the explicit Gaussian working model in our appendix, with equally informative views and shared symmetric covariance, this sum yields the likelihood-ratio ranking. This provides a conditional design rationale, not a claim of universal optimality.

W4 Missing citations to closely related work

In our updated introduction and related work sections, we explicitly cite and distinguish Oh et al. (ACL 2026), Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities; Zhang et al. (ICML 2026), Agentic Confidence Calibration; and Zhang et al. (2026), Agentic Uncertainty Quantification (AUQ). HTC learns a supervised calibrator and was not evaluated. AUQ propagates verbalized confidence/explanations through memory and triggers reflection that can change actions. TURF is passive with respect to the within-trajectory agent: adaptive sampling changes only how many executions are scored. AUQ therefore requires a separate controlled interventional evaluation, not post-hoc scoring of our fixed pools. As AUQ does not have released code, we actually tried to reimplement and adapt it to our settings, but found that the settings are very difficult and the adaptation is nontrivial.

We added paired comparisons with three passive-UQ adaptations from several recent 2026 papers: AgentUQ (ACL 2026), TRACER (ICML 2026), and TER (2026 arXiv preprint), across B={2,3,4,6,8,12,16}. Below, entries are TURF-Fixed minus baseline in AUROC/AUPR percentage points (AUPR is average precision). Each row uses identical tasks for both methods, fixed across budgets, with equal-cell averaging; different rows have different task intersections.

Baseline/adaptationCellsB=4B=8
AgentUQ-Obs-NLL54+13.08/+9.57+14.75/+11.27
TRACER-Local54+17.35/+12.77+18.73/+13.75
TER-NativeJudge54+13.65/+11.89+13.57/+12.19

For these three baselines, macro gains remain positive across all seven budgets.

However, we want to clarify that these are adapted implementations, not unmodified reproductions of the authors’ full experimental pipelines. We can not directly use their pipelines, and the problem settings are different.

  • AgentUQ-Obs-NLL: We directly reused the author-released observation-rescoring prompt builder. Our harness supplies recorded histories and native-backbone token likelihoods, sums observation-token NLL within each rollout, and averages across the first B rollouts.
  • TRACER-Local: We directly reused the scoring and aggregation routines from the authors' released code. We replaced Vertex embeddings with frozen local MPNet embeddings to allow fair comparison. We adapt recorded action/observation inputs and average the first B rollout scores; this is not the original embedding configuration.
  • TER-NativeJudge: As the authors did not release their code, we reimplemented the published TER definition. Our implementation uses the generating backbone as judge, but not more advanced LLMs as judge as used in the paper.

Interpretation. These results evaluate the transfer of three recent uncertainty estimators to our datasets and fixed trajectory-pool protocol. AgentUQ-Obs-NLL is one estimator within the AgentUQ framework, while TRACER and TER involve disclosed embedding and judging adaptations. Although implementation and numerical checks were performed, the current comparisons do not isolate how much these adaptations contribute to the observed gaps. We therefore interpret the results as evidence for the evaluated variants, rather than general superiority over the authors’ original systems.

Dear authors,

I really appreciate your professional rebuttal. I've checked the revised PDF, and it seems to have been improved, certainly compared to the initial version. Some of my concerns about the presentation/writing (W1/W4) have been addressed, and I will update my rating from 1 (clear reject) to 2 (weak reject); also raised the sub-category rating the Presentation and Informativeness from 1 to 2.

I am sorry, but I am still leaning towards rejection due to the limited advantage of the proposed component (W3), which is also pointed out by another reviewer (7vBT), and the experimental rigor.

Anyway, thanks for your thorough rebuttal to address my concern.

Best, Reviewer q1QX

Comments on the paper

1 comment
AuthorsOpenReview ↗

Withdrawal of Our Submission

Dear Reviewers,

We have decided to withdraw our submission. Nevertheless, we sincerely appreciate the time and effort you devoted to reviewing our paper. Your thoughtful reviews and responses to our rebuttals have helped us improve the manuscript from multiple perspectives. We will also carefully consider the remaining concerns and address them in our future revision.

Thank you again for your valuable feedback and constructive suggestions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.