- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(−1)
Read review
Paper summary
The paper proposes TURF, a method for ranking the failure risk of LLM agents from repeated executions. It combines variation in observable execution outcomes with semantic variation in terminal responses, using Gaussianized ranks to fuse the two signals. The method does not train a predictor on task correctness labels. Its adaptive version uses Bayesian evidence of agreement with the first execution to decide how many trajectories to inspect. The theoretical analysis establishes properties of the scoring components and a Bayesian guarantee for the stopping rule. Experiments cover six benchmarks, three language models, three random seeds, and 15 baselines. Averaged over the five main trajectory budgets, TURF-Fixed improves AUROC and AUPR over SGD by 5.19 and 2.99 percentage points. TURF-Adaptive uses 20.90% fewer trajectories for scoring, with respective losses of 0.52 and 0.39 points relative to TURF-Fixed. The evaluation measures failure ranking for previously recorded task results, with correctness labels held fixed across budgets and policies.
Critical strengths
-
Direct evidence for the value of execution outcomes. The paper provides a convincing test of whether execution outcomes add useful information beyond final responses. The comparison in Figure 5 and Appendix F.5 keeps the tasks, trajectory prefixes, count statistic, and semantic scores the same, and removes the additional WebShop severity signal from both versions. Using execution outcomes improves AUROC and AUPR by 1.59 and 1.24 percentage points, respectively, averaged over the five main budgets. The largest gains occur on the database and shopping tasks. This experiment directly supports the paper's central motivation and is an important reason for my positive assessment.
-
Careful comparisons with the baselines. I also appreciate the effort to check whether the results depend on how the baselines are adapted. Appendix E evaluates 13 baseline families both on final responses and by aggregating uncertainty across steps, using the same task set for every method. In the averages reported in Table 2, TURF-Fixed still exceeds SGD-Final by 5.58 percentage points in AUROC and 3.42 points in AUPR. This comparison addresses two plausible concerns: that the main results favor a particular baseline adaptation, or that methods are being compared on different subsets of tasks. It makes the empirical evidence considerably more persuasive.
Critical weaknesses
-
The benefit of several components of the full method remains modest. In the core scoring ablation at , adding the semantic score to the count score improves AUROC and AUPR by only 0.16 and 0.53 percentage points. The AUROC confidence interval includes zero, and the AUPR result does not survive the reported Holm correction. Stouffer fusion also performs very close to simply averaging the two ranks (Appendix F.2). The count score is therefore already close to the complete scorer. In addition, the adaptive scoring ablations retain the stopping decisions made by the full method, so they do not establish the value of semantic computation across the entire procedure. Given the additional NLI computation, the practical benefit of the complete design needs a clearer demonstration.
-
The WebShop comparison does not fully separate the contribution of the general scorer from an additional severity signal. This signal uses purchase status, request coverage, and price. Removing it lowers TURF-Fixed's AUROC from 0.7020 to 0.6454 at (Appendix H.3). The paper clearly discloses this adjustment, and the controlled experiment in Appendix F.5 excludes it. However, I could not find a comparison in which a strong semantic baseline receives the same severity score and the same rule for combining it with its uncertainty score. This leaves an important question about how much of the main WebShop advantage comes from the general TURF scorer and how much comes from the additional rules.
Questions for authors
-
Could you compare the full method with a version using only execution outcomes for both scoring and Bayesian stopping, at the same average trajectory budget? The current scoring ablation retains the full method's stopping decisions. Reporting NLI overhead alongside this comparison would also help assess whether the semantic component is worthwhile.
-
How does a strong semantic baseline perform on WebShop when given the same severity score and combination rule as TURF? Comparing them on the same tasks and trajectory budgets would help separate the contribution of the general scorer from the additional WebShop rules.
-
Keeping the archived correctness labels fixed, how do the stopping decisions and ranking results change if semantic availability is determined only from the trajectories inspected so far? Please recompute the reference ranks as well. This would clarify the effect of using the full trajectory pool for availability checks, as described in Sections A.4 and B.3.
2 comments
Thank you for the detailed rebuttal and additional experiments. While some concerns have been addressed, I remain unconvinced by the methodological novelty and the benefits of the key components.
I am also concerned about the extensive rewriting of the manuscript during rebuttal, which I believe goes beyond the scope of reasonable revisions at this stage.
Thank you for the careful assessment. We agree that semantics should be evaluated jointly with stopping and computational overhead, and that WebShop requires severity-matched controls.
All AUROC/AUPR differences below are absolute changes on the percentage scale, not relative percentage improvements.
Q1. Outcome-only scoring and stopping at matched trajectory cost
We compared full TURF-Adaptive with outcome-only scoring and independently recomputed outcome-only Bayesian stopping across six datasets, three backbones, and three seeds (54 cells). Entries below are full minus outcome-only; AUPR denotes average precision.
At B=16, paired 95% intervals are [−0.01%, 1.79%]/[0.36%, 1.76%], supporting a modest AUPR benefit while the AUROC difference remains inconclusive. At B=8, the intervals are [0.00%, 1.66%]/[0.35%, 1.54%]. B=4 tests scoring alone because early stopping begins at prefix five. Intervals use 5,000 paired query resamples conditional on frozen scores/ranks, with equal-cell macro averaging.
Contribution of semantics to stopping. Holding the fused scoring rule fixed, we separately compared two-view and outcome-only stopping using the same retrospective trajectory-matching procedure. At B=16, using semantics in stopping improves AUROC/AUPR by +0.91%/+0.92%, with paired 95% intervals of [0.38%, 1.43%]/[0.55%, 1.25%]. This isolates the contribution of semantics to sampling decisions rather than to the scoring rule.
Computational overhead. A single directed NLI comparison took approximately 12 ms on an A100 GPU when processed individually. This is component timing, not complete per-task runtime. Matching trajectory counts does not equalize total computation, and we do not claim an overall deployment-cost advantage over the outcome-only version.
Q2. Identical WebShop severity adjustment
We applied the same outer WebShop severity score and combination rule to TURF and Semantic Graph Density (SGD), a strong semantic baseline, on matched tasks across all nine backbone–seed cells. Entries below are TURF-Fixed minus severity-matched SGD.
B=4 provides descriptive estimates only. At B=8, the secondary, unadjusted 95% intervals are [−0.55%, 5.35%] for AUROC and [0.01%, 1.30%] for AUPR. At B=16, both intervals include zero: [−1.30%, 4.01%]/[−0.11%, 1.10%]. All point estimates favor TURF after matching the WebShop severity adjustment. The intervals accommodate substantial positive effects but also a small negative effect.
Q3. Prefix-local semantic availability and recomputed reference ranks
The historical availability check used all 16 trajectories, allowing an invalid later response to disable semantics at an earlier prefix. We reran the evaluation using only trajectories inspected so far to determine availability, retaining the original validity criteria and outcome-only fallback.
Across all 54 cells and B={2,3,4,6,8,12,16}, we kept correctness labels, outcomes, trajectory order, and stopping parameters fixed while recomputing stopping decisions, scores, and reference ranks, including WebShop’s outer ranks. Entries below are prefix-local minus historical results.
All changed stops are later; counts refer to task–cell records. Fixed B=16 is exactly unchanged. The correction produced very small observed changes in this replay, with all displayed intervals crossing zero.