- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents LLM-as-a-Verifier, a training-free framework for evaluating agent trajectories. It combines probability-weighted scoring with finer score levels, repeated evaluation, and separate evaluation criteria. It also introduces a Probabilistic Pivot Tournament to reduce the number of candidate comparisons. Experiments cover coding, robotic trajectory evaluation, and medical tasks, with additional applications to progress monitoring and reinforcement learning. The central claim is that improving verification can help agents select better solutions and obtain more useful feedback.
Critical strengths
S1. The paper addresses a practical bottleneck: agents often generate successful trajectories but fail to select them reliably. Reporting both single-candidate performance and oracle selection performance makes this motivation concrete.
S2. The framework works with existing language and vision-language models without additional verifier training. Its application across coding, robotics, and medical tasks gives it practical relevance.
S3. The experiments examine several useful aspects of verification, including scoring granularity, repeated evaluation, ranking budgets, and caching. The extensions to progress monitoring and reinforcement learning broaden the potential applications.
Critical weaknesses
W1. Counting ties as errors in Appendix D overstates the selection benefit of continuous scoring. In Table 3, random tie-breaking would give the discrete baseline an expected accuracy of 80.35% with one evaluation, versus 80.1% for the continuous method. The main results also need an explicit tie-breaking rule.
W2. The three-criterion ensemble uses more verifier calls than the individual criteria. The key missing control is repeated evaluation with one criterion under the same total budget, which would separate the benefit of different criteria from that of additional evaluations.
W3. PPT needs a comparison with V1 using the same scorer, criteria, and total evaluation budget, plus a random-pivot baseline. Its pivot selection also relies on scores alone, despite the claim of uncertainty-aware allocation.
W4. VOC measures agreement with chronological order and can remain high when a trajectory stalls before completion. Progress tracking should be evaluated against intermediate completion labels and on trajectories with stalls, regressions, and failures.
Questions for authors
Q1. How are ties handled in Figures 3 and 8 and in final trajectory selection? How do the comparisons change when ties are resolved randomly?
Q2. Under the same total verification budget, how does the three-criterion ensemble compare with repeated evaluation using one criterion?
Q3. How does PPT compare with V1 and random-pivot selection when all methods use the same scorer, evaluation criteria, and repetition settings?
3 comments
Response to Reviewer xVwX (Part 2)
W3 & Q3: PPT vs. V1 and random-pivot selection
Ring-based vs. random pivot selection. To isolate the effect of pivot selection, we evaluate ring-selected pivots against randomly chosen pivots on the existing 20-candidate pools (Terminus-2 harness, 89 tasks). All variants share the same verifier, evaluation criteria, repeated-evaluation settings, and number of pivot comparisons, so they differ only in how the five pivots are selected. Results are mean ± std over three seeds, each with a different random ring or set of random pivots. Ring-based selection is better than random selection on every seed, improving the success rate by 3 points.
| Method | Pairs queried | TB 2.0 success rate |
|---|---|---|
| Pass@1 | — | 52.64% |
| Ring pass only (no pivot rounds) | 1,060 | 62.55 ± 1.72% |
| Random pivots (pivot rounds) | 5,550 | 63.30 ± 2.83% |
| Ring-based pivots (PPT) | 5,550 | 66.29 ± 1.95% |
Quality of the selected pivots. We also examine the pivots themselves on the 70 swing tasks, whose pools contain both passing and failing trajectories. Ring-based selection raises the fraction of passing pivots by 14.3 points, so PPT spends its verification budget on candidates that are more likely to be correct.
| Metric | Random pivots | Ring-based pivots (PPT) |
|---|---|---|
| Fraction of pivots that are passing trajectories | 50.8 ± 0.9% | 65.1 ± 0.6% |
| Tasks with at least one passing pivot | 84.8 ± 2.2% | 91.0 ± 1.6% |
| Tasks where all 5 pivots pass | 14.8 ± 2.2% | 36.2 ± 0.8% |
| Tasks with no passing pivot (lower is better) | 15.2 ± 2.2% | 9.0 ± 1.6% |
Comparison with V1 and the budget–accuracy trade-off. On the same pools, PPT with only achieves 66.17% using 4,723 pairs, surpassing the best V1 result (65.85%) while using 32.5% fewer comparisons. Raising the number of pivots to gives 67.13% using 9,630 pairs, close to full round-robin (67.42%) with 26.5% fewer comparisons.
| Method | Total pairs queried | Accuracy |
|---|---|---|
| V1 (5N budget) | 7,000 | 65.85% |
| PPT (k=3) | 4,723 | 66.17% |
| PPT (k=5) | 6,610 | 66.29% |
| PPT (k=9) | 9,630 | 67.13% |
| Full round-robin | 13,111 | 67.42% |
W4: VOC and progress tracking on failed trajectories
We agree that VOC measures agreement with chronological order, and that its absolute value does not certify task completion. We therefore do not use VOC as an absolute measure of progress. We use it only for relative comparison: to rank parallel trajectories on the same task against each other and to identify the runs whose progress has plateaued. To test whether VOC supports this relative use, we evaluated it on trajectories with stalls, regressions, and failures, using 500 paired successful and failed trajectories from the same Terminal-Bench V2 tasks.
| Trajectory outcome | Spearman VOC (rank correlation) |
|---|---|
| Successful | 0.848 ± 0.012 |
| Failed | 0.769 ± 0.016 |
LLM-as-a-Verifier achieves a Spearman VOC of 0.848 on successful trajectories, compared with 0.769 on failed trajectories from the same tasks, a gap of 0.08. On successful trajectories, the verifier score rises consistently as the agent advances, which gives a high rank correlation with step order. On failed trajectories, the score more often plateaus when the agent stalls and drops after erroneous actions, and these flat or decreasing segments lower the correlation. As you point out, the VOC of a failed trajectory can still be high in absolute terms (0.769), which is exactly why we do not threshold it. What matters for our use is the relative ordering: on the same task, runs that stall or regress receive a lower VOC than runs that keep progressing.
Progress-aware scheduling relies only on this relative comparison. A run is paused when its VOC stops improving and falls below that of the other parallel runs on the same task (Appendix K, Algorithm 2). On Terminal-Bench V2, pausing stalled trajectories with VOC improves reward per GPU-hour by 2.2×, compared with 1.2× for a matched-rate random-stopping baseline, which shows that the relative VOC signal identifies the runs that are worth pausing.
We hope these clarifications and additional results address your concerns, and we hope you will consider updating your score in light of these improvements.
Thank you for the clarifications and additional experiments. While some concerns remain, I will raise my rating to 3.
Response to Reviewer xVwX (Part 1)
Dear Reviewer xVwX,
Thank you for your thoughtful review and constructive feedback. We have updated the paper in line with your comments. Changes are marked in green in the revised manuscript. Below, we address the comments in your review.
Concrete steps we took based on your feedback:
W1 & Q1: Tie handling
Ties are assigned 0.5 credit throughout the paper, which is the expected accuracy under random tie-breaking:
Accuracy=P(sc>si)+21P(sc=si),
where sc and si are the scores of the correct and incorrect trajectories. This protocol already applies to Figures 3 and 8 and to the closed-model table in Appendix D (Table 3 in the original submission, Table 5 in the revision). The strict-preference description in the original text only applied to the continuous verifier, as in Figure 3: it produces zero ties, so strict accuracy and accuracy under random tie-breaking are identical. For the discrete baseline, the reported accuracies already include the half credit, so they should not be adjusted further. We have revised Appendix D to state this protocol explicitly.
Figure 3 is unchanged, since continuous scores produce no ties and strict accuracy equals accuracy under random tie-breaking. For Figure 8, we now additionally report the discrete judge under both protocols:
Random tie-breaking improves the discrete judge relative to counting ties as incorrect. Even so, the continuous verifier reaches 74.7% with a single evaluation, matching the discrete judge's 74.7% after 16 repeated evaluations. This demonstrates that continuous scoring provides comparable discrimination with substantially fewer evaluations.
W2 & Q2: Three-criterion ensemble vs. one criterion
Under 16 repeated evaluations, we obtain:
We find that a single criterion reaches at most 76.38%, while the ensemble over separate prompts reaches 78.29%. Placing all three criteria in one combined prompt reduces accuracy to 74.34%.