- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Verdict Audit is a four-stage procedure for comparing LLM judges on agent trajectories. It examines output ranking and threshold choice, robustness to misleading trajectory content, information recoverable from hidden states, and whether interventions can selectively correct errors. Threshold tuning substantially reduces the balanced-accuracy gap between Qwen3-8B and Qwen3-14B, while stress tests reveal robustness differences whose ordering reverses across domains. For Qwen3.6-27B, hidden-state probes improve ranking over the verdict margin, but neither a balanced-accuracy gain nor selective correction is established.
The paper presents useful robustness findings and interprets its statistical results carefully. However, the added model-selection value of the internal stages remains insufficiently demonstrated, the probing comparison is limited by the chosen output readout, and important experimental details are missing. Together, these issues make the contribution of the full four-stage procedure less convincing than the output-level and robustness analyses on their own.
Critical strengths
The paper addresses a practical problem: selecting judges for long agent trajectories when aggregate accuracy can obscure important differences. The controlled false-completion experiments are useful, particularly the reversal of relative vulnerability between RuVerBench and AgentRewardBench.
The statistical interpretation is generally careful. The authors distinguish a failure to detect a difference from evidence of equivalence, acknowledge limited power, and separate readable internal information from deployable error correction. Trajectory-grouped evaluation and matched intervention controls are also strengths.
Critical weaknesses
The added decision value and practical scope of the full audit remain unclear.
- The paper presents Verdict Audit as a procedure for choosing judges, but the clearest actionable findings come from Stages 1 and 2: threshold tuning changes the performance comparison, and stress tests reveal robustness tradeoffs. Table 3 does not demonstrate a deployment decision that changes because of Stages 3 or 4. While Stage 3 identifies additional information in Qwen3.6, Stage 4 does not establish selective error correction.
More importantly, the audit focuses exclusively on binary PASS/FAIL verdicts. In practical agent evaluation, knowing that a task failed is often insufficient. Understanding what went wrong, where the failure occurred, and why can be much more useful. Related work such as TRAIL, Who&When, and AdaMAST explores error taxonomies and failure attribution. Although these are not direct baselines, the paper does not discuss how Verdict Audit relates to such diagnostic approaches or whether it could evaluate more informative judgments. Given that the individual techniques are already established, the limited demonstrated benefit for judge selection and the narrow focus on binary verdicts weaken the contribution of the proposed framework.
-
The probe results support a narrower conclusion than a general gap between internal and output-level judgment quality. Section 4.3 constructs the output score from first-token probabilities for fixed verdict labels. The probe is therefore compared with one particular output readout. The conditional analysis and matched label-budget experiments are useful, but do not establish whether the advantage persists with alternative prompts, verdict labels, full-label sequence scoring, or trained output-only baselines using richer output features. The evidence supports information beyond the chosen margin; it leaves open how much of that advantage depends on the output scoring protocol.
-
The experiments do not resolve the practical value of the internal analyses with sufficient precision. At a label budget of 200, the reported balanced-accuracy change from adding the Qwen3.6 probe is +0.073, with a confidence interval of [−0.050, +0.184]. The paper states that the design can detect gains of roughly 0.15–0.23, leaving smaller but potentially useful improvements unresolved. Thus, the lack of an established gain should not be read as evidence that probing has little practical value; rather, the experiments are not precise enough to settle that question. This limits the strength of the deployment conclusions drawn from the internal stages. AgentRewardBench provides useful cross-domain evidence for selected Case A stress tests, but does not address this limitation for Case B.
-
The experimental specification has major reproducibility gaps. Section 3.4 states that the exact sampled interchange layers are reported in the appendix, but I could not locate that list in Appendix B.8. Appendix B.4 describes the probe family and grouped evaluation, but does not provide enough detail to reconstruct the candidate sites and regularization-selection procedure. Table 10 reports cohort sizes without identifying the exact items or the number of unique trajectories, which matters because checks from the same trajectory are dependent. I also could not find a repository or experimental-artifact link in the provided submission. These omissions make it difficult to reproduce the comparisons and assess their effective sample sizes.
Questions for authors
-
Table 3 describes how each stage changes the interpretation of the models. Can you identify a concrete deployment decision that changes after Stages 3 or 4, beyond the evidence from threshold tuning and stress tests? What would justify their additional cost?
-
Does the Qwen 3.6 probe advantage persist under alternative prompts and verdict labels, full-label sequence scoring, and trained output-only baselines evaluated with the same trajectory splits and label budgets?
-
Given the reported detectable balanced-accuracy gains of approximately 0.15–0.23, what practically relevant effect size was the study intended to resolve? Could additional evaluation narrow the uncertainty around the probe’s decision-level benefit?
-
Could you provide the exact interchange layers, candidate probe sites, regularization-selection procedure, prompts, seeds, and intervention configurations? Section 3.4 explicitly points to an appendix layer specification that I could not locate.
-
Could you provide an anonymized artifact link with evaluation scripts, item IDs, trajectory IDs, and split assignments for the reported cohorts, including the 700 rubric checks, 73 shared omission cases, and 200 AgentRewardBench items?
-
Could Verdict Audit be extended to evaluate error categories and failure localization, rather than only binary PASS/FAIL decisions? How do the authors see its relationship to approaches such as TRAIL, Who&When, and AdaMAST?
3 comments
Thank you for your reply and the work you have done. I have reviewed the attached materials and can say the following:
- The revised explanation makes the role of Stages 3 - 4 clearer: they help assess whether internal methods are worth adding to the evaluation pipeline. This partly addresses my concern. However, the reported total of $592.70 does not show how much these stages add to the cost. Could you provide a separate estimate for Stages 3-4?
- Stronger output baselines. The baseline with margin, input length, and rubric category has indeed been added, and its AUC of 0.745 can be reproduced from the archive.
- The issue of statistical precision remains: the interval [−0.050, +0.184] is too wide to determine the practical benefit of the probe. No additional data has been provided to resolve smaller effects.
- The main missing experimental settings are now provided, and I consider this issue resolved.
- Reproducibility. The archive largely addresses my concern: 115 reported quantities can be recomputed from the attached data, and identifiers and split assignments are available. However, 42 quantities are provided as saved analysis outputs, probe activations are not included, and the exact checkpoint revision of Qwen 3.6 is unknown. Therefore, full reproduction of experiments remains limited. Please explicitly state these limitations in the paper and soften the wording about full reproducibility.
- Connection with failure localization. I consider this issue closed; extending the approach to failure localization remains a possible direction for future work.
Thank you for reviewing the revised materials and confirming that several concerns have been resolved. We have added further analyses and documentation to address your remaining questions.
1. Cost of Stages 3–4. Stage 3 reuses activations collected during Stage 1, so probe fitting requires no additional GPU inference. For Stage 4, our experimental records identify approximately $115.18 spent on causal interventions across the four judges, included within the earlier campaign total. Appendix G now reports this breakdown.
3. Statistical precision. Your concern was whether the available evidence could establish a practical improvement from probing. We revisited the existing label-budget experiments across all four judges and accounted for dependence across their 189 shared trajectories.
The dependence-adjusted pooled balanced-accuracy difference is −0.008 [−0.077, +0.060], improving precision by 1.70 times over the original Qwen3.6 estimate. Three of four judges have negative point estimates. This provides stronger evidence that recoverable internal information does not consistently translate into better decisions across these judges.
A smaller gain of +0.05 remains possible. We therefore quantified the evidence needed to detect such an improvement. Our 80%-power calculation estimates approximately 767 labeled checks per judge, compared with the 200 used. Appendix F now presents the pooled analysis, dependence adjustment, and the assumptions behind this sample-size estimate. The new analysis gives us a more precise result from existing data and establishes the scale of evaluation needed to resolve smaller effects.
5. Reproducibility. We have expanded the supplementary archive to include the original verdict-position activations for all four judges, alongside experimental scripts, prompts, configurations, item and trajectory IDs, and split assignments. Reviewers can now rerun the probe comparisons from saved model states rather than relying solely on reported analysis outputs. We have also refined the manuscript's Reproducibility Statement to distinguish the analyses supported by these materials from the remaining requirements for full GPU reproduction.
We hope these additional results and materials address your remaining concerns about experimental cost, statistical precision, and reproducibility. We appreciate your careful review and consideration of these revisions.
Thank you for recognizing our robustness findings and statistical controls. We have substantially revised the Introduction, Related Work, Results, and appendices to address your concerns.
1. Why Stages 3–4 matter. Stages 1–2 help teams choose a judge and test whether its decisions follow recorded evidence. But when mistakes remain, teams face another decision. Can internal information help detect those mistakes, and can it be used to correct them? Stage 3 tests whether internal states identify errors beyond what the outputs already reveal. Stage 4 tests whether interventions correct wrong decisions without disrupting correct ones.
Our experiments show why these distinctions matter. Qwen3.6 contains recoverable information, but its probe does not establish better pooled error detection than accessible outputs. Steering also changes correct and incorrect PASS judgments together. Neither result justifies deploying the tested additions. The later stages therefore help teams avoid investing in internal methods without evidence of additional benefit. They become relevant when a team considers such methods, rather than being mandatory for every evaluation (Sections 4.3–4.4, Discussion).
2. Stronger output baselines. We added predictors using the verdict margin, input length, and rubric category on identical trajectory-grouped splits. For Qwen3.6, the strongest accessible baseline achieves AUC 0.745 versus 0.797 for the probe. The paired difference of +0.050 [−0.004, +0.101] does not establish an overall advantage (Appendix F, Table 8). We do not extend this conclusion to untested prompts, labels, or full-label scoring.
3. Statistical precision. At 200 labels, the estimated balanced-accuracy gain is +0.073 [−0.050, +0.184]. Our added power analysis shows detectable gains of approximately 0.15–0.23. Smaller useful improvements remain unresolved and would require more independent trajectories. We do not interpret this as evidence that probing cannot help (Appendix F).
4–5. Reproducibility. Appendix B now specifies the exact interchange-layer grids, fixed probe depths, regularization penalty C=0.05, and grouped fitting. Appendix F provides intervention sites and controls. Appendix A documents the 700 checks from 189 trajectories, 73 omission checks from 53 trajectories, and 200 AgentRewardBench checks from 188 trajectories. We have also uploaded anonymized code with prompts, configurations, seeds, item and trajectory IDs, and split assignments.
6. Failure localization. TRAIL, Who&When, and AdaMAST study where and why an agent failed. Verdict Audit asks whether the judge recognized and used evidence of that failure. These are complementary questions, as explained in the revised Related Work. Categorical and localized judgments are possible extensions, but are not claimed as validated here.
We hope the additional experiments, clearer methodological details, and revised discussion address your concerns about the value of the internal stages, the strength of our comparisons, and reproducibility. We appreciate your consideration of these revisions.