- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks which better predicts an agent’s failure: the agent’s own confidence, or an outside LLM judge that reads the agent’s transcript. Across five settings, the authors give each a hand-assigned “visibility” score (how readable errors are from the transcript) and find that the judge’s advantage over self-confidence rises steadily with that score. Code is their key example: correctness there is objective, yet the judge gains almost nothing, which they take to mean visibility matters more than objectivity. They also find that combining several judges does no better than the best single judge. Also the conclusion is that choosing between judge and self-confidence based on trajectory length matches the best possible choice.
Critical strengths
The code example is informative -- even with objective correctness (unit tests), an external judge reading the source does no better than self-assessment (AUROC .59 vs .58), because many bugs only appear at runtime. This is a useful caution for deploying LLM judges.
Critical weaknesses
W1. The visibility scores look chosen after the fact. The paper says the scores were assigned from task structure before any results were seen, but several things undercut that:
- Airline and retail are hard to tell apart structurally. Airline gets all zeros and retail all twos, yet both use the identical -bench setup (policy document, tool calls, database outputs). The real difference is that airline has more complex policies and more tasks where the correct response is to refuse.
- The airline judges point the wrong way. Two judges score well below chance (AUROC 0.39 and 0.46), and the aggregate is 0.43. That suggests judges systematically misread correct runs, for example treating a correct refusal as a failure.
- Two tables disagree. Table 1 and appendix Table 3 give Code and Who&When-deep different sub-scores that happen to add up to the same totals.
W2. Statistical significance and the concern that the paper’s own numbers don’t support the “law.” Two of the five points measure something different. The Who&When points compare the best of four judges, picked after seeing results, against an unclear baseline rather than against self-assessment. Their values also change between sections (0.31 and 0.45 in Table 2 vs. 24.1% and 41.3% in Section 5(iii)), and the Who&When-deep point rests on only 16 examples with no confidence interval. When restricted to cases scored by all four judges (Table 4), code’s gain (+.086) exceeds retail’s (+.039), so the ordering breaks, and neither extreme remains significant.
W3. The routing result fairness concerns. The depth cutoff is tuned on the same three domains it is evaluated on. Routing each example individually, which is what a real deployment would need, gives essentially no gain (+.006, not significant). Depth also conflicts with the paper’s own scores, since the longer Who&When Hand-Crafted traces are rated less visible, not more. Finally, the paper reports “97% of oracle,” but its routing score equals the oracle’s exactly.
Questions for authors
- Do the judges see the airline policy? What is judge accuracy on tasks whose correct outcome is a refusal?
- Self-assessment is elicited in a separate call where the acting model rates its own finished trajectory, so the “self vs. judge” gap may partly reflect self-preference bias (models favor their own outputs) rather than visibility. Could the authors specify which models generated each domain’s trajectories and add a control in which each model rates both its own and other models’ trajectories?
- Tables 1 and 3 give different sub-scores for Code and Who&When-deep that happen to sum to the same totals. Could the authors describe the scoring protocol? Could the authors clarify which table is correct and whether the ordering holds when results-blind raters redo the scoring?
- Please report the ordering under complete panels with a task-clustered bootstrap. Does code still fall below retail?