acceptodds
Withdrawn from ICLR 2027

TraceJudgeBench: A Unified Benchmark for Validating LLM-as-a-Judge on Agent Execution Traces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM judges are now a standard way to evaluate LLM-based agents, but the methods and the benchmarks used to validate them do not agree with each other. Each defines its own error taxonomy, annotation scheme, and error distribution, so neither a judge's score nor a benchmark's verdict carries over to another. We present TraceJudgeBench, a meta-analysis that harmonizes ten judge-validation datasets and three unannotated trace corpora into one frame: 846 agent execution traces kept in their native formats, with every label mapped into a shared two-level taxonomy of ten fine-grained and five coarse categories. The frame lets existing and new evaluation methods, each with its own taxonomy, be cross-validated against one another and against the benchmarks they come from. We annotate 231 further traces to fill the gaps public sources leave open: API and system failures, environment failures, and clean executions with no error at all. Applying the frame to a prompted LLM, TRAIL, AEGIS, and a decomposed ensemble judge on one backbone shows what the separate benchmarks hide. Three of the four judges are blind by construction to categories their native schemes cannot name. Every judge reports errors on clean traces: false-error rates run from 27% to 74% for the three established judges and reach 100% for the ensemble run without abstention, and the prompted baseline stays in that range across three backbones. False-error also grows with trace length: on clean labels that two human annotators confirmed trace by trace, the prompted baseline and TRAIL rise 44 and 50 points from short to long clean traces, an association that the data cannot yet separate from task type. A judge validated on short traces carries no guarantee on long ones, and a corpus concentrated at one length cannot show it. The traces, taxonomy mappings, judge-output mapping code, and analysis scripts are included as supplementary material.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-judge, agent evaluation, benchmarks, error taxonomy, multi-agent systems, execution traces, false positives, judge validation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_U8c6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TraceJudgeBench, a meta-analysis that harmonizes ten LLM-as-a-judge datasets and three unannotated trace corpora into a single frame. The native or original labels were mapped into a shared two-level taxonomy of ten fine-grained and five coarse categories. Therefore this new taxonomy frame establishes a unification of labels between the judge datasets enabling old and new evaluation methods to be cross-validated against one another and against the original benchmarks.

Critical strengths
  1. The paper addresses an important and timely problem of unifying fragmented LLM-as-a-judge benchmarks through the conversion to L1/L2 taxonomy labels that enable cross-validation across previously distinct judge benchmarks.
Critical weaknesses
  1. The author should elaborate on the judgment methods in Table 1, as they currently seem a bit simplistic. Additionally, some more details about the different judges and elaboration on the ensemble judge would help.
  2. The motivation of the work is good, but the study is narrow as it is only conducted on the typical LLM-as-a-judge approach for agent traces that are within context length limit. Authors should consider extending the scope to cover more advanced judge methods like agentic judges meant for traces that exceed context length limit. Agentic judges are more relevant in the community today since a majority of the agent traces are very long especially for skill-based agents which are gaining traction in the recent year (see more details on this in the question section below).
Questions for authors
  1. Why the ensemble method is omitted from Table 2?
  2. Can you clarify and elaborate more on the different judges that is being compared especially some details on AEGIS, TRAIL, and the ensemble judge such that the paper is self-contained.
  3. The authors claimed that for clean-trace false errors grow with trace length which is not surprising at all when the judge used is a typical standard LLM-as-a-judge method. However, it remains questionable whether this finding generalizes to more sophisticated judge methods designed for very long traces such as agentic judges which can intelligently use tools to inspect part of the trace.
  4. Some parts of the paper such as Section 5 is hard to read and can benefit from a rewrite.
Reviewer_kDX8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TraceJudgeBench, a benchmark for validating LLM judges on agent execution traces. It harmonizes ten judge-validation datasets and three trace corpora into a shared taxonomy, preserves native trace formats, and adds annotations for underrepresented cases. Experiments evaluate four judge designs and identify category coverage gaps, high false-error rates on clean traces, and an association between trace length and false-error rates.

Critical strengths

The paper addresses an important problem: agent judges are commonly evaluated using incompatible taxonomies and datasets. The unified error categories provide a useful basis for cross-source evaluation.

The inclusion of clean executions and infrastructure failures broadens the evaluation beyond commonly studied reasoning and planning errors. Evaluating whether judges incorrectly flag valid executions is practically relevant.

The authors provide several useful validation analyses, including human auditing, mapping perturbations, bootstrap intervals, and source-specific analyses. The distinction between scheme-defined zeros and empirical failures is also helpful.

Critical weaknesses
  1. Category alignment does not establish consistent error semantics across sources. The benchmark largely inherits source annotations and maps them into refined categories. However, sources differ in error annotations, as stated in the main text. Directly mapping existing error descriptions does not by itself resolve differences and establish consistent error semantics. This directly affects the validity of cross-source precision and recall.

  2. The operational annotation criteria, particularly for clean traces, are insufficiently specified. Although the authors provide L2 category definitions and independently audit clean labels with two annotators, the manuscript does not clearly explain how boundary cases such as reasonable exploration, harmless redundancy, and transient external failures are treated. These distinctions determine whether a trajectory qualifies as clean and directly affect the headline false-error rates.

  3. The evidence for length-related degradation is limited by the size and reliability of the long clean-trace subsets. As the authors acknowledge, undetected errors in long traces may inflate false-error rates. The small samples within comparable task environments further increase sensitivity to label noise, weakening the interpretation of the length gap as judge degradation.

  4. The experimental coverage across backbones is limited, and the evaluated judge configurations require clarification. Backbone transfer is examined only for the prompted baseline, leaving the robustness of the other judge designs and their relative performance unclear. Although implementations are available in the released code, the judge configurations used in the reported experiments are not sufficiently clear. Explicitly documenting the prompts, model settings, and aggregation rules would improve reproducibility and interpretation.

Questions for authors

Given the limited accuracy of the evaluated LLM judges, were there additional safeguards used to ensure annotation accuracy when LLM-proposed labels were reviewed by a single annotator? Was this annotation procedure independently validated beyond the clean-trace audit?

How did the authors verify that the judge prompts accurately reflect the decision criteria for each label? Were the prompts refined through prompt engineering or validated through ablation studies?

Reviewer_6EyzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses fragmentation of benchmarks used to validate LLM-based agent-trace judges, each with its own taxonomy and error distribution. The authors introduce TraceJudgeBench, 846 native-format traces from ten judge-validation datasets and three trace corpora, with labels mapped into a shared taxonomy of ten L2 and five L1 categories. The authors also add 231 new traces with two annotators. Evaluating a prompted LLM, TRAIL, AEGIS and an ensemble on one backbone, the authors find false-error ranges from 27.2% to 73.5% for the established judges (100% for the ensemble), rising by 44 and 50 points on long clean traces for the prompted baseline and TRAIL.

Critical strengths
  • The paper tackles a highly relevant problem, of validating trace judges outside their own benchmark. I especially found the audited clean-trace analysis useful, which no single source supports, and over-flagging persists across three backbones. Preserving the original traces while making the category mappings explicit provides a good basis for comparing their coverage and identifying categories that the original judge could not express.
  • I like the fact that the paper calibrates its numbers against a random judge at matched label volume (Table 9) and a paired bootstrap on the top two judges (Table 13).
Critical weaknesses
  • From my reading, the error labels are not exhaustive, while scoring is multi-label one-vs-rest. Several sources annotate only the first or decisive failure, and each free-text mistake_reason maps to a single L2 category. In the evaluation, a judge reporting a second, real error is counted as a false positive, so the low precision that Section 5.2 highlights may just be incomplete gold mappings.
  • As a result, it is unclear how well the benchmark discriminates judges at L2, its main use. Only 3 of 10 categories separate the top two judges (Table 13), the ensemble tracks a random judge (Section 5.2).
  • The paper is difficult to follow. The abstract and Section 5 are dense with numbers and caveats, and no clear takeaways. The main result (Table 1) is a large table of three-decimal values without a summary, and the reader has to go to the appendix to understand which judge does what well. The paper would benefit from stating a few clear takeaways and moving the coverage figures (Figures 3 and 4) into the main text.
Questions for authors
  1. Can the authors audit a sample of predictions counted as false positives on error traces and report what fraction are real errors missing from the gold labels? Also, how do results change without the 30 meta-only error traces (Section 3.4)?
  2. Could the authors discuss how a practitioner should choose between judges, given that most per-category differences are not separated and ensemble is not well motivated from the evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.