- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces TraceJudgeBench, a meta-analysis that harmonizes ten LLM-as-a-judge datasets and three unannotated trace corpora into a single frame. The native or original labels were mapped into a shared two-level taxonomy of ten fine-grained and five coarse categories. Therefore this new taxonomy frame establishes a unification of labels between the judge datasets enabling old and new evaluation methods to be cross-validated against one another and against the original benchmarks.
Critical strengths
- The paper addresses an important and timely problem of unifying fragmented LLM-as-a-judge benchmarks through the conversion to L1/L2 taxonomy labels that enable cross-validation across previously distinct judge benchmarks.
Critical weaknesses
- The author should elaborate on the judgment methods in Table 1, as they currently seem a bit simplistic. Additionally, some more details about the different judges and elaboration on the ensemble judge would help.
- The motivation of the work is good, but the study is narrow as it is only conducted on the typical LLM-as-a-judge approach for agent traces that are within context length limit. Authors should consider extending the scope to cover more advanced judge methods like agentic judges meant for traces that exceed context length limit. Agentic judges are more relevant in the community today since a majority of the agent traces are very long especially for skill-based agents which are gaining traction in the recent year (see more details on this in the question section below).
Questions for authors
- Why the ensemble method is omitted from Table 2?
- Can you clarify and elaborate more on the different judges that is being compared especially some details on AEGIS, TRAIL, and the ensemble judge such that the paper is self-contained.
- The authors claimed that for clean-trace false errors grow with trace length which is not surprising at all when the judge used is a typical standard LLM-as-a-judge method. However, it remains questionable whether this finding generalizes to more sophisticated judge methods designed for very long traces such as agentic judges which can intelligently use tools to inspect part of the trace.
- Some parts of the paper such as Section 5 is hard to read and can benefit from a rewrite.