- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors studies how the evidence (traces, task-time skills, generated files, and executable behavior) for evaluating the agent trajectories should be exposed to the evaluator. The proposed pipeline has a MetaAgent that routes each trajectories to one or two task-specific micro-judges, and a structured OutcomeJudge that aggregates their findings into a schema-defined output. The paper reported the wide range of experimental results transparently.
Critical strengths
None
Critical weaknesses
-
The performance of judge is too limited. The balanced accuracy in Table 3 shows nearly the chance (50%), and most of results are lower than JudgeSkill. Furthermore, the experimental results only provide single-run, without variance of CI.
-
As the short reference list suggests, the paper does not include appropriate citations where the text needs them. While reading, it is often unclear whether a particular term comes from prior work or was proposed by the authors themselves, because no citation is given. Furthermore, the related work section is missing much of the comparison against similar works.
Questions for authors
- How does the proposed pipeline differ from prior agentic evaluator, AutoJudge [1], which also route trajectories to specialized judges and aggregate their findings?
[1] Zhidkovskaya, Alina Borisovna, et al. "AutoJudge: Automatic Generation of LLM-Based Judges from Execution Traces." Workshop on Failure Modes of Agentic AI at ICML 2026.