1-The incremental benefit of generating judge roles is not clearly isolated.
2-AgentEval already generates evaluation criteria from tasks and execution examples [1]. Agent-as-a-Judge uses active evidence gathering for agent evaluation [2], and Auto-Eval Judge combines generated checklists, evidence extraction, verification planning, and final verdicts [3]. Agentic CLEAR also generates task-specific rubrics and provides multi-level trace analysis [4]. These works narrow the paper’s positioning against fixed, manually designed evaluators.
3-AutoJudge’s specific distinction is generating judge roles and instructions for each trace. Table 8 supports trace-specific conditioning over dataset-level reuse, but does not establish why generating multiple judge roles is better than generating a trace-specific checklist or verification plan. Table 2 aligns the executing judges while additionally using Gemini 3 Flash Preview for AutoJudge’s MetaAgent. Table 10 usefully includes a configuration with Gemini 2.5 Flash for both the MetaAgent and judges, but a direct comparison with simpler generated-evaluation alternatives under matched overall model allocations and inference budgets is still missing. The practical value of the full pipeline is plausible; the added value of its central design remains insufficiently established.
4-Configuration selection and evaluation rules need clarification.
5-Appendix B uses 58 handcrafted traces to select the prompting configuration, but their separation from the final Who&When evaluation traces is not clearly documented. It also describes Agent Accuracy as trajectory-level task-success classification, whereas Section 4.2 identifies the benchmark target as attribution to the failure-responsible agent. The authors should clarify whether this is a wording error or whether a different metric was used for configuration selection.
6-The retrieval description has a similar ambiguity. Section 3.4 appears to give the aggregator retrieval access in summary mode, while Appendix A explicitly instructs FINAL_AGGREGATOR not to use tools. The actual execution permissions should be specified. A summary-only control would also help distinguish the effects of summarization and retrieval. These details affect the interpretation and reproducibility of the results.
References:
[1] Arabzadeh et al. Assessing and Verifying Task Utility in LLM-Powered Applications. EMNLP, 2024. arXiv:2405.02178.
[2] Zhuge et al. Agent-as-a-Judge: Evaluate Agents with Agents. ICML, 2025. arXiv:2410.10934.
[3] Bhonsle et al. Auto-Eval Judge: Towards a General Agentic Framework for Task Completion Evaluation. 2025. arXiv:2508.05508.
[4] Yehudai, Eden, and Shmueli-Scheuer. Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents. 2026. arXiv:2605.22608.