acceptodds
Under review as a conference paper at ICLR 2027

From Traces to Judges: Automatic Design of Evaluation Pipeline for LLM-Based Multi-Agent Systems

Abstract

Evaluating multi-agent systems from their execution traces is hard: traces are long, failure modes are diverse, and existing methods rely on fixed, benchmark-specific judges or manual prompt engineering, so each new benchmark needs a new judge. We introduce AutoJudge, a framework that constructs an adaptive pool of LLM judges inside a fixed evaluation graph. Given a trace, an evaluation taxonomy, and an output schema, a MetaAgent writes the roles and instructions of a set of intermediate judges and a final aggregation judge, enabling input-adaptive evaluation without task-specific training or judge design. To handle long traces, we propose a dual strategy combining full-trace evaluation with a summary-based mode and controlled step-level access to the original trace. Across seven benchmarks with heterogeneous taxonomies, AutoJudge matches or exceeds the published benchmark-specific evaluator on 14 of 19 within-benchmark metrics with a fixed, low-cost judging backbone and no benchmark-specific judge design, while specialized judges with stronger backbones remain ahead on several metrics. Results are stable across repeated runs, and the framework runs at inference time without training. Our code is available at https://anonymous.4open.science/r/AutoJudge-B0E0.

Primary area:
interpretability and explainable AI
Keywords:
llm, llm-as-judge, multi agent systems, failure attribution in agents, evaluation pipelines, scalable evaluation, agentic AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2DSFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a framework, AutoJudge, which is training-free framework to evaluate multi-agent systems. This framework provides both full-trace evaluation and summary-based evaluation. Experimental results demonstrate its practical applicabilibty to different evaluation benchmarks.

Critical strengths
  • This paper proposes and addresses a practical evaluation problem, which reduce the manual judgement design effort across different LLM-based Multi-Agent systems.

  • The overall evaluation covers multi-type benchmarks and also provides different baseline comparisons. And the experimental results demenstrate the effectiveness of the proposed method.

Critical weaknesses
  • W1: The contribution of trace-adaptive judge generation is not fully isolated. While in Table 2, the paper matches the judging backbone, AutoJudge additionally uses Gemini 3 Flash Preview as its MetaAgent. Comparisons against fixed judge pools with matched models are needed to distinguish the benefit source;

  • W2: The evaluation protocol is not very clear. The paper does not specify whether the 58 traces used for configuration selection are separate from the final test set. Agent Accuracy is also described inconsistently between the main paper and Appendix B.

Questions for authors

See weakness.

Reviewer_mELEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AutoJudge generates an evaluation pipeline from an agent execution trace, an evaluation taxonomy, and an output schema. A MetaAgent writes the roles and instructions of intermediate judges and an aggregation judge, which combines their outputs into the final evaluation. The execution graph remains fixed. The method supports full-trace evaluation and a summary mode with retrieval of individual original steps. Experiments cover seven benchmarks and compare trace-specific generation, reusable judge pools, and existing evaluation pipelines.

Critical strengths

1-The framework addresses a practical evaluation problem. Separating the taxonomy and output schema from the generated judge instructions makes the system adaptable to different attribution and error-detection tasks. The summary-and-retrieval option also addresses the cost of processing long traces.

2-The evaluation covers several benchmarks with different annotation schemes. The Who&When comparison includes reruns with a shared judging backbone, and the paper reports cases where existing evaluators remain stronger. The per-trace versus reusable-pool comparison, model ablations, and cost estimates provide useful evidence about the complete system.

Critical weaknesses

1-The incremental benefit of generating judge roles is not clearly isolated.

2-AgentEval already generates evaluation criteria from tasks and execution examples [1]. Agent-as-a-Judge uses active evidence gathering for agent evaluation [2], and Auto-Eval Judge combines generated checklists, evidence extraction, verification planning, and final verdicts [3]. Agentic CLEAR also generates task-specific rubrics and provides multi-level trace analysis [4]. These works narrow the paper’s positioning against fixed, manually designed evaluators.

3-AutoJudge’s specific distinction is generating judge roles and instructions for each trace. Table 8 supports trace-specific conditioning over dataset-level reuse, but does not establish why generating multiple judge roles is better than generating a trace-specific checklist or verification plan. Table 2 aligns the executing judges while additionally using Gemini 3 Flash Preview for AutoJudge’s MetaAgent. Table 10 usefully includes a configuration with Gemini 2.5 Flash for both the MetaAgent and judges, but a direct comparison with simpler generated-evaluation alternatives under matched overall model allocations and inference budgets is still missing. The practical value of the full pipeline is plausible; the added value of its central design remains insufficiently established.

4-Configuration selection and evaluation rules need clarification.

5-Appendix B uses 58 handcrafted traces to select the prompting configuration, but their separation from the final Who&When evaluation traces is not clearly documented. It also describes Agent Accuracy as trajectory-level task-success classification, whereas Section 4.2 identifies the benchmark target as attribution to the failure-responsible agent. The authors should clarify whether this is a wording error or whether a different metric was used for configuration selection.

6-The retrieval description has a similar ambiguity. Section 3.4 appears to give the aggregator retrieval access in summary mode, while Appendix A explicitly instructs FINAL_AGGREGATOR not to use tools. The actual execution permissions should be specified. A summary-only control would also help distinguish the effects of summarization and retrieval. These details affect the interpretation and reproducibility of the results.

References:

[1] Arabzadeh et al. Assessing and Verifying Task Utility in LLM-Powered Applications. EMNLP, 2024. arXiv:2405.02178.

[2] Zhuge et al. Agent-as-a-Judge: Evaluate Agents with Agents. ICML, 2025. arXiv:2410.10934.

[3] Bhonsle et al. Auto-Eval Judge: Towards a General Agentic Framework for Task Completion Evaluation. 2025. arXiv:2508.05508.

[4] Yehudai, Eden, and Shmueli-Scheuer. Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents. 2026. arXiv:2605.22608.

Questions for authors

1-What does trace-specific judge-role generation add beyond generated criteria, checklists, and verification plans, as explored in AgentEval, Auto-Eval Judge, and Agentic CLEAR?

2-Building on the model ablations in Table 10, how does AutoJudge compare with those alternatives under the same overall model allocation and comparable inference budgets?

3-How are the 58 configuration-selection traces separated from the final evaluation traces?

4-Is Appendix B’s description of Agent Accuracy a wording error, or was a task-success metric used instead of failure-responsible-agent attribution for configuration selection?

5-Can the aggregator retrieve original steps in summary mode, or is retrieval restricted to intermediate judges? How does performance change when retrieval is disabled?

6-Which stages are included in Appendix E’s aggregate costs—judge generation, summarization, retrieval-related calls and tokens, aggregation, and retries—and what are the corresponding costs for the baseline pipelines?

Reviewer_9BvmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AutoJudge, a practical, inference-time framework for automatic generation of LLM based judges from execution traces. The framework also supports summarization and retrieval for long traces. Experiments on seven benchmarks show that AutoJudge matches or outperforms the reported benchmark-specific baselines on 14 of 19 metrics.

Critical strengths
  1. The problem is useful in practice. Evaluating long multi-agent execution traces is challenging, and a framework that can handle different evaluation tasks for each benchmark could be valuable.

  2. The experiments coverage is broad. The paper discusses seven benchmarks with different evaluation targets. The paper also compares different judge generation strategies and includes experiments with the same judge backbone. The results provide some evidence that the framework can work across different tasks.

Critical weaknesses
  1. Limited science and technical depth. The method combines existing techniques, including LLM-generated prompts, multiple specialized judges, a final aggregation judge, and summarization with retrieval. The overall evaluation graph is fixed, and the MetaAgent only generates judge roles and instructions. The paper is more about a practical engineering workflow than a new research method. The paper does not clearly explain what new insight can be learned beyond existing LLM evaluation frameworks.

  2. The experiments are not able to fully support the core contribution. The core claimed contribution is that generating judges for each trace improves evaluation. However, Table 8 compares this approach against a reusable judge pool generated from only three traces, which may not be a strong baseline. Table 2 uses the same judge backbone, but AutoJudge also requires a separate MetaAgent to generate its judges. The paper does not provide a comparison with a strong fixed judge pool under the same overall inference budget. It is unclear how much of the improvement is from adaptive judge generation itself. Some results also are weak, such as the 10.2% step accuracy on AgenTracer Code compared with 18.85% for the published baseline.

Questions for authors
  1. Can the authors compare AutoJudge with a well-designed fixed judge pool using the same models, number of judges, and overall inference budget? This would help to understand if generating judges for each trace provides a real advantage.

  2. How much does each part of the framework contribute? Does dynamically generate judge roles improve performance beyond using fixed roles with dynamically generated instructions? How much does retrieval help compared with summarization alone?

  3. Appendix B describes Agent Accuracy as task-success classification, but Section 4.2 uses it to measure whether the responsible agent is correctly identified. Could the authors clarify which definition was used in the ablation experiments? Section 3.4 suggests that the final judge can revisit the original trace, but Appendix A says it should not use tools. How does this work in summary mode?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.