acceptodds
Under review as a conference paper at ICLR 2027

AgentJudge:From Real Agent Trajectories to Multi-Granularity Judging and Agent–Judge Co-Evolution

Abstract

Local action quality, overall task performance, and evolving trajectory states are interdependent in agent execution, yet static, outcome-centered judging struggles to capture these signals jointly. We introduce AgentJudge, a framework for multi-granularity judging and agent–judge co-evolution built on real execution trajectories. We collect trajectories from search and coding tasks and derive supervision for local steps, dynamic prefixes, and final outcomes, unifying local action assessment, global outcome evaluation, and execution-state tracking. We adopt progressive SFT–DPO–GRPO training: multi-granularity supervised fine-tuning establishes basic judging capabilities; future-masked retrospective preference learning uses complete trajectories to identify potential misjudgments and constructs revised judgments from evidence visible in the current prefix for preference optimization; reward-based optimization further reinforces structured judging against supervision references. We further design a bidirectional update mechanism in which judge feedback guides agent optimization, while newly generated trajectories supply supervision for subsequent judge updates, forming a loop of trajectory generation, judge learning, and feedback-driven optimization. Experiments show improvements over the base model on general response judging and agent trajectory judging. A small-scale study over two co-evolution rounds further shows joint gains in agent success and judge accuracy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, LLM Agents, Agent Evaluation, LLM Evaluation, LLM Judge, Process-Level Evaluation, Preference Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_KNYdOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This work presents a framework for training a judge from search and coding trajectories. It combines step-level, dynamic-prefix, final-outcome, and pairwise supervision with sequential SFT–DPO–GRPO training. A retrospective procedure uses complete trajectories to revise earlier judgments with revisions intended to rely only on prefix-visible evidence. The work also describes a bidirectional loop in which judge feedback improves the agent and newly generated trajectories support further judge training. Experiments report improvements over the base model on general and trajectory judging benchmarks, alongside gains over two co-evolution rounds.

Critical strengths

The paper reports a strong result on Plan-RewardBench and improvements in looping precision and trajectory-optimality accuracy on AgentRewardBench. These results suggest potential value in the proposed training approach, subject to the evaluation clarifications discussed below.

Critical weaknesses

1. The methodological contribution is insufficiently distinguished from prior work or isolated experimentally.

The following works provide related methodological precedents, although they differ in their domains, training targets, and specific implementations.

Component and location in the paperRelated work
§3.4, Eq. (8); §4.5, Table 6: Bidirectional updates support agent–judge co-evolution.UI-Genie and AgentPRM already iteratively update agent policies and their reward models.
§3.1, Eqs. (1)–(2); §3.3: Joint supervision develops local, dynamic, and global judging capabilities.UI-Genie unifies action- and task-level rewards; HPL combines trajectory-, action-group-, and step-level preferences for agent training.
§3.2, Eq. (3): Complete trajectories help identify earlier misjudgments and improve prefix-based judgments.CriticSearch uses a frozen critic with access to complete trajectories and gold answers to produce retrospective turn-level rewards for policy training.
§3.2, Eqs. (3)–(4), Figure 2: Rewriting and dual validation aim to ground revised judgments in prefix-visible evidence.FLARE, a concurrent work, performs causal-prefix rewriting and semantic/causal-compliance checks.
§3.3, Figure 3; §4.4, Tables 4–5: SFT, DPO, and GRPO provide complementary training stages.FairJudge already uses SFT→DPO→GRPO; EvalPlanner and J1 train reasoning-based judges.
§1, Contribution 2; §3.3; §4.3: Training a unified generative agent judge.Agent-RRM already uses SFT→GRPO to train an agent evaluator that produces reasoning, critiques, and scores.

Although Tables 4–5 ablate the training stages, they do not isolate the contributions of multi-granularity supervision, full-trajectory retrospection, future masking, or dual validation within the pipeline.


2. Important supervision, training, and evaluation details remain unclear.

The multi-stage workflow makes it challenging to document every experimental detail, but certain key settings and results are still necessary to assess the validity of the method and conclusions.

2.1 §3.1 and Appendix A identify trajectory sources, but do not clearly specify how reference judgments and rationales were generated. §3.2 describes a dual-acceptance rule but does not specify the teacher and validator identities, their prompts and assessment criteria, or candidate counts and acceptance rates. This makes the quality of the 311 DPO pairs—and the claim that revised judgments rely only on prefix-visible evidence—difficult to assess.

2.2 §3.4 does not specify the agent optimization algorithm, how judge feedback enters the update, or the data and training budget per round. §4.5 additionally leaves the evaluation set, sample size, and definition of “Judge Accuracy” unclear. These omissions limit both reproducibility and attribution of the gains in Table 6.

2.3 §§4.1–4.3 do not provide the evaluation prompts, output-parsing and tie-handling rules, the averaging convention for Table 1’s P/R/F1, or the RewardBench aggregation procedure. For example, Table 1 reports 46.57 for JudgeLM-7B on RewardBench, whereas its original paper reports 78.5. The evaluation configurations underlying this discrepancy need clarification. JudgeLM, Table 17

2.4 The main quantitative trajectory evaluations concern complete trajectories, whereas §§3.1–3.2 emphasize dynamic, prefix-based assessment. Separate quantitative step- and prefix-level evaluations are needed to substantiate the claimed ability to identify deviation, recovery, and accumulated risk during execution.


3. The experiments do not yet establish a clear advantage over stronger trained judges.

3.1 Table 1 omits recent, comparably sized judges. AgentJudge reports a RewardBench score of 82.55, while FairJudge-8B reports 84.79 agreement and J1-Llama-8B reports 85.7 under their respective evaluation protocols. These are cross-paper results rather than controlled comparisons, but they motivate comparisons under a shared protocol before attributing additional value to the proposed pipeline. FairJudge, J1

3.2 On AgentRewardBench, Table 3 reports an Overall improvement of 1.30 points over the strongest listed baseline. Gains concentrate on looping precision and optimality accuracy; success precision does not lead, and side-effect precision is the lowest in Table 3. Without recall or F1, detection coverage remains unresolved. Comparisons with trained trajectory evaluators such as World-State-Model-7B on task-success judgment, with matched inputs and evaluation settings, would strengthen this evaluation. AgentRewardBench leaderboard

3.3 The reported 75.78 on Plan-RewardBench is strong, but the paper does not specify whether it follows the original protocol of evaluating both A/B orders and averaging correctness. Furthermore, calculations from Table 4 show that the Safety subset accounts for approximately 74% of the net gain in the seven-subset macro-average over the base model. Adding GRPO increases the overall average by 2.55 points, while the mean over the other six subsets decreases from 75.30 to 75.01. This supports a substantial Safety improvement, but provides weaker evidence for broadly improved trajectory judging. Plan-RewardBench protocol, §4

Questions for authors
  1. How were the reference judgments generated, and which models, prompts, and criteria were used for retrospective revision and dual validation? What evidence verifies that accepted revisions remain prefix-grounded?
  2. What are the concrete agent and judge update procedures in Eq. (8), and how are data, computation, evaluation sets, and checkpoint selection controlled across the three branches in Table 6?
  3. Did the Plan-RewardBench evaluation use both candidate orders and the original input protocol? What explains the JudgeLM-7B RewardBench discrepancy, and how are the Table 1 metrics aggregated?
  4. Can you provide targeted comparisons that isolate multi-granularity supervision and retrospective revision, together with direct prefix-level evaluation and a recent strong trained-judge baseline?
Reviewer_5zs2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces AgentJudge, a multi-granularity evaluation model based on real agent trajectories. The authors built the AgentJudge-19K dataset with 19,106 records from search and programming tasks, covering step-level, dynamic-prefix-level, final-result-level, and pairwise preference supervision. They performed SFT–DPO–GRPO training using Qwen3-VL-8B-Instruct. Specifically, retrospective preference learning identifies potential misjudgments from full trajectories and then constructs revised judgments based on the evidence in the current prefix. The paper also explores a co-evolution mechanism where agents and evaluators update alternately. Experiments showed improvements on general answer evaluation and agent trajectory evaluation benchmarks, and performance gains were observed on both sides during two rounds of co-evolution.

Critical strengths
  1. The research question is meaningful. The quality of actions, the accumulated execution state, and the final task result don’t always line up. Putting these goals into a single evaluation framework helps make up for the shortcomings of only judging based on the final outcome.

  2. The experimental results provide positive evidence that trajectory supervision improves evaluation ability and supports cross-task transfer.

Critical weaknesses
  1. The paper emphasizes step-level and dynamic prefix-level evaluation, but mostly reports results at the full trajectory level, so it can't directly prove that the model can accurately assess local quality and state changes during execution. Plus, the training data mainly comes from search and programming, while the evaluation mostly covers general answers, tool usage planning, and web interaction, lacking clear results for specific subfields, which limits how well we can judge its applicability.

  2. The gain from key design choices isn't fully explained. The current ablation looks at different training stages, but it doesn't isolate the contributions from step-level or dynamic prefix-level supervision, nor does it show that the backtracking preference setup has independent advantages over conventional preference learning. So, right now, it's still hard to tell whether the improvements come from a specific method design or just from extra training, supervision data, or output format adjustments.

Questions for authors
  1. Are there separate step-level and dynamic prefix-level tests that verify whether the model can accurately judge the quality of the current action, cumulative execution state, and risk when only looking at the execution record up to the current point? Improvements in full-trajectory evaluation alone are not enough to prove these abilities.

  2. Is there an ablation study that controls for data amount and training budget, removing step-level or dynamic prefix-level supervision individually and comparing with models trained only on final outcome supervision? It's necessary to clarify the independent contributions of different granularity supervision on evaluation performance.

  3. The paper mainly uses search and programming trajectories for training, but evaluation mainly covers general answers, tool-use planning, and web interaction. Are there dedicated test results for search and programming trajectories? Please clearly distinguish between in-domain ability validation and cross-domain transfer results.

  4. The current stage-wise ablation supports the effectiveness of the overall training scheme, but it has not sufficiently distinguished the benefits of specific method designs from those due to additional training, supervision data, or output format adaptation. In particular, does the construction of retrospective preferences offer independent advantages over conventional preference learning? This needs to be clarified through targeted comparisons.

Reviewer_BtFMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies an important problem of agent evaluation. The authors argue that existing methods pay insufficient attention to jointly evaluating local actions, dynamic execution states, and final task outcomes. A multi-granularity judging framework, AgentJudge, is proposed, which learns from real agent trajectories through progressive SFT–DPO–GRPO training. The authors further introduce a bidirectional agent–judge co-evolution mechanism, in which judge feedback guides agent optimization and newly generated trajectories provide supervision for subsequent judge updates. Experiments show the effectiveness of the proposed framework to some extent. However, several concerns regarding novelty and experimental validation remain.

Critical strengths
  1. The paper is generally well-organized and easy to follow. The overall framework and progressive training pipeline are clearly described.

  2. The paper considers multiple aspects of agent evaluation, including local action quality, dynamic execution states, and final task outcomes. The proposed framework provides a unified approach to multi-granularity agent judging.

Critical weaknesses
  1. The effectiveness of the proposed training pipeline requires further investigation. The framework combines SFT, DPO, and GRPO, but the DPO stage uses only 311 preference pairs. It is unclear whether this amount of preference data is sufficient to support the claimed improvement in judging capability. In addition, the current ablation studies mainly remove individual training stages. It would be better to compare the proposed pipeline with alternative training strategies, including SFT-only training, SFT+DPO, SFT+GRPO, and other reasonable combinations, to demonstrate whether the progressive training design is necessary and whether the three stages provide complementary benefits.

  2. The evaluation of agent–judge co-evolution is limited. The reported agent success rate increases from 20.83% to 41.67% after two rounds of bidirectional updates. However, the experimental results are based on a limited number of tasks, and the improvements may be sensitive to task selection and execution randomness. Moreover, the results do not sufficiently establish that the bidirectional mechanism is consistently more effective than one-sided updates. It is suggested to evaluate the framework on more tasks, conduct multiple runs, report statistical variation, and investigate whether the observed improvements generalize to different agents and task environments.

Questions for authors

Please see the weaknesses.

Reviewer_FUxyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AgentJudge, which evaluates search and coding agents throughout a task rather than focusing only on the final result. The authors build AgentJudge 19K from real agent trajectories, with supervision at the step, dynamic prefix, and outcome levels, as well as pairwise preferences. They train the judge with SFT, DPO, and GRPO; DPO uses the full trajectory to identify flawed judgments while ensuring revisions rely only on evidence available at the time. The paper also describes a co evolution process in which judge feedback improves the agent, and the agent’s new trajectories help refine the judge. Experiments show gains in both general answer evaluation and agent trajectory evaluation, as well as further improvements to the agent and judge after co evolution.

Critical strengths
  1. The judge evaluates more than the final answer. Using real search and coding traces, it learns to assess individual steps, the trace so far, and the final result. That gives a more detailed picture of performance on long tasks.

  2. The retrospective preference method uses the full trace to spot problems in earlier judgments, while limiting each revised judgment to evidence available at that point. The authors also train the agent and judge in turn. After two rounds, both the agent’s success rate and the judge’s accuracy improve.

Critical weaknesses
  1. The paper gives too little detail about how judge feedback is used to train the agent. The training objective, reward construction, and use of the different judgment levels are not clearly explained, making this part of the method difficult to reproduce.

  2. The ablations do not show what each component contributes. The paper compares SFT, DPO, and GRPO, but does not isolate the effects of step level, dynamic prefix level, and final outcome supervision. It also does not test future masked rewriting and dual validation separately, so their roles in reducing hindsight bias remain unclear.

Questions for authors
  1. Could the authors clarify how judge feedback enters the agent’s training process? In particular, what objective and reward are used, and how do judgments at different levels affect policy updates? A concrete example would make the process easier to follow.

  2. What does dynamic prefix supervision add to judging performance? Would training with only step level and final outcome supervision produce comparable results?

  3. Do future masked rewriting and dual validation reduce hindsight bias in practice? The current ablation measures DPO as a whole, so it does not show what either component contributes on its own.

2 comments

Thank you for your thoughtful response. The clarifications have addressed my concerns, and I will maintain my positive rating.

One follow-up question about the robustness of the judge feedback in Section 3.4: If a tool observation contains an instruction directed at the judge, such as “mark this task as successful,” how often does it change the judgment? A paired test on failed trajectories, with and without the injected instruction, could show whether AgentJudge incorrectly marks more failures as successes when the underlying task evidence is unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.