- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This work presents a framework for training a judge from search and coding trajectories. It combines step-level, dynamic-prefix, final-outcome, and pairwise supervision with sequential SFT–DPO–GRPO training. A retrospective procedure uses complete trajectories to revise earlier judgments with revisions intended to rely only on prefix-visible evidence. The work also describes a bidirectional loop in which judge feedback improves the agent and newly generated trajectories support further judge training. Experiments report improvements over the base model on general and trajectory judging benchmarks, alongside gains over two co-evolution rounds.
Critical strengths
The paper reports a strong result on Plan-RewardBench and improvements in looping precision and trajectory-optimality accuracy on AgentRewardBench. These results suggest potential value in the proposed training approach, subject to the evaluation clarifications discussed below.
Critical weaknesses
1. The methodological contribution is insufficiently distinguished from prior work or isolated experimentally.
The following works provide related methodological precedents, although they differ in their domains, training targets, and specific implementations.
| Component and location in the paper | Related work |
|---|---|
| §3.4, Eq. (8); §4.5, Table 6: Bidirectional updates support agent–judge co-evolution. | UI-Genie and AgentPRM already iteratively update agent policies and their reward models. |
| §3.1, Eqs. (1)–(2); §3.3: Joint supervision develops local, dynamic, and global judging capabilities. | UI-Genie unifies action- and task-level rewards; HPL combines trajectory-, action-group-, and step-level preferences for agent training. |
| §3.2, Eq. (3): Complete trajectories help identify earlier misjudgments and improve prefix-based judgments. | CriticSearch uses a frozen critic with access to complete trajectories and gold answers to produce retrospective turn-level rewards for policy training. |
| §3.2, Eqs. (3)–(4), Figure 2: Rewriting and dual validation aim to ground revised judgments in prefix-visible evidence. | FLARE, a concurrent work, performs causal-prefix rewriting and semantic/causal-compliance checks. |
| §3.3, Figure 3; §4.4, Tables 4–5: SFT, DPO, and GRPO provide complementary training stages. | FairJudge already uses SFT→DPO→GRPO; EvalPlanner and J1 train reasoning-based judges. |
| §1, Contribution 2; §3.3; §4.3: Training a unified generative agent judge. | Agent-RRM already uses SFT→GRPO to train an agent evaluator that produces reasoning, critiques, and scores. |
Although Tables 4–5 ablate the training stages, they do not isolate the contributions of multi-granularity supervision, full-trajectory retrospection, future masking, or dual validation within the pipeline.
2. Important supervision, training, and evaluation details remain unclear.
The multi-stage workflow makes it challenging to document every experimental detail, but certain key settings and results are still necessary to assess the validity of the method and conclusions.
2.1 §3.1 and Appendix A identify trajectory sources, but do not clearly specify how reference judgments and rationales were generated. §3.2 describes a dual-acceptance rule but does not specify the teacher and validator identities, their prompts and assessment criteria, or candidate counts and acceptance rates. This makes the quality of the 311 DPO pairs—and the claim that revised judgments rely only on prefix-visible evidence—difficult to assess.
2.2 §3.4 does not specify the agent optimization algorithm, how judge feedback enters the update, or the data and training budget per round. §4.5 additionally leaves the evaluation set, sample size, and definition of “Judge Accuracy” unclear. These omissions limit both reproducibility and attribution of the gains in Table 6.
2.3 §§4.1–4.3 do not provide the evaluation prompts, output-parsing and tie-handling rules, the averaging convention for Table 1’s P/R/F1, or the RewardBench aggregation procedure. For example, Table 1 reports 46.57 for JudgeLM-7B on RewardBench, whereas its original paper reports 78.5. The evaluation configurations underlying this discrepancy need clarification. JudgeLM, Table 17
2.4 The main quantitative trajectory evaluations concern complete trajectories, whereas §§3.1–3.2 emphasize dynamic, prefix-based assessment. Separate quantitative step- and prefix-level evaluations are needed to substantiate the claimed ability to identify deviation, recovery, and accumulated risk during execution.
3. The experiments do not yet establish a clear advantage over stronger trained judges.
3.1 Table 1 omits recent, comparably sized judges. AgentJudge reports a RewardBench score of 82.55, while FairJudge-8B reports 84.79 agreement and J1-Llama-8B reports 85.7 under their respective evaluation protocols. These are cross-paper results rather than controlled comparisons, but they motivate comparisons under a shared protocol before attributing additional value to the proposed pipeline. FairJudge, J1
3.2 On AgentRewardBench, Table 3 reports an Overall improvement of 1.30 points over the strongest listed baseline. Gains concentrate on looping precision and optimality accuracy; success precision does not lead, and side-effect precision is the lowest in Table 3. Without recall or F1, detection coverage remains unresolved. Comparisons with trained trajectory evaluators such as World-State-Model-7B on task-success judgment, with matched inputs and evaluation settings, would strengthen this evaluation. AgentRewardBench leaderboard
3.3 The reported 75.78 on Plan-RewardBench is strong, but the paper does not specify whether it follows the original protocol of evaluating both A/B orders and averaging correctness. Furthermore, calculations from Table 4 show that the Safety subset accounts for approximately 74% of the net gain in the seven-subset macro-average over the base model. Adding GRPO increases the overall average by 2.55 points, while the mean over the other six subsets decreases from 75.30 to 75.01. This supports a substantial Safety improvement, but provides weaker evidence for broadly improved trajectory judging. Plan-RewardBench protocol, §4
Questions for authors
- How were the reference judgments generated, and which models, prompts, and criteria were used for retrospective revision and dual validation? What evidence verifies that accepted revisions remain prefix-grounded?
- What are the concrete agent and judge update procedures in Eq. (8), and how are data, computation, evaluation sets, and checkpoint selection controlled across the three branches in Table 6?
- Did the Plan-RewardBench evaluation use both candidate orders and the original input protocol? What explains the JudgeLM-7B RewardBench discrepancy, and how are the Table 1 metrics aggregated?
- Can you provide targeted comparisons that isolate multi-granularity supervision and retrospective revision, together with direct prefix-level evaluation and a recent strong trained-judge baseline?
Thank you for your thoughtful response. The clarifications have addressed my concerns, and I will maintain my positive rating.