- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes EvoJudge, a framework that improves a frozen LLM judge by learning two libraries of reusable guidance from labeled examples: procedural skills describing how to evaluate responses and task-specific rubrics specifying evaluation criteria. The framework generates and consolidates this guidance, periodically checks validation performance, and uses selective rollback and corrective feedback when performance degrades. At inference time, retrieved skills and a task-type rubric guide the judge without updating model parameters. Experiments on VerifyBench-Hard, RewardBench2, and RM-Bench-Hard report accuracy improvements across multiple judge backbones. Additional analyses examine component contributions, periodic validation, cross-model transfer, and generated output length.
Critical strengths
- Practical improvement without model parameter updates EvoJudge learns reusable evaluation guidelines that can be applied to frozen judge models. It offers a practical paradigm for evaluator tuning in scenarios where fine-tuning is unavailable or undesirable, extending the potential applicability of this method beyond the specific models tested in the work.
- Empirical gains across multiple evaluation settings The proposed method improves the performance of two Qwen-based judges on all three benchmarks. For Qwen3.5-35B-A3B, it also outperforms the evaluated baseline and skill-based alternatives on each benchmark. These results demonstrate that the framework can enhance judgment accuracy.
Critical weaknesses
The independent contribution of the dual-evolution design is insufficiently established. The paper lacks budget-matched comparisons with directly optimizing the judge prompt, such as GEPA, or evolving a unified experience library. These comparisons are important because the gains could arise from additional guidance and supervised feedback rather than the proposed separation into skills and rubrics. Moreover, Section 4.4 describes inference-time component ablations, but the library counts in Table 6 leave unclear whether these settings use components of the same evolved libraries or independently evolved ones. This ambiguity limits the ablations’ ability to establish the value of the central design.
The coverage and robustness of the learned experience under heterogeneous task distributions remain unclear. The experiments use 200 induction examples and 100 validation examples per benchmark, without sufficiently examining sensitivity to sample selection, supervision size, or task composition. In a heterogeneous distribution, this limited sample may underrepresent rare evaluation requirements, and guidance useful for some categories may interfere with others. Aggregate improvements and cross-model transfer do not establish robustness to these conditions. This limits the evidence that the learned experience can reliably support diverse judging tasks beyond the particular sampled settings.
Questions for authors
Under the same backbone, labeled-data access, and optimization budget, how does EvoJudge compare with optimizing only the judge prompt, for example using GEPA, or evolving a unified experience library containing both procedures and criteria?
How sensitive are the results to the number and selection of induction examples and to the composition of mixed-task data? Per-category results across different sampling seeds and budgets would help assess coverage of underrepresented tasks. Please also clarify whether induction, validation, and test sets are separated by original question ID, keeping related response variants within the same split.
What is the end-to-end cost when including experience generation and consolidation, repeated validation, retrieval, and additional input tokens? Relative to the relevant baselines, how many subsequent judgments are required to amortize the upfront evolution cost?