- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper splits an LLM-judge prompt into a criterion (what to detect) and a scoring schema (how to decide), then uses GEPA to compare optimization regimes that edit these parts separately, jointly, or in phases. Across four frozen 8B–31B judges and twelve binary judging tasks, it reports that this decomposition beats optimizing the prompt as one block, and that it lifts an 8B judge on a financial-compliance benchmark from 0.70 to 0.92 F1.
Critical strengths
- Splitting the judge prompt into a criterion that defines what to judge and a scoring schema that defines how to decide is reasonable and intuitive, and the optimized result is human-readable, auditable rule text.
- Experiments show that GEPA-based prompt optimization clearly improves over zero-shot prompting, especially for weak judges and in the FinProof financial-compliance setting, where F1 rises from 0.70 to 0.92.
Critical weaknesses
- The method applies GEPA without modification, and its joint mode, which edits the criterion and the scoring schema in turn, is essentially GEPA's existing round-robin optimization over multiple prompt modules.
- Compared with plain GEPA on the whole prompt, decomposition adds only 0.003 and 0.004 F1 on gemma-4-31b and granite-guardian-4.1-8b, which is below the paper's own run-to-run noise, and at matched compute plain GEPA is even slightly better.
- The paper never tests the teacher models directly as judges, even though qwen3-next-80b-a3b activates only 3B parameters per token and could be a stronger judge at similar cost; the headline FinProof and ATBench results also use three times the search budget with no plain GEPA control at that budget.
- Since only GEPA is tested and the criterion-plus-scoring-schema format is the native prompt template of Granite Guardian, it is unclear whether the benefit of decomposition holds for other optimizers or judge prompts.
Questions for authors
See Weaknesses.
2 comments
Thanks for the authors' response. Since my concerns are not fully addressed, I decide to maintain my original rating.
Thank you for the review. Please find below the responses to the concerns raised in the review::
W1 - Existing GEPA mechanism : That is correct. Joint uses GEPA’s existing round-robin component updates [R1]. The blanket single-string characterization in §2 was not precise for GEPA and MIPRO and the revised related-work section corrects it. Our contribution is applying criterion–schema decomposition to prompt optimization for more reliable judgments from frozen, single-call models, with comparisons of component compatibility and optimization schedules.
W2 - Small gains and matched budgets :
W3 - Teachers and high-budget controls.
W4 - Template and optimizer generality : The criterion/schema headings are native to Guardian, whose released helper already accepts both components [R5]. Joint and combined start from the same rendered seed prompt and are instructed to keep the same section headers, so neither regime gets a template the other lacks. Table 1's general Granite and Gemma judges extend model coverage. Our conclusions are limited to the tested models, this template and GEPA.
Please let us know if any further clarification would be helpful.