- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes the new method GraphRubric, a lightweight graph controller conditioned on the input case that optimizes rubric presentation to improve agreement between judge scores and human labels. Rubrics are encoded as graph nodes that exchange information through graph attention; GRPO-style sampling converts within-group rewards into advantages for directed edges in their execution order. Authors use three DeepSeek training seeds with fixed initialization, supplemented by single seed GLM and Gemini experiments. The primary metric is MAP/R3, which evaluates each case’s highest-probability path three times; policy-sampled MC8 is secondary. Across three datasets (Summeval, HANNA, USR), GraphRubric shows generally favorable performance relative to the random and empty-rubric controls and the reported validation-selected fixed-path baselines.
Critical strengths
Very detailed reproducibility information in the appendix, all datasets are open, and prompts and parameters are described.
The research question idea is sound and well developed. The practical advantage of GraphRubric is that they optimize not the criteria themselves, but the configuration of how they are used together. At the same time, the LLM judge's weights are not changed, so after training the method can be used with a standard API. The method is well suited for offline calibration of the evaluation prompt for a frequently used rubric.
Critical weaknesses
Would be interesting to add also not graph-based baselines. For example, GEPA with a fixed set of initial rubrics / GEPA with a limited operation space (this is a direct comparison of search algorithms within the same solution space). It would also help to report the training costs and compare the final results of the proposed methods.
Questions for authors
—
Fixed-text GEPA comparison: shared search space, alignment, and token costs
We thank the reviewer for the positive assessment of GraphRubric and its Graph-GRPO training mechanism, and for the practical value of calibrating rubric configurations with a frozen judge. Your suggestion to add a non-graph prompt-optimization baseline and compare both predictive performance and training cost motivated the following experiments.
GEPA in the same selection and ordering space. Following your suggestion, we added a restricted version of the official GEPA optimizer (v0.1.4): the original text of every rubric item is fixed, and the optimizer can only select a nonempty subset and reorder its items. Both methods therefore share the same legal rubric-path space, current DeepSeek judge, scoring prompts, and original data partitions. GEPA starts from all rubric items in their original order, uses training examples for reflection, and selects one global fixed path by MSE on the full validation set.
The GraphRubric training described in the paper uses only the training split. To give both methods access to the same labeled data in this comparison, we additionally trained GraphRubric for one pass over the combined training and validation splits. Both methods were evaluated on the same original test set.
Test results. GraphRubric uses MAP/R3; GEPA evaluates its selected fixed path three times per case. We compute squared errors for each call and average across target dimensions, the three evaluations, and test cases. USR uses dimension-range-normalized MSE over five dimensions, excluding overall. Lower is better.
Training budget and cost. We count one case-by-candidate evaluation as one logical scoring unit. GEPA's cumulative official evaluation thresholds match the scoring slots in GraphRubric's training pass: 5,120 for SummEval-4 (320 × 16), 24,640 for HANNA (770 × 32), and 11,550 for USR (462 × 25).
GEPA checks its stopping threshold at iteration boundaries, accounting for the additional 84 evaluations on USR.
Token estimates charge every counted scoring slot the input-plus-output usage of its matching successful receipt. Each method is accounted for independently; final TEST evaluation is excluded.
Including reflection token, GEPA's estimated token consumption under this logical-scoring accounting is 1.44–1.50 times that of GraphRubric.