acceptodds
Under review as a conference paper at ICLR 2027

GraphRubric: Learning Rubric Selection and Ordering for LLM-as-a-Judge

Abstract

Evaluation rubrics specify what an LLM judge should assess, yet prior work shows that rubric selection and ordering also affect its scores, even when prompts are semantically equivalent. We introduce GraphRubric, a lightweight graph controller conditioned on the input case that optimizes rubric presentation to improve agreement between judge scores and human labels. Rubrics are encoded as graph nodes that exchange information through graph attention; GRPO-style sampling converts within-group rewards into advantages for directed edges in their execution order. On SummEval, HANNA, and USR, three DeepSeek training seeds with fixed initialization reduce the primary error metric by 13.6%, 20.5%, and 5.8%, respectively, relative to uniform sampling over all nonempty paths. We also evaluate GLM and Gemini as judges. Across these datasets, GraphRubric shows generally favorable performance relative to the random and empty-rubric controls and the reported validation-selected fixed-path baselines. The method offers a prompt-engineering approach with potential applications to agent skill and harness optimization, reward-system construction, and user-data screening and cleaning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-Judge, Prompt Optimization, Graph Neural Networks, Reinforcement Learning, Automated Evaluation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_qGzHOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes the new method GraphRubric, a lightweight graph controller conditioned on the input case that optimizes rubric presentation to improve agreement between judge scores and human labels. Rubrics are encoded as graph nodes that exchange information through graph attention; GRPO-style sampling converts within-group rewards into advantages for directed edges in their execution order. Authors use three DeepSeek training seeds with fixed initialization, supplemented by single seed GLM and Gemini experiments. The primary metric is MAP/R3, which evaluates each case’s highest-probability path three times; policy-sampled MC8 is secondary. Across three datasets (Summeval, HANNA, USR), GraphRubric shows generally favorable performance relative to the random and empty-rubric controls and the reported validation-selected fixed-path baselines.

Critical strengths

Very detailed reproducibility information in the appendix, all datasets are open, and prompts and parameters are described.

The research question idea is sound and well developed. The practical advantage of GraphRubric is that they optimize not the criteria themselves, but the configuration of how they are used together. At the same time, the LLM judge's weights are not changed, so after training the method can be used with a standard API. The method is well suited for offline calibration of the evaluation prompt for a frequently used rubric.

Critical weaknesses

Would be interesting to add also not graph-based baselines. For example, GEPA with a fixed set of initial rubrics / GEPA with a limited operation space (this is a direct comparison of search algorithms within the same solution space). It would also help to report the training costs and compare the final results of the proposed methods.

Questions for authors

—

1 comment
AuthorsOpenReview ↗

Fixed-text GEPA comparison: shared search space, alignment, and token costs

We thank the reviewer for the positive assessment of GraphRubric and its Graph-GRPO training mechanism, and for the practical value of calibrating rubric configurations with a frozen judge. Your suggestion to add a non-graph prompt-optimization baseline and compare both predictive performance and training cost motivated the following experiments.

GEPA in the same selection and ordering space. Following your suggestion, we added a restricted version of the official GEPA optimizer (v0.1.4): the original text of every rubric item is fixed, and the optimizer can only select a nonempty subset and reorder its items. Both methods therefore share the same legal rubric-path space, current DeepSeek judge, scoring prompts, and original data partitions. GEPA starts from all rubric items in their original order, uses training examples for reflection, and selects one global fixed path by MSE on the full validation set.

The GraphRubric training described in the paper uses only the training split. To give both methods access to the same labeled data in this comparison, we additionally trained GraphRubric for one pass over the combined training and validation splits. Both methods were evaluated on the same original test set.

Test results. GraphRubric uses MAP/R3; GEPA evaluates its selected fixed path three times per case. We compute squared errors for each call and average across target dimensions, the three evaluations, and test cases. USR uses dimension-range-normalized MSE over five dimensions, excluding overall. Lower is better.

MethodSummEval-4HANNAUSR
GraphRubric0.7715281.4465890.157155
Restricted GEPA0.9475691.4867450.155429
GEPA-selected pathCADBABCEFDCBAE

Training budget and cost. We count one case-by-candidate evaluation as one logical scoring unit. GEPA's cumulative official evaluation thresholds match the scoring slots in GraphRubric's training pass: 5,120 for SummEval-4 (320 × 16), 24,640 for HANNA (770 × 32), and 11,550 for USR (462 × 25).

DatasetGraph-GRPO scoring slotsGEPA counted evaluationsGEPA reflection calls
SummEval-45,1205,120600
HANNA24,64024,6403,790
USR (seed 31000 for GEPA)11,55011,6341,367

GEPA checks its stopping threshold at iteration boundaries, accounting for the additional 84 evaluations on USR.

Token estimates charge every counted scoring slot the input-plus-output usage of its matching successful receipt. Each method is accounted for independently; final TEST evaluation is excluded.

MethodSummEval-4HANNAUSR
GraphRubric (Graph-GRPO)4.191M38.842M8.163M
Restricted GEPA6.250M58.123M11.745M

Including reflection token, GEPA's estimated token consumption under this logical-scoring accounting is 1.44–1.50 times that of GraphRubric.

Reviewer_rnVkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to align LLM-as-a-judge scores with human judgements. It frames this as supervised prompt calibration: human scores are the target, and the objective is to make the judge's numerical scores as close to them as possible. This is similar to writing an SOP for a task based on human feedback.

The proposed method, GraphRubric, represents each rubric dimension as a node in a graph. It learns which rubric definitions to include in the judge prompt and in what order. The judge itself stays frozen.

Critical strengths
  • Important problem. Making LLM judge scores match human scores more closely is a practical and widely relevant problem.
  • No model changes. The method needs no fine-tuning or weight access, so it works with closed-API judges.
  • Simple, structured approach. It learns only the selection and ordering of rubric definitions. The graph-based model gives a systematic way to capture relationships between rubrics.
  • Consistent gains over baselines. GraphRubric improves over the following baselines on all three DeepSeek datasets:
    • EMPTY: no rubric definitions are provided, but the judge still scores all required dimensions.
    • Uniform Nonempty: at least one rubric dimension definition is included.
    • VAL-fixed: a fixed prompt chosen on validation data.
    • The full path (all rubric definitions present) is also part of the comparison.
Critical weaknesses

Prompt optimization methods. None of the standard prompt optimizers are compared:

Calibration methods. Other judge-calibration approaches are also missing:

Model adaptation. There is no open-weight LoRA comparison. We could show how much alignment direct adaptation of the judge achieves versus only optimizing rubric selection and ordering, and would clarify the performance-versus-adaptation-cost trade-off. LoRA is a stronger-access comparison rather than a strict baseline for the paper's frozen/closed-API setting.

2. Cost not analysed

Training requires many judge calls, but the paper does not show whether this is more cost-effective than competing approaches. A comparison of judge calls or cost against the baselines above would help.

3. Unclear benefit of case-specific routing

In some experiments GraphRubric selects the same path for every test case. This makes it unclear whether case-specific routing is adding value over a single well-chosen fixed path.

4. Incomplete ablations

The experiments do not fully isolate the contributions of:

  • graph attention
  • case conditioning
  • the training objective

5. Uneven improvements

GraphRubric does not beat the simpler controller variants in every setting. For example, the Additive Graph Policy is slightly better on USR.

6. Limited robustness analysis

More independent initialisations are needed, along with an analysis of judge-call variability, to show the gains are stable and not noise.

Questions for authors

Baselines and comparisons

  1. Prompt optimisation. How does GraphRubric compare with established prompt-optimisation methods such as ProTeGi/APO, OPRO, or GEPA under a similar evaluation budget? Please add these baselines, or explain why they are not applicable.
  2. Judge calibration. Can you compare GraphRubric with supervised calibration approaches such as SAJA, LLM-Rubric, or FELIX, using the same human-labelled data?
  3. LoRA. For an open-weight judge, how does GraphRubric compare with LoRA fine-tuning on the same human-labelled data, in terms of alignment quality and training/inference cost? This would be a stronger-access reference point and should be labelled as such, not as a strict baseline for the frozen/closed-API setting.
  4. Cost. How does performance compare with alternative methods once the total number and cost of judge and optimisation calls are included? Please report judge calls or cost against performance for all methods.

Understanding the method

  1. Case-specific routing. In some experiments GraphRubric selects the same rubric path for all test cases. How much of the gain comes from case-specific routing? Please analyse when routing changes the selected path and when it helps, and compare against a single well-chosen fixed path.
  2. Component contribution. Can you provide ablations that separately measure the benefit of graph attention, case conditioning, the product branch, and the training objective?
  3. Uneven improvements. Simpler controller variants sometimes match or beat GraphRubric (e.g., the Additive Graph Policy is slightly better on USR). Can you discuss when the added complexity is justified?

Reliability

  1. Robustness. Are the reported improvements stable across different model initialisations and repeated judge calls? Please report variance across seeds and judge-call repeats.
2 comments
AuthorsOpenReview ↗

Clarifying the research scope, baseline comparisons, component ablations, and reliability

Thank you for the constructive review. New experiments use DeepSeek-V4.1-Flash-1017 with unchanged decoding settings, plus Qwen3-8B. S/H/U denote SummEval-4/HANNA/USR. TEST MSE uses MAP/R3 (three scores of each MAP path) or MC8 (eight sampled paths), averaging per-call errors. USR uses range normalization.

Q1, Q2. Optimization targets and comparisons. GraphRubric selects/orders fixed rubric text before judge scoring. ProTeGi/OPRO/GEPA optimize prompt text; SAJA/LLM-Rubric calibrate extracted scores/probabilities, while FELIX also discovers features, Three methods serve different prompt optimization stages. GraphRubric accesses the judge only through the API, which makes the method more versatile. Compared to SAJA-type methods, it can also retain the ability to calibrate scores for each rubric.

GraphRubric can prioritize evaluation dimensions through reward weights. On S, weights .4/.3/.2/.1 for coherence/consistency/fluency/relevance change dimension-wise MC8 MSE from .8375/.6488/1.4436/.6083 to .6594/.6280/1.5993/.8578; TEST MAP choices change from B×80 to B×38, DB×42. This expresses human priorities through rubric selection and ordering before scoring, whereas SAJA-style calibration operates on extracted scores.

We added restricted official GEPA v0.1.4 (fixed rubric text; subset/order search) and a multi-output SAJA-RF adaptation (200 trees; TRAIN-only fitting/grouped CV). GraphRubric now makes one pass over TRAIN+VAL, matching GEPA’s label access.

Q3. Weight adaptation. On matched Qwen3-8B data, S/H MSE is .7219/.8451 for frozen judge+GraphRubric (MAP/R3) and .5142/.3434 for judge LoRA with fixed full rubric (R3). LoRA training took 19.8/54.3 minutes.

Q4. Alignment and cost. Cells: TEST/R3 MSE; counted scoring slots; estimated tokens (M). Costs cover optimization only. GraphRubric/GEPA are single runs; SAJA MSE averages three feature collections, with first-run cost.

MethodSHU
GraphRubric.7715; 5120; 4.1911.4466; 24640; 38.842.1572; 11550; 8.163
Restricted GEPA.9476; 5120; 6.2501.4867; 24640; 58.123.1554; 11634; 11.745
SAJA-RF.4178; 240; .202.3803; 580; .958.0660; 330; .240

GEPA thresholds match 5,120/24,640/11,550 scoring slots; completing the last iteration adds 84 on U, Each counted slot is charged matching receipt input+output usage. As GEPA uses extra tokens for 600/3,790/1,367 reflection calls, its giving 1.44–1.50× GraphRubric’s token cost. SAJA-RF needs one fixed-full-rubric TRAIN evaluation per case (240/580/330), followed by local calibration; GraphRubric samples K paths per case.

Q5. Case-specific routing. GraphRubric learns a case-conditioned path distribution, which can express shared global preferences or input-specific MAP routes. Cases sharing the highest-probability path can therefore have different probability distributions.

The paper primarily reports one pass over TRAIN to limit judge API calls. We added multi-epoch training to examine how further optimization changes routing and cross-case distributions.

DatasetDistinct MAP paths: 1 epoch → multi (3 inits)MAP/R3: 1 → multiMC8: 1 → multi
S1/1/1 → 3/1/1.7764 → .7816.8903 → .7852
H3/2/1 → 2/1/11.5104 → 1.43521.5611 → 1.4501
U1/3/2 → 3/4/5.1598 → .1567.1611 → .1607

Mean pairwise case-distribution TV changes from .0304/.1078/.0062 to .0691/.1027/.0880. These measurements show that full path distributions vary across cases. The MSE results also distinguish routing diversity from accuracy gains.

Q6, Q8. Components and robustness. We added component ablations with three independent initialization seeds, other experimental settings remain the same. Case-zero zeros case inputs to both branches; No-GAT removes message passing; A (Additive Graph Policy) removes the product branch. L-path uses intra-group advantages for full-path RL training.

Single-epoch MAP/R3, mean±SDSHU
GraphRubric.7764±.01201.5104±.0733.1598±.0064
Case-zero.7764±.01201.5549±.0675.1620±.0043
No-GAT.7764±.01201.5105±.0559.1600±.0038
A.7764±.01201.5638±.0871.1602±.0082
L-path.7764±.01201.5248±.0806.1750±.0018

We additionally compared 1/2/3 GAT layers with one initialization. MC8 is .9253/.9345/.8956 on S, 1.5837/1.5999/1.5530 on H, and .1613/.1622/.1607 on U. MAP/R3 favors three layers on H, two on U, and ties on S. Three layers thus give the lowest MC8 across these datasets.

Q7. Uneven benefits. MiniLM context/response token ratios for S/H/U are 6.26/2.20/12.95. HANNA's longer candidate stories may provide richer rubric-relevant features for case×rubric interactions, consistent with its larger benefit over A. This is a representation hypothesis; fields are independently encoded. GraphRubric-versus-A MSE comparisons across the three datasets likewise indicate task-dependent benefits from the product branch.

Thank you for the additional experiments. The new baselines, cost comparisons, and ablations address several of my concerns. However, two main concerns remain:

  1. Benefit over simpler calibration. SAJA-RF achieves much lower error on all three datasets with far fewer judge calls. Although it works at a different stage, it addresses the same goal of matching human scores. It is therefore still not clear what advantage GraphRubric will offer over the simpler approach.

  2. Need for the proposed components. The new results do not clearly show why graph attention and case-specific routing are needed. Removing graph attention barely changes the accuracy across the three datasets. the reported results do not clearly establish whether case-specific rubric selection improves accuracy over using a single fixed rubric path.

The restricted GEPA comparison provides useful evidence for GraphRubric’s effectiveness and lower optimization token cost within the same rubric-selection space.

I appreciate the additional work, but these concerns remain. Therefore, I maintain my weak-reject rating.

Reviewer_RXFGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GraphRubric, a framework for optimizing rubric selection and ordering in LLM-as-a-Judge. It uses a graph attention network and a case–rubric interaction branch to learn rubric representations, followed by a sequential decoder to generate rubric paths. The controller is trained using a GRPO-inspired edge-level feedback mechanism, where rewards are computed based on the agreement between LLM judge scores and human labels.

Critical strengths
  1. This paper study an interesting evaluation problem. Rubric selection and ordering can affect LLM judge performance. Learning to optimize rubric paths is a useful direction for improving LLM-based evaluation.
  2. This proposed framework combines graph-based rubric representations with a sequential decoder, and the edge-level feedback mechanism provides a way to optimize rubric paths without evaluating all possible combinations.
Critical weaknesses
  1. Although the method aims to select different rubric paths for different inputs, Appendix D table 8 shows that many test cases end up using the same MAP path. For example, all 80 SummEval test cases share the same path within each training seed. This makes it unclear whether the model actually learns case-dependent selection or simply finds a good fixed path.
  2. The proposed edge-level feedback is a key part of the method, but this paper does not compare it with standard path-level group-relative optimization. It is unclear whether assigning rewards to individual edges is more effective than directly optimizing complete paths. Also, the same edge may appear in different paths, so averaging their rewards may not accurately reflect the contribution of that edge.
  3. The ablation studies are not sufficient to explain where the improvements come from. This paper jointly optimizes rubric selection and ordering, but does not evaluate their individual contributions. In addition, the effects of GAT and the interaction branch are not fully separated. Therefore, it is difficult to tell which components are actually responsible for the performance gains.
Questions for authors
  1. Why do so many test cases share the same MAP path? Can authors provide stronger evidence that case-conditioned selection is better than learning a fixed path?
  2. How does the proposed edge-level feedback compare with standard path-level GRPO under the same settings?
  3. Can authors add ablations to separately evaluate selection, ordering, GAT, and the interaction branch?
1 comment
AuthorsOpenReview ↗

Case-conditioned routing, training-objective comparisons, and selection/ordering ablations

Thank you for the constructive review. New experiments use DeepSeek-V4.1-Flash-1017. S/H/U denote SummEval-4/HANNA/USR. TEST MAP/R3 evaluates each case's MAP path three times; MC8 evaluates eight policy-sampled paths. Both average per-call squared errors. USR uses range-normalized MSE.

Q1. Shared MAP paths and case conditioning. Analysis of our earlier exhaustive SummEval-4 VAL scores found that optimal rubric paths can vary across cases and need not be unique. GraphRubric learns case-conditioned path distributions, and different distributions can share the same highest-probability path. Its objective is agreement with human labels; routing diversity and accuracy are distinct properties.

The submitted one-epoch setting limits judge API calls. We added multi-epoch training with VAL checkpoint selection. Mean pairwise case-distribution TV changes from .0304/.1078/.0062 to .0691/.1027/.0880, shows that the probability of the path distribution does change with the input of the case. Model MSEs below average three independent initializations.

DatasetMAP counts, 1 → multi (3 inits)MAP/R3, 1 → multiVAL-fixed R3 rangeMC8, 1 → multi
S1/1/1 → 3/1/1.7764 → .7816.8160–.8753.8903 → .7852
H3/2/1 → 2/1/11.5104 → 1.43521.5312–1.57671.5611 → 1.4501
U1/3/2 → 3/4/5.1598 → .1567.1548–.1583.1611 → .1607

VAL-fixed exhaustively scores all nonempty ordered paths on sampled VAL cases, selects the path with the lowest average VAL MSE, and evaluates it on TEST/R3. The table reports minimum–maximum TEST MSE across 3 VAL draws on S/H/U. Each draw approximately matches one epoch's logical scoring budget; multi-epoch training uses additional budget.

Even at the same budget, VAL-fixed is sensitive to the validation draw, making the selected path difficult to interpret as a stable deployment preference. S selects B or BD; H selects AECBF, ABECF or CAFBD. As rubric count grows, exhaustive search covers fewer VAL cases within the budget: 60/9/25 on S/H/U. H improves despite fewer MAP routes, further illustrating the distinction between routing diversity and accuracy.

Q2. Edge-level versus path-level feedback. We added **L-path **: online sampling of K paths, calculating the relative advantage within the group to create rewards, and using the rewards for training on the full paths.

Each cell in the table is MAP/R3 / MC8.

Training ruleSHU
Graph-GRPO.7764/.89031.5104/1.5611.1598/.1611
L-path.7764/.91681.5248/1.6037.1750/.1692

Q3. Selection, ordering and components. We isolated selection and ordering. Joint Graph-GRPO learns both; selection-only learns nonempty subsets in canonical order; ordering-only retains all rubrics and learns permutations. Training settings are matched within each dataset. Cells are MAP/R3 / MC8.

Path spaceSHU
Joint.777431/.8956161.557407/1.553009.157449/.161053
Selection-only1.010764/.9169701.671930/1.665546.155598/.168771
Ordering-only.854167/.9740021.700682/1.728119.161322/.161116

We also added component ablations with three independent initializations, other experimental settings remain the same. Case-zero zeros case inputs to both branches; No-GAT removes message passing; A (Additive Graph Policy) removes the product branch.

Single-epoch MAP/R3, mean±SDSHU
GraphRubric.7764±.01201.5104±.0733.1598±.0064
Case-zero.7764±.01201.5549±.0675.1620±.0043
No-GAT.7764±.01201.5105±.0559.1600±.0038
A.7764±.01201.5638±.0871.1602±.0082

In an additional single-initialization comparison of 1/2/3 GAT layers, three layers give the lowest MC8 on all datasets; MAP/R3 ties on S, favors three layers on H and two on U. Together, these controls separate the effects of rubric selection, ordering, message passing and the interaction branch.

Reviewer_Xr12OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

GraphRubric learns to select and order rubric descriptions in a prompt for a frozen LLM judge, using agreement with human scores as supervision. A case-conditioned GAT and a direct multiplicative case–rubric branch parameterize a sequential policy over nonempty ordered subsets. Training pools sampled terminal rewards by structural edge and applies a group-relative log-probability surrogate.

The paper evaluates three datasets and three API judges, with three training-order/sampling seeds for DeepSeek. It reports improvements over random rubric presentations, an empty-rubric control, a validation-selected fixed path, and several controller ablations.

Critical strengths
  • Useful and well-defined problem. Optimizing rubric presentation without modifying judge weights is practically relevant. Keeping score dimensions and output requirements fixed across paths makes the comparison interpretable.
  • Substantial implementation detail. The appendices document exact prompts, initialization, request parameters, missing-data handling, continuation probabilities, and optimization settings.
Critical weaknesses

1. The primary deployment results largely reflect global path selection, not case-conditioned routing

The path diagnostics substantially weaken the central motivation. In Table 8, GraphRubric selects exactly one path for all 80 SummEval test cases in every seed. In Table 9, it likewise selects one global path for all HANNA cases in two of three seeds. USR distributions show extremely small between-case total variation.

Consequently, much of the headline MAP/R3 performance can be obtained by deploying one fixed prompt after training. Differences from Case-zero do not, by themselves, establish a benefit from input-specific routing: conditioning can alter the optimization trajectory and the globally preferred path without producing adaptive deployment.

2. The fixed-path baseline is not a strong use of the available calibration budget

VAL-fixed exhaustively evaluates every path on relatively few cases—only nine for HANNA—whereas GraphRubric distributes its calls across 580 training cases. This is a valid comparison with exhaustive selection, but not a convincing comparison with efficient fixed-prompt optimization. Its weakness is partly imposed by the chosen budget allocation.

3. The optimization contribution is not adequately validated

Equation 8 pools rewards across occurrences of the same structural edge, despite those occurrences having different visited sets and conditional action probabilities. Equal weighting over observed edges and averaging over their occurrences further separates this update from a standard trajectory policy-gradient estimator.

The authors appropriately call this a surrogate rather than claiming an unbiased gradient. Nevertheless, the paper provides neither analysis nor an ablation showing why this surrogate is preferable. The assertion that pooling reduces noise is plausible but unmeasured; pooling can also obscure prefix-dependent effects.

4. Architectural comparisons do not isolate graph message passing

Contextual-MLP removes both the GAT and GraphRubric’s multiplicative interaction branch. Its performance gap therefore cannot be attributed specifically to graph message passing. A graph-free controller retaining the same product branch and comparable parameter capacity is needed.

The additive graph comparison does test the product branch, but its gains are modest on SummEval and absent in the mean USR MAP/R3 result. These results support a narrower architectural conclusion than the paper’s overall framing suggests.

Questions for authors
  1. The three DeepSeek runs share a fixed initialization and vary only training order and path sampling. Do the MAP/R3 differences persist under other initializations, and what are paired bootstrap confidence intervals that resample held-out source groups for GraphRubric against Case-zero, EMPTY, and VAL-fixed?
  2. GraphRubric and Case-zero both select path B for every SummEval test case in seeds 31000 and 31001, yet receive different MAP/R3 scores. The mean gains over Case-zero are only about 0.0082 MSE on SummEval and 0.0016 on USR. What is the repeated-call variability of an identical prompt, and which of these gaps exceed it?
  3. GLM and Gemini are each reported for a single seed. On Gemini HANNA, MAP/R3 improves over EMPTY by only 0.0031 MSE. Do these cross-judge comparisons remain once that repeated-call variability is taken into account?
1 comment
AuthorsOpenReview ↗

Bootstrap confidence intervals, independent initializations, and judge repeatability

Thank you for identifying this issue. MAP/R3 averages per-call squared errors over three evaluations; USR uses range-normalized MSE.

Q1–Q2. Identical paths, uncertainty, and new initializations. We agree: identical case/path prompts under the same judge configuration have the same expected loss. The original SummEval differences on identical paths arose from separately collected scores, not different routing decisions. For the original GraphRubric runs, the standard deviation of whole-TEST MSE across three repeated calls per case was .0104–.0153/.0129–.0328/.00025–.00084 on SummEval/HANNA/USR (ranges across runs).

Original-paper bootstrap. From saved R3 losses for seeds 31000/31001/31002, we average each case's loss across runs, then jointly resample the test set for both methods, retaining all their cases. We use 20,000 draws over 20/19/21 groups (80/190/132 cases): news articles, writing prompts, and dialogue-history components. Both CI tables report Δ̂ = MSE(comparator) − MSE(GraphRubric) with 95% percentile CIs; positive values favor GraphRubric. On average, GraphRubric has a consistent advantage.

Original comparatorSummEvalHANNAUSR
Case-zero.0082 [-.0041,.0222].0830 [.0522,.1167].0016 [-.0015,.0051]
EMPTY.0760 [.0083,.1580].4782 [.3733,.5902].0276 [.0180,.0375]
VAL-fixed.0861 [.0031,.1663].1223 [.0574,.1905].0025 [-.0051,.0114]

Independent-initialization replication. After the recorded DeepSeek backend fingerprint changed, we trained three new controllers with distinct initial weights and fixed training-order/path-sampling seeds. Each initialization collected its own judge feedback. Case-zero retains all GraphRubric modules and zeros case input to both branches.

To address identical-prompt scoring differences, we adopted shared scoring: The three R3 repeats remain separate calls.

New DeepSeek TEST MAP/R3SummEvalHANNAUSR
GraphRubric.7764±.01201.5104±.0733.1598±.0064
Case-zero.7764±.01201.5549±.0675.1620±.0043
Additive A.7764±.01201.5638±.0871.1602±.0082
EMPTY.83061.8327.1718

The corresponding paired bootstrap results are:

New comparatorSummEvalHANNAUSR
Case-zero.0000 [.0000,.0000].0445 [.0250,.0633].0022 [-.0017,.0060]
Additive A.0000 [.0000,.0000].0534 [.0268,.0822].0004 [-.0025,.0035]
EMPTY.0542 [-.0311,.1397].3224 [.2465,.3980].0119 [.0037,.0208]

We also added extra experiments with Qwen3-8B to back up the effectiveness of GraphRubric. Single-epoch, greedy-decoding controls give:

Qwen3 TEST MAP/R3SummEvalHANNAUSR
GraphRubric.7219.8451.1261
Case-zero.7219.9816.1267
No-GAT.7219.9622.1262

Q3. Cross-judge gains and expectation. R3 estimates expected per-call squared error. On Gemini HANNA, GraphRubric and EMPTY use different rubric paths, unlike the identical-path DeepSeek comparison. Their MSEs are 2.010429/2.013450, giving Δ̂ = +.003021 with a paired source-group 95% CI of [-.0562,.0625], consistent in sign with the clearer positive gains observed under GLM.

GLM's corresponding gains over EMPTY on SummEval/HANNA/USR are .5243 [.3944,.6632] / .1547 [.1007,.2115] / .0276 [.0179,.0393], providing clearer evidence of positive gains.

Comments on the paper

1 comment
Area_Chair_WrSzOpenReview ↗

Please review the authors’ responses and join the discussion

Dear Reviewers,

The authors have responded to all reviews. Please read their responses and the other reviewers’ comments, then post a brief follow-up indicating which concerns, if any, remain and whether the responses have changed your assessment. If your concerns have been resolved, a short confirmation would also be helpful.

Thank you for your time and for contributing to the discussion.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.