acceptodds
Under review as a conference paper at ICLR 2027

Do LLM Judges Faithfully Follow the Rubric? Measuring and Improving Rubric-Following in Pairwise Evaluation

Abstract

LLM judges are widely used to evaluate open-ended generation, and rubrics are increasingly employed to guide their evaluations using explicit criteria. This helps only if the judge faithfully follows the rubric, an assumption that is rarely tested. We test it on rubric-conditioned pairwise judges, which choose the better of two responses according to a given rubric. We find that when the rubric contradicts the judge's own preference, the judge ignores the rubric and keeps its verdict on up to 23.6% of judgments across four judges and four preference datasets. In most cases, the more confident the judge is before seeing the rubric, the more likely it is to ignore it. To address this, we propose RubricLens, which improves rubric-following by optimizing only the judge's instruction. For each pair of responses, it synthesizes two rubrics, each favoring a different response, so that one of them always conflicts with the judge's own preference. It then optimizes the instruction with GEPA to make the judge faithfully follow both rubrics. Across four judges and four preference datasets, RubricLens raises agreement with the rubric when it conflicts with the judge's own preference in every judge-dataset pair, from 86.2% to 93.7% on average, and lowers the rate at which the judge keeps its own choice to at most 12.0%. The same instruction improves rubric-following on expert-written rubrics without further tuning. RubricLens also outperforms fine-tuned rubric-conditioned judges in every setting, without changing model weights.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-Judge, Rubric-based Evaluation, Instruction Following, Knowledge Conflict

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GZJUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores a fundamental but rarely systematically tested hypothesis in the field of "Large Model as a Judge" (LLM-as-a-judge): given explicit rubrics, do pairwise judges truly adhere faithfully to those rubrics rather than relying on the model's own internal prior preferences? To address this question, this paper makes three main contributions:

  • Proposes the "Stubbornness Rate" (S): First, it defines the judge's prior preference ypy_p by the consistent choice of a judge J0J_0 without rubrics in a bidirectional presentation order. Then, it constructs conflicting rubrics RyˉpR_{\bar{y}_p} that favor prior rejections ypy_p, thereby quantifying the proportion of judges that disregard rubrics and adhere to their own priors. Experiments show that four mainstream open-source evaluators ignore conflicting scoring rules in 6.2%–23.6% of their judgments across four preference datasets. Furthermore, on open-ended preference datasets, the higher the evaluator's prior confidence, the more likely it is to violate the scoring rules.

  • We propose the RubricLens cue optimization framework: without modifying the model weights, we use a generative model (gpt-5.6-luna) to synthesize two opposing single-criteria scoring rules for each preference answer pair (each biased towards one of the two candidate answers). This is combined with the reflective cue evolution algorithm GEPA (using gpt-5.6-sol as the reflective model) to optimize the evaluator instructions Ir⋆I_r^\star, forcing the model to strictly adhere to the given evaluation criteria under both bidirectional order and positive/negative rules.

  • Validating generalization across datasets and rule types: After tuning the instruction only on the synthetic scoring rules in UltraFeedback, RubricLens improved the average consistency (Aflip\mathcal{A}_{\text{flip}}) under flipped rules from 86.2% to 93.7% across all 16 "judge-dataset" combinations, reducing the stubbornness rate to a maximum of 12.0%; at the same time, when the instruction was migrated to RubricBench with zero samples, it also improved the original consistency by 1.2–4.0 percentage points.

Critical strengths
  • The problem is precisely addressed and has high practical value: Current communities heavily utilize scoring rules to standardize LLM evaluators or provide reinforcement learning rewards (Rubrics as Rewards), but generally only report raw consistency Aorig\mathcal{A_{\text{orig}}}. This paper astutely points out that Aorig\mathcal{A_{\text{orig}}} is highly saturated on existing datasets (0.959–1.000), failing to distinguish whether the model "truly understands and follows the rules" or "just happens to choose the correct answer based on its prior knowledge." By introducing flip consistency Aflip\mathcal{A}_{\text{flip}} and persistence rate S\mathcal{S}, this paper provides a more discriminative benchmark perspective for evaluating the evaluator's rule-following ability.

  • The experimental design was meticulous, effectively eliminating common confounding factors:

    • Eliminating positional bias: All indicators were calculated and averaged under two presentation orders: (yA,yB)(y_A, y_B) and (yB,yA)(y_B, y_A). The prior set D0\mathcal{D}_0 was strictly limited to samples selected consistently by the ruleless evaluator under both orders, avoiding positional bias from contaminating the measurement of prior preferences.

    • Eliminating lexical surface matching shortcuts: Appendix H rigorously tested the results on RubricBench using TF-IDF bag-of-words (BoW) cosine similarity, demonstrating that the optimized instruction Ir⋆I_r^\star did not increase the proportion of highly overlapping responses (maintaining at 0.567–0.577), and the gain was even greater on samples without BoW preference, effectively ruling out the assumption of "judging solely based on surface word overlap".

Critical weaknesses
  • Lack of comparison with other baselines

    • Lack of comparison with a manually enhanced prompt baseline: The paper only compares the RubricLens-optimized instruction Ir⋆I_r^\star with the minimalist original instruction IrI_r (e.g., "Determine which answer better satisfies the following scoring criteria") in the experimental section. IrI_r contains almost no adversarial bias, emphasizes "rule-based" approaches, or ignores external prior conflicts—these are the meta-cue constraints. This leads to a serious experimental confounding factor: Is the performance improvement brought by RubricLens due to the specific strategies automatically searched by GEPA (such as complex trade-off mechanisms), or simply due to the regularization effect of "longer prompts, stronger emphasis, or the introduction of general rule warnings" (i.e., the general prompt complexity/length effect)? Without a manually enhanced prompt baseline carefully crafted by human experts, the necessity of automated prompt optimization cannot be demonstrated.

    • Lack of comparison with other baselines that do not alter model weights (such as in-context learning examples or general thought chain prompts): While RubricLens' core advantage is its non-altering model weights, many mature, lightweight techniques beyond simple system prompts are not included in the comparison within the "non-altering model weights" prompt engineering paradigm:

      • In-context learning (Few-Shot Prompting): Providing several input-output demonstration samples in the prompt where "the details conflict with the prior but successfully conform to the details," rather than modifying the instruction itself.

      • General/open-source advanced reasoning/chain-of-thought prompts: For example, standard thought chain templates ("Let's think step by step and strictly follow the rubric even if it contradicts your intuition").

  • The Justification and Potential Negative Effects of "Forced Adherence to Conflicting Rules" (Reward Hacking Risk) on Datasets Containing Objective Facts/Logical Errors (RewardBench, RM-Bench)

    • In RewardBench (math-prm math problems and hep-code problems) and RM-Bench (concise style answers injected with subtle factual errors), manually rejected answers yˉh\bar{y}_h contain explicit objective errors (incorrect solutions, flawed code, or factual fallacies).

    • According to the constraints in Appendix F (Figure 5), the synthesizer gpt-5.6-luna must write a "general, positive, good answer trait like that a rigorous evaluator would write" standard RyˉhR_{\bar{y}_h} for incorrect answers yˉh\bar{y}_h, and the incorrect answer must explicitly meet this standard while the correct answer must explicitly not. When two answers in RM-Bench are styled to be identical (both concise), the synthesizer, in order to favor the incorrect answer, often only grasps some marginal formatting, structural, or phrasing features, or indirectly caters to the incorrect content itself.

    • More importantly, the optimized instruction Ir⋆I_r^\star shown in Figure 3 explicitly commands the evaluator: "Do not automatically favor answers that are factually more correct, more complete... unless the scoring criteria explicitly require these qualities; even if an answer has serious flaws in aspects not mentioned in the scoring criteria, it must be selected as long as it better meets the literal standard." In practical applications, human-written criteria rarely exhaustively list all implicit bottom lines (for example, a criterion requiring "summarize in a list" usually implicitly assumes "no fabricated facts"). If an evaluator optimized with RubricLens is used for reinforcement learning reward modeling (RLHF/RLVR), this deliberate blindness to factual errors can easily induce serious reward hacking problems. Although the paper briefly mentions in its limitations that "it is assumed that even if the rules are wrong, they should still be followed," it lacks a specific analysis of the form of the rules for synthesizing conflicts in the objective task, and does not discuss how to distinguish between "reasonable anti-prior criteria" and "degenerate marginal criteria."

Questions for authors
  • Regarding the comparative validation of manually enhanced prompts: The control group instruction IrI_r in the paper is too basic and fails to examine the effect of strong prompts written by human experts. Did the authors test manually written general enhanced prompts (e.g., a 200-300 word manual anti-stubbornness prompt, rather than instructions automatically searched through GEPA) on the validation or test sets, which also include "emphasizing the rule of thumb, eliminating prior interference, and paying attention to multiple standard trade-offs"? Compared to these zero-cost manually enhanced prompts, what is the marginal performance gain of RubricLens through automated search?

  • Regarding comparisons with other baselines that do not change model weights (such as Few-Shot ICL or general thought chain templates): In similar baselines that keep model weights unchanged, why not try introducing a small number of successful examples of compliance with conflict rules (In-Context Learning / Few-Shot) or standard general defensive inference templates (such as CoT prompts with strong alignment warnings) into the prompts? When computational costs and inference latency are comparable, where does the core performance advantage of GEPA's automatically generated instruction Ir⋆I_r^\star compared to these general enhancement hint schemes that do not require tuning (especially the improvement in flip consistency Aflip\mathcal{A}_{\text{flip}}) lie?

  • Regarding the boundaries of "blindly conforming to rules" and the risk of reward hacking in objective error tasks:

    • In the code/mathematical subset of RewardBench and RM-Bench, rejected answers often contain objective logical, arithmetic, or factual errors. According to the cue word constraints in Appendix F, the synthesizer gpt-5.6-luna must write a "positive, general, and winning" criterion for these incorrect answers. However, human-written scoring rules often contain unspoken bottom lines (such as "factual correctness must be guaranteed"). If a RubricLens-optimized judge, forced to "ignore factual flaws not mentioned in the rules," is directly applied to reinforcement learning reward modeling (RLHF/RLVR), will this induce serious reward hacking problems? Under what circumstances should the judge have the ability to "reject" or "conversely overturn flawed rules"?
3 comments

We thank the reviewer for the careful reading and the concrete questions. In response, we added two non-training baselines, a prompt-engineered instruction and a few-shot instruction, and clarified the role of conflicting rubrics and the risk of reward hacking.

Q1/Q2. Comparison with other non-training baselines

In the revised version of the paper, we added two non-training baselines (Sec. 4.1, Table 2). The first is a detailed instruction (IrPEI_r^{\text{PE}}) written by an LLM acting as a prompt engineer from a task description that follows the specification in your question; the full task description is in Appendix H. The second is a few-shot instruction (IrFSI_r^{\text{FS}}) that adds four examples in which the rubric conflicts with the judge's prior. We did not test the suggested CoT prompt separately, because every judge in the paper already reasons with a 1,024-token thinking budget before giving its verdict.

The table below compares the original instruction (IrI_r), the two new baselines (IrFSI_r^{\text{FS}} and IrPEI_r^{\text{PE}}), and RubricLens (Ir⋆I_r^\star) on the four datasets. Each value is averaged over the four judges; the per-judge results are in Table 2 (Sec. 4.1) of the revised paper.

InstructionUltraFeedback (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)HelpSteer3 (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)RewardBench (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)RM-Bench (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)
IrI_r0.895 / 0.1240.934 / 0.0710.779 / 0.2220.839 / 0.171
IrFSI_r^{\text{FS}} (few-shot)0.926 / 0.0860.948 / 0.0550.827 / 0.1720.857 / 0.152
IrPEI_r^{\text{PE}} (prompt-engineered)0.940 / 0.0690.960 / 0.0410.880 / 0.1220.890 / 0.117
Ir⋆I_r^\star (RubricLens)0.958 / 0.0490.965 / 0.0360.915 / 0.0850.910 / 0.096

RubricLens achieves the highest flipped agreement and the lowest stubbornness rate on every dataset, about +0.02+0.02 / −0.02-0.02 over IrPEI_r^{\text{PE}} on average. Since IrPEI_r^{\text{PE}} and Ir⋆I_r^\star are of similar length, the difference between them is unlikely to be explained by prompt length.

Q3. Following conflicting rubrics and the risk of reward hacking

On RewardBench and RM-Bench, the conflicting rubrics act as a stress test of whether a rubric can override even the judge's sense of correctness: a conflicting rubric favors the response that the label rejects, which on these datasets is the incorrect answer (Sec. 2.2). As you point out, such rubrics may differ from those used in practice, and we leave building a benchmark with more realistic rubrics to future work.

Regarding the risk of reward hacking, we believe that a judge that does not follow the rubric may not be safer. If the judge mixes in its own preferences that are not in the rubric, it becomes hard to tell whether a verdict comes from the rubric or from the judge's preference, and thus what a policy is exploiting. With a judge that follows the rubric, by contrast, the cause of reward hacking can at least be traced back to the rubric, which can then be revised. We agree that following a flawed rubric can be risky; when a judge should reject such a rubric is beyond the scope of this work and left to future work (Sec. 6).

Following a question from Reviewer jDSD (Q2) on stability across runs, we repeated the construction of every stochastic instruction three times for each judge: for IrFSI_r^{\text{FS}} and IrPEI_r^{\text{PE}}, a new set of five candidates with a new validation-based selection, and for Ir⋆I_r^\star, a new GEPA run. Table 2 of the revised paper now reports the mean over the three runs. We also repeated the ablation, reasoning-budget, and transfer experiments three times.

Most of our claims hold with three runs: RubricLens improves over IrI_r in all 16 judge–dataset cells and on RubricBench for all four judges, in every run, and it outperforms IrFSI_r^{\text{FS}} in flipped agreement in all 16 cells. One claim in our answer to Q1/Q2 needs correction. The margin of RubricLens over IrPEI_r^{\text{PE}} is smaller than in the single run, about +0.005+0.005 / −0.006-0.006 on average rather than +0.02+0.02 / −0.02-0.02. RubricLens is ahead on UltraFeedback, HelpSteer3, and RewardBench, but the two are tied on RM-Bench, so it is no longer best on every dataset. We have corrected Sec. 4.2 accordingly. Nevertheless, GEPA's feedback loop still adds a small gain on average over this carefully written instruction.

Thank you again for your time and thoughtful feedback. Please let us know if any concerns remain or if there is anything we could clarify further.

Thank you for providing the additional baseline experiments. However, marginal technical gain over basic prompt engineering (IrPEI_r^{\text{PE}}): The newly added baseline reveals that a simple zero-shot prompt-engineered instruction (IrPEI_r^{\text{PE}})—requiring no synthetic data generation or GEPA optimization—already captures 71.4% to 83.9% of the total improvement in Aflip\mathcal{A}_{\text{flip}} across all datasets (leaving only a 0.5% marginal gain on HelpSteer3), indicating that the vast majority of the reported gain comes from simply replacing the overly simplistic baseline IrI_r rather than the proposed RubricLens pipeline.

Reviewer_NAeUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a twofold research: 1) a method and quality metrics that quantifies the degree of rubric–prior conflict for a given judge and preference dataset; 2) RubricLens — a simple method that combines synthetic rubric data with prompt optimization. RubricLens is a prompt-optimization method for improving rubric-following based on the rubric–prior conflict idea. For every training pair, it synthesizes two rubrics that favor opposite responses, and tunes the judge’s instruction so that the judge follows whichever rubric it is given. RubricLens needs no human-written rubrics: it is tuned only on rubrics synthesized for UltraFeedback, yet improves rubric-following on three held-out preference datasets, HelpSteer3, RewardBench and RM-Bench, and on RubricBench, whose rubrics are written by experts.

Critical strengths

The authors attempted to formalize the judge's level of resistance to preference shifts.

The authors propose evaluating not just agreement with human preferences, but the judge's ability to apply a criterion even when it contradicts its initial choice. This is a useful additional aspect of LLM-as-a-Judge reliability.

The generalization ability for the RubricLens is demonstrated on held-out datasets and expert rubrics.

Critical weaknesses

In the first part (stubbornness) the synthetic rubrics raise questions. The term human labels is used to denote three things at once: GPT-4 ratings, human preferences, and the objective correctness of answers. These are signals of fundamentally different nature, and they can not be treated as equivalent. In the synthetic pipeline (where an LLM generates a criterion) the criterion by design determines the winner, and another LLM must then pick that winner. Thus, the stubbornness metric may reflect not only the model's resistance to criteria, but also errors or ambiguity in the synthetic rubrics themselves. It is therefore unclear what the proposed stubbornness rate actually measures: whether it is genuinely the judge's inability to follow a rubric, or the ambiguity and errors of the criteria themselves. Manual validation is needed to verify that the conflicting rubric is a valid and unambiguous criterion.

In real practice, a rubric for a task contains many criteria, not one, whereas here in the experiments each synthetic rubric consists of a single isolated binary criterion. There is no analysis of how quality depends on the number of criteria or on their interactions.

Questions for authors

Why did you use GEPA with greedy parent selection instead of the original Pareto sampling?

How necessary is a complex optimization loop with GEPA at all, if the final prompt essentially boils down to a detailed instruction to strictly follow the rubric and ignore extraneous characteristics of the answer? It seems it would make sense to also have an experienced prompt engineer (i mean human) do this and compare the results with RubricLens.

2 comments

We thank the reviewer for the careful reading and the concrete questions. In response, we clarified what stubbornness measures on each type of dataset, added a prompt-engineered baseline to test the need for the GEPA loop, and added rubric verification and multi-criterion rubrics to the limitations.

W1. What stubbornness measures

In the revision, Sec. 2.2 divides the datasets into two groups by how the label is produced: open-ended (UltraFeedback, HelpSteer3), where the label judges which response is better, and verifiable (RewardBench, RM-Bench), where the label is fixed by correctness. Sec. 2.2–2.3 also explain how the results should be interpreted differently for each group.

As you suggest, manual validation would make the measurement more reliable. We leave developing the synthetic rubrics into a more precise, human-verified benchmark to future work and state this in Sec. 6 of the revision. The current measurement assumes that the synthetic rubrics are valid. As a result, even when the judge rejects a rubric because it is ambiguous or its target is wrong, the case is counted as stubbornness. Stubbornness may therefore somewhat overestimate how often a judge ignores a valid rubric.

W2. Single-criterion rubrics

With a single criterion, the conflict between the judge's prior and the criterion can be observed most clearly, so this work focuses on single-criterion rubrics. With several criteria, we expect conflicts to arise not only between the prior and the criteria but also among the criteria themselves. Studying this would require a separate problem setup and a more detailed analysis. We added this point to the Limitations (Sec. 6).

Q1. Greedy parent selection in GEPA

RubricLens uses GEPA's reflective mutation, which reads failure cases and rewrites the instruction, with greedy parent selection. In an early tuning run, greedy selection gave a slightly higher validation score than Pareto sampling (+0.007), so we used it in all experiments. The GEPA paper reports that Pareto sampling helps the search escape local optima, because it also improves candidates that perform best on only some validation examples. This helps when different candidates are strong on different examples, but in our task, most candidates got most validation examples right, so the differences between them were small. Pareto sampling therefore likely offers little benefit here, and greedy selection, which spends the search budget on the best candidate, appears to be more effective.

Q2. Necessity of the GEPA loop

We added the suggested comparison to the revision. To obtain a carefully written instruction under controlled conditions, we asked a frontier LLM (gpt-5.6-sol) to act as an experienced prompt engineer. We gave it a detailed task description that explains the conflict between the rubric and the judge's prior and lists the requirements, sampled five instructions, and kept the one with the highest validation agreement for each judge (IrPEI_r^{\text{PE}}, Sec. 4.1). The task description is quoted in full in Appendix H. We agree that a comparison with instructions written by experienced human prompt engineers would be valuable, and we plan to include it in future work.

The table below compares the original instruction (IrI_r), the prompt-engineered baseline (IrPEI_r^{\text{PE}}), and RubricLens (Ir⋆I_r^\star) on the four datasets. Each value is averaged over the four judges; the per-judge results are in Table 2 (Sec. 4.1) of the revised paper.

InstructionUltraFeedback (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)HelpSteer3 (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)RewardBench (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)RM-Bench (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)
IrI_r0.895 / 0.1240.934 / 0.0710.779 / 0.2220.839 / 0.171
IrPEI_r^{\text{PE}} (prompt-engineered)0.940 / 0.0690.960 / 0.0410.880 / 0.1220.890 / 0.117
Ir⋆I_r^\star (RubricLens)0.958 / 0.0490.965 / 0.0360.915 / 0.0850.910 / 0.096

IrPEI_r^{\text{PE}} raises flipped agreement by +0.056+0.056 over IrI_r on average, about three quarters of the gain of RubricLens (+0.075+0.075). Still, RubricLens achieves higher flipped agreement and a lower stubbornness rate than IrPEI_r^{\text{PE}} in all 16 judge–dataset cells (Table 2). Both methods have an LLM write the instruction, so the additional gain appears to come from GEPA's feedback loop, which rewrites the instruction from failure cases.

Following a question from Reviewer jDSD (Q2) on stability across runs, we repeated the construction of every stochastic instruction three times for each judge: for IrFSI_r^{\text{FS}} and IrPEI_r^{\text{PE}}, a new set of five candidates with a new validation-based selection, and for Ir⋆I_r^\star, a new GEPA run. Table 2 of the revised paper now reports the mean over the three runs. We also repeated the ablation, reasoning-budget, and transfer experiments three times.

Most of our claims hold with three runs: RubricLens improves over IrI_r in all 16 judge–dataset cells and on RubricBench for all four judges, in every run, and it outperforms IrFSI_r^{\text{FS}} in flipped agreement in all 16 cells. One claim in our answer to Q2 needs correction. The margin of RubricLens over IrPEI_r^{\text{PE}} is smaller than in the single run: IrPEI_r^{\text{PE}} raises flipped agreement over IrI_r by +0.059+0.059 on average, close to the +0.064+0.064 of RubricLens, rather than about three quarters of it. RubricLens is ahead on UltraFeedback, HelpSteer3, and RewardBench, the two are tied on RM-Bench, and RubricLens is better in 14 of the 16 cells rather than all 16. We have corrected Sec. 4.2 accordingly. Nevertheless, the GEPA loop still adds a small gain on average over this carefully written instruction.

Thank you again for your time and thoughtful feedback. Please let us know if any concerns remain or if there is anything we could clarify further.

Reviewer_jDSDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether pairwise LLM judges follow an explicit rubric when it conflicts with their rubric-free preference. It introduces a stubbornness measure and constructs opposing rubrics to expose such conflicts. It then iteratively optimizes the judge’s prompt without updating model weights. Experiments across four judges demonstrate reduced stubbornness, with improvements transferring to held-out datasets and expert-written rubrics.

Critical strengths

The paper presents a coherent and complete investigation of a practically relevant problem. It identifies rubric–preference conflicts, develops a quantitative measure of stubbornness, and introduces a training-free prompt optimization approach to mitigate the issue. Evaluation across multiple judges, held-out datasets, and expert-written rubrics provides useful evidence for the effectiveness and transferability of the approach.

Critical weaknesses
  1. The stubbornness measure depends on whether each synthetic rubric genuinely favors its intended response, but the paper provides limited evidence validating this correspondence. If a criterion is ambiguous or its intended winner is incorrectly assigned, a reasonable disagreement with that assignment may be counted as stubbornness. This introduces uncertainty into the estimated extent of rubric noncompliance.

  2. The experiments do not sufficiently separate the benefit of making the task requirements explicit from that of iterative prompt optimization. The optimized instructions are substantially more detailed than the brief baseline instruction, leaving the additional benefit over a carefully designed manual prompt, including its transferability, insufficiently established.

Questions for authors
  1. How do you verify that each generated rubric is satisfied by its intended response and not by the other? Particularly for cases classified as stubborn, how do you distinguish genuine adherence to the judge’s prior from disagreement caused by an ambiguous rubric or an incorrect target assignment?

  2. The prior is determined by agreement across two presentation orders, which checks sensitivity to response order. I did not find a corresponding analysis of sampling uncertainty under identical inputs and a fixed order. What decoding settings are used, and how stable are the prior and stubbornness estimates across repeated evaluations?

  3. The baseline instruction is brief. Compared with a carefully designed manual instruction that explicitly prioritizes the rubric over overall response quality, how much additional improvement does iterative prompt optimization provide? Do these additional gains also transfer to held-out datasets and expert-written rubrics?

3 comments

We thank the reviewer for the careful reading and the concrete questions. Following your suggestions, we added a detailed prompt-engineered baseline and are repeating the construction of the tuned and baseline instructions, and we discuss the validity of the synthetic rubrics as a limitation.

W1/Q1. Validity of synthetic rubrics

As you point out, the current measurement assumes that each synthetic rubric unambiguously favors its target response, and it does not separate a judge that holds on to its prior from one that rejects an ambiguous or mis-targeted rubric. The latter is also counted as stubbornness, so the absolute level of stubbornness may be somewhat overestimated. Since every judge and instruction is evaluated on the same rubrics, however, we believe the measurement remains meaningful for comparing judges and for comparing a judge before and after tuning. The revision states this limitation in Sec. 6, and we leave re-measuring stubbornness on human-verified rubrics to future work. We believe the contribution of this work lies in constructing conflicts between the rubric and the judge's prior at scale from existing preference data, and in making it possible to measure quantitatively how often judges fail to follow such rubrics.

W2/Q3. Gains of automatic optimization over a detailed manual prompt

We added the suggested comparison to the revision. To obtain a carefully written instruction under controlled conditions, we asked a frontier LLM (gpt-5.6-sol) to act as an experienced prompt engineer. We gave it a detailed task description that explains the conflict between the rubric and the judge's prior and lists the requirements, sampled five instructions, and kept the one with the highest validation agreement for each judge (IrPEI_r^{\text{PE}}, Sec. 4.1). The task description is quoted in full in Appendix H. A comparison with instructions written by human prompt engineers would further complement this baseline, and we leave it to future work.

The table below compares the original instruction (IrI_r), the prompt-engineered baseline (IrPEI_r^{\text{PE}}), and RubricLens (Ir⋆I_r^\star) on the four datasets. Each value is averaged over the four judges; the per-judge results are in Table 2 (Sec. 4.1) of the revised paper.

InstructionUltraFeedback, tuning (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)HelpSteer3† (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)RewardBench† (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)RM-Bench† (Aflip\mathcal{A}_\text{flip} ↑ / S\mathcal{S} ↓)
IrI_r0.895 / 0.1240.934 / 0.0710.779 / 0.2220.839 / 0.171
IrPEI_r^{\text{PE}} (prompt-engineered)0.940 / 0.0690.960 / 0.0410.880 / 0.1220.890 / 0.117
Ir⋆I_r^\star (RubricLens)0.958 / 0.0490.965 / 0.0360.915 / 0.0850.910 / 0.096

†: held-out datasets not used for tuning.

IrPEI_r^{\text{PE}} raises flipped agreement by +0.056+0.056 over IrI_r on average, about three quarters of the gain of RubricLens (+0.075+0.075). Still, RubricLens achieves higher flipped agreement and a lower stubbornness rate than IrPEI_r^{\text{PE}} in all 16 judge–dataset cells (Table 2). Both methods have an LLM write the instruction, so the additional gain appears to come from GEPA's feedback loop, which rewrites the instruction from failure cases.

Q2. Decoding settings and stability across repeated runs

All judges decode with temperature 0. After reasoning within a fixed budget, the final verdict is restricted by constrained decoding to a single letter in {A,B}\{A, B\}. Because decoding is greedy, the evaluation itself involves no sampling.

The main remaining source of variation is how the instructions are built. IrPEI_r^{\text{PE}}, Ir⋆I_r^\star, and IrFSI_r^{\text{FS}} each come from a stochastic procedure. We are repeating each of these constructions for every judge, and will add a comment with the results as soon as the experiments are complete.

AuthorsOpenReview ↗

Update: results of the repeated runs (Q2)

As promised in our answer to Q2, and following your question on stability across runs, we repeated the construction of every stochastic instruction three times for each judge: for IrFSI_r^{\text{FS}} and IrPEI_r^{\text{PE}}, a new set of five candidates with a new validation-based selection, and for Ir⋆I_r^\star, a new GEPA run. Table 2 of the revised paper now reports the mean over the three runs. We also repeated the ablation, reasoning-budget, and transfer experiments three times.

Most of our claims hold with three runs: RubricLens improves over IrI_r in all 16 judge–dataset cells and on RubricBench for all four judges, in every run, and it outperforms IrFSI_r^{\text{FS}} in flipped agreement in all 16 cells. One claim in our answer to Q3 needs correction. The margin of RubricLens over IrPEI_r^{\text{PE}} is smaller than in the single run: IrPEI_r^{\text{PE}} raises flipped agreement over IrI_r by +0.059+0.059 on average, close to the +0.064+0.064 of RubricLens, rather than about three quarters of it. RubricLens is ahead of IrPEI_r^{\text{PE}} on UltraFeedback, HelpSteer3, and RewardBench, the two are tied on RM-Bench, and RubricLens is better in 14 of the 16 cells rather than all 16. We have corrected Sec. 4.2 accordingly. Nevertheless, GEPA's feedback loop still adds a small gain on average over this carefully written instruction.

Thank you again for your time and thoughtful feedback. Please let us know if any concerns remain or if there is anything we could clarify further.

Thank you for the clarifications and additional experiments. The stronger prompt baseline and repeated runs address most of my concerns about the comparison and stability.

I continue to appreciate the paper’s coherent investigation of rubric–prior conflicts and its transfer results on held-out datasets and expert-written rubrics. I therefore maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.