- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explores a fundamental but rarely systematically tested hypothesis in the field of "Large Model as a Judge" (LLM-as-a-judge): given explicit rubrics, do pairwise judges truly adhere faithfully to those rubrics rather than relying on the model's own internal prior preferences? To address this question, this paper makes three main contributions:
-
Proposes the "Stubbornness Rate" (S): First, it defines the judge's prior preference by the consistent choice of a judge without rubrics in a bidirectional presentation order. Then, it constructs conflicting rubrics that favor prior rejections , thereby quantifying the proportion of judges that disregard rubrics and adhere to their own priors. Experiments show that four mainstream open-source evaluators ignore conflicting scoring rules in 6.2%–23.6% of their judgments across four preference datasets. Furthermore, on open-ended preference datasets, the higher the evaluator's prior confidence, the more likely it is to violate the scoring rules.
-
We propose the RubricLens cue optimization framework: without modifying the model weights, we use a generative model (gpt-5.6-luna) to synthesize two opposing single-criteria scoring rules for each preference answer pair (each biased towards one of the two candidate answers). This is combined with the reflective cue evolution algorithm GEPA (using gpt-5.6-sol as the reflective model) to optimize the evaluator instructions , forcing the model to strictly adhere to the given evaluation criteria under both bidirectional order and positive/negative rules.
-
Validating generalization across datasets and rule types: After tuning the instruction only on the synthetic scoring rules in UltraFeedback, RubricLens improved the average consistency () under flipped rules from 86.2% to 93.7% across all 16 "judge-dataset" combinations, reducing the stubbornness rate to a maximum of 12.0%; at the same time, when the instruction was migrated to RubricBench with zero samples, it also improved the original consistency by 1.2–4.0 percentage points.
Critical strengths
-
The problem is precisely addressed and has high practical value: Current communities heavily utilize scoring rules to standardize LLM evaluators or provide reinforcement learning rewards (Rubrics as Rewards), but generally only report raw consistency . This paper astutely points out that is highly saturated on existing datasets (0.959–1.000), failing to distinguish whether the model "truly understands and follows the rules" or "just happens to choose the correct answer based on its prior knowledge." By introducing flip consistency and persistence rate , this paper provides a more discriminative benchmark perspective for evaluating the evaluator's rule-following ability.
-
The experimental design was meticulous, effectively eliminating common confounding factors:
-
Eliminating positional bias: All indicators were calculated and averaged under two presentation orders: and . The prior set was strictly limited to samples selected consistently by the ruleless evaluator under both orders, avoiding positional bias from contaminating the measurement of prior preferences.
-
Eliminating lexical surface matching shortcuts: Appendix H rigorously tested the results on RubricBench using TF-IDF bag-of-words (BoW) cosine similarity, demonstrating that the optimized instruction did not increase the proportion of highly overlapping responses (maintaining at 0.567–0.577), and the gain was even greater on samples without BoW preference, effectively ruling out the assumption of "judging solely based on surface word overlap".
-
Critical weaknesses
-
Lack of comparison with other baselines
-
Lack of comparison with a manually enhanced prompt baseline: The paper only compares the RubricLens-optimized instruction with the minimalist original instruction (e.g., "Determine which answer better satisfies the following scoring criteria") in the experimental section. contains almost no adversarial bias, emphasizes "rule-based" approaches, or ignores external prior conflicts—these are the meta-cue constraints. This leads to a serious experimental confounding factor: Is the performance improvement brought by RubricLens due to the specific strategies automatically searched by GEPA (such as complex trade-off mechanisms), or simply due to the regularization effect of "longer prompts, stronger emphasis, or the introduction of general rule warnings" (i.e., the general prompt complexity/length effect)? Without a manually enhanced prompt baseline carefully crafted by human experts, the necessity of automated prompt optimization cannot be demonstrated.
-
Lack of comparison with other baselines that do not alter model weights (such as in-context learning examples or general thought chain prompts): While RubricLens' core advantage is its non-altering model weights, many mature, lightweight techniques beyond simple system prompts are not included in the comparison within the "non-altering model weights" prompt engineering paradigm:
-
In-context learning (Few-Shot Prompting): Providing several input-output demonstration samples in the prompt where "the details conflict with the prior but successfully conform to the details," rather than modifying the instruction itself.
-
General/open-source advanced reasoning/chain-of-thought prompts: For example, standard thought chain templates ("Let's think step by step and strictly follow the rubric even if it contradicts your intuition").
-
-
-
The Justification and Potential Negative Effects of "Forced Adherence to Conflicting Rules" (Reward Hacking Risk) on Datasets Containing Objective Facts/Logical Errors (RewardBench, RM-Bench)
-
In RewardBench (math-prm math problems and hep-code problems) and RM-Bench (concise style answers injected with subtle factual errors), manually rejected answers contain explicit objective errors (incorrect solutions, flawed code, or factual fallacies).
-
According to the constraints in Appendix F (Figure 5), the synthesizer gpt-5.6-luna must write a "general, positive, good answer trait like that a rigorous evaluator would write" standard for incorrect answers , and the incorrect answer must explicitly meet this standard while the correct answer must explicitly not. When two answers in RM-Bench are styled to be identical (both concise), the synthesizer, in order to favor the incorrect answer, often only grasps some marginal formatting, structural, or phrasing features, or indirectly caters to the incorrect content itself.
-
More importantly, the optimized instruction shown in Figure 3 explicitly commands the evaluator: "Do not automatically favor answers that are factually more correct, more complete... unless the scoring criteria explicitly require these qualities; even if an answer has serious flaws in aspects not mentioned in the scoring criteria, it must be selected as long as it better meets the literal standard." In practical applications, human-written criteria rarely exhaustively list all implicit bottom lines (for example, a criterion requiring "summarize in a list" usually implicitly assumes "no fabricated facts"). If an evaluator optimized with RubricLens is used for reinforcement learning reward modeling (RLHF/RLVR), this deliberate blindness to factual errors can easily induce serious reward hacking problems. Although the paper briefly mentions in its limitations that "it is assumed that even if the rules are wrong, they should still be followed," it lacks a specific analysis of the form of the rules for synthesizing conflicts in the objective task, and does not discuss how to distinguish between "reasonable anti-prior criteria" and "degenerate marginal criteria."
-
Questions for authors
-
Regarding the comparative validation of manually enhanced prompts: The control group instruction in the paper is too basic and fails to examine the effect of strong prompts written by human experts. Did the authors test manually written general enhanced prompts (e.g., a 200-300 word manual anti-stubbornness prompt, rather than instructions automatically searched through GEPA) on the validation or test sets, which also include "emphasizing the rule of thumb, eliminating prior interference, and paying attention to multiple standard trade-offs"? Compared to these zero-cost manually enhanced prompts, what is the marginal performance gain of RubricLens through automated search?
-
Regarding comparisons with other baselines that do not change model weights (such as Few-Shot ICL or general thought chain templates): In similar baselines that keep model weights unchanged, why not try introducing a small number of successful examples of compliance with conflict rules (In-Context Learning / Few-Shot) or standard general defensive inference templates (such as CoT prompts with strong alignment warnings) into the prompts? When computational costs and inference latency are comparable, where does the core performance advantage of GEPA's automatically generated instruction compared to these general enhancement hint schemes that do not require tuning (especially the improvement in flip consistency ) lie?
-
Regarding the boundaries of "blindly conforming to rules" and the risk of reward hacking in objective error tasks:
- In the code/mathematical subset of RewardBench and RM-Bench, rejected answers often contain objective logical, arithmetic, or factual errors. According to the cue word constraints in Appendix F, the synthesizer gpt-5.6-luna must write a "positive, general, and winning" criterion for these incorrect answers. However, human-written scoring rules often contain unspoken bottom lines (such as "factual correctness must be guaranteed"). If a RubricLens-optimized judge, forced to "ignore factual flaws not mentioned in the rules," is directly applied to reinforcement learning reward modeling (RLHF/RLVR), will this induce serious reward hacking problems? Under what circumstances should the judge have the ability to "reject" or "conversely overturn flawed rules"?
3 comments
Following a question from Reviewer jDSD (Q2) on stability across runs, we repeated the construction of every stochastic instruction three times for each judge: for and , a new set of five candidates with a new validation-based selection, and for , a new GEPA run. Table 2 of the revised paper now reports the mean over the three runs. We also repeated the ablation, reasoning-budget, and transfer experiments three times.
Most of our claims hold with three runs: RubricLens improves over in all 16 judge–dataset cells and on RubricBench for all four judges, in every run, and it outperforms in flipped agreement in all 16 cells. One claim in our answer to Q1/Q2 needs correction. The margin of RubricLens over is smaller than in the single run, about / on average rather than / . RubricLens is ahead on UltraFeedback, HelpSteer3, and RewardBench, but the two are tied on RM-Bench, so it is no longer best on every dataset. We have corrected Sec. 4.2 accordingly. Nevertheless, GEPA's feedback loop still adds a small gain on average over this carefully written instruction.
Thank you again for your time and thoughtful feedback. Please let us know if any concerns remain or if there is anything we could clarify further.
Thank you for providing the additional baseline experiments. However, marginal technical gain over basic prompt engineering (): The newly added baseline reveals that a simple zero-shot prompt-engineered instruction ()—requiring no synthetic data generation or GEPA optimization—already captures 71.4% to 83.9% of the total improvement in across all datasets (leaving only a 0.5% marginal gain on HelpSteer3), indicating that the vast majority of the reported gain comes from simply replacing the overly simplistic baseline rather than the proposed RubricLens pipeline.
We thank the reviewer for the careful reading and the concrete questions. In response, we added two non-training baselines, a prompt-engineered instruction and a few-shot instruction, and clarified the role of conflicting rubrics and the risk of reward hacking.
Q1/Q2. Comparison with other non-training baselines
In the revised version of the paper, we added two non-training baselines (Sec. 4.1, Table 2). The first is a detailed instruction (IrPE) written by an LLM acting as a prompt engineer from a task description that follows the specification in your question; the full task description is in Appendix H. The second is a few-shot instruction (IrFS) that adds four examples in which the rubric conflicts with the judge's prior. We did not test the suggested CoT prompt separately, because every judge in the paper already reasons with a 1,024-token thinking budget before giving its verdict.
The table below compares the original instruction (Ir), the two new baselines (IrFS and IrPE), and RubricLens (Ir⋆) on the four datasets. Each value is averaged over the four judges; the per-judge results are in Table 2 (Sec. 4.1) of the revised paper.
RubricLens achieves the highest flipped agreement and the lowest stubbornness rate on every dataset, about +0.02 / −0.02 over IrPE on average. Since IrPE and Ir⋆ are of similar length, the difference between them is unlikely to be explained by prompt length.
Q3. Following conflicting rubrics and the risk of reward hacking
On RewardBench and RM-Bench, the conflicting rubrics act as a stress test of whether a rubric can override even the judge's sense of correctness: a conflicting rubric favors the response that the label rejects, which on these datasets is the incorrect answer (Sec. 2.2). As you point out, such rubrics may differ from those used in practice, and we leave building a benchmark with more realistic rubrics to future work.
Regarding the risk of reward hacking, we believe that a judge that does not follow the rubric may not be safer. If the judge mixes in its own preferences that are not in the rubric, it becomes hard to tell whether a verdict comes from the rubric or from the judge's preference, and thus what a policy is exploiting. With a judge that follows the rubric, by contrast, the cause of reward hacking can at least be traced back to the rubric, which can then be revised. We agree that following a flawed rubric can be risky; when a judge should reject such a rubric is beyond the scope of this work and left to future work (Sec. 6).