- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether LLM judges are consistent in their judgment of individual rules and their judgment of entire rulesets. Their experiments show that this is not the case, and that LLM judges tend to be more lenient when presented with entire rulesets, compared to if they are presented with rules individually. The paper further provides two ablations showing that this is both due to weaknesses in a model's reasoning, and in the way rulesets are written. Lastly, the paper shows that models can be trained to be more consistent in their judgments.
Critical strengths
The paper has two clear strengths:
- Problem importance and execution: The paper investigates a very important problem. Given the proliferation of LLM judges, it is crucial to know more about their weaknesses and biases. Given this problem, the paper does a solid job at coming up with the right experiments, and correct ablations to disentangle their results from potential confounders.
- Writing style: A special praise for the writing style of the paper. I really enjoyed reading it. The paper is very clearly structured, I had no problem following the main line of arguments, and all the expected parts of the paper are present.
Critical weaknesses
My main worry revolves around how novel the results of the paper are. There are some related works that I believe are very relevant and that are currently not discussed in the paper. I would appreciate to get a comparison of the author's work with the following papers:
- Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation, Yu et al., 2026
- RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following, Pan et al., 2026. Section 5.1 might contain some related experiments.
- DynaGuard: A Dynamic Guardian Model With User-Defined Policies, Hoover et al., 2025. They also do GRPO and their section 3.2 might contain similar argumentation than this paper.
I currently stay cautious with my rating and go with weak rejection. If the authors can sufficiently rebut these concerns, I'm willing to raise my score.
Questions for authors
- If I understand correctly, the paper currently only measures consistency between single-rule and rule-set judgments. Wouldn't it make sense to use a dataset that has a ground-truth? That way you can also measure which of the two strategies is "more correct", and whether fine-tuning does not only improve consistency, but also preserves correctness.
- I noticed that you are using an OpenAI dataset and that the two OpenAI models are performing best (i.e., they have the lowest inconsistency rate). Is it possible that this is because OpenAI models have already seen the questions during their pre-/post-training? I guess this wouldn't be the end of the world, because your primary goal is not to measure how different models compare against each other, but it would still be interesting to see a re-run of the experiments on a more neutral dataset.
5 comments
Addressing W1: Novelty, Point (c)
W1: Novelty
W1(c):
Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation, Yu et al., 2026
As noted in the ICLR Reviewer Policy, this work is contemporaneous. Our work differs from theirs in significant ways, which we describe below.
Setting: Yu et al. focus on the rubric-based grading setting, evaluating on tasks like HealthBench, FLASK, and ResearchQA. Rubrics are usually sample-specific, are much shorter than constitutions, have relatively unambiguous ground truths, and don’t involve hierarchies. In contrast, we study the constitutional alignment setting, where we have a fixed but much larger ruleset, which may or may not have hierarchies, that is applied to every scenario. This difference in setting impacts the kind of questions we ask in our analysis.
Significant aspects of our analysis of inconsistency not covered by Yu et al.:
- Yu et al. use agreement metrics, some of which are similar to our measure of inconsistency, but they do not evaluate which direction (individual rule judgments or full ruleset judgments) causes more or less lenient verdicts. Our finding that the shift in verdicts is mostly towards leniency has significant safety consequences in the constitutional alignment setting.
- Our Figure 4(b) demonstrates that models are very good at aggregating verdicts of individual rules produced in the same call towards a ruleset-level judgment. Ruleset-level judgments aren’t considered by Yu et al., but they are very important for the constitutional alignment setting, as evidenced by our leniency findings.
- Yu et al. only examine rubrics of sizes 2, 4, and 8 in their scaling analysis (see their Figure 3), whereas we analyze ruleset sizes from 2 to 57 rules (see our Figure 3). We show that inconsistency continues to get much worse (~40-80% eventually) than what is observed at the sizes tested by Yu et al., and that this trend is true of models much larger than those they test, contradicting their “hypothesis that stronger models have implicitly learned partial invariance through scale”.
- We investigate the case with ruleset hierarchies, again a very important consideration for the constitutional alignment setting (see Appendix B, Figure 7 and Figure 8), which their analysis does not.
Significant differences in our interventions:
- Yu et al. treat rubric interference entirely as a training issue. As we argue in Section 2.2, inconsistency can result from both reasoning errors in the judge model and poor ruleset design. We argue that ruleset designers should explicitly decide what kinds of rule interactions are allowed and which intervention mechanism suits their setting.
- Our results from RuleForge show that inconsistency can be reduced even with the judge model held fixed by adding explicit hierarchies that ruleset designers can review.
- Our training also differs from theirs. We use GRPO while they use OPSD. Their approach succeeds on ResearchQA, about doubling their exact match variation of rubric interference, but has modest results for HealthBench and FLASK (less than 0.20 exact match improvement), with sometimes little to no difference between the trained and base models (see their Table 3). In comparison, our GRPO experiments halve the inconsistency rate for Gemma-4-31B and reduce the inconsistency rate to 0.063 for Qwen3-8B.
Reliability of results:
- All of our results are backed by rigorous statistical practices and include uncertainty quantification to help readers understand our claims and their limitations.
- We evaluate a more diverse and recent set of open- and closed-weight models, including Olmo-3-32B, Gemma-4-31B, GPT-OSS-120B, Muse-Glimmer-30B, GPT-5.6 Luna, and Claude Haiku 5.5.
- In the abstract, Section 3.2, and Table 1 of their paper, Yu et al. mention ‘subsetting’ as one of the four metrics for rubric interference they analyze. However, we were unable to find results for this metric in the main text or appendix of their paper.
Action: We have revised the related work section to discuss these works and our novelty in comparison.
Addressing Q1 & Q2
Q1: Ruleset Choice
If I understand correctly, the paper currently only measures consistency between single-rule and rule-set judgments. Wouldn't it make sense to use a dataset that has a ground-truth? That way you can also measure which of the two strategies is "more correct", and whether fine-tuning does not only improve consistency, but also preserves correctness.
There are only 2 official, updated, and publicly available constitutional AI rulesets: Claude’s Constitution and OpenAI’s Model Spec are the only two publicly available constitutions that are actively deployed and being updated in public releases. We choose OpenAI’s model spec because it contains an explicit hierarchy and clearer rule headers than Claude’s Constitution.
Constitutional alignment doesn’t have a simple ground truth: Constitutional alignment is inherently subjective in what compliance or violation entails. As mentioned in the related works section, prior work has shown that interpretive ambiguity is inherent in constitutional judgments, and instead evaluation has centered on calculating agreement between judge models. Benchmarks with labels do exist for synthetic rulesets, but we wanted to study realistic constitutions and rules used in deployment. Therefore, we center our paper on OpenAI’s model spec. In the revised PDF, we have clarified our choice of OpenAI’s model spec and the lack of ground truth in constitutional alignment as a footnote in Section 2.
Action: We have elaborated on our choice of OpenAI’s model spec and the lack of ground truth in constitutional alignment in Section 2.
Q2: Re-running Experiments on Neutral Dataset
I noticed that you are using an OpenAI dataset and that the two OpenAI models are performing best (i.e., they have the lowest inconsistency rate). Is it possible that this is because OpenAI models have already seen the questions during their pre-/post-training? [...] it would still be interesting to see a re-run of the experiments on a more neutral dataset.
Model specs from the same family may help but don’t remove inconsistency: We appreciate the concern for possible data contamination. However, we do not believe contamination is an issue for the inconsistency metric. Even if OpenAI has trained gpt-oss-120B and GPT-5.6-Luna on the OpenAI model spec, their models still show concerningly high inconsistency rates (>0.4, i.e., 2 of every 5 scenarios will get judged inconsistently) on our simplified version of OpenAI’s model spec (see Figure 3). As discussed in the limitations section of our original PDF, we believe if models cannot consistently reason about simplified rulesets, they are unlikely to consistently reason about more complex rulesets. If OpenAI hasn’t trained on the OpenAI model spec, then there is no issue of data contamination.
Additionally, note that models from other families also exhibit inconsistency. We have updated the revised PDF with results for Muse-Glimmer-30B and Claude Haiku 5.5. Notably, in Figure 3, Muse-Glimmer-30B has inconsistency rates very similar to GPT-5.6-Luna, which has the lowest inconsistency rates, as you pointed out.
OpenAI models haven’t been trained on WeirdChat: The held-out set of scenarios includes WeirdChat scenarios, which were released after both gpt-oss-120b and GPT-5.6-Luna’s training cutoffs.
Neutral constitutional alignment rulesets don’t exist: We agree it would be interesting to evaluate over a neutral constitutional alignment ruleset, but as we mentioned in Q1: Ruleset Choice, there are only two publicly released and actively maintained constitutional rulesets. We acknowledge this is a limitation, but we want to use a realistic ruleset.
Our experiments are computationally costly: Obtaining all results required approximately 2.8k GPU-hours and $1k in API costs. While we would like to conduct a thorough investigation into a neutral ruleset (assuming such a ruleset exists) it would be cost-prohibitive. We have added GPU-hours and API costs to Appendix A.1.
Action: We have updated the revised PDF with results for Muse-Glimmer-30B and Claude Haiku 5.5.
Action: We have clarified the insignificance of data contamination in Section 2.
Action: We have listed the GPU-hours and cost required for all experiments in the paper in Appendix A.1.
What do you think?
Thank you again for taking the time to review the paper! We believe your feedback has helped strengthen our paper. Have we addressed your concerns with the paper? If not, what further clarification or modifications could we make to improve it?
Addressing W1: Novelty, Points (a) and (b)
Thank you so much for taking the time to give us a thorough review. We are very grateful and glad that you found multi-rule inconsistency to be a “very important problem” substantiated by our experiments and “very clearly structured” writing.
W1: Novelty
Per-Paper Response: Thank you for bringing these works to our attention. We discuss below why our paper provides substantial differences in framing, experiments, and methodology compared to these papers.
ICLR Reviewer Policy: We would like to kindly mention that the ICLR Reviewer Policy mentions that a lack of comparisons to papers published on or after July 17, 2026, or published exclusively at a non-peer-reviewed venue like arXiv “cannot be a basis for rejection.” DynaGuard is the only paper that isn’t considered “contemporaneous” by this criterion, and we believe the scope of our work is very different (see discussion below). Nevertheless, we appreciate the reviewer bringing our attention to more recent works, and we have updated our related works to include clarifications on how we differ.
W1(a):
Framing: DynaGuard creates DynaBench, a dataset to evaluate judge models’ abilities to determine compliance with basic, user-defined rules. They then train models to do better on DynaBench. We are different, focusing on inconsistency in judgments made by models when judging natural language rulesets during rule-based alignment, which they do not discuss.
Experiments & Methods: The only substantive overlap is their Section 3.2, as you pointed out. DynaGuard assumes that models “perform best at judging one rule at a time,” but doesn’t substantiate the assumption. They then train models to output reference labels created by a strong teacher in DynaGuard, whereas we train a judge to have better self-consistency, so our optimization targets are different (see Appendix F of our revised PDF). Our other experiments, such as our in-depth analysis of inconsistency, leniency, hierarchical evaluations, and RuleForge, are not present in DynaGuard and instead fill a crucial gap left by the unsubstantiated assumption. Additionally, DynaGuard evaluates against the ground truth of DynaBench. In the constitutional alignment setting, we don’t assume that a simple ground truth exists (see Q1: Ruleset Choice below).
W1(b):
Framing: Pan et al. evaluate LLM judges’ abilities to accurately determine if a response satisfies a rubric and create a dataset for measuring this ability. We are substantially different since we thoroughly evaluate inconsistency (see the framing discussion in Yu et al. (2026)).
Experiments & Methods: The only experiment that overlaps with ours is Section 5.1, as you pointed out, where they see whether checklist-level (we call this same-call judgment) or rubric-level (we call this separate-call rule judgments) prompting is better for Qwen2.5-32B-Instruct and GPT-4.1. We measure inconsistency as the difference between full ruleset judgments and separate-call rule judgments. They compare against ground truth generated by their dataset, finding rubric-level judgments are better.
Similar to DynaGuard, our framing and experiments greatly differ from theirs. Constitutional alignment doesn’t have a simple ground truth (see the discussion of Q1: Ruleset Choice below), so we measure inconsistency instead. We thoroughly analyze inconsistency, provide judge model and ruleset solutions, and operate over a more recent, diverse set of scenarios and models (see discussion of Yu et al. (2026) below).