acceptodds
Under review as a conference paper at ICLR 2027

Safety is Contextual, LLM-Judges are Not: Navigating the Rigid Priors of Evaluators

Abstract

LLMs-as-judges are the primary way to evaluate safety at scale. Despite their importance, LLM-judges themselves are rarely evaluated beyond human agreement in simple, static benchmarks. We therefore investigate two under-explored but crucial properties of LLMs-as-judges: their **sensitivity** to in context-information, and their **steerability** to differing safety definitions, which may not align with their internal safety priors. We evaluate the safety judging abilities of 13 generalist LLMs and safety-specific judges, and investigate the impact of novel in-context information and changing safety definitions. We find that while LLM-judges can learn from new information, they are broadly unlikely to adjust their evaluations if the context or safety definition contradicts their prior.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
llm-judges, evaluation, safety, context, steerability, alignment

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_BrvJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper meta-evaluates LLM safety judges beyond standard human-agreement benchmarks, formalizing two overlooked properties: context-sensitivity (whether a judge updates its prediction when given novel in-context information about the prompt) and steerability (whether a judge follows a modified safety policy rather than its internal safety prior). Evaluating 13 judges (generalist and safety-specific, open- and closed-source) on three datasets — including a new multilingual human-annotated dataset (French, Arabic, Japanese, Korean), a new NovelPrompts dataset containing post-cutoff concepts, and Sorry-BENCH — the authors find that judges can absorb in-context information mainly when their priors are weak (prediction flips correlate negatively with term frequency in pre-training corpora), but are broadly unable to adapt to modified safety policies, with steerability improving substantially only when the task is masked as an arbitrary classification task. The authors release NovelPrompts and an evaluation framework.

Critical strengths
  • Well-motivated and clearly formalized problem: the paper identifies that human-agreement benchmarks miss two practically crucial properties — sensitivity (for temporal drift/novel concepts) and steerability (for policy customization across cultures/domains) — and gives both a clean formalization and a principled Bayesian-prior interpretation of the results.
  • Broad, well-controlled empirical study: 13 diverse judges, human-annotated multilingual data, controlled conditions (no/irrelevant/correct context; base/variant/absurd policies; 5 seeds), with findings that are largely consistent across three datasets and supported by supplementary analyses (e.g., classification-masking, prior-strength correlation).
  • Actionable findings and artifacts: the result that steering fails even when definitions are clear, that sensitivity/steerability/accuracy are distinct properties, and that masking safety as classification roughly doubles steerability, are directly useful to practitioners; datasets and framework are released.
Critical weaknesses
  • The steerability evaluation covers a narrow slice of the policy space: only two definition variants (plus an absurd toy policy), no systematic variation of "distance" from the judge's prior (e.g., single-category changes, lenient direction, severity-level redefinitions). The strong claim that judges are "broadly not steerable" may overgeneralize from these two variants, especially given prior work (e.g., Weng et al., 2026) found some adaptation to simple strict-to-lenient changes.
  • Statistical rigor is thin: datasets are small (NovelPrompts n=194; MultilingualPrompts n=779), outcomes are binary flips, and most comparisons are reported without significance tests or confidence intervals; reported Pearson correlations (e.g., 0.55–0.65 for steerability across definitions) lack CIs, and no correction for multiple comparisons across 13 judges.
  • The prior-strength proxy (word frequency in FineWeb) is coarse and judge-agnostic; it cannot capture per-model training data or knowledge, so the central Bayesian interpretation (sensitivity ∝ weak prior) is only weakly evidenced, as the authors themselves partially acknowledge.
  • The setup is simpler than production judging (binary safe/unsafe on prompts, single policy), and the practical recommendations (choose sensitive/steerable judges) are not validated in any deployment-like scenario; the classification-masking "mitigation" is presented without discussing its risks.
Questions for authors
  • Have you tested intermediate policy distances (e.g., changing a single category, the lenient direction, or severity thresholds)? Does steerability degrade monotonically with distance from the judge's prior, and how do you reconcile your conclusion with Weng et al.'s finding that some judges adapt to simple definition changes?
  • Given the small sample sizes, can you report confidence intervals/significance tests for the flip-rate differences and correlations, and show that judge rankings (e.g., Fig. 3) are stable under subsampling?
  • Can you validate the prior-strength proxy with a judge-specific measure (e.g., perplexity, explicit knowledge elicitation), and show the negative correlation with word frequency holds when controlling for prompt ambiguity/difficulty?
  • What are the failure modes of the classification-masking trick in practice (e.g., loss of severity-sensitive judgment, adversarial reframing), and would you recommend it for production safety pipelines?
6 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for your comprehensive review and suggestions. We respond to each comment below and also refer to improvements we have made in the re-uploaded paper (changes are marked in blue).

W1/Q1: Steerability evaluation covers a narrow slice of the policy variants. Findings may overgeneralise from these two variants, especially given prior work.

Thank you for highlighting this, we agree that we overgeneralised by saying that safety judges were broadly unsteerable, and have revised our third contribution to: “We show that LLM safety-judges can be difficult to steer, and instead often rely on their internal safety boundary to judge, despite being instructed otherwise.”. Our key point was not to show that judges are never steerable, rather to show that providing a safety policy does not ensure that a judge will apply it faithfully.

Our variants already include the kinds of change you mention, and we have clarified this in §5 and §C.4. In particular, we clarified that Sorry-BENCH uses a different safety policy as it tests refusals, so we make different kinds of policy changes, further expanding the space of variants we investigate.

  • Severity threshold: in MultilingualPrompts and NovelPrompts, variant B makes low-severity (neutral, informational) mentions of any category unsafe.
  • Single-category changes in both directions: variant A makes self-harm content safe (more lenient) and sexual content unsafe (stricter), and variant B in Sorry-BENCH changes which categories are unsafe.
  • Lenient direction: on SORRY-Bench, variant A makes compliance with requests in two formerly unsafe categories safe.

We also include a no-definition control, the same rules under a classification framing, and an absurd sports policy.

We agree that Weng et al. [1] provide important evidence of adaptation, and our findings are partly consistent with theirs. Their study focuses on strict-to-lenient threshold shifts, applied to 4 judges evaluating agent trajectories. Our work complements it by testing category-specific lenient and stricter changes, severity-threshold changes, and the same rules under safety vs. arbitrary-classification framings. Table 6 shows that the direction of the change matters. Under safety judging, adaptation on examples whose gold label changes is lowest for the stricter variant B (5.7-10.1% on MultilingualPrompts and NovelPrompts), intermediate for the mixed variant A (13.7-39.1%), and highest for the purely lenient SORRY-Bench variant A (57.5%). In the lenient direction, our judges adapt more, in line with Weng et al [1]. Adaptation drops when the policy becomes stricter. This is not because stricter rules are harder to apply: under classification framing, adaptation to variant B on MultilingualPrompts rises from 9.3% to 65.0%. We have added this comparison to §5.

Finally, while our findings that classification steerability is over twofold higher than when judging safety (Figure 6) and that judges can follow the absurd sports definition (Figure 20) point to preliminary evidence that steerability degrades with distance from the judge’s prior, we have not conducted a systematic sweep of safety policies nor have we explicitly quantified distance to each model’s prior. We have now mentioned this in our limitations to ensure it is clear we are not making this claim.

W2/Q3: Statistical rigour.

Regarding datasets: we would push back that our datasets are of considerable size for safety datasets, especially considering they are human-generated, human-annotated, and natively multilingual. For instance, established benchmarks like SimpleSafetyTests [2] and XSTest [3] contain 100 and 450 prompts respectively. We also complement our work with Sorry-BENCH (n=1000) and our synthetic sports dataset (n=240) and report additional uncertainty estimates as described below.

Regarding binary flips: we do not think binary flips are non-rigorous in themselves, but we agree they do not show the full picture, and have now reported additional metrics on the direction of prediction changes (e.g., Table 3 and 6), as we detail in our response to Reviewer AXrG.

Regarding significance tests and CIs: we consistently report standard deviation across 5 random seeds, but we have now also added further statistical analysis:

  • Frequency vs. sensitivity: reported statistical significance Spearman ρ = −0.26, p < 0.001 on our key Figure 2; ρ = −0.15 unbinned (Figure 15); ρ = −0.14, p = 0.045 on OLMo-2's corpus (Figure 14).
  • Knowledge vs. sensitivity: logistic slopes with 95% CIs, e.g., β = −0.10 [−0.15, −0.05] (Figure 3; all settings in Figure 18).
  • Steerability: 95% CIs in Figures. 6 and 22, softened wording regarding steerability correlations to say that “steerability to the two definitions tends to be positively correlated across judges (Pearson r = 0.55–0.65 across the three datasets, p = 0.02–0.06” in Section 5 and E5.

[continued in the next comment]

AuthorsOpenReview ↗

Response Part 2

Regarding sensitivity rankings: we followed your suggestion and tested whether the rankings in Figure 4 are stable under subsampling. We resampled prompts with replacement 2,000 times (within each dataset and language) and recomputed flip rates and ranks each time. The extremes of the ranking are robust (e.g., Nemotron is in the top three of all four experiments in 91% of resamples), while mid-ranked judges have overlapping rank intervals. Rankings also replicate across datasets (Spearman ρ=0.80\rho = 0.80, 95% bootstrap interval [0.41, 0.90]) and are concordant across all four experiments (Kendall’s W=0.88,p<0.001W = 0.88, p<0.001). We summarise this in a footnote in §4.2.

Regarding multiple comparisons: our main claims rest on tests pooled across judges (the frequency correlation, the knowledge regression, the sign test across datasets) rather than on per-judge tests, which are descriptive. Where we report the same analysis across several settings, we describe the consistency of its direction rather than claiming significance in each.

Thank you for your help ensuring our findings are statistically robust.

W3: The prior strength proxy is judge-agnostic.

You are absolutely correct that our prior strength proxy is judge-agnostic. While we do believe that judges have similar priors due to large overlap in their pre-training datasets (e.g., mostly draw on internet-scrapes like CommonCrawl and its derivatives [4,5]), we agree that it would strengthen our claim to explore each judge’s specific prior knowledge, as it is true that judges may have different priors and levels of understanding over the concepts mentioned in the prompts, and this may cause variability in their sensitivity to the same context. To do this, we performed an additional experiment eliciting each judge’s understanding of the key concept mentioned in the prompt, and comparing this to the context definition that the human annotators had provided. We sampled 10 responses at high temperature to get a more calibrated idea of each judge’s understanding. Our results show that judge prior knowledge is also correlated to context-sensitivity across our 4 setups. Judges change their predictions significantly more often for concepts they know less well. For example, in NovelPrompts, each additional point of understanding lowers the odds of a flip by about 10% (β=−0.10\beta=-0.10, 95% CI [-0.15,-0.05], p<0.001p<0.001), and the fit for each individual judge slopes downward for 9 of the 10 judges. We believe this new analysis considerably strengthens our claim, and have added it to the main body of the paper (Figure 3), with full experimental details and supplementary results in Appendix D4.

W4a: The setup is simpler than production judging.

Thank you for raising this point. Our primary concern is practitioners using LLM judges to evaluate a candidate model before deploying it, for example, estimating its policy-violation rate, comparing release candidates, or deciding whether it satisfies a deployment policy. This closely reflects the evaluation procedures reported by major providers:

  • OpenAI uses LLM graders to classify candidate-model completions as not_unsafe [6],
  • Cohere uses LLM judges to identify unsafe generated content [7],
  • Anthropic automatically scores responses before deployment [8], and
  • Meta evaluates violation and false-refusal rates using safety classifiers [9].

Our findings directly apply to these evaluations: if the judge is insensitive to required context or cannot follow the intended policy, the resulting pre-deployment safety measurements may provide false confidence. We released NovelPrompts and our evaluation framework to support pre-deployment testing of whether a selected judge incorporates necessary contextual information and follows a supplied policy.

We do not claim to have solved end-to-end production judging. Our methods provide actionable pre-deployment tests that reveal whether a judge actually uses the supplied policy and context or instead defaults to its latent safety boundaries--a failure mode that standard human-agreement metrics do not detect. If a judge is subsequently used in a live environment, it must be tested against its particular deployment scenario, policy, languages, content distribution, and available context.

[continued in the next comment]

AuthorsOpenReview ↗

Response Part 3

W4b/Q4: What are the failure modes of classification masking?

Thank you for highlighting this interesting point. Classification masking also introduces important risks. A judge asked for safe or unsafe can fall back to its latent safety boundary when the supplied policy is incomplete or absent. Arbitrary labels such as Class 1 and Class 2 remove that semantic fallback. The policy must therefore comprehensively define the relevant harm categories, content types, and edge cases.

Practitioners could evaluate the Class 1/Class 2 formulation on a representative development set, inspect the cases it misses, and iteratively refine the policy and prompt template. This requires ensuring that the evaluation set represents the intended deployment distribution and preserving a separate held-out test set to avoid overfitting. We present classification masking and judge selection as empirically supported directions for improving pre-deployment model evaluation, not as complete solutions for production judging. We have made a new section in our Appendix (E3.1: Recommendations for use of classification masking in practice) where we now discuss this explicitly.

Thank you again for your engagement with our work. We hope our improvements of statistical rigour, paper revisions on steerability wording and classification masking, and our new experiment eliciting judge knowledge, will give you confidence to increase your score. Please let us know if you have any further comments or concerns.

[1] Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test https://arxiv.org/pdf/2605.06161

[2] SIMPLESAFETYTESTS: A Test Suite for Identifying Critical Safety Risks in Large Language Models https://arxiv.org/pdf/2311.08370v2

[3] XSTEST: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models https://aclanthology.org/2024.naacl-long.301.pdf

[4] LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/html/2302.13971v1#S2.SS1

[5] Language Models are Few-Shot Learners https://arxiv.org/html/2005.14165v4#S2.SS2

[6] GPT-5 System Card https://deploymentsafety.openai.com/gpt-5

[7] Command A: An Enterprise-Ready Large Language Model cohere.com/research/papers/command-a-technical-report.pdf

[8] Claude Sonnet 4.5 System Card assets.claude.com/963373e433e489a87a10c823c52a0a013e9172dd.pdf

[9] The Llama 3 Herd of Models https://arxiv.org/abs/2407.21783

Reviewer_mRQwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates LLM judges as they relate to safety classification, in particular their ability to take into account new information in-context in a way that meaningfully affects their classification decisions. The authors focus on two aspects: context sensitivity (do judges take into account supplemental factual information with the prompt) and steerability (can judges be adapted to a new safety policy post-hoc via prompting). The experiments show that judges are by default somewhat able to take into account new factual knowledge in-context but are not easily adaptable to new safety policies, except when replacing the safety classification task labels with neutral numeric labels. To conduct experiments, the authors also release two new safety datasets which are designed to measure sensitivity and steerability.

Critical strengths
  • this line of research is around creating dynamic safety judges that can by quickly updated to adopt new policies or information, which is an important practical concern for safely deploying LLMs in dynamic environments
  • the finding that judges can be made more steerable by replacing the safe/unsafe labels with a generic number class, while not very surprising, is relevant to the field and can be easily adopted by practioners
Critical weaknesses
  • the proposed NovelPrompts dataset contains several incoherent, incomplete, or borderline examples that are labeled as harmful but whose harmfulness seems debatable. Safety datasets ought to be unambiguous to avoid artificially reducing classification performance due to unclear examples and to reduce noise in the evaluation. The low quality of this dataset leads me to question any results building on it.
  • a core claim of the paper "judges only use context when their priors are weak" is not supported by the evidence shown. This claim relies primarily on a downwards trend in Figure 2b, which is not supported by the underlying data for at least three reasons: 1) even a straight line would lie entirely within the error bars, 2) the OOV class is extremely small with N=12, leading to significant noise, and 3) the pre-training corpus FineWeb-2 is not used by most of the models in the study, making it a questionable proxy for the actual strength of prior knowledge of the surveyed models.
Questions for authors
  • in NovelPrompts, what do the csv columns actually represent and which are model-derived vs human labels? Some names like "safety_assessment_without_context" are not self-explanatory.
  • L392: if judges only change about 5% of their predictions, how can their accuracy drop by 15%?
  • Do you have clearer data to support the headline claim in 4.2?
  • In L380 you state that the "results suggest that your safety definition broadly aligns with those the LLMs were trained with". This could be verified in a straightforward way by comparing your baseline policy with the safety policies of the models in the study. Have you investigated how your baseline policy mu_s compares to the LLM providers' safety policies?
  • What is your explanation for why even the best judges only reach an accuracy of max 69% on MultilingualPrompts given that other safety benchmarks typically report very high safety classification accuracy (often above 95%)?
6 comments
AuthorsOpenReview ↗

Response Part 1

Thank you very much for your thoughtful review of our work. We respond to your comments below. We have updated our manuscript in light of your feedback, highlighting all changes in blue.

W1: The NovelPrompts dataset appears of low quality with debatable harmfulness labels.

Thank you for raising this, we agree that dataset quality is very important, particularly in safety, and we may not have sufficiently documented the steps we took to ensure this. Each of the prompts was professionally and human generated by an external data company specialising in the curation of safety data (Alice.io), then quality checked by a trained team of safety data annotators. Synthetic completions from 5 LLMs were then obtained, and the same team of annotators annotated both the prompts and completions following clear and detailed safety guidelines, which we detailed in Appendix C1 and have now also detailed in the official dataset README, with links to the full annotation instructions and data collection request. Annotation was thus done rigorously and systematically according to our policy. Any disagreement with the harmfulness of the prompts is likely due to disagreement with our policy. We provide granular safety categories and severity labels for this exact reason: practitioners can relabel or filter the dataset according to their policy and requirements.

Regarding apparent incoherence or ambiguity, this is intentional. The dataset is designed to be very difficult without proper understanding of the context. As such it contains references to niche events, uses obscure slang, and contains unusual phrasing. For example, the "LeBron" prompt in Table 4 looks nonsensical without context, but with the annotators' explanation it is a request involving the sexual exploitation of a child. We believe this is a strength of our dataset, and allows safety judges to be stress-tested.

We hope this alleviates concerns on our dataset and thank you for helping us improve clarity of the documentation, as we believe the dataset is a truly unique resource and hope it will benefit the community.

W2/Q3: Not enough support for the claim that judges only use context when their priors are weak.

Thanks for raising this, we agree that the evidence we presented was lacking in some aspects. We have now done more experiments and analyses to clarify the three main pieces of evidence we have to support this point and revised the paper accordingly. We have also rephrased the claim to “Judges use context more when their priors are weak” to more precisely reflect our results.

A) Judge sensitivity on a prompt is correlated to its frequency in two large pre-training corpora.

We find that judge sensitivity on each prompt is correlated to how frequent the concepts mentioned in the prompts are in large pre-training datasets. We believe this is a valuable analysis as pre-training frequency can be associated with the strength of the judges’ priors (beyond their understanding). While you are correct that FineWeb-2 may not used by most models in our study, pre-training datasets are often quite similar, as they mostly draw on internet-scrapes like CommonCrawl and its derivatives (e.g., FineWeb) [1,2]. We have improved the rigour of this analysis by:

  • Replacing our main Figure 2 with a grouped analysis across both MultilingualPrompts and NovelPrompts (n=569). This allows us to address your point about the low number of OOV in MultilingualPrompts and increases the power of our analysis. The observed trend is the same, with a statistically significant Spearman correlation of -0.26 (also now reported on the plot).
  • We also added a scatter plot (Figure 15, Appendix D4) which shows the relationship between prediction flips and prompt frequency without any binning to ensure statistical robustness. Again, context sensitivity is significantly negatively correlated to prompt frequency.
  • We report the NovelPrompts results on another large and representative training corpus (Olmo-2’s training data [3], mostly English) and find similar trends (ρ = −0.14, p = 0.045, Figure 14).

B) Judge sensitivity is correlated to their own prior knowledge of each prompt.

As suggested by multiple reviewers, we thought it would be useful to supplement this analysis with a per-judge investigation, as it is true that judges may have different priors and levels of understanding over the concepts mentioned in the prompts, and this may cause variability in their sensitivity to the same context. To do this, we performed an additional experiment eliciting each judge’s understanding of the key concept mentioned in the prompt, and comparing this to the context definition that the human annotators had provided. We sampled 10 responses at high temperature to get a more calibrated idea of each judge’s understanding.

[continued in the next comment]

AuthorsOpenReview ↗

Response Part 2

Our results show that judge prior knowledge is also correlated to context-sensitivity across our 4 setups. Judges change their predictions significantly more often for concepts they know less well. For example, in NovelPrompts, each additional point of understanding lowers the odds of a flip by about 10% (β=−0.10\beta=-0.10, 95% CI [-0.15,-0.05], p<0.001), and the fit for each individual judge slopes downward for 9 of the 10 judges. We believe this new analysis considerably strengthens our claim, and have added it to the main body of the paper (Figure 3), with full experimental details and supplementary results in Appendix D4.2.

C) Judges are more sensitive on NovelPrompts than MultilingualPrompts.

Finally, we have clarified in the paper that the differences in flip rates between NovelPrompts and MultilingualPrompts (19% vs. 5%) also support our claim, as NovelPrompts words are much rarer (⅓ are OOV in FineWeb vs. less than 3% in MultilingualPrompts, Figure 11, new) and less well understood by the judges (Figure 17, new). 13 out of 13 judges flip their predictions more in NovelPrompts than MultilingualPrompts (Figure 11, new).

Q1: What do the csv columns represent in NovelPrompts and which are model-derived vs human labels?

The README details what each column in the csv represents. For instance, safety_assessment_without_context is yes if annotators can judge the safety of the response without understanding the novel concept. All columns are human labels except for the response column which contains the model completions. We have added this note to the README.

Q2: L392: if judges only change about 5% of their predictions, how can their accuracy drop by 15%?

Our sentence was unclear. We revised it to: "under definitions sas_a and sbs_b the judges only change an average of 5% of their predictions when they should be changing 14% to 21% [...] their accuracy [...] drops by up to 0.13". While judges perform worse, not all suffer such big accuracy drops. We have also added a new table showing per judge breakdown of steerability results (Table 7), and another breaking down prediction flips by correctness (Table 6). These should more clearly present the relationship between prediction changes and performance.

Q4: How does our baseline policy compare to the LLM providers’ safety policies?

Thank you for this suggestion. We investigated the model-specific documentation for all 13 judges and checked how it compares to our base policy. We have incorporated this analysis into our Appendix E4 and also expanded footnote 6.

Unfortunately, we cannot make any stronger claims on other labs’ safety policies, as the safety policies and annotation guidelines are generally not available publicly. Technical reports vary considerably: some identify broad categories ([4-8],[11-12]), whereas others only state that the model was trained for “safety” without publishing any taxonomy [9][10][13]. Even when reports use similar category names, they rarely disclose definitions. Moreover, the presence of the category names in evaluation sections does not directly imply that the same categories were used for training.

The three safety-specific guardrail models differ in that they are policy-conditioned classifiers, where the safety policy is given as part of the input (i.e, μₛ in our experiments). The example or default taxonomies published for Nemotron Safety Guard, Llama Guard 4, and GPT-OSS-Safeguard [14-16] mention all five broad areas in μₛ, so we can conclude that a similar policy was in-distribution to test their adaptability, but we cannot quantify the overlap.

Thus, while our results that judge accuracy is essentially the same with or without μ_s in the prompt (within ±0.02 across datasets, §E.1) would suggest there is considerable overlap, no stronger claim about their exact training-time policies can be established from public information. This is why our claim is phrased as “suggests” rather than asserting policy equivalence.

Why do even the best judges only reach an accuracy of 69% on MultilingualPrompts given that other safety benchmarks typically report very high classification accuracy?

We have intentionally made MultilingualPrompts a difficult benchmark. It requires niche multilingual knowledge across different language scripts and families. Moreover, its safe prompts are written to resemble unsafe ones (in the style of exaggerated-safety tests such as XSTest, C2). This ensures that we can discriminate between judges and identify failures that more standard benchmarks would miss.

Thank you again for your time reviewing our work. We hope our responses, in particular clarifications on the NovelPrompts dataset and our revised evidence to support our claim on prior knowledge and context will give you confidence to raise your score. In any case, we are happy to discuss further.

AuthorsOpenReview ↗

References

[1] LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/html/2302.13971v1#S2.SS1

[2] Language Models are Few-Shot Learners https://arxiv.org/html/2005.14165v4#S2.SS2

[3] OLMo 2 Furious, Team OLMo https://arxiv.org/abs/2501.00656

[4] GPT-5.2 System Card, OpenAI https://cdn.openai.com/pdf/3a4153c8-c748-4b71-8e31-aecbde944f8d/oai%5F5%5F2%5Fsystem-card.pdf

[5] GPT-5 System Card, OpenAI https://cdn.openai.com/gpt-5-system-card.pdf

[6] Claude Sonnet 4.5 System Card, Anthropic https://www-cdn.anthropic.com/963373e433e489a87a10c823c52a0a013e9172dd/Claude%20Sonnet%204.5%20System%20Card.pdf

[7] Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet Model Card Addendum, Anthropic https://www-cdn.anthropic.com/c7822cdc35ad788ec87e14b3a9d45010f1f86c38.pdf

[8] Command A: An Enterprise-Ready Large Language Model, Cohere https://arxiv.org/abs/2504.00698

[9] C4AI Command R Model Card, Cohere For AI https://huggingface.co/CohereLabs/c4ai-command-r-v01

[10] Qwen3-235B-A22B-Instruct-2507 Model Card, Qwen https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507

[11] The Llama 3 Herd of Models, Dubey et al. https://arxiv.org/abs/2407.21783

[12] GPT-OSS-120B & GPT-OSS-20B Model Card, OpenAI https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai%5Fgpt-oss%5Fmodel%5Fcard.pdf

[13] Tiny Aya Technical Report, Cohere Labs https://arxiv.org/abs/2603.11510

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response Part 1

Thank you very much for your thorough review and helpful comments. We respond to each of your points below, and also refer to changes we have made in the re-uploaded paper (changes marked in blue).

W1: Prediction changes do not measure successful adaptation.

We agree with your point, this is why we complement our reports of prediction changes with performance changes. Having found that judges do not appear particularly brittle (Figure 7), we believe prediction changes are useful signal whether they occur on initially incorrect or correct predictions. But we completely agree that a high flip rate by itself is not necessarily desirable and have now clarified this when we introduce the metrics. Having said this, we see the value in your suggestion of also breaking down changes by correctness, and we performed additional analyses which we present below. These broadly show that, as expected, prediction flips occur considerably more often in the incorrect to correct direction.

W1a/Q2: Prediction flips in context-sensitivity.

We have added an additional figure to our appendix (Figure 12) which shows that across all 4 dataset setups, context flips occur much more often on initially incorrect predictions than on initially correct ones (2.5 - 6.8 times more often depending on the judge). This explains why context on average leads to performance improvements. Table 3 further breaks this down and shows that, as you suggested, most predictions that do not change with context were already correct 87%, 76%, 78%, and 84% on prompts and completions on MultilingualPrompts and NovelPrompts respectively, but that some predictions stay wrong despite correct context (13-23% of all predictions), indicating a failure to use this information.

W1b/Q1: Prediction flips in steerability.

In light of your comments, we further analysed steerability prediction flips and added a new table (Table 6) which breaks down prediction flips in each setup by whether they were an adaptation to the new definition or a spurious (incorrect) flip, and which also reports how many predictions stay correct under the base and the new definition. It shows that while judges sometimes make spurious flips under the policy changes, this is at a much lower rate. For instance, in classification, the percentage of adaptation flips is between 17% and 72%, while the percentage of spurious flips is between 6 and 13% (with an exception for MultilingualPrompts completions, where it is 26%, which we discuss below). We have also revised our main figure (Figure 6) so the dotted line is labelled “expected flip rate” and the y axis “Prediction flips / gold label changes”. We agree it could have been misleading as it was earlier, thank you for this suggestion. Altogether, by showing performance changes (Table 5), flip rate changes (Figure 6), and the direction of flip rate changes (Table 6), we clearly show that most of our judges adapt very poorly to the safety variants, but framing the task as classification can improve this adaptation.

W1c: Exception in Table 3 where classification framing reduces balanced accuracy on MultilingualPrompts prompt-plus-completion evaluation under variant B.

Thank you for catching this. Upon verification, we realised there was a typo in this table, and balanced accuracy on variant B MultilingualPrompts completions should have been 59.4 instead of 49.4. This is nonetheless a surprising decrease relative to the other results and we agree it deserves further analysis. This decrease is driven by 5 of the 12 judges, and appears due to the fact that these 5 judges over-predict completions as being unsafe, even refusals (GPT-5: 73%, GPT-5 Mini: 86%, Llama 3.1 70B: 77%). We think this is because under variant B (strict safety), some of the classification judges still categorise refusals as mentioning a safety category. On the other hand, in the safety judging setup, it is likely that most judges already have the prior that refusals cannot be unsafe. We have now added this analysis to the Appendix (E3) along with per-judge steerability results (Table 7) for full transparency. While this example highlights the importance of properly defining edge cases, especially when judges cannot fall back on their safety priors (as we now discuss in Section E3.1), we do not think it invalidates our point that classification framing generally improves the steerability of most judges and setups we test.

[continued in the next comment]

AuthorsOpenReview ↗

Response Part 2

W2a/Q3: Evidence for rigid learned priors in context experiments and model-specific knowledge checks.

We agree that our evidence as presented initially could have been improved to more rigorously support our claim that judges only use context when their priors are weak. We have now conducted further experiments and analyses to clarify our three main pieces of supportive evidence and revised the paper accordingly. We have also rephrased the claim to “Judges use context more when their priors are weak” to more precisely reflect our results.

A) Judge sensitivity is correlated to their frequency in two large pre-training corpora.

We find that judge sensitivity on each prompt is correlated to how frequent the concepts mentioned in the prompts are in large pre-training datasets. We believe this is a valuable analysis as pre-training frequency can be associated with the strength of the judges’ priors (beyond their understanding), and even though FineWeb-2 may not be used by the models we study, it is likely to be similar to their pre-training datasets, as these mostly draw on internet-scrapes like CommonCrawl and its derivatives [1-2]. We have now further improved the rigour of this analysis by:

  • Replacing our main Figure 2 with a grouped analysis across both MultilingualPrompts and NovelPrompts (n=569). This allows us to address your point about the low number of OOV in MultilingualPrompts and increases the power of our analysis. The observed trend is the same, with a statistically significant Spearman correlation of -0.26 (also now reported on the plot).
  • We also added a scatter plot (Figure 15, Appendix D4) which shows the relationship between prediction flips and prompt frequency without any binning to ensure statistical robustness. Again, context sensitivity is significantly negatively correlated to prompt frequency.
  • We report the NovelPrompts results on another large and representative training corpus (Olmo-2’s training data [3], mostly English) and find similar trends (ρ = −0.14, p = 0.045, Figure 14).

B) Judge sensitivity is correlated to their own prior knowledge of each prompt.

As suggested by multiple reviewers, we thought it would be useful to supplement this analysis with a per-judge investigation, as it is true that judges may have different priors and levels of understanding over the concepts mentioned in the prompts, and this may cause variability in their sensitivity to the same context. To do this, we performed your suggested experiment of eliciting each judge’s understanding of the key concept mentioned in the prompt, and comparing this to the context definition that the human annotators had provided. We sampled 10 responses at high temperature to get a more calibrated idea of each judge’s understanding. Our results show that judge prior knowledge is also correlated to context-sensitivity across our 4 setups. Judges change their predictions significantly more often for concepts they know less well. For example, in NovelPrompts, each additional point of understanding lowers the odds of a flip by about 10% (β=−0.10\beta=-0.10, 95% CI [−0.15,−0.05][-0.15,-0.05], p<0.001p<0.001), and the fit for each individual judge slopes downward for 9 of the 10 judges. We believe this new analysis considerably strengthens our claim, and have added it to the main body of the paper (Figure 3), with full experimental details and supplementary results in Appendix D4.

C) Judges are more sensitive on NovelPrompts than MultilingualPrompts.

Finally, we have clarified in the paper that the differences in flip rates between NovelPrompts and MultilingualPrompts (19% vs. 5%) are also evidence of our claim, as NovelPrompts words are much rarer (⅓ are not in FineWeb vs. less than 3% in MultilingualPrompts, Figure 11, new) and less well understood by the judges (Figure 17, new). 13 out of 13 judges flip their predictions more in NovelPrompts than MultilingualPrompts (Figure 11, new).

[continued in the next comment]

AuthorsOpenReview ↗

Response Part 3

W2b/Q3: Neutral classification prompt changes both the output labels and the presentation of the decision rules.

You are correct to point out that the neutral classification prompt not only changes the output labels but also makes minor changes to the presentation of the decision rules. We did this at the time to ensure the definitions in the classification framing were clear, but we agree it is important to verify that differences originate from the output label change, as we claimed. To do this, we re-ran the safety judging experiments with the classification template except replacing categories 1 and 2 with “SAFE” and “UNSAFE” on all models. We found that most judges were very robust to different presentations of the definition, with agreement values between 95 and 99%. Tiny-AYA was the only judge to appear more brittle, with agreement of 87%. It is also the worst-performing model, which likely explains its brittleness. Overall, this suggests that presentation alone has a limited effect on judge predictions, so it is likely changes in output space are responsible for most of the differences we observe in steerability with classification. The full analysis and table is now in Section C4 of the Appendix, thank you for the suggestion.

W2c: Overclaims regarding rigid priors in Section 5.

We understand your point, and have softened our language to be more precise. We replaced “isolate” with “probe” l.410, and rephrased l.483 - 485 with “these results suggest that lack of steerability is unlikely to be explained by misunderstanding of the definition or of the data sample, or general model brittleness alone”.

Thank you again for your efforts reviewing our work, and your help making our paper more statistically rigorous and precise. We hope our responses, in particular our new analyses on the direction of prediction changes and our new experiments on the link between each judge’s prior knowledge and their context-sensitivity, will give you confidence to raise your score. We are happy to discuss any remaining points further.

[1] LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/html/2302.13971v1#S2.SS1

[2] Language Models are Few-Shot Learners https://arxiv.org/html/2005.14165v4#S2.SS2

[3] OLMo 2 Furious, Team OLMo https://arxiv.org/abs/2501.00656

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.