- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper meta-evaluates LLM safety judges beyond standard human-agreement benchmarks, formalizing two overlooked properties: context-sensitivity (whether a judge updates its prediction when given novel in-context information about the prompt) and steerability (whether a judge follows a modified safety policy rather than its internal safety prior). Evaluating 13 judges (generalist and safety-specific, open- and closed-source) on three datasets — including a new multilingual human-annotated dataset (French, Arabic, Japanese, Korean), a new NovelPrompts dataset containing post-cutoff concepts, and Sorry-BENCH — the authors find that judges can absorb in-context information mainly when their priors are weak (prediction flips correlate negatively with term frequency in pre-training corpora), but are broadly unable to adapt to modified safety policies, with steerability improving substantially only when the task is masked as an arbitrary classification task. The authors release NovelPrompts and an evaluation framework.
Critical strengths
- Well-motivated and clearly formalized problem: the paper identifies that human-agreement benchmarks miss two practically crucial properties — sensitivity (for temporal drift/novel concepts) and steerability (for policy customization across cultures/domains) — and gives both a clean formalization and a principled Bayesian-prior interpretation of the results.
- Broad, well-controlled empirical study: 13 diverse judges, human-annotated multilingual data, controlled conditions (no/irrelevant/correct context; base/variant/absurd policies; 5 seeds), with findings that are largely consistent across three datasets and supported by supplementary analyses (e.g., classification-masking, prior-strength correlation).
- Actionable findings and artifacts: the result that steering fails even when definitions are clear, that sensitivity/steerability/accuracy are distinct properties, and that masking safety as classification roughly doubles steerability, are directly useful to practitioners; datasets and framework are released.
Critical weaknesses
- The steerability evaluation covers a narrow slice of the policy space: only two definition variants (plus an absurd toy policy), no systematic variation of "distance" from the judge's prior (e.g., single-category changes, lenient direction, severity-level redefinitions). The strong claim that judges are "broadly not steerable" may overgeneralize from these two variants, especially given prior work (e.g., Weng et al., 2026) found some adaptation to simple strict-to-lenient changes.
- Statistical rigor is thin: datasets are small (NovelPrompts n=194; MultilingualPrompts n=779), outcomes are binary flips, and most comparisons are reported without significance tests or confidence intervals; reported Pearson correlations (e.g., 0.55–0.65 for steerability across definitions) lack CIs, and no correction for multiple comparisons across 13 judges.
- The prior-strength proxy (word frequency in FineWeb) is coarse and judge-agnostic; it cannot capture per-model training data or knowledge, so the central Bayesian interpretation (sensitivity ∝ weak prior) is only weakly evidenced, as the authors themselves partially acknowledge.
- The setup is simpler than production judging (binary safe/unsafe on prompts, single policy), and the practical recommendations (choose sensitive/steerable judges) are not validated in any deployment-like scenario; the classification-masking "mitigation" is presented without discussing its risks.
Questions for authors
- Have you tested intermediate policy distances (e.g., changing a single category, the lenient direction, or severity thresholds)? Does steerability degrade monotonically with distance from the judge's prior, and how do you reconcile your conclusion with Weng et al.'s finding that some judges adapt to simple definition changes?
- Given the small sample sizes, can you report confidence intervals/significance tests for the flip-rate differences and correlations, and show that judge rankings (e.g., Fig. 3) are stable under subsampling?
- Can you validate the prior-strength proxy with a judge-specific measure (e.g., perplexity, explicit knowledge elicitation), and show the negative correlation with word frequency holds when controlling for prompt ambiguity/difficulty?
- What are the failure modes of the classification-masking trick in practice (e.g., loss of severity-sensitive judgment, adversarial reframing), and would you recommend it for production safety pipelines?
6 comments
Response Part 2
Regarding sensitivity rankings: we followed your suggestion and tested whether the rankings in Figure 4 are stable under subsampling. We resampled prompts with replacement 2,000 times (within each dataset and language) and recomputed flip rates and ranks each time. The extremes of the ranking are robust (e.g., Nemotron is in the top three of all four experiments in 91% of resamples), while mid-ranked judges have overlapping rank intervals. Rankings also replicate across datasets (Spearman , 95% bootstrap interval [0.41, 0.90]) and are concordant across all four experiments (Kendall’s ). We summarise this in a footnote in §4.2.
Regarding multiple comparisons: our main claims rest on tests pooled across judges (the frequency correlation, the knowledge regression, the sign test across datasets) rather than on per-judge tests, which are descriptive. Where we report the same analysis across several settings, we describe the consistency of its direction rather than claiming significance in each.
Thank you for your help ensuring our findings are statistically robust.
W3: The prior strength proxy is judge-agnostic.
You are absolutely correct that our prior strength proxy is judge-agnostic. While we do believe that judges have similar priors due to large overlap in their pre-training datasets (e.g., mostly draw on internet-scrapes like CommonCrawl and its derivatives [4,5]), we agree that it would strengthen our claim to explore each judge’s specific prior knowledge, as it is true that judges may have different priors and levels of understanding over the concepts mentioned in the prompts, and this may cause variability in their sensitivity to the same context. To do this, we performed an additional experiment eliciting each judge’s understanding of the key concept mentioned in the prompt, and comparing this to the context definition that the human annotators had provided. We sampled 10 responses at high temperature to get a more calibrated idea of each judge’s understanding. Our results show that judge prior knowledge is also correlated to context-sensitivity across our 4 setups. Judges change their predictions significantly more often for concepts they know less well. For example, in NovelPrompts, each additional point of understanding lowers the odds of a flip by about 10% (, 95% CI [-0.15,-0.05], ), and the fit for each individual judge slopes downward for 9 of the 10 judges. We believe this new analysis considerably strengthens our claim, and have added it to the main body of the paper (Figure 3), with full experimental details and supplementary results in Appendix D4.
W4a: The setup is simpler than production judging.
Thank you for raising this point. Our primary concern is practitioners using LLM judges to evaluate a candidate model before deploying it, for example, estimating its policy-violation rate, comparing release candidates, or deciding whether it satisfies a deployment policy. This closely reflects the evaluation procedures reported by major providers:
- OpenAI uses LLM graders to classify candidate-model completions as not_unsafe [6],
- Cohere uses LLM judges to identify unsafe generated content [7],
- Anthropic automatically scores responses before deployment [8], and
- Meta evaluates violation and false-refusal rates using safety classifiers [9].
Our findings directly apply to these evaluations: if the judge is insensitive to required context or cannot follow the intended policy, the resulting pre-deployment safety measurements may provide false confidence. We released NovelPrompts and our evaluation framework to support pre-deployment testing of whether a selected judge incorporates necessary contextual information and follows a supplied policy.
We do not claim to have solved end-to-end production judging. Our methods provide actionable pre-deployment tests that reveal whether a judge actually uses the supplied policy and context or instead defaults to its latent safety boundaries--a failure mode that standard human-agreement metrics do not detect. If a judge is subsequently used in a live environment, it must be tested against its particular deployment scenario, policy, languages, content distribution, and available context.
[continued in the next comment]
Response Part 3
W4b/Q4: What are the failure modes of classification masking?
Thank you for highlighting this interesting point. Classification masking also introduces important risks. A judge asked for safe or unsafe can fall back to its latent safety boundary when the supplied policy is incomplete or absent. Arbitrary labels such as Class 1 and Class 2 remove that semantic fallback. The policy must therefore comprehensively define the relevant harm categories, content types, and edge cases.
Practitioners could evaluate the Class 1/Class 2 formulation on a representative development set, inspect the cases it misses, and iteratively refine the policy and prompt template. This requires ensuring that the evaluation set represents the intended deployment distribution and preserving a separate held-out test set to avoid overfitting. We present classification masking and judge selection as empirically supported directions for improving pre-deployment model evaluation, not as complete solutions for production judging. We have made a new section in our Appendix (E3.1: Recommendations for use of classification masking in practice) where we now discuss this explicitly.
Thank you again for your engagement with our work. We hope our improvements of statistical rigour, paper revisions on steerability wording and classification masking, and our new experiment eliciting judge knowledge, will give you confidence to increase your score. Please let us know if you have any further comments or concerns.
[1] Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test https://arxiv.org/pdf/2605.06161
[2] SIMPLESAFETYTESTS: A Test Suite for Identifying Critical Safety Risks in Large Language Models https://arxiv.org/pdf/2311.08370v2
[3] XSTEST: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models https://aclanthology.org/2024.naacl-long.301.pdf
[4] LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/html/2302.13971v1#S2.SS1
[5] Language Models are Few-Shot Learners https://arxiv.org/html/2005.14165v4#S2.SS2
[6] GPT-5 System Card https://deploymentsafety.openai.com/gpt-5
[7] Command A: An Enterprise-Ready Large Language Model cohere.com/research/papers/command-a-technical-report.pdf
[8] Claude Sonnet 4.5 System Card assets.claude.com/963373e433e489a87a10c823c52a0a013e9172dd.pdf
[9] The Llama 3 Herd of Models https://arxiv.org/abs/2407.21783
Response Part 1
Thank you for your comprehensive review and suggestions. We respond to each comment below and also refer to improvements we have made in the re-uploaded paper (changes are marked in blue).
Thank you for highlighting this, we agree that we overgeneralised by saying that safety judges were broadly unsteerable, and have revised our third contribution to: “We show that LLM safety-judges can be difficult to steer, and instead often rely on their internal safety boundary to judge, despite being instructed otherwise.”. Our key point was not to show that judges are never steerable, rather to show that providing a safety policy does not ensure that a judge will apply it faithfully.
Our variants already include the kinds of change you mention, and we have clarified this in §5 and §C.4. In particular, we clarified that Sorry-BENCH uses a different safety policy as it tests refusals, so we make different kinds of policy changes, further expanding the space of variants we investigate.
We also include a no-definition control, the same rules under a classification framing, and an absurd sports policy.
We agree that Weng et al. [1] provide important evidence of adaptation, and our findings are partly consistent with theirs. Their study focuses on strict-to-lenient threshold shifts, applied to 4 judges evaluating agent trajectories. Our work complements it by testing category-specific lenient and stricter changes, severity-threshold changes, and the same rules under safety vs. arbitrary-classification framings. Table 6 shows that the direction of the change matters. Under safety judging, adaptation on examples whose gold label changes is lowest for the stricter variant B (5.7-10.1% on MultilingualPrompts and NovelPrompts), intermediate for the mixed variant A (13.7-39.1%), and highest for the purely lenient SORRY-Bench variant A (57.5%). In the lenient direction, our judges adapt more, in line with Weng et al [1]. Adaptation drops when the policy becomes stricter. This is not because stricter rules are harder to apply: under classification framing, adaptation to variant B on MultilingualPrompts rises from 9.3% to 65.0%. We have added this comparison to §5.
Finally, while our findings that classification steerability is over twofold higher than when judging safety (Figure 6) and that judges can follow the absurd sports definition (Figure 20) point to preliminary evidence that steerability degrades with distance from the judge’s prior, we have not conducted a systematic sweep of safety policies nor have we explicitly quantified distance to each model’s prior. We have now mentioned this in our limitations to ensure it is clear we are not making this claim.
Regarding datasets: we would push back that our datasets are of considerable size for safety datasets, especially considering they are human-generated, human-annotated, and natively multilingual. For instance, established benchmarks like SimpleSafetyTests [2] and XSTest [3] contain 100 and 450 prompts respectively. We also complement our work with Sorry-BENCH (n=1000) and our synthetic sports dataset (n=240) and report additional uncertainty estimates as described below.
Regarding binary flips: we do not think binary flips are non-rigorous in themselves, but we agree they do not show the full picture, and have now reported additional metrics on the direction of prediction changes (e.g., Table 3 and 6), as we detail in our response to Reviewer AXrG.
Regarding significance tests and CIs: we consistently report standard deviation across 5 random seeds, but we have now also added further statistical analysis:
[continued in the next comment]