- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how disagreements among harm classifiers affect 3 downstream applications: content moderation, alignment intervention selection, and preference alignment. It finds disagreement in content labels and shows that the choice of classifier can change which alignment intervention appears most effective. However, models trained with DPO on preferences derived from different classifiers exhibit relatively small differences in aggregate harmfulness, even when the classifiers disagree considerably. These models nevertheless differ in their generated content: greater classifier sensitivity to features such as political content or identity mentions is associated with less frequent generation of those features after alignment. The findings highlight that harm classifier selection has task-dependent consequences and that aggregate safety metrics can obscure meaningful behavioral differences.
Critical strengths
The paper addresses a practically important question. Rather than stopping at comparisons of classification accuracy or agreement, it investigates whether classifier disagreements actually affect content moderation, alignment intervention selection, and preference training. This gives the study clear practical relevance.
The experimental coverage and robustness checks are fairly comprehensive. The paper examines multiple classifiers, six alignment interventions, and different models, and uses preference-label flipping experiments to assess the effects of disagreement. The appendix further investigates factors such as dataset size, shared prompts, random candidate selection, and full-parameter DPO, going beyond observations from a single experimental configuration.
The findings are valuable and counterintuitive. The experiments show substantial disagreement among harm classifiers, with classifier choice affecting content moderation labels and which alignment intervention appears most effective. Although DPO models trained on preferences from different classifiers show relatively small differences in aggregate harmfulness, they differ in the content they generate: greater classifier sensitivity to particular features is associated with less frequent generation of those features after alignment.
Critical weaknesses
The paper offers a strong empirical study, but further methodological exploration would strengthen its contribution. We would welcome an investigation of methods for addressing the issues identified by the findings.
A small Harm Distance does not sufficiently support the conclusion that the effects are “negligible.” Harm Distance first computes each model’s harm rate across all generated responses under each classifier, then averages the absolute differences between models across classifiers. Whether a difference of “less than three percentage points” is negligible depends on the baseline harm rate, the severity of the harms, and the scale of deployment; it cannot be determined from the absolute difference alone. The paper establishes that differences are small under the selected metric, rather than demonstrating that their practical safety implications are necessarily minor.
Differences in what the classifiers are designed to detect make the meaning of “disagreement” insufficiently clear. The paper groups toxicity, hate speech, and broader safety detection tools under the umbrella of harm classifiers, despite substantive differences in their target constructs. For example, the Safety Pretraining prompt incorporates certain political and religious topics, as well as professional advice, into its safety scoring criteria. By contrast, the HarmBench prompt restricts detection to harassment, bullying, hate speech, and encouragement of self-harm, explicitly excluding other harmful behaviors. Consequently, some observed disagreements may reflect classifiers correctly performing different tasks, rather than making different judgments about the same harm. Although the paper acknowledges this possibility, it does not systematically distinguish differences in task scope, differences in value judgments, and classification errors. This limits the interpretation of both the sources of disagreement and their downstream implications.
Questions for authors
Could the small differences in aggregate harmfulness mask larger or systematic differences within specific harm categories? Would a category-level analysis reveal downstream effects that are not captured by Harm Distance?
2 comments
Rebuttal Table 3: Base Model Harm Distances
3. Harm distance between base model and models aligned on preference datasets chose by each harm classifier.
| Model | Harm distance from base (seed 1) | Harm distance from base (seed 2) |
|---|---|---|
| Aligned on preference data selected by: | ||
| Perspective | 3.1 [2.8, 3.3] | 3.0 [2.8, 3.3] |
| OpenAI | 4.3 [4.1, 4.6] | 4.4 [4.1, 4.6] |
| RoBERTa | 3.9 [3.6, 4.2] | 3.9 [3.6, 4.2] |
| ToxiGen HateBERT | 4.7 [4.4, 5.0] | 4.6 [4.3, 4.9] |
| HateXplain | 3.2 [3.0, 3.5] | 3.4 [3.2, 3.7] |
| Llama Guard | 3.2 [2.9, 3.4] | 3.2 [3.0, 3.5] |
| Simple Judge | 2.8 [2.6, 3.1] | 2.7 [2.5, 3.0] |
| Safety Pretraining Judge | 5.5 [5.2, 5.8] | 5.3 [5.0, 5.6] |
| Constitutional Judge | 4.1 [3.8, 4.4] | 4.0 [3.7, 4.3] |
| HarmBench Judge | 3.7 [3.4, 4.0] | 3.7 [3.4, 3.9] |
| 100% of pairs flipped: | ||
| Perspective 100% flipped | 5.3 [5.0, 5.7] | — |
| OpenAI 100% flipped | 4.6 [4.3, 4.9] | — |
Thank you for recommending acceptance and for highlighting that our findings are practical and counterintuitive. We agree that our interpretations of magnitude and disagreement can be clarified further and have revised the paper to better reflect this.
Are small harm distances really negligible in practice?
We agree with your distinction between small distances and negligible consequences, and we have modified the wording throughout to emphasize that classifier choice can affect aspects of model behavior beyond aggregate harmfulness, particularly content features, without implying it is negligible.
We separately strengthen our findings through a variety of ablations on the metric aggregation including maximum distance, prompt-level aggregation, and normalization by classifier base rates, as well as per-classifier harm rates (new Appendix Figure 12). We confirm that the magnitude of harm distance varies with aggregation, but harm distances are relatively small across these ablations (Revised Section 4.2, Figure 3). For example, in the per-classifier analysis, harmful-response rates for the same model vary by 40-52 percentage points across classifiers, while the median variation across aligned models under the same classifier is only 1.4 percentage points.
While we cannot compare a baseline harm rate without privileging a single operationalization of harm, we now report distances between the base and aligned models (2.7–5.5 percentage points), alongside distances between differently aligned models (at most 2.6 percentage points; Appendix Table 9). Each aligned model therefore differs more from the base model than any two aligned models differ from each other. However, these comparisons do not establish that the differences are negligible in practice. We have revised our wording to distinguish the magnitude of the measured differences from their potential consequences. We additionally contextualize our claim that harm distances under 20 are small by reiterating the metric range (0-100) and the examples of response pairs at different values of the harm distance metric.
Do small harm distances obscure differences between harm categories?
We agree that aggregate measures may obscure differences, such as those between content features. We present our harm distance results because this broader metric measures the intended goal of the preference alignment task: to make models less harmful.
We examine six content features to understand differences in generated text that aggregate harm rates may obscure (Revised Section 4.3). Here, we find that models aligned with harm classifiers that are more sensitive to particular content features generate those features in responses less frequently (Revised Figure 5). By redirecting attention from the intended goals to the unintended consequences of preference alignment here, we identify a difference that the harm distance metric overlooked. While the content features are not exactly equivalent to harm categories, they still offer insight into downstream effects related to suppression of certain topics like mentions of identity. A more granular analysis of harm categories and severity would be valuable in future work.
But harm classifiers operationalize distinct constructs!
We agree that much of the disagreement between classifiers might reflect legitimate distinctions in their intended scope. We note that we intentionally include some classifiers whose constructs overlap, such as LLM judges that use paraphrased versions of the same prompt, and off-the-shelf classifiers that use similar definitions of toxicity.
Our goal is not to determine which classifiers are correct under a universal definition of harm. Instead, we ask what happens when classifiers that operationalize harm in different ways are used for similar goals, which does occur in practice (Fasching and Lelkes 2025; Achara and Chhabra 2025). For instance, the Perspective API is being phased out at the end of this year and may be substituted with OpenAI Moderation API in many uses even though they are supposed to do different things. We have edited the framing to clarify that it's important to understand disagreement even beyond classifiers that share scope.
Addressing the raised issues.
We agree that developing methods to address the issues our paper raises is important! In this work, we focus on establishing when and how classifier differences propagate downstream, but we recognize the need for follow-up work that seeks to mitigate problems. We've described some practical tensions and extensions to work on in our discussion.