acceptodds
Under review as a conference paper at ICLR 2027

Disagreement in, Disagreement Out? Uneven Downstream Effects of Harm Classifier Differences

Abstract

Harm classifiers are machine learning models that are primarily used in three settings: content moderation, where they scale up the detection of harmful text; intervention selection, where they identify the most effective alignment interventions; and preference alignment, where they help train models to be harmless. Prior work has established that popular harm classifiers disagree substantially with each other in terms of which pieces of text are harmful. However, it is not yet known whether these differences actually matter where harm classifiers are used. We find that for content moderation, some pairs of harm classifiers can disagree in a majority of cases and half the pairs disagree on at least 27% of text; and for intervention selection, five of six alignment interventions rank first under at least one of ten harm classifiers, meaning the choice of harm classifier can determine which intervention appears best. On the other hand, for preference alignment, we find that harm classifiers have little effect on the aggregate harmfulness of models aligned on their preference signal. Even when two harm classifiers disagree on 55% of texts, the responses of the associated pair of aligned models differ in average harm rates by only 3 percentage points. However, these models do not behave identically: if harm classifiers are more likely to flag text containing content features like political content, the associated aligned models are also less likely to generate that content feature. Our results establish that the choice of harm classifier, which is often seen as an inconsequential choice of convenience, is in fact a critical design decision that can have dramatic effects in some settings (like content moderation), but negligible impacts in others (like harm reduction in preference alignment).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
harm detection, content moderation, toxicity detection, safety, evaluation, alignment, preference learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QzHROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how disagreements among harm classifiers affect 3 downstream applications: content moderation, alignment intervention selection, and preference alignment. It finds disagreement in content labels and shows that the choice of classifier can change which alignment intervention appears most effective. However, models trained with DPO on preferences derived from different classifiers exhibit relatively small differences in aggregate harmfulness, even when the classifiers disagree considerably. These models nevertheless differ in their generated content: greater classifier sensitivity to features such as political content or identity mentions is associated with less frequent generation of those features after alignment. The findings highlight that harm classifier selection has task-dependent consequences and that aggregate safety metrics can obscure meaningful behavioral differences.

Critical strengths

The paper addresses a practically important question. Rather than stopping at comparisons of classification accuracy or agreement, it investigates whether classifier disagreements actually affect content moderation, alignment intervention selection, and preference training. This gives the study clear practical relevance.

The experimental coverage and robustness checks are fairly comprehensive. The paper examines multiple classifiers, six alignment interventions, and different models, and uses preference-label flipping experiments to assess the effects of disagreement. The appendix further investigates factors such as dataset size, shared prompts, random candidate selection, and full-parameter DPO, going beyond observations from a single experimental configuration.

The findings are valuable and counterintuitive. The experiments show substantial disagreement among harm classifiers, with classifier choice affecting content moderation labels and which alignment intervention appears most effective. Although DPO models trained on preferences from different classifiers show relatively small differences in aggregate harmfulness, they differ in the content they generate: greater classifier sensitivity to particular features is associated with less frequent generation of those features after alignment.

Critical weaknesses

The paper offers a strong empirical study, but further methodological exploration would strengthen its contribution. We would welcome an investigation of methods for addressing the issues identified by the findings.

A small Harm Distance does not sufficiently support the conclusion that the effects are “negligible.” Harm Distance first computes each model’s harm rate across all generated responses under each classifier, then averages the absolute differences between models across classifiers. Whether a difference of “less than three percentage points” is negligible depends on the baseline harm rate, the severity of the harms, and the scale of deployment; it cannot be determined from the absolute difference alone. The paper establishes that differences are small under the selected metric, rather than demonstrating that their practical safety implications are necessarily minor.

Differences in what the classifiers are designed to detect make the meaning of “disagreement” insufficiently clear. The paper groups toxicity, hate speech, and broader safety detection tools under the umbrella of harm classifiers, despite substantive differences in their target constructs. For example, the Safety Pretraining prompt incorporates certain political and religious topics, as well as professional advice, into its safety scoring criteria. By contrast, the HarmBench prompt restricts detection to harassment, bullying, hate speech, and encouragement of self-harm, explicitly excluding other harmful behaviors. Consequently, some observed disagreements may reflect classifiers correctly performing different tasks, rather than making different judgments about the same harm. Although the paper acknowledges this possibility, it does not systematically distinguish differences in task scope, differences in value judgments, and classification errors. This limits the interpretation of both the sources of disagreement and their downstream implications.

Questions for authors

Could the small differences in aggregate harmfulness mask larger or systematic differences within specific harm categories? Would a category-level analysis reveal downstream effects that are not captured by Harm Distance?

2 comments

Thank you for recommending acceptance and for highlighting that our findings are practical and counterintuitive. We agree that our interpretations of magnitude and disagreement can be clarified further and have revised the paper to better reflect this.

Are small harm distances really negligible in practice?

We agree with your distinction between small distances and negligible consequences, and we have modified the wording throughout to emphasize that classifier choice can affect aspects of model behavior beyond aggregate harmfulness, particularly content features, without implying it is negligible.

We separately strengthen our findings through a variety of ablations on the metric aggregation including maximum distance, prompt-level aggregation, and normalization by classifier base rates, as well as per-classifier harm rates (new Appendix Figure 12). We confirm that the magnitude of harm distance varies with aggregation, but harm distances are relatively small across these ablations (Revised Section 4.2, Figure 3). For example, in the per-classifier analysis, harmful-response rates for the same model vary by 40-52 percentage points across classifiers, while the median variation across aligned models under the same classifier is only 1.4 percentage points.

While we cannot compare a baseline harm rate without privileging a single operationalization of harm, we now report distances between the base and aligned models (2.7–5.5 percentage points), alongside distances between differently aligned models (at most 2.6 percentage points; Appendix Table 9). Each aligned model therefore differs more from the base model than any two aligned models differ from each other. However, these comparisons do not establish that the differences are negligible in practice. We have revised our wording to distinguish the magnitude of the measured differences from their potential consequences. We additionally contextualize our claim that harm distances under 20 are small by reiterating the metric range (0-100) and the examples of response pairs at different values of the harm distance metric.

Do small harm distances obscure differences between harm categories?

We agree that aggregate measures may obscure differences, such as those between content features. We present our harm distance results because this broader metric measures the intended goal of the preference alignment task: to make models less harmful.

We examine six content features to understand differences in generated text that aggregate harm rates may obscure (Revised Section 4.3). Here, we find that models aligned with harm classifiers that are more sensitive to particular content features generate those features in responses less frequently (Revised Figure 5). By redirecting attention from the intended goals to the unintended consequences of preference alignment here, we identify a difference that the harm distance metric overlooked. While the content features are not exactly equivalent to harm categories, they still offer insight into downstream effects related to suppression of certain topics like mentions of identity. A more granular analysis of harm categories and severity would be valuable in future work.

But harm classifiers operationalize distinct constructs!

We agree that much of the disagreement between classifiers might reflect legitimate distinctions in their intended scope. We note that we intentionally include some classifiers whose constructs overlap, such as LLM judges that use paraphrased versions of the same prompt, and off-the-shelf classifiers that use similar definitions of toxicity.

Our goal is not to determine which classifiers are correct under a universal definition of harm. Instead, we ask what happens when classifiers that operationalize harm in different ways are used for similar goals, which does occur in practice (Fasching and Lelkes 2025; Achara and Chhabra 2025). For instance, the Perspective API is being phased out at the end of this year and may be substituted with OpenAI Moderation API in many uses even though they are supposed to do different things. We have edited the framing to clarify that it's important to understand disagreement even beyond classifiers that share scope.

Addressing the raised issues.

We agree that developing methods to address the issues our paper raises is important! In this work, we focus on establishing when and how classifier differences propagate downstream, but we recognize the need for follow-up work that seeks to mitigate problems. We've described some practical tensions and extensions to work on in our discussion.

AuthorsOpenReview ↗

Rebuttal Table 3: Base Model Harm Distances

3. Harm distance between base model and models aligned on preference datasets chose by each harm classifier.

ModelHarm distance from base (seed 1)Harm distance from base (seed 2)
Aligned on preference data selected by:
Perspective3.1 [2.8, 3.3]3.0 [2.8, 3.3]
OpenAI4.3 [4.1, 4.6]4.4 [4.1, 4.6]
RoBERTa3.9 [3.6, 4.2]3.9 [3.6, 4.2]
ToxiGen HateBERT4.7 [4.4, 5.0]4.6 [4.3, 4.9]
HateXplain3.2 [3.0, 3.5]3.4 [3.2, 3.7]
Llama Guard3.2 [2.9, 3.4]3.2 [3.0, 3.5]
Simple Judge2.8 [2.6, 3.1]2.7 [2.5, 3.0]
Safety Pretraining Judge5.5 [5.2, 5.8]5.3 [5.0, 5.6]
Constitutional Judge4.1 [3.8, 4.4]4.0 [3.7, 4.3]
HarmBench Judge3.7 [3.4, 4.0]3.7 [3.4, 3.9]
100% of pairs flipped:
Perspective 100% flipped5.3 [5.0, 5.7]—
OpenAI 100% flipped4.6 [4.3, 4.9]—
Reviewer_RupfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a systematic benchmark of harm classifiers to evaluate their disagreement on three tasks: content moderation, intervention detection, and preference alignment. Classifiers are chosen from different families and architectures to have a representative sample of systems and the analysis is performed on a meta-dataset of HS messages. Results show that models disagree substantially in content moderation and intervention detection but when it comes to preference alignment the disagreement is dramatically reduced.

Critical strengths
  • The idea of systematically analyze harmful classifiers across tasks is relevant;
  • the "preference alignment" task is the most interesting one because it potentially connects with post-training safety interventions developed by LLM devs.
Critical weaknesses
  • There are many design choices that can be considered a bit arbitrary. Two examples: 1. the harm distance metric (section 3.4) is introduced but not motivated against traditional metrics like Kappa Cohen and the 10-dimensional vector is presented out-of-the-blue (why 10 and not 20 or 100?); 2. the lexical analysis relies on synthetic, non-validated approaches as all lexica are generated through GPT models;
  • Results do not acknowledge existing literature on HLV. There is a wide literature on subjectivity in HS perception that is not taken into account. E.g., human labels are chosen by majority vote without considering differences in evaluation. Without accounting for this source of subjectivity, the findings become more contestable.
Questions for authors
  • Why did you create a new score metric to measure models symilarity?
  • how did you assess statistical significant results in each step of your experimental setting?
2 comments

Thank you for your helpful review and for your interest in the preference alignment task. We agree that our methodological choices deserved further justification and validation and have addressed this through revisions to our paper.

How did we choose the harm distance metric for model similarity?

Our goal was to compare the aggregate harmfulness of responses generated by different models, as assessed by the same set of classifiers. We sought a metric that (1) captured differences in the magnitude of score pairs for each classifier; (2) was intuitive to interpret the score. We ruled out the following

  • Cohen’s Kappa: compares two raters labeling the same items with some constrained value. In contrast, our harm distance metric compares the responses generated by two models and uses harm classifiers as intermediate evaluators.
  • Cosine similarity: treats two vectors with proportional harm rates as identical and fails criterion 1
  • Jensen-Shannon divergence: designed for probability distributions and fails criterion 2
  • Wasserstein distance: compares distributions of harm rates and fails criterion 1

We thus choosed harm distance, which is the scaled L1 distance between vectors of the harm rates of all the classifiers, which gives an interpretable measure of average percentage-point differences in harmfulness. We clarify this in 3.4.

How did we choose 10 classifiers in the harm distance metric?

We select ten classifiers to be consistent with the rest of our experiments, which seek to cover classifiers with a range of operationalizations of harm, but acknowledge that using harm rates from a different number of classifiers is a reasonable approach. To test whether the number of classifiers impacts our findings, we run ablations that use subsets of five and two classifiers (Section 4.2, Revised Figure 3). Using 2, 5, or 10 classifiers produces comparable harm distances (see Rebuttal Table 4), suggesting our finding is not sensitive to this.

Lexical analysis is validated and not entirely GPT-based.

We would like to clarify that we do not use GPT-generated lexica and describe our methods and validation for the content feature analysis in Appendix D.

  • Lexical Hate: Our lexical hate word list is derived from Davidson et al. (2017), a prominent early attempt at classifying harm in text. We split, deduplicate, and filter this existing lexicon to come up with our word list.
  • Political Content: We use GPT-5 Nano to label 54,000 texts for political content and manually validate its labeling on 100 examples (0.82 F1). We then distill its labels into a Qwen3-8B classifier (0.80 F1, Appendix Table 6).

Missing literature on human label variation.

We agree that human label variation is very important to understand the subjectivity of harm judgments, and we have included literature on perspectivist annotation, human label variation, and pluralistic alignment in new Appendix I.

Our work is motivated by the increasing use of automated harm classifiers to make critical judgments across contexts. We recognize that classifier disagreement may reflect disagreement in underlying human annotations, but add that classifiers’ operationalizations of harm acquire power by shaping decisions and model behavior in practice, in a way that human judgments may not. We focus on the former, so our experimental design emphasizes downstream consequences of choosing one classifier over another. Future work could examine how variation in human and classifier judgments propagates differently through RLHF and RLAIF pipelines.

Assessing statistical reliability.

We account for statistical uncertainty in different ways across the paper:

  • Confidence Intervals: We now report bootstrapped 95% confidence intervals for all relevant findings. For example, the relationship between classifier agreement and harm distance remains negative in both controlled experiments (r = −0.94, 95% CI [−0.94, −0.92]) and naturalistic comparisons (r = −0.78, 95% CI [−0.82, −0.73]; Figure 2).
  • Intervention Selection Robustness: We tested whether differences in top-choice interventions persist under 10,000 prompt bootstraps (Appendix F). Six of ten classifiers select the same top intervention in over 95% of bootstrap samples, yet these six classifiers identify four different interventions as best. We also find differences in their full rankings, with pairwise rank correlations ranging from −0.94 to 1.00 (median 0.43).
  • DPO Training Noise: Since models vary across training runs, we trained two models on each preference dataset for 20 aligned models in total. We use the variation between the two models trained on the same preference dataset to contextualize the distances between models aligned on different datasets by demarcating how much arises through training noise.
AuthorsOpenReview ↗

Rebuttal Table 4: Varying Number of Classifiers

4. Harm distances computed with different numbers of harm classifiers.

ComparisonHarm Classifier Agreement (%)10 Classifiers5 Classifiers2 Classifiers
Perspective vs. 0% Flipped (Seed Replicate)1000.140.160.26
Perspective vs. Safety Pretraining Judge452.422.173.60
Perspective vs. 100% Flipped08.427.118.45

10 classifiers: all

5 classifiers: Perspective, OpenAI, Llama Guard, Safety Pretraining Judge, HarmBench Judge

2 classifiers: Perspective, Safety Pretraining Judge

Reviewer_dNZpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how the choice of a harmful classifier can affect the downstream tasks of content moderation, intervention selection, and preference alignment.

Critical strengths
  1. Prior work usually looked at classifier disagreement in isolation; this study tested several classifiers across three different downstream tasks in one unified framework.

  2. Content-feature analysis adds interpretability. Rather than just reporting aggregate harm distances, they dig into what differs between aligned models, showing that classifiers suppress specific content features (political content, identity mentions) even when aggregate harm looks similar. This is a practically useful finding for deploying these systems in different domains and applications.

Critical weaknesses
  1. Positioning of the paper- The impact of moderation system choice on classification outcomes is already a known problem (Fasching & Lelkes, 2025, as cited by the authors). The community is also aware that harm classifiers disagree and encode different notions of harm, while LLM judges are sensitive to prompt wording. Thus, the paper mainly confirms known issues across three tasks, leaving its novelty unclear. The finding on content-feature suppression in DPO is more interesting, but lacks in-depth explanation and strong validation. Overall, this is a well-executed empirical study, but its contribution would be stronger with a framework, benchmark, or practical solution that advances the field.

  2. The work averages scores across several classifiers to measure how similar two models are, but this aggregation may hide important differences. A recent study, Operationalizing Pluralistic Values in Large Language Model Alignment (AAAI 2026), empirically shows that preserving individual signals rather than aggregating them can reveal meaningful differences. Thus, the conclusion that classifiers barely matter may be an artifact of the measurement approach. This needs further validation using per-classifier results or alternative aggregation methods.

Questions for authors

I would like to clarify a few points based on my understanding, to make sure I did not miss anything.

DPO process: classifiers label ToxiGen responses, build preference pairs, and then train the model via DPO. The same classifiers then provide a harmful label to the aligned model’s generations on different benchmarks; in short, classifiers create the training signal, and classifiers measure the outcome. It’s the same models all the way through. Could you let me know if any human validation is involved here? Or please correct me if my understanding of DPO here is incorrect!

Fig. 10 shows aligned models’ harm rates, but I did not see the base model’s rate. A small harm distance partly means DPO barely moved the models. Though I found that the synthetic flip result partly answers this, the base-vs-aligned comparison might be better for clarification. Was anything related to this mentioned in the paper?

Table 9 shows the breakdown for intervention selection for each classifier. Is there a similar breakdown for harm distance in the DPO alignment setting, showing how much aligned model pairs differ under each individual classifier rather than only the aggregate harm distance?

3 comments

Thank you for your feedback and for highlighting our preference alignment finding. We have substantially strengthened that analysis and made it the central finding in our revised paper.

What is novel beyond classifier disagreement and prompt sensitivity?

As you highlighted, while we already know harm classifiers disagree and are sensitive to prompts, our central contribution is to establish that downstream consequences of classifier disagreement vary across tasks and depend on which outcomes are measured. Our findings on content moderation validate what others have found (and we kept that and intervention selection for completeness), but our analyses show that classifier choice produces surprisingly small differences in aggregate harmfulness after alignment, while differentially impacting content produced by aligned models. We have revised to emphasize our contributions and novelty:

  • Restructuring Results: We consolidated content moderation and intervention selection into 4.1, rewrote 4.2 to establish that classifier disagreement produces small harm distances across a variety of setups, and rewrote 4.3 to examine how classifier sensitivity to content features result in those features being generated less by aligned models (Revised Figure 5).

  • Findings Robustness: We expanded ablations in the Appendix and moved them to our main findings, characterizing how our novel finding generalizes across a range of post-training configurations used in practice. We tested nine variations in data construction, optimization, and metrics (Revised Figure 3 and line 409). Across four data construction and optimization ablations, we found that the naturalistic harm distance changes <2pp. Furthermore, we found negative correlations between classifier sensitivity and feature generation for all content features (r = −0.47 to −0.76, Revised Figure 5), and validated this relationship by inspecting preference datasets (Revised Appendix Figures 15–16).

  • Introduction: We distinguished tasks where classifiers directly impact decisions vs. shape model behavior. We rewrote our contributions to highlight the unexpected lack of difference in preference alignment and emphasize that they instead impact the frequency of content features in generated text.

Could averaging across classifiers hide differences?

We agree aggregation can obscure variation and report per-classifier rates and alternate aggregations:

  • Per-Classifier Results: We reported harm rates for every aligned model according to each classifier (new Appendix Figure 12). For a fixed aligned model, we found harm rates vary by 40-52 percentage points across classifiers, while within-classifier variation across aligned models is ~1.4 percentage points.
  • Maximum-Distance Aggregation: Taking the maximum difference across classifiers produces larger distances (Revised Figure 3), but privileges a single evaluator.
  • Prompt-Level Aggregation: We computed absolute differences at the prompt level before averaging (Section 4.2, Revised Figure 3). Even models aligned to exactly the same dataset have a distance of 6pp, so the increase cannot be solely due to classifier choice.
  • Base-Normalized Aggregation: We normalized differences by each classifier’s base rate before averaging across them. After normalization, the naturalistic distance is only 11% of the distance produced by completely reversing preferences vs. 29% under the original metric (Section 4.2).

While some alternatives reveal larger differences, the small differences we see after DPO are not solely an artifact of our aggregation method.

Did DPO meaningfully change the models, and can we inspect individual distances?

DPO does change aggregate harmfulness. We now report:

  • Breakdown of Harm Distances for Every Pair: No pair of aligned models has a harm distance of more than 2.6 (Revised Appendix Figure 13, also Rebuttal Tables 1, 2 below).
  • Base Model Harm Distances: The harm distance between the base model and aligned models have harm distances of at least 2.7 (Revised Appendix Table 9, also Rebuttal Table 3 below). Each aligned model differs more from the base model than any two aligned models differ from each other, indicating that DPO has a substantive effect on measured aggregate harmfulness.

Was DPO validated by humans?

No human validation was performed. Our study examines how differences between automated preference signals propagate through RLAIF pipelines that replace human annotation (Bai et al., 2022; Lee et al., 2024). Understanding how classifier choice shapes model behavior is valuable regardless of whether models satisfy human preferences. We agree human eval would help assess the practicality of these differences and leave it to future work. As discussed in new Appendix I, human judgments are also subject to disagreement, motivating future work comparing the effects of human vs. AI disagreement on preference alignment.

AuthorsOpenReview ↗

Rebuttal Tables 1 and 2: Breakdown of Harm Distances for Every Pair of Models After DPO

1. Harm Distance between models aligned on Perspective and OpenAI preference datasets and the models aligned on the versions of those datasets with a certain percent of those preference pairs flipped.

Preference Pairs FlippedPerspectiveOpenAI
0%0.10.2
10%0.92.0
20%1.42.6
30%1.73.8
40%2.54.3
50%3.14.3
60%3.54.8
70%3.75.1
80%5.15.9
90%6.07.4
100%8.48.9

2. Harm distance between pairs of models aligned on preference datasets chose by each harm classifier.

Preference Data Harm ClassifierPerspectiveOpenAIRoBERTaToxiGen HateBERTHateXplainLlama GuardSimple JdgSafety Pretraining JdgConstitutional JdgHarmBench Jdg
Perspective0.11.30.81.70.60.50.52.41.10.9
OpenAI1.30.20.60.81.31.41.61.10.50.9
RoBERTa0.80.60.11.10.80.91.21.70.50.6
ToxiGen HateBERT1.70.81.10.21.51.61.91.10.91.2
HateXplain0.61.30.81.50.20.50.62.20.90.6
Llama Guard0.51.40.91.60.50.10.62.41.00.7
Simple Jdg0.51.61.21.90.60.60.32.61.30.9
Safety Pretraining Jdg2.41.11.71.12.22.42.60.31.41.8
Constitutional Jdg1.10.50.50.90.91.01.31.40.10.5
HarmBench Jdg0.90.90.61.20.60.70.91.80.50.2

All values represent harm distance (0–100), as labeled in the original figure.

AuthorsOpenReview ↗

Rebuttal Table 3: Base Model Harm Distances

3. Harm distance between base model and models aligned on preference datasets chose by each harm classifier.

ModelHarm distance from base (seed 1)Harm distance from base (seed 2)
Aligned on preference data selected by:
Perspective3.1 [2.8, 3.3]3.0 [2.8, 3.3]
OpenAI4.3 [4.1, 4.6]4.4 [4.1, 4.6]
RoBERTa3.9 [3.6, 4.2]3.9 [3.6, 4.2]
ToxiGen HateBERT4.7 [4.4, 5.0]4.6 [4.3, 4.9]
HateXplain3.2 [3.0, 3.5]3.4 [3.2, 3.7]
Llama Guard3.2 [2.9, 3.4]3.2 [3.0, 3.5]
Simple Judge2.8 [2.6, 3.1]2.7 [2.5, 3.0]
Safety Pretraining Judge5.5 [5.2, 5.8]5.3 [5.0, 5.6]
Constitutional Judge4.1 [3.8, 4.4]4.0 [3.7, 4.3]
HarmBench Judge3.7 [3.4, 4.0]3.7 [3.4, 3.9]
100% of pairs flipped:
Perspective 100% flipped5.3 [5.0, 5.7]—
OpenAI 100% flipped4.6 [4.3, 4.9]—

Comments on the paper

1 comment

We thank all reviewers for their thoughtful feedback and for recognizing the practicality, breadth, and counterintuitive findings in our work. We have substantially revised the paper to clarify its novel contributions, strengthened their validation, and addressed methodological concerns.

Our central contribution is not that harm classifiers disagree, but that differences between them propagate unevenly across different downstream tasks. While classifier choice substantially impacts moderation labels (classifier pairs disagree on up to 55% of texts) and intervention selections (five of six interventions ranked best by at least one classifier), it has surprisingly small differences in aggregate harmfulness after preference alignment (less than three percentage points). However, similar aggregate harmfulness hides other differences in what models generate. We show that when a classifier is more likely to flag text containing a particular feature, such as violence or political content, the model aligned using that classifier generates content containing that feature less often. This negative trend holds across all six content features (r = −0.47 to −0.75). We rewrote Sections 4.1-4.3 and reframed the introduction to emphasize and investigate these novel findings in response to dNZp's novelty concerns.

We have also strengthened the robustness and interpretation of our preference alignment results in 4.2 and 4.3. Our revisions include:

  • Four experimental variations covering data construction and optimization, with naturalistic harm distances changing by less than two percentage points (Revised Figure 3, moved out of Appendix per dNZp's validation concerns).
  • Five alternative aggregation methods and disaggregated results demonstrating that small differences in aggregate harmfulness after preference alignment are not solely an artifact of our metric. Base-rate normalization even reduces the harm distance of naturalistic models from 29% to 11% of the controlled model range (Revised Section 4.2, added per dNZp and QzHR's aggregation concerns and Rupf's metric concerns).
  • Validation that DPO has an effect, showing that the harm distances between base and aligned models (2.7–5.5 points) exceed every harm distance between differently aligned models (maximum 2.6; Appendix Table 9 added per dNZp and QzHR's baseline concerns).
  • Additional analysis of content features linking classifier sensitivities to preference dataset composition for five of six features (Appendix Figures 15–16, added per QzHR's aggregation concerns). We thus also distinguished small differences in aggregate harmfulness from potentially consequential differences in model behavior.
  • Additional statistical validation with bootstrapped 95% confidence intervals on all relevant results, in response to Rupf's reliability concerns.
  • Additional human label variation literature, particularly on perspectivism and pluralism (Appendix I, in response to Rupf's prior work concerns). We explain below why we believe this literature addresses a slightly different problem from our work, which is why our initial submission excluded it.

Finally, we have clarified existing validation (DPO training across two seeds, robustness checks on intervention selection results) the selection of our L1-based harm distance metric, reiterated content feature labeling and validation, and expanded our discussion of human label variation and differing harm classifier scopes.

Together, these revisions reiterate that the effects of harm classifier choice depend on both the downstream task and which aspects of its outcomes are measured. This has critical implications for how harm classifiers are selected, evaluated, and repurposed across ML development pipelines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.