acceptodds
Under review as a conference paper at ICLR 2027

Same Words, Different Intent: Hostility-Conditioned Contrastive Learning for Robust Multilingual, Multimodal Hate Speech Detection

Abstract

As digital hostility rapidly shifts from text to regional voice notes and multimedia broadcasts, automated moderation systems are failing to keep pace. Deprived of true semantic understanding, current multimodal classifiers act as brittle keyword spotters. They routinely misclassify political sarcasm, non-hateful distress, and pro-group solidarity as toxic abuse simply because these statements contain shared identity terms. Escaping this shortcut-learning paradigm requires nuanced multimodal training, yet there is a critical scarcity of such resources for diverse, low-resource languages. To break this bottleneck, we introduce a massive synthetic hate speech corpus spanning English and nine Indian languages, uniquely aligned across three modalities: text, speech, and text-in-image. Leveraging this foundational resource, we propose a novel hostility-conditioned composite contrastive objective over SONAR and SeamlessM4T encoders. By structuring the latent space around fine-grained hostility categories and integrating minimal-edit counterfactual pairs, we compel the model to learn the true intent behind the language, differentiating genuine malice from benign linguistic overlap. The empirical gains are substantial. On a demanding diagnostic probe isolating these exact linguistic failure modes, our intervention increases accuracy from 78.4% to 95.1%. Furthermore, our approach outperforms the state-of-the-art multimodal baseline across all ten languages. Crucially, this robust representation transfers zero-shot to independent, real-world benchmarks (X-MuTeST and ADIMA). This meaningfully bridges the gap between synthetic training environments and real-world deployment, offering a potential approach for multilingual AI safety.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hatespeech, text-to-speech, hostility, contrastive learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XTyXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multilingual and multimodal hate speech detection framework that combines hostility-conditioned contrastive learning with counterfactual data augmentation. Experiments demonstrate improved classification performance across languages, substantial gains on a counterfactual diagnostic benchmark, and positive zero-shot transfer results on real-world datasets.

Critical strengths
  1. The paper addresses an important problem in multilingual hate speech detection by reducing reliance on misleading lexical cues and improving intent-level discrimination.

  2. The evaluation covers ten languages, includes targeted counterfactual experiments and ablation studies, and demonstrates generalization to independent real-world datasets.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework mainly combines existing contrastive learning and counterfactual augmentation techniques. The ablation results also suggest that most improvements come from counterfactual data rather than the proposed objective.

  2. Limited real-world robustness evaluation. Both training counterfactuals and diagnostic samples are synthetically generated around similar confound categories, potentially overestimating robustness. The improvements on real-world datasets are considerably smaller.

  3. Unclear multimodal contribution. The improvement over the text-only baseline is relatively small, while most audio training data is synthesized. The effectiveness of multimodal fusion under realistic speech conditions remains insufficiently demonstrated.

  4. Limited linguistic diversity. The evaluation primarily focuses on nine Indian languages and English, without considering languages from other linguistic regions. This limits the evidence for the claimed multilingual generalizability of the proposed method.

  5. Although the authors acknowledge that some figures were generated using AI, several figures are too small and blurry, and some contain incomplete borders. These figures should be carefully revised to improve their clarity, readability, and overall presentation quality.

Questions for authors
  1. How does the proposed hostility-conditioned contrastive objective compare with standard supervised contrastive learning when using the same counterfactual training data?

  2. How well does the method generalize to human-written counterfactual examples, given that both training and diagnostic samples are synthetically generated around similar confound categories?

  3. How much does the audio modality contribute beyond text alone, particularly when evaluated on real speech with sarcasm, emotional cues, or implicit hostility?

  4. How well does the proposed method generalize to languages beyond the Indian linguistic context, such as Arabic, Chinese, or European languages?

4 comments

We thank the reviewer for the thoughtful feedback.

Q1. Comparison with standard SupCon on the same CF data

We compare standard SupCon and our objective using the same CF data, encoder architecture, optimization settings, Stage-3 evaluation protocol, and six shared seeds. Table R1 reports the complete controlled comparison, including CE, standard SupCon, composite SupCon, the addition of decorrelation, and the hostility-shuffle control.

Table R1. Controlled objective comparison (mean ± SD over six seeds). Probe-FPR is the percentage of benign probe anchors predicted as hateful. Pair-acc measures the percentage of (anchor, CF) pairs for which both members are classified correctly. ZS denotes zero-shot evaluation and LP denotes linear-probe evaluation.

ObjectiveTest F1Probe accProbe-FPR ↓Pair-accADIMA LPX-MuTeST LPADIMA ZSX-MuTeST ZS
CE79.09±0.1393.37±0.2112.07±0.4486.7±0.473.01±0.7477.94±0.8156.24±1.3366.74±1.18
Standard SupCon79.46±0.2094.07±0.6210.33±1.4088.2±1.273.43±0.5076.28±1.1756.65±2.1766.60±1.07
Composite SupCon79.43±0.1294.37±0.809.40±1.7588.7±1.674.50±0.7278.31±0.7857.30±2.3464.63±1.36
Ours (Composite SupCon + decorrelation)79.44±0.1294.84±0.808.44±1.9189.7±1.674.70±0.9978.86±0.5756.81±1.4765.56±1.37
Ours, hostility shuffled78.65±0.3793.34±0.6211.33±2.2886.8±1.274.54±0.8478.53±0.7357.09±1.9065.44±1.37

The results show that our method achieves comparable in-distribution test F1 to standard SupCon (79.44 vs. 79.46), while improving probe accuracy from 94.07 to 94.84 and reducing benign-anchor FPR from 10.33% to 8.44%. More importantly, it improves linear-probe transfer by 2.58 points on X-MuTeST (p<0.001p<0.001) and 1.27 points on ADIMA (p=0.033p=0.033) over standard SupCon. Adding decorrelation to composite SupCon provides further improvements on both benchmarks.

We also report the zero-shot results: standard SupCon performs approximately one point better on X-MuTeST (66.60 vs. 65.56; p=0.046p=0.046). We retain this result for a balanced comparison.

Finally, shuffling hostility labels within the binary hate/non-hate classes reduces probe accuracy from 94.84 to 93.34 and test F1 from 79.44 to 78.65. This supports the contribution of the hostility information to the learned representations.

W1 Novelty: Contribution beyond counterfactual data

We agree that counterfactual (CF) data provides the largest individual improvement in our experiments. However, the contribution of this work is broader than introducing CF augmentation or supervised contrastive learning independently. It lies in combining a multilingual multimodal resource, controlled hostility-focused counterfactual generation, and a representation objective that preserves fine-grained hostility information.

First, we develop a multilingual multimodal resource covering English and nine Indian languages across text, audio, and images. This addresses an important gap in existing hate-speech research, which remains limited in its coverage of Indian languages and modalities. The resource also provides a common basis for studying how hostility is expressed across different languages and forms of communication.

Second, we introduce a controlled counterfactual generation strategy. The generated examples retain shared topics and key terms while varying hostility. This enables controlled comparisons of model behaviour and helps examine whether detectors rely on hostility-related information or superficial lexical and topical cues. The use of synthetic speech further extends this evaluation to audio without requiring identifiable voice recordings of people producing hateful content.

Third, we propose a hostility-conditioned contrastive objective. Standard binary supervised contrastive learning groups all samples with the same hate/non-hate label, potentially overlooking meaningful hostility distinctions within each class. Our objective uses both the binary label and the fine-grained hostility category to organize representations. This encourages the model to distinguish, for example, neutral criticism from insults and different forms of explicit hostility, even when they share the same binary label.

The controlled comparison in Table R1 (in Q1) further establishes the contribution of our learning objective. With the same CF data and matched experimental settings, composite SupCon improves ADIMA and X-MuTeST linear-probe F1 over standard SupCon by 1.07 and 2.03 points, respectively. Adding decorrelation further improves these results to 74.70 and 78.86, corresponding to gains of 1.27 and 2.58 points over standard SupCon. The proposed objective also achieves the lowest benign-anchor FPR (8.44%) among the compared objectives. Moreover, shuffling hostility labels within binary classes reduces probe accuracy from 94.84% to 93.34%, showing that the hostility information contributes to the learned representations. These results demonstrate that the proposed learning objective adds value beyond the CF data itself.

We do not attribute the overall gains to the contrastive objective alone. Removing CF data reduces probe accuracy by 16.59 points and zero-shot X-MuTeST F1 by 3.33 points (p<0.001p<0.001). Rather, the results show that CF data provides substantial training benefit, while the proposed objective contributes additional improvements by preserving fine-grained hostility structure.

The broader contribution is therefore the integration of data, generation, and learning: a multilingual multimodal resource, a controlled way to construct hostility-focused examples, and a representation objective designed to preserve distinctions that binary supervision can overlook. Together, these components support more systematic and inclusive hate-speech research, particularly for under-resourced Indian languages and modalities, while providing a basis for developing and evaluating more reliable detection systems.

Q2 / W2: Generalisation to human-written examples

We evaluate zero-shot on HateCheck (Röttger et al., 2021) and Multilingual HateCheck (Röttger et al., 2022), which contain expert-written examples, including counter-speech, negation, neutral and positive identity mentions, abuse against non-protected targets, and reclaimed slurs. These cases provide a complementary evaluation to our targeted synthetic probe.

The results show that all systems trained on the original corpus, including the Ranjan et al. baseline, over-flag some templated sentences that explicitly mention protected groups. Within our controlled family, CF data and hostility conditioning provide the strongest results on neutral and positive identity mentions. However, absolute false-positive rates remain.

We further examine whether the same CF-generation framework can address this specific confound. We add protected-identity mentions and supportive/defending statements through the same generation, translation, and filtering pipeline. No HateCheck data is used. With this extension:

  • HateCheck-EN F1 improves from 59.1 to 62.2 p=0.026p=0.026.
  • Positive identity accuracy improves from 19.5 to 30.2.
  • The result exceeds the Ranjan et al. baseline on HateCheck-EN and on the aggregate Multilingual HateCheck score.
  • On the same extended training data, our objective improves neutral identity accuracy by 6.0 points over CE and 6.3 points over standard SupCon.

The extension also improves the original probe and retains in-distribution F1. We report the remaining identity-term over-flagging and the lack of improvement on counter-speech that quotes hateful language in order to reject it.

The external benchmarks provide additional evidence beyond the synthetic probe. X-MuTeST contains human-written posts, while ADIMA provides real speech. CF data improves zero-shot X-MuTeST F1 by 3.33 points p<0.001p<0.001, and the hostility-conditioned objective improves linear-probe transfer over both CE and standard SupCon.

We also clarify the composition of the English source data: 75% comes from MetaHate and consists of real, human-written social-media posts with human labels. The synthetic components are primarily the translations and speech renderings, rather than the original content.

References

  • P Röttger, B Vidgen, D Nguyen, Z Talat, H Margetts, and J Pierrehumbert. 2021. [HateCheck: Functional Tests for Hate Speech Detection Models]. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 41–58.
  • P Röttger, H Seelawi, D Nozza, Z Talat, and B Vidgen. 2022. [Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models]. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 154–169.

Q3 / W3: Contribution of audio, especially on real speech

We evaluate text-only, audio-only, and fused variants of the same pipeline using the same data and six-seed protocol.

ModelTest F1Probe accProbe-FPRADIMA ZS (real speech)ADIMA LPX-MuTeST LP
Text-only79.32±0.1091.24±0.4316.356.57±1.1070.77±0.8578.30±0.45
Audio-only61.44±0.3394.50±0.355.667.13±1.6373.94±0.7773.69±0.30
Text + audio (ours)79.44±0.1294.84±0.808.456.81±1.4774.70±0.9978.86±0.57

The results highlight complementary contributions from the two modalities.

  1. Audio adds robustness beyond text. The fused model improves probe accuracy by 3.60 points over text-only p<0.001p<0.001 and reduces benign-anchor FPR from 16.3% to 8.4%. The probe audio uses the same neutral prosody description for every item, so this improvement comes from the speech representation rather than label-dependent emotional or prosodic cues.

  2. Audio is particularly informative on real speech. On ADIMA, the audio-only model achieves 67.13 zero-shot F1, compared with 56.57 for text-only. With in-domain linear-probe evaluation, fusion performs best, improving over text-only by 3.93 points p=0.001p=0.001.

  3. Zero-shot fusion does not fully exploit audio on ADIMA. We investigate whether the fused model over-relies on text by retraining Stage 3 with modality dropout p=0.3p=0.3. This leaves ADIMA zero-shot performance essentially unchanged (56.67 ± 1.82), as well as the other metrics. The results suggest a domain shift between synthetic speech/ASR and real speech, rather than a simple modality-selection issue. A linear probe on the real-speech domain improves the fused result.

Q4 / W4: Languages beyond India

We evaluate zero-shot transfer on Multilingual HateCheck (Röttger et al., 2022) in Arabic (AR), Mandarin (ZH), German (DE), French (FR), Spanish (ES), Italian (IT), Dutch (NL), Polish (PL), and Portuguese (PT), in addition to romanised, code-mixed Hindi (HI-Roman). The nine non-Indic languages have no training data in our framework.

ModelARZHDEFRESITNLPLPTHI-Roman
Ranjan et al.54.057.658.657.857.257.059.758.957.754.2
Ours (original CF)55.657.355.257.256.358.058.256.156.751.7
Ours + identity CF57.360.057.159.158.060.160.257.458.951.3

With the protected-identity CF extension, our method exceeds the baseline on seven of the nine non-Indic languages, without training data in those languages.

Threshold-free ranking quality is also informative: AUROC is approximately 0.78–0.80 on the seven European languages, 0.75 on Mandarin, and 0.68 on Arabic. Thresholded F1 remains affected by identity-term over-flagging, as discussed above.

We scope the claim accordingly: these results demonstrate zero-shot transfer on the evaluated languages, rather than universal multilingual generalization. The framework uses language-independent representations and translation/speech components that support a broad language range, but demonstrating reliable performance in additional languages requires appropriate training and evaluation data.


W5: Figures

We will improve the figures with complete borders and legible fonts.

Reviewer_Q33sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors approach the problem of binary hate speech (hate vs no-hate) detection for text + audio data. They design a three-step supervised training setup:

  1. train a text-audio encoder EE consisting of projection networks, on top of frozen pre-trained model embeddings, to predict 6 hostility classes;
  2. train EE using contrastive learning to shape its latent space to separate 12 composite classes cic_i;
  3. freeze EE and train a classifier head on top to predict hate speech labels.

The authors also construct a dataset using minimal-edit counterfactual augmentation. They start from English data and expand it to 9 additional Indian languages using MT for text and generate matched audio samples with TTS models, going from 35.8k English-language instances to over 1M instances in the 10 total languages across three modalities (text, audio, images). The authors use this data to carry out experiments against a state-of-the-art baseline (Ranjan et al., 2025a) and two unimodal baselines and find that their model outperforms all of them.

They also experiment in a cross-domain setting to evaluate their approach on out-of-distribution data (X-MuTeST and ADIMA) and find once again that their framework outperforms the Ranjan et al. (2025a) baseline.

Critical strengths

The problem and approach are relevant and well motivated in Section 2. The methodology in Section 3 is well explained.

The overall approach seems to produce concrete performance gains over Ranjan et al. (2025a), and the counterfactual training approach seems quite effective based on the ablation results in Table 4. The fact that the results also seem to transfer on real OOD benchmarks is especially impressive (Tables 5 and 6). In addition, on the diagnostic test set, counterfactual augmentation with plain cross-entropy already achieves 91.73% accuracy, compared with 95.07% for the full framework (Table 7).

Critical weaknesses

The main issue of this submission is its transparency and reproducibility. In general, more details on how the dataset was created are required, since it is one of the main contributions, along with the training approach tailored around it. The full framework improves over CAD-CE by only 0.07 percentage points in macro-F1 on the main test set (Table 9), whereas it improves diagnostic-set accuracy by 3.34 percentage points (Table 7). Therefore, being able to inspect the generated data is particularly important. On this, no full examples are provided in the paper (Figure 1 seems like a diagram simplification). In addition, no information is provided as to how the hostility labels are obtained (they are not in the MetaHate and ToxiGen data).

The authors do not make the data and/or code available at submission and only promise to publish them upon publication. I think for such a submission having access to the data and the code is an integral part of the reviewing process. The code would help clarify a few aspects that are not discussed in the paper. Some specific issues:

  • Lines 197–198: What are the "original seed sentences" and from which dataset are they excluded? The generated counterfactuals or the diagnostic set?
  • Lines 240-242: The relationship between LSupCon\mathcal{L}_{\mathrm{SupCon}} and \mathcal{L\}_{\mathrm{base}} is not clear.
  • Lines 300-308: The prompts and generation settings used to produce the prosody descriptions and synthesized speech are not specified.
  • Lines 312-313: Still on the topic of opaqueness, the authors claim "Dataset quality is verified through dual validation: human evaluation demonstrating high inter-annotator agreement and naturalness, alongside rigorous quantitative metrics." However, the claimed human validation is not substantiated with an evaluation protocol, inter-annotator agreement scores, or naturalness ratings.
  • Lines 372-376: The separate instruction used to generate the diagnostic test set is not provided, and the procedure used to enforce non-overlap with training counterfactuals is insufficiently described.
Questions for authors

Questions:

How are the hostility labels obtained?

What is the reason behind the performance difference between CAD-CE and the full model being 3.34 percentage points in accuracy in Table 7, but only 0.07 percentage points in macro-F1 in Table 9?

Lines 63-64: Are the example predictions from a model trained on the data the authors are using? Or are they just plausible examples designed by the authors?

Presentation:

The methodology is explained clearly in Section 3 but it is not easy to match with the experiments, because of the opaqueness of the data creation process.

Line 63: "Figure 1)" --> remove the stray parenthesis

Line 129: Figure 2 is not referenced anywhere in the text.

Line 214-215: cite layer normalization and dropout papers

Line 242: remove one of "discourage" / "penalize"

Maybe move Figure 3 up by two lines, since the justification in lines 283-284 is a bit ugly.

Line 297: provide a URL footnote to Sarvam AI.

Lines 343–346: The authors claim that "all primary results" report mean and SD across six seeds, but Table 3 does not provide standard deviations. The only standard deviations for Section 6 are provided in the prose for the proposed method, but not for the baselines.

5 comments

We thank the reviewer for the careful assessment.

Q1. How are the hostility labels obtained, and how reliable are they?

The six hostility categories are automatically assigned using Qwen2.5-7B-Instruct, with greedy decoding and a fixed JSON output format. Annotation is performed on English source sentences from the training and validation sets only. The resulting labels are propagated to their translations and corresponding modalities using the source sample ID. Test samples are never annotated or used to construct training labels.

The six categories are none, criticism, insult, threat, violence, and dehumanization. The full annotation prompt and detailed reliability analyses are provided in our response to Reviewer 6HmC (W2/Q2). For completeness, the main evidence is:

  • Human-rated reference: The labels show a positive association with ToxiGen's crowd-rated toxicity scores (Spearman ρ=0.676\rho=0.676, n=7,289n=7{,}289).
  • Independent annotator: A separate model, Mistral-Small-24B-Instruct, gives Cohen's κ=0.58\kappa=0.58 on a 1,500-item sample (1,472 items with valid labels from both models).
  • Noise robustness: Randomly corrupting 25% of hostility labels does not materially affect performance. Shuffling hostility labels within the binary hate/non-hate classes reduces probe accuracy by 1.50 points and test F1 by 0.79 points, supporting the value of the hostility information.

Q2. Why does the CAD-CE improvement differ between the diagnostic set and the main test set?

We revisited this comparison and conducted a matched-protocol evaluation. The original Table 7 comparison used a CAD-CE model trained end-to-end, without the Stage-3 frozen-encoder evaluation used by our framework. The revised comparison provides a more appropriate estimate of the objective's contribution.

Under the matched protocol:

  • Our probe improvement over CAD-CE is 1.48 points (95% CI [0.19, 2.91]).
  • The test-F1 improvement is 0.34 points p=0.001p=0.001.

The two evaluations also measure different aspects of performance. The 1,500-instance probe deliberately concentrates on the targeted counterfactual confounds, whereas the 53,830-instance test set reflects the broader source-data distribution, where these confounds are less frequent. Counterfactual samples account for approximately 3.5% of the training pool. Consequently, their effect is much more visible on a targeted diagnostic set than on aggregate test performance. The counterfactual data itself shows the same pattern: adding it changes test F1 by only −0.08 (not significant) but improves probe accuracy by 16.6 points.

We will use the controlled comparison reported in Table R1 (W1) of our response to Reviewer 6HmC and update the earlier comparison accordingly. This avoids attributing differences caused by the training protocol to the proposed objective.


Q3. Are the examples in Figure 1 actual model predictions?

The original Figure 1 is an illustrative schematic, not a set of model predictions. We will make this distinction explicit and replace the schematic with actual predictions from the trained models.

The revised examples report the fraction of six seeds predicting hate, shown as no-CF model → our model:

  • "Why do Democrats always think they can just wing it on healthcare?" (6/6 → 0/6)
  • "I'm just sick of this." (6/6 → 1/6)
  • "Republicans keep talking about tax cuts, but what's the point if it doesn't help the middle class?" (6/6 → 0/6)

This makes the examples directly traceable to model outputs. We will identify the examples and report the prediction frequencies in the revised figure.

Q4. Experimental clarity and reproducibility

We appreciate the reviewer’s request for greater clarity on the data-generation and experimental procedures. We provide the specific details below.

Source sentences and filtering (Lines 197–198 and 372–376). The seven original seed sentences are hand-written examples used as generation prototypes; they are not taken from an existing dataset. Generated samples that reproduce these prototypes are excluded from both the training counterfactual set and the diagnostic probe. Duplicate filtering uses character-level difflib.SequenceMatcher similarity with a threshold of 0.85, not cosine similarity. The diagnostic probe is additionally checked against all training counterfactual sentences. The maximum observed character-level similarity is 0.83.

Loss construction (Lines 240–242). We clarify the relationship between LSupConL_{\mathrm{SupCon}} and LbaseL_{\mathrm{base}} by defining the per-anchor contrastive loss. With sij=zi⊤zj/τs_{ij}=z_i^\top z_j/\tau, we use

ℓi(Z1,Z2)=−1∣P(i)∣∑p∈P(i)sip+log⁡∑a∈A(i)exp⁡(sia).\ell_i(Z_1,Z_2)=-\frac{1}{|P(i)|}\sum_{p\in P(i)}s_{ip}+\log\sum_{a\in A(i)}\exp(s_{ia}).

Here, P(i)={j:cj=ci}P(i)=\{j:c_j=c_i\} contains the positive samples sharing the composite label with anchor ii. If no such positive is present in the batch, or the anchor's hostility label is unknown, we fall back to P(i)={j:yj=yi}P(i)=\{j:y_j=y_i\}, where yiy_i is the binary hate/non-hate label. Self-comparisons are excluded from P(i)P(i) and A(i)A(i) in the intra-modal terms. We then define

LSupCon(Z1,Z2)=1B∑i=1Bℓi(Z1,Z2),L_{\mathrm{SupCon}}(Z_1,Z_2)=\frac{1}{B}\sum_{i=1}^{B}\ell_i(Z_1,Z_2),

and combine the three contrastive terms as

Lbase=13[LSupCon(ZT,ZT)+LSupCon(ZA,ZA)+LSupCon(ZT,ZA)].L_{\mathrm{base}}=\frac{1}{3}\left[L_{\mathrm{SupCon}}(Z^T,Z^T)+L_{\mathrm{SupCon}}(Z^A,Z^A)+L_{\mathrm{SupCon}}(Z^T,Z^A)\right].

Thus, LSupConL_{\mathrm{SupCon}} defines the contrastive objective for a pair of representation sets, while LbaseL_{\mathrm{base}} averages its text, audio, and cross-modal applications.

Prosody and speech generation (Lines 300–308). We use Llama-3.1-8B-Instruct to generate prosody descriptions from the English sentence alone; the model never sees the hate/non-hate label. The prompt instructs the model to describe acoustic properties of the delivery rather than the meaning or morality of the text, choosing pitch and pace from fixed options and the tone from a fixed list (e.g., calm, assertive, frustrated). Generation uses temperature 0.7, top-p 0.9, and a maximum output length of 120 tokens. We then synthesize speech using Indic-Parler-TTS, with a fixed speaker for each language. Importantly, all audio for the original counterfactual pairs and the diagnostic probe uses the same fixed neutral prosody description. Prosody therefore cannot reveal the counterfactual label, and the probe cannot be solved through prosodic cues.

Diagnostic-probe construction (Lines 372–376). The probe uses the same generator prompt as the training counterfactuals, with only the style instruction replaced by: "Ensure the vocabulary, tone, slang, registers, and sentence lengths are highly diverse. Use colloquialisms, snarky idioms, varied sentence lengths, and synonyms to make this set linguistically independent from training phrasing." Its examples are filtered against the complete training counterfactual set using the same character-level similarity criterion described above.

Dataset quality (Lines 312–313).

Together, these details specify the source selection, filtering, loss construction, generation settings, and validation procedures underlying the experiments.


Q5. Human validation and dataset quality

We agree that the original wording about human validation was insufficiently supported by a documented protocol and quantitative agreement scores. The dataset received qualitative feedback from 6–7 members proficient in the relevant languages, who reviewed the linguistic appropriateness and consistency of the examples. This provides useful expert feedback, but it should not be presented as a formal inter-annotator evaluation.

We will therefore revise the claim to distinguish the qualitative expert review from the automatic checks. These include translation consistency, semantic similarity, back-transcription checks, and speech-quality measurements. We will report the corresponding metrics and avoid claiming quantified human agreement or naturalness ratings that were not measured.


Q6. Code, prompts, and reproducibility

We agree that transparent documentation is important for this dataset-driven study. The generation and annotation procedures, including the prompt instructions, model settings, filtering criteria, and representative source-to-counterfactual examples, are specified above. We also report the experimental configurations and per-seed results. The generated dataset itself will remain under controlled research access because it contains harmful content; we do not intend unrestricted redistribution.

Q7. Presentation and reporting corrections

We thank the reviewer for identifying these presentation issues. We will correct the typographical errors, add the missing references and URLs, improve figure placement and cross-referencing, and ensure that the figures and tables are consistent with the reported experiments. We will also include the standard deviations for all baselines in Table 3. The six-seed results are:

ModelOverall accuracyOverall F1
Ranjan et al.77.35 ± 0.2577.33 ± 0.25
Text-only78.74 ± 0.3078.70 ± 0.29
Audio-only62.09 ± 0.2661.25 ± 0.43
Ours79.48 ± 0.1279.44 ± 0.12
Reviewer_6HmCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses lexical shortcut learning in multilingual and multimodal hate speech detection. The authors introduce a synthetic dataset covering ten languages and three modalities, and propose a framework combining counterfactual augmentation with hostility-conditioned supervised contrastive learning. The method achieves substantial improvements on targeted diagnostic examples and is further evaluated on external datasets. While the problem is important and the results are promising, the independent contribution of the proposed method and the strength of the generalization evidence remain unclear.

Critical strengths
  1. The paper addresses an important problem in hate speech detection, focusing on distinguishing genuine hostility from non-hateful expressions with similar lexical cues.
  2. The counterfactual augmentation achieves substantial improvements on the diagnostic set, and the multilingual dataset has potential value for low-resource language research.
Critical weaknesses
  1. The independent contribution of the proposed contrastive learning method is not sufficiently demonstrated. Much of the improvement appears to come from counterfactual augmentation, while the comparison with CAD-CE does not fully isolate the effect of the proposed objective.
  2. The construction and validation of the six hostility categories are insufficiently explained. Since these labels directly determine the contrastive learning supervision, their reliability is essential to the validity of the proposed method.
  3. The evidence for real-world robustness remains limited. Improvements on external datasets are relatively modest and inconsistent across languages. Moreover, the lack of direct comparisons with CAD-CE makes it difficult to determine whether the generalization gains come from the proposed method or primarily from data augmentation.
Questions for authors
  1. Table 7 suggests that the proposed contrastive objective provides additional gains over CAD-CE. However, Appendix D.1 indicates differences in the training procedures. Could the authors provide a comparison under identical training settings, including standard supervised contrastive learning, to better isolate the contribution of the proposed objective?

  2. The six hostility categories are central to the proposed method, but Sections 3–4 do not clearly explain how these labels are obtained. Could the authors clarify whether they are manually annotated or automatically generated, and provide evidence of their annotation quality and consistency across languages?

  3. Section 4 describes generating translations and multiple modalities from the same source utterances, while Table 2 reports the dataset splits. Was the split performed at the source-utterance level before generating these variants? If not, how did the authors ensure that related instances do not appear across training and test sets?

  4. Table 5 compares the full model with the Ranjan et al. baseline on external datasets, but does not include CAD-CE. Could the authors provide this comparison to clarify whether the transfer gains come from the proposed contrastive objective? Additionally, how are ADIMA's original abuse/profanity labels mapped to the hate-speech task?

  5. The paper aims to reduce lexical shortcuts and false positives on non-hateful content, but the evaluation mainly reports accuracy and F1. Could the authors provide false-positive analyses on non-hateful examples containing identity-related terms, or evaluate confound categories beyond those used during counterfactual training?

7 comments

W1 / Q1: Contribution of the objective under identical training conditions

The main paper compares our complete framework with conventional end-to-end classification baselines to evaluate the overall benefit of the proposed pipeline. This comparison addresses a complementary question: whether the complete framework improves over standard classification approaches. To further isolate the contribution of the hostility-conditioned objective itself, we now extend the evaluation with a controlled comparison against CE and standard supervised contrastive learning (SupCon), as suggested by the reviewer.

All methods in this additional comparison use identical training data, encoder architecture, initialization, optimization settings, training budget, and Stage-3 evaluation protocol, with six shared random seeds. Only the objective changes. For CE, the encoder is trained jointly with a classification head before applying the same frozen-encoder Stage-3 evaluation used for the other methods.

Table R1. Controlled objective comparison (mean ± SD over six seeds). Probe-FPR is the percentage of benign probe anchors predicted as hateful. Pair-acc measures the percentage of (anchor, CF) pairs for which both members are classified correctly. ZS denotes zero-shot evaluation and LP denotes linear-probe evaluation.

ObjectiveTest F1Probe accProbe-FPR ↓Pair-accADIMA LPX-MuTeST LPADIMA ZSX-MuTeST ZS
CE79.09±0.1393.37±0.2112.07±0.4486.7±0.473.01±0.7477.94±0.8156.24±1.3366.74±1.18
Standard SupCon79.46±0.2094.07±0.6210.33±1.4088.2±1.273.43±0.5076.28±1.1756.65±2.1766.60±1.07
Composite SupCon79.43±0.1294.37±0.809.40±1.7588.7±1.674.50±0.7278.31±0.7857.30±2.3464.63±1.36
Ours (Composite SupCon + decorrelation)79.44±0.1294.84±0.808.44±1.9189.7±1.674.70±0.9978.86±0.5756.81±1.4765.56±1.37
Ours, hostility shuffled78.65±0.3793.34±0.6211.33±2.2886.8±1.274.54±0.8478.53±0.7357.09±1.9065.44±1.37

The results demonstrate that the proposed objective provides benefits beyond both conventional CE and binary SupCon:

  • Over CE, our method improves probe accuracy by 1.48 points (95% CI [0.19, 2.91]), test F1 by 0.34 points (p=0.001p=0.001), and ADIMA linear-probe F1 by 1.69 points (p=0.009p=0.009). It also reduces benign-anchor FPR from 12.07% to 8.44%.
  • Over standard SupCon, our method achieves comparable in-distribution F1 (79.44 vs. 79.46), while improving ADIMA linear-probe F1 by 1.27 points (p=0.033p=0.033) and X-MuTeST linear-probe F1 by 2.58 points (p<0.001p<0.001).
  • The hostility information itself matters. Shuffling hostility labels within each binary class, while retaining the same objective and training pipeline, reduces probe accuracy by 1.50 points (95% CI [0.44, 2.61]) and test F1 by 0.79 points (p=0.004p=0.004). This shows that the benefit is not simply due to introducing an additional contrastive-loss component.

W1 / Q1: Contribution of the objective under identical training conditions (continued)

Why does the proposed objective provide this benefit?

Standard SupCon defines positive pairs using only the binary hate/non-hate label. Consequently, examples belonging to the same binary class are pulled together, even when their underlying hostility differs. For example, a non-hateful insult, criticism, and neutral statement may all be treated as positives. Similarly, implicit hate and explicit dehumanization can be grouped together.

Our objective instead defines positives using the joint (binary label, hostility) category. Samples with the same binary label but different hostility categories are treated as negatives. This encourages the representation to retain fine-grained hostility distinctions within each binary class, which is particularly useful for separating lexically similar examples with different underlying intent.

We examine this mechanism directly by measuring how well hostility can be decoded within each binary class from the frozen representations:

EncoderWithin non-hateWithin hate
Raw SONAR + Seamless features34.035.4
CE40.4 ± 1.639.4 ± 0.6
Standard SupCon39.1 ± 1.738.3 ± 0.3
Ours42.7 ± 1.340.9 ± 0.5
Ours, without Stage 141.9 ± 1.540.3 ± 0.9
Ours, hostility shuffled34.7 ± 1.532.4 ± 1.1

Our representations improve within-class hostility decodability by 3.1 macro-F1 points over standard SupCon (p<0.001p<0.001) and 1.9 points over CE (p=0.002p=0.002). Shuffling the hostility labels removes this improvement, bringing decodability close to the raw-feature level. The same representation ordering is reflected in the improved linear-probe transfer to ADIMA and X-MuTeST.

Together, these results provide evidence for the intended role of the objective: rather than learning only a binary separation between hateful and non-hateful examples, it preserves additional hostility structure that supports more discriminative representations and improved transfer.

The CF data remains the largest contributor to robustness, improving probe accuracy by 16.59 points over the corresponding no-CF setting (95% CI [12.96, 20.27]). Our objective complements this contribution by providing further improvements under the same CF data and controlled training conditions.

W2 / Q2: How the six hostility labels are obtained and their reliability

We clarify the annotation procedure and provide complementary evidence for the reliability and usefulness of the hostility labels.

Annotation procedure. We use Qwen2.5-7B-Instruct with greedy decoding and a JSON schema to annotate the English source sentences in the training and validation splits. The test set is never annotated, and hostility labels are not used at inference. Each source label is propagated to its corresponding translations and modalities using the shared sample_id.

The prompt extracts six attributes: target_type, hostility, speaker_stance, explicitness, contains_profanity, and confidence. Only hostility enters the composite key used for contrastive learning. The six hostility categories are none, criticism, insult, threat, violence, and dehumanization.

(A) Agreement with independent human-provided ratings. The ToxiGen subset includes crowd-worker ratings of toxicity and intent on a 1–5 scale. These ratings are independent of our annotation pipeline. Mean human-rated toxicity is lowest for none (1.54), intermediate for criticism (2.81), and highest for the four hostile categories (3.30–4.05). Across 7,289 samples, the LLM labels correlate with human-rated toxicity ((\rho=0.676)) and intent ((\rho=0.665)), with (p<10^{-10}) for both. These correlations provide evidence of convergent validity with independent human ratings.

(B) Independent LLM annotation. We use Mistral-Small-24B-Instruct, a larger model from a different family, to annotate 1,500 randomly selected samples with the same prompt. The agreement is Cohen's (\kappa=0.58), with 68.7% raw agreement. When the labels are grouped into three levels (non-hostile, insult, and severe), agreement reaches (\kappa=0.60), with 76.6% raw agreement. Most disagreements occur between adjacent categories, such as none and criticism or insult and dehumanization, rather than between clearly hostile and non-hostile cases.

(C) Cross-lingual annotation audit. We examine whether labels should instead be independently regenerated for each translation. Re-annotation gives moderate agreement with the English source labels for Hindi ((\kappa=0.50)) and Bengali ((\kappa=0.54)), but lower agreement for Tamil ((\kappa=0.20)). In contrast, repeated greedy annotation of English samples gives near-perfect consistency ((\kappa=0.994)). This supports annotating the English source once and propagating its label, rather than relying on a 7B model to assign fine-grained labels independently to low-resource Indic translations. This procedure assumes meaning-preserving translation, which we assess separately using COMET and back-transcription metrics.

(D) Robustness to label noise. Randomly corrupting 25% of the training hostility labels leaves performance essentially unchanged: probe accuracy is (95.06\pm0.98), compared with (94.84\pm0.80) without corruption, while test F1 changes from 79.44 to 79.34. A 50% corruption causes a reduction of approximately 1.2 probe points. In contrast, shuffling hostility labels within each binary class reduces probe accuracy by 1.50 points and test F1 by 0.79 points. Thus, the objective tolerates substantial random annotation noise while benefiting from the semantic structure captured by the hostility labels.

Why hostility is the key attribute. Hostility directly defines the contrast in our CF pairs, which share topics and key terms but differ in hostility. Profanity and explicitness capture surface form rather than intent (NMI with binary hate: 0.16–0.18, compared with 0.17 for hostility). Speaker stance is also unreliable: hateful proportions are similar for condemns (61%) and endorses (67%). Combining all attributes fragments the positive sets, increasing occupied combinations from 14 with hostility alone to 823 with all five attributes; consequently, 12% of anchors in a batch of 1,024 lack an in-batch positive and fall back to the binary objective.

Target type as a complementary refinement. Adding a coarse target category (none/self, individual, or group) alongside hostility improves probe accuracy by 0.91 points (p=0.004p=0.004), including a 1.70-point gain on self-directed examples. It also reduces benign false positives from 8.4% to 6.9%, without changing in-distribution F1 or linear-probe transfer. This shows that target type can complement hostility without replacing it as the central contrastive attribute.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.