- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a multilingual and multimodal hate speech detection framework that combines hostility-conditioned contrastive learning with counterfactual data augmentation. Experiments demonstrate improved classification performance across languages, substantial gains on a counterfactual diagnostic benchmark, and positive zero-shot transfer results on real-world datasets.
Critical strengths
-
The paper addresses an important problem in multilingual hate speech detection by reducing reliance on misleading lexical cues and improving intent-level discrimination.
-
The evaluation covers ten languages, includes targeted counterfactual experiments and ablation studies, and demonstrates generalization to independent real-world datasets.
Critical weaknesses
-
Limited methodological novelty. The proposed framework mainly combines existing contrastive learning and counterfactual augmentation techniques. The ablation results also suggest that most improvements come from counterfactual data rather than the proposed objective.
-
Limited real-world robustness evaluation. Both training counterfactuals and diagnostic samples are synthetically generated around similar confound categories, potentially overestimating robustness. The improvements on real-world datasets are considerably smaller.
-
Unclear multimodal contribution. The improvement over the text-only baseline is relatively small, while most audio training data is synthesized. The effectiveness of multimodal fusion under realistic speech conditions remains insufficiently demonstrated.
-
Limited linguistic diversity. The evaluation primarily focuses on nine Indian languages and English, without considering languages from other linguistic regions. This limits the evidence for the claimed multilingual generalizability of the proposed method.
-
Although the authors acknowledge that some figures were generated using AI, several figures are too small and blurry, and some contain incomplete borders. These figures should be carefully revised to improve their clarity, readability, and overall presentation quality.
Questions for authors
-
How does the proposed hostility-conditioned contrastive objective compare with standard supervised contrastive learning when using the same counterfactual training data?
-
How well does the method generalize to human-written counterfactual examples, given that both training and diagnostic samples are synthetically generated around similar confound categories?
-
How much does the audio modality contribute beyond text alone, particularly when evaluated on real speech with sarcasm, emotional cues, or implicit hostility?
-
How well does the proposed method generalize to languages beyond the Indian linguistic context, such as Arabic, Chinese, or European languages?
4 comments
W1 Novelty: Contribution beyond counterfactual data
We agree that counterfactual (CF) data provides the largest individual improvement in our experiments. However, the contribution of this work is broader than introducing CF augmentation or supervised contrastive learning independently. It lies in combining a multilingual multimodal resource, controlled hostility-focused counterfactual generation, and a representation objective that preserves fine-grained hostility information.
First, we develop a multilingual multimodal resource covering English and nine Indian languages across text, audio, and images. This addresses an important gap in existing hate-speech research, which remains limited in its coverage of Indian languages and modalities. The resource also provides a common basis for studying how hostility is expressed across different languages and forms of communication.
Second, we introduce a controlled counterfactual generation strategy. The generated examples retain shared topics and key terms while varying hostility. This enables controlled comparisons of model behaviour and helps examine whether detectors rely on hostility-related information or superficial lexical and topical cues. The use of synthetic speech further extends this evaluation to audio without requiring identifiable voice recordings of people producing hateful content.
Third, we propose a hostility-conditioned contrastive objective. Standard binary supervised contrastive learning groups all samples with the same hate/non-hate label, potentially overlooking meaningful hostility distinctions within each class. Our objective uses both the binary label and the fine-grained hostility category to organize representations. This encourages the model to distinguish, for example, neutral criticism from insults and different forms of explicit hostility, even when they share the same binary label.
The controlled comparison in Table R1 (in Q1) further establishes the contribution of our learning objective. With the same CF data and matched experimental settings, composite SupCon improves ADIMA and X-MuTeST linear-probe F1 over standard SupCon by 1.07 and 2.03 points, respectively. Adding decorrelation further improves these results to 74.70 and 78.86, corresponding to gains of 1.27 and 2.58 points over standard SupCon. The proposed objective also achieves the lowest benign-anchor FPR (8.44%) among the compared objectives. Moreover, shuffling hostility labels within binary classes reduces probe accuracy from 94.84% to 93.34%, showing that the hostility information contributes to the learned representations. These results demonstrate that the proposed learning objective adds value beyond the CF data itself.
We do not attribute the overall gains to the contrastive objective alone. Removing CF data reduces probe accuracy by 16.59 points and zero-shot X-MuTeST F1 by 3.33 points (). Rather, the results show that CF data provides substantial training benefit, while the proposed objective contributes additional improvements by preserving fine-grained hostility structure.
The broader contribution is therefore the integration of data, generation, and learning: a multilingual multimodal resource, a controlled way to construct hostility-focused examples, and a representation objective designed to preserve distinctions that binary supervision can overlook. Together, these components support more systematic and inclusive hate-speech research, particularly for under-resourced Indian languages and modalities, while providing a basis for developing and evaluating more reliable detection systems.
Q2 / W2: Generalisation to human-written examples
We evaluate zero-shot on HateCheck (Röttger et al., 2021) and Multilingual HateCheck (Röttger et al., 2022), which contain expert-written examples, including counter-speech, negation, neutral and positive identity mentions, abuse against non-protected targets, and reclaimed slurs. These cases provide a complementary evaluation to our targeted synthetic probe.
The results show that all systems trained on the original corpus, including the Ranjan et al. baseline, over-flag some templated sentences that explicitly mention protected groups. Within our controlled family, CF data and hostility conditioning provide the strongest results on neutral and positive identity mentions. However, absolute false-positive rates remain.
We further examine whether the same CF-generation framework can address this specific confound. We add protected-identity mentions and supportive/defending statements through the same generation, translation, and filtering pipeline. No HateCheck data is used. With this extension:
- HateCheck-EN F1 improves from 59.1 to 62.2 .
- Positive identity accuracy improves from 19.5 to 30.2.
- The result exceeds the Ranjan et al. baseline on HateCheck-EN and on the aggregate Multilingual HateCheck score.
- On the same extended training data, our objective improves neutral identity accuracy by 6.0 points over CE and 6.3 points over standard SupCon.
The extension also improves the original probe and retains in-distribution F1. We report the remaining identity-term over-flagging and the lack of improvement on counter-speech that quotes hateful language in order to reject it.
The external benchmarks provide additional evidence beyond the synthetic probe. X-MuTeST contains human-written posts, while ADIMA provides real speech. CF data improves zero-shot X-MuTeST F1 by 3.33 points , and the hostility-conditioned objective improves linear-probe transfer over both CE and standard SupCon.
We also clarify the composition of the English source data: 75% comes from MetaHate and consists of real, human-written social-media posts with human labels. The synthetic components are primarily the translations and speech renderings, rather than the original content.
References
- P Röttger, B Vidgen, D Nguyen, Z Talat, H Margetts, and J Pierrehumbert. 2021. [HateCheck: Functional Tests for Hate Speech Detection Models]. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 41–58.
- P Röttger, H Seelawi, D Nozza, Z Talat, and B Vidgen. 2022. [Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models]. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 154–169.
Q3 / W3: Contribution of audio, especially on real speech
We evaluate text-only, audio-only, and fused variants of the same pipeline using the same data and six-seed protocol.
| Model | Test F1 | Probe acc | Probe-FPR | ADIMA ZS (real speech) | ADIMA LP | X-MuTeST LP |
|---|---|---|---|---|---|---|
| Text-only | 79.32±0.10 | 91.24±0.43 | 16.3 | 56.57±1.10 | 70.77±0.85 | 78.30±0.45 |
| Audio-only | 61.44±0.33 | 94.50±0.35 | 5.6 | 67.13±1.63 | 73.94±0.77 | 73.69±0.30 |
| Text + audio (ours) | 79.44±0.12 | 94.84±0.80 | 8.4 | 56.81±1.47 | 74.70±0.99 | 78.86±0.57 |
The results highlight complementary contributions from the two modalities.
-
Audio adds robustness beyond text. The fused model improves probe accuracy by 3.60 points over text-only and reduces benign-anchor FPR from 16.3% to 8.4%. The probe audio uses the same neutral prosody description for every item, so this improvement comes from the speech representation rather than label-dependent emotional or prosodic cues.
-
Audio is particularly informative on real speech. On ADIMA, the audio-only model achieves 67.13 zero-shot F1, compared with 56.57 for text-only. With in-domain linear-probe evaluation, fusion performs best, improving over text-only by 3.93 points .
-
Zero-shot fusion does not fully exploit audio on ADIMA. We investigate whether the fused model over-relies on text by retraining Stage 3 with modality dropout . This leaves ADIMA zero-shot performance essentially unchanged (56.67 ± 1.82), as well as the other metrics. The results suggest a domain shift between synthetic speech/ASR and real speech, rather than a simple modality-selection issue. A linear probe on the real-speech domain improves the fused result.
Q4 / W4: Languages beyond India
We evaluate zero-shot transfer on Multilingual HateCheck (Röttger et al., 2022) in Arabic (AR), Mandarin (ZH), German (DE), French (FR), Spanish (ES), Italian (IT), Dutch (NL), Polish (PL), and Portuguese (PT), in addition to romanised, code-mixed Hindi (HI-Roman). The nine non-Indic languages have no training data in our framework.
| Model | AR | ZH | DE | FR | ES | IT | NL | PL | PT | HI-Roman |
|---|---|---|---|---|---|---|---|---|---|---|
| Ranjan et al. | 54.0 | 57.6 | 58.6 | 57.8 | 57.2 | 57.0 | 59.7 | 58.9 | 57.7 | 54.2 |
| Ours (original CF) | 55.6 | 57.3 | 55.2 | 57.2 | 56.3 | 58.0 | 58.2 | 56.1 | 56.7 | 51.7 |
| Ours + identity CF | 57.3 | 60.0 | 57.1 | 59.1 | 58.0 | 60.1 | 60.2 | 57.4 | 58.9 | 51.3 |
With the protected-identity CF extension, our method exceeds the baseline on seven of the nine non-Indic languages, without training data in those languages.
Threshold-free ranking quality is also informative: AUROC is approximately 0.78–0.80 on the seven European languages, 0.75 on Mandarin, and 0.68 on Arabic. Thresholded F1 remains affected by identity-term over-flagging, as discussed above.
We scope the claim accordingly: these results demonstrate zero-shot transfer on the evaluated languages, rather than universal multilingual generalization. The framework uses language-independent representations and translation/speech components that support a broad language range, but demonstrating reliable performance in additional languages requires appropriate training and evaluation data.
W5: Figures
We will improve the figures with complete borders and legible fonts.
We thank the reviewer for the thoughtful feedback.
Q1. Comparison with standard SupCon on the same CF data
We compare standard SupCon and our objective using the same CF data, encoder architecture, optimization settings, Stage-3 evaluation protocol, and six shared seeds. Table R1 reports the complete controlled comparison, including CE, standard SupCon, composite SupCon, the addition of decorrelation, and the hostility-shuffle control.
Table R1. Controlled objective comparison (mean ± SD over six seeds). Probe-FPR is the percentage of benign probe anchors predicted as hateful. Pair-acc measures the percentage of (anchor, CF) pairs for which both members are classified correctly. ZS denotes zero-shot evaluation and LP denotes linear-probe evaluation.
The results show that our method achieves comparable in-distribution test F1 to standard SupCon (79.44 vs. 79.46), while improving probe accuracy from 94.07 to 94.84 and reducing benign-anchor FPR from 10.33% to 8.44%. More importantly, it improves linear-probe transfer by 2.58 points on X-MuTeST (p<0.001) and 1.27 points on ADIMA (p=0.033) over standard SupCon. Adding decorrelation to composite SupCon provides further improvements on both benchmarks.
We also report the zero-shot results: standard SupCon performs approximately one point better on X-MuTeST (66.60 vs. 65.56; p=0.046). We retain this result for a balanced comparison.
Finally, shuffling hostility labels within the binary hate/non-hate classes reduces probe accuracy from 94.84 to 93.34 and test F1 from 79.44 to 78.65. This supports the contribution of the hostility information to the learned representations.