- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose BEAST a large scale benchmark for evaluating robustness and continual adaptation of abusive language classifier under linguistic distribution shift. The benchmark comprises of 6.04 million naturally occuring posts , collected from a 3.5 years single online source. The authors order them as by sorting them according to their negative log-likelihood under a 5-gram language model trained on the initial data. That leads to 20 evaluation intervals from relatively familiar to increasingly unfamiliar. The authors utilize LLMs to extract semantic attributes from the posts. These are the mapped to binary abuse labels using a deterministic moderation policy.The benchmark is evaluated on BERT-Large, ModernBERT-Large, and Qwen3.5 models under static training, periodic retraining, and retrieval-augmented adaptation. The experimental results show that increasing model size and periodic retraining improve average classification performance. However, it does not prevent substantial degradation in performance as linguistic novelty increases. The retreival augmented configuration performs worse than its static counterpart, while requiring more computational overhead.
Critical strengths
Strength:
-
This paper addresses an important problem of ensuring reliable abusive-language detection under evolving linguistic distributions.
-
It provides a large scale benchmark comprising 6.04 million naturally occurring posts collected over 3.5 years, providing much larger evaluation setting than existing continual abusive-language detection benchmarks.
-
The paper uses a prequential protocol(test-then-adapt); so every model is tested on each new step before it can train on it. This lets static and adaptive models be compared fairly as the posts become less familiar.
-
Besides classification performance, the authors also report computational costs and backward transfer, providing useful insights into the efficiency, and forgetting behaviour of adaptation methods. The experiments include both encoder and decoder-based language models and compare static training, periodic retraining, and retrieval-augmented adaptation, providing useful baseline results for future research.
Critical weaknesses
-
The benchmark does not actually measure how abusive language changes over time. The paper's own chronological test (Appendix D) shows no drop in performance over the 3.5 years. The authors rather sort posts by how unfamiliar they are to a 5-gram model trained on the initial batch. It does not order them as per timeline; rather it's an out-of-distribution ranking. E.g., an unusual post from 2022, could appear at the end of the benchmark; while a fairly common post from 2024, could appear at the beginning of the benchmark. High NLL does not suggest linguistic evolution or newly emerging abuses; rather they might suggest rare words, unusual syntax, or uncommon spelling variations. This raises concerns regarding whether the degradation is due to chronological linguistic evolution or due to the artificially constructed novelty.
-
The authors use Gemini 3.5 Flash to generate semantic annotations, which are then converted into binary labels using a deterministic policy. However, the authors do not validate these labels against human annotations. An annotation model may also experience difficulty in reliably interpreting linguistically uncommon expressions and coded abusive language. Therefore, annotation errors could also increase with novelty and contribute to the performance degradation.
-
The RAG performs substantially worse than its static counterpart; with mean PR-AUC decreasing from 0.7093 to 0.4141. However, the paper does not sufficiently investigate the reasons for this degradation. The experiments use a single embedding model and a fixed number of retrieved examples. Therefore, it is difficult to conclude whether the poor performance of RAG is an inherent limitation of retrieval-based adaptation or shortcomings of the particular implementation.
-
Although the paper proposes a benchmark for continual learning; the evaluation is limited to static models, periodic retraining, and a single RAG configuration. Established continual-learning approaches, such as experience replay, regularization-based methods, and cumulative retraining, are not evaluated in this paper. Given the paper's emphasis on continual adaptation, the absence of these baselines limits the strength of its experimental conclusions.
-
There are also some issues related to formatting and presentation: a)Table 1 (page 2) extends beyond the normal text margins. b) The mathematical expression defining the abuse policy in Appendix B (page 13) extends beyond text margin. c)Some tables, particularly Table 8 (page 13), are difficult to read due to small font sizes and densely packed information.d) In Figure 7(b) (page 17), the legends refer to Mean Perplexity and Median Perplexity; however, both the y-axis and figure caption refer to negative log-likelihood (NLL). The terminology should be consistent throughout the paper.
Questions for authors
-
Given that the chronological evaluation shows no clear performance drop, how do the authors justify that the NLL-based ordering captures evolving abusive language rather than simply unfamiliar or difficult examples?
-
Can the authors validate the LLM-generated labels against human annotations? Does annotation quality remain consistent as linguistic novelty increases?
-
Why RAG perform significantly worse than the static model? Is this due to the choice of embedding model, retrieval quality or the number of retrieved examples?
-
How would known continual-learning methods (e.g. experience replay, regularization-based methods) perform on BEAST?