- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether large language models exhibit different bias-detection capabilities across formal institutional language and informal social-media language.
Critical strengths
The paper correctly highlights that model performance on informal harmful-language detection does not necessarily transfer to formal legal, institutional, and policy documents. This is a meaningful concern because mistakes in such contexts can have substantial consequences.
Critical weaknesses
-
Source–register confounding limits the main conclusion. the primary benchmark does not isolate linguistic register from several other variables. As described in Appendix B, all informal examples come from Social Bias Frames, whereas the formal examples come from six different institutional corpora. These partitions differ systematically in source, era, topic, sentence construction, and label distribution. For example, Section 4.4 reports that Social Bias Frames contains 87.3% harmful examples, while GovReport contains 92.6% harmless examples and the Comparative Agendas Project is entirely antibias. Consequently, the main finding that harmful recall is 45.1 percentage points higher on informal text may reflect differences in the linguistic expression, severity, and subject matter of harmful content, rather than register alone. The authors explicitly acknowledge this limitation and introduce matched-pair controls. This is a significant strength. However, the pooled harmful effect decreases from approximately 0.45 in the natural benchmark to 0.136 in the matched-pair analysis. This attenuation suggests that a substantial portion of the observed disparity is associated with differences beyond the controlled register manipulation.
-
The GRDC framework consists of Grouped Recall Divergence (GRD), Directional Consistency (DC), Register Polarity Flip (RPF), Confusion Drift (CD), and Capability-Conditional Drift (CCD). Although these metrics form a useful diagnostic organization, their methodological novelty is limited. GRD is an average absolute recall difference across groups, RPF checks whether signed class-wise recall differences have opposing signs, CD compares conditional confusion probabilities, and CCD applies Spearman correlation to overall performance and disparity. These are useful descriptive measures, but they largely reorganize existing evaluation statistics rather than introduce fundamentally new methods. More seriously, Equation (5), seems mathematically incorrect.
Questions for authors
-
Equation (5) appears to divide the absolute mean recall shift by itself, making DC equal to 1 whenever defined. Is this a typographical error? What formula was implemented to obtain the reported DC results?
-
Appendix Q shows that Mistral's harmful PRE reverses direction and several individual confidence intervals include zero. How should this be reconciled with the claim of a consistent Register Polarity Flip across all eight models?
2 comments
2. GRDC Framework
The main contribution of the GRDC framework is the framework’s formulation itself and the application to register-aware bias detection. We do not claim that each GRDC component is a fundamentally new statistical method. As mentioned in Section 3 of the paper, the GRDC framework is intended to be a standardized diagnostic organization of class conditional performance, rather than five entirely independent statistical inventions.
We argue that the contribution of the GRDC is to to bring together the five components and use them to characterize a failure pattern that aggregate performance can obscure. This organization matters because aggregate performance can hide deployment-critical errors. Mean macro-F1 differs by only 0.024 between formal and informal text, yet harmful recall changes from 0.85 on informal text to 0.40 on formal text, and harmless recall moves from 0.52 on informal text to 0.82 on formal text. The small aggregate difference therefore conceals two large and operationally distinct failures.
| Diagnostic | What it reveals | Actionable consequence |
|---|---|---|
| GRD | Magnitude of the register-specific recall gap | Determines whether aggregate evaluation is adequate or register-stratified testing is required |
| DC | Whether the direction is systematic across models | Distinguishes a recurring benchmark-wide failure from an isolated model error |
| RPF | Whether class errors move in opposite directions | Reveals when aggregate scores cancel missed harms against false alarms |
| CD | Which confusion cells cause the divergence | Identifies whether mitigation should target false negatives, false positives, or protective-language recognition |
| CCD | Whether divergence changes with model capability | Tests whether selecting a stronger model is likely to resolve the problem; conclusions are tentative with eight models |
3. DC Equation
The intended DC definition in the paper is the absolute mean signed recall shift divided by the mean of the per-model absolute recall shifts:
The correct implementation that we use computes the absolute mean signed RRS divided by the mean absolute RRS. We thank the reviewer for pointing out the rendering error made in the DC equation. In the revised manuscript, we have correctly displayed this equation with explicit summation parentheses and define the denominator appropriately. Most importantly, the reported DC values, including the antibias value of 0.159, reflect this correctly implemented formula.
Questions
Q1.
This response about the formula for the DC equation is already addressed under Weakness 3 above. In the now revised version of the manuscript, we have made sure that the equation typography is unambiguous.
Q2.
The natural benchmark results and matched-pair experiment results refer to two distinct analyses. The directional consistency claim about the eight models is made on the basis of polarity flip results from the full BRIDGE benchmark experiments. On the natural BRIDGE data, the harmful and harmless recall directions are consistent across all of the eight evaluated models, and this yields DC=1.0 for those two natural-split classes. On the other hand, in the matched rewrite experiment that was conducted, we are considering a different measure. As reported in Appendix Q in the paper, the harmful PRE value is positive for seven models and negative for Mistral (−0.100), and the harmless PRE value is negative for all eight models. Regardless, the matched-pair PRE remains +0.136 for harmful items and −0.094 for harmless items. We will make this distinction explicit when discussing the effects and results of matched-pair experiments specifically, which are different from the 8/8 directional consistency observed on the full benchmark.
We thank the reviewer again for raising these important concerns. We believe BRIDGE’s main value lies in providing a large and openly available benchmark that exposes a deployment-relevant failure mode that is usually overlooked by informal bias evaluations. In the revised version of the paper we have made sure to further clarify the points and equations stated above, and discuss all the limitations transparently.
We thank the reviewer for the careful assessment and for identifying issues in the paper with causal interpretation and the presentation of the GRDC metric. We address source confounding, the scope of GRDC, the DC equation, and the matched-pair results below.
Weaknesses
1. Source and register confounding
We thank the reviewer for pointing out the source and register confounding in the natural benchmark. The available natural data unfortunately does not permit a complete topic and source controlled estimate for us to analyze since register is genuinely confounded with the source and era. However, as described in the paper, our main findings capture performance across the source mixtures represented in BRIDGE.
The matched-pair analysis mentioned in Section 6.2 of the paper addresses a narrower question that is focused on just the register gap by comparing formal and informal versions of the same seed. We argue that the findings for the pooled harm from approximately +0.45 (for the full BRIDGE dataset) to +0.136 (for the matched experiment) is actually informative. In our findings, source and content composition do contribute to the larger gap shown in the natural split of the data. However, the matched harmful PRE still remains positive and the harmless PRE still remains negative, with both reported confidence intervals excluding zero. These findings provide direct evidence that register contributes to the polarity flip after semantic content and source-related attributes are held fixed. Ultimately, our results show that BRIDGE supports both a mixed performance finding with sources and a register only associated contribution to the direction shifts.
Furthermore, we conducted an additional analysis experiment to clarify the scope of the results and support the main findings. In terms of the distributions of the sources themselves, this analysis acts as a source-sensitivity check to ensure that our observed results are not substantially confounded. The natural harmful and harmless directions found in the results still remain when each formal source is removed in turn. The harmful informal-formal gap ranges from [+0.188 to +0.552] across these analyses where one formal source is left out each turn. Similarly, the harmless gap remains negative, from [−0.290 to −0.319]. These results show that no single formal source drives the pooled natural result in our main findings and that the findings of the paper are significant for the imbalanced natural distributions of the data.
We argue that the above analysis and the matched pair results still provide strong evidence that register contributes to the main findings of the paper concerning the direction shift. We do acknowledge the strong causal language used in the paper, and in the now revised version of the manuscript, we have made sure to be more precise and moderate any strong language. We will make sure to add further analysis and the existing matched-pair results to the main paper and clarify that BRIDGE supports a register-associated contribution.