acceptodds
Under review as a conference paper at ICLR 2027

BRIDGE: A Bias Register-Integrated Dataset for Generalized Evaluation of LLM Bias Detection

Abstract

Large Language Models(LLMs) are increasingly being used to identify discriminatory or harmful language in institutional settings, where text is often formal, legalistic, and policy-oriented. However, existing bias evaluation frameworks for LLMs largely rely on informal, synthetic, crowd-sourced or social-media-style text (BBQ, StereoSet, CrowS-Pairs, ToxiGen, Social Bias Frames), creating a structural mismatch between benchmark registers and deployment registers. Across formal and informal registers, we observe a systematic failure mode, which we call the Register Polarity Flip (RPF): eight frontier LLMs under-detect harmful bias in formal text while over-flagging neutral informal text as harmful, with consistent directionality across models. To study this failure mode, we introduce BRIDGE (Bias Register-Integrated Dataset for Generalized Evaluation), a 19,420-sentence benchmark spanning formal institutional text and informal social media text under a unified three-way taxonomy of harmful, harmless, and antibias content assembled from seven open-source corpora spanning 150 years of discourse. We also introduce GRDC, a five-metric diagnostic suite for group-conditional bias evaluation, capturing recall divergence, directional consistency, polarity flips, confusion drift, and capability-conditional drift. The RPF pattern replicates on an independent external dataset (BABE + Measuring Hate Speech) and is not meaningfully mitigated by few-shot prompting. Because source, era, topic, and genre are correlated in the BRIDGE benchmark, we further conduct a bidirectional matched-pair control (Paired Register Effect: PRE) which shows that the same register polarity direction persists. These findings show that current bias benchmarks can misrepresent model reliability for formal-text bias detection and motivate register-aware evaluation protocols that report stratified performance across deployment-relevant registers before LLMs are used in institutional auditing workflows.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Bias Detection in LLMs, Linguistic Register, Formal vs Informal Text, Bias Benchmarking, Model Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tmnYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether large language models exhibit different bias-detection capabilities across formal institutional language and informal social-media language.

Critical strengths

The paper correctly highlights that model performance on informal harmful-language detection does not necessarily transfer to formal legal, institutional, and policy documents. This is a meaningful concern because mistakes in such contexts can have substantial consequences.

Critical weaknesses
  1. Source–register confounding limits the main conclusion. the primary benchmark does not isolate linguistic register from several other variables. As described in Appendix B, all informal examples come from Social Bias Frames, whereas the formal examples come from six different institutional corpora. These partitions differ systematically in source, era, topic, sentence construction, and label distribution. For example, Section 4.4 reports that Social Bias Frames contains 87.3% harmful examples, while GovReport contains 92.6% harmless examples and the Comparative Agendas Project is entirely antibias. Consequently, the main finding that harmful recall is 45.1 percentage points higher on informal text may reflect differences in the linguistic expression, severity, and subject matter of harmful content, rather than register alone. The authors explicitly acknowledge this limitation and introduce matched-pair controls. This is a significant strength. However, the pooled harmful effect decreases from approximately 0.45 in the natural benchmark to 0.136 in the matched-pair analysis. This attenuation suggests that a substantial portion of the observed disparity is associated with differences beyond the controlled register manipulation.

  2. The GRDC framework consists of Grouped Recall Divergence (GRD), Directional Consistency (DC), Register Polarity Flip (RPF), Confusion Drift (CD), and Capability-Conditional Drift (CCD). Although these metrics form a useful diagnostic organization, their methodological novelty is limited. GRD is an average absolute recall difference across groups, RPF checks whether signed class-wise recall differences have opposing signs, CD compares conditional confusion probabilities, and CCD applies Spearman correlation to overall performance and disparity. These are useful descriptive measures, but they largely reorganize existing evaluation statistics rather than introduce fundamentally new methods. More seriously, Equation (5), seems mathematically incorrect.

Questions for authors
  1. Equation (5) appears to divide the absolute mean recall shift by itself, making DC equal to 1 whenever defined. Is this a typographical error? What formula was implemented to obtain the reported DC results?

  2. Appendix Q shows that Mistral's harmful PRE reverses direction and several individual confidence intervals include zero. How should this be reconciled with the claim of a consistent Register Polarity Flip across all eight models?

2 comments

We thank the reviewer for the careful assessment and for identifying issues in the paper with causal interpretation and the presentation of the GRDC metric. We address source confounding, the scope of GRDC, the DC equation, and the matched-pair results below.

Weaknesses

1. Source and register confounding

We thank the reviewer for pointing out the source and register confounding in the natural benchmark. The available natural data unfortunately does not permit a complete topic and source controlled estimate for us to analyze since register is genuinely confounded with the source and era. However, as described in the paper, our main findings capture performance across the source mixtures represented in BRIDGE.

The matched-pair analysis mentioned in Section 6.2 of the paper addresses a narrower question that is focused on just the register gap by comparing formal and informal versions of the same seed. We argue that the findings for the pooled harm from approximately +0.45 (for the full BRIDGE dataset) to +0.136 (for the matched experiment) is actually informative. In our findings, source and content composition do contribute to the larger gap shown in the natural split of the data. However, the matched harmful PRE still remains positive and the harmless PRE still remains negative, with both reported confidence intervals excluding zero. These findings provide direct evidence that register contributes to the polarity flip after semantic content and source-related attributes are held fixed. Ultimately, our results show that BRIDGE supports both a mixed performance finding with sources and a register only associated contribution to the direction shifts.

Furthermore, we conducted an additional analysis experiment to clarify the scope of the results and support the main findings. In terms of the distributions of the sources themselves, this analysis acts as a source-sensitivity check to ensure that our observed results are not substantially confounded. The natural harmful and harmless directions found in the results still remain when each formal source is removed in turn. The harmful informal-formal gap ranges from [+0.188 to +0.552] across these analyses where one formal source is left out each turn. Similarly, the harmless gap remains negative, from [−0.290 to −0.319]. These results show that no single formal source drives the pooled natural result in our main findings and that the findings of the paper are significant for the imbalanced natural distributions of the data.

Formal source excludedHarmful gapHarmless gap
Comparative Agendas Project+0.441−0.304
GovReport+0.442−0.319
LSC Eviction Laws+0.441−0.300
Mapping Inequality+0.499−0.312
On the Books: Jim Crow Laws+0.552−0.290
Pile of Law+0.188−0.299

We argue that the above analysis and the matched pair results still provide strong evidence that register contributes to the main findings of the paper concerning the direction shift. We do acknowledge the strong causal language used in the paper, and in the now revised version of the manuscript, we have made sure to be more precise and moderate any strong language. We will make sure to add further analysis and the existing matched-pair results to the main paper and clarify that BRIDGE supports a register-associated contribution.

2. GRDC Framework

The main contribution of the GRDC framework is the framework’s formulation itself and the application to register-aware bias detection. We do not claim that each GRDC component is a fundamentally new statistical method. As mentioned in Section 3 of the paper, the GRDC framework is intended to be a standardized diagnostic organization of class conditional performance, rather than five entirely independent statistical inventions.

We argue that the contribution of the GRDC is to to bring together the five components and use them to characterize a failure pattern that aggregate performance can obscure. This organization matters because aggregate performance can hide deployment-critical errors. Mean macro-F1 differs by only 0.024 between formal and informal text, yet harmful recall changes from 0.85 on informal text to 0.40 on formal text, and harmless recall moves from 0.52 on informal text to 0.82 on formal text. The small aggregate difference therefore conceals two large and operationally distinct failures.

DiagnosticWhat it revealsActionable consequence
GRDMagnitude of the register-specific recall gapDetermines whether aggregate evaluation is adequate or register-stratified testing is required
DCWhether the direction is systematic across modelsDistinguishes a recurring benchmark-wide failure from an isolated model error
RPFWhether class errors move in opposite directionsReveals when aggregate scores cancel missed harms against false alarms
CDWhich confusion cells cause the divergenceIdentifies whether mitigation should target false negatives, false positives, or protective-language recognition
CCDWhether divergence changes with model capabilityTests whether selecting a stronger model is likely to resolve the problem; conclusions are tentative with eight models

3. DC Equation

The intended DC definition in the paper is the absolute mean signed recall shift divided by the mean of the per-model absolute recall shifts:

DC(c)=∣1∣M∣∑m∈MRRS(m,c)∣1∣M∣∑m∈M∣RRS(m,c)∣.\mathrm{DC}(c) = \frac{\left|\frac{1}{|M|}\sum_{m \in M}\mathrm{RRS}(m,c)\right|}{\frac{1}{|M|}\sum_{m \in M}|\mathrm{RRS}(m,c)|}.

The correct implementation that we use computes the absolute mean signed RRS divided by the mean absolute RRS. We thank the reviewer for pointing out the rendering error made in the DC equation. In the revised manuscript, we have correctly displayed this equation with explicit summation parentheses and define the denominator appropriately. Most importantly, the reported DC values, including the antibias value of 0.159, reflect this correctly implemented formula.

Questions

Q1.

This response about the formula for the DC equation is already addressed under Weakness 3 above. In the now revised version of the manuscript, we have made sure that the equation typography is unambiguous.

Q2.

The natural benchmark results and matched-pair experiment results refer to two distinct analyses. The directional consistency claim about the eight models is made on the basis of polarity flip results from the full BRIDGE benchmark experiments. On the natural BRIDGE data, the harmful and harmless recall directions are consistent across all of the eight evaluated models, and this yields DC=1.0 for those two natural-split classes. On the other hand, in the matched rewrite experiment that was conducted, we are considering a different measure. As reported in Appendix Q in the paper, the harmful PRE value is positive for seven models and negative for Mistral (−0.100), and the harmless PRE value is negative for all eight models. Regardless, the matched-pair PRE remains +0.136 for harmful items and −0.094 for harmless items. We will make this distinction explicit when discussing the effects and results of matched-pair experiments specifically, which are different from the 8/8 directional consistency observed on the full benchmark.

We thank the reviewer again for raising these important concerns. We believe BRIDGE’s main value lies in providing a large and openly available benchmark that exposes a deployment-relevant failure mode that is usually overlooked by informal bias evaluations. In the revised version of the paper we have made sure to further clarify the points and equations stated above, and discuss all the limitations transparently.

Reviewer_3J5VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The submission addresses LLM bias detection in formal and informal texts. It starts by noting that many widely used benchmarks target informal texts, which might bias LLM performance. The main contributions are BRIDGE, a dataset with annotated samples of both text types, and GRDC, a multi-metric diagnostic tool. The evaluation includes eight LLMs, three recent frontier models and five smaller open-weight models. The zero- and few-shot results, presented densely and mostly narratively, indicate that important differences between formal and informal texts appear regardless of the model used.

Critical strengths

The submission addresses bias detection in formal versus informal texts, starting from the hypothesis that existing research concentrates on the latter type of text. This is an interesting angle that should be explored. The BRIDGE dataset builds on topically and temporally diversified sources for its formal subset. The annotation protocol is generally sound, and including the antibias class is a good modeling choice. The proposal of multiple metrics that measure complementary bias facets is appreciated. The evaluation shows that LLMs treat bias differently in informal and formal texts, which matters for practical deployment.

Critical weaknesses

Claim and positioning: The title overclaims the contribution. While the proposed bias detection pipeline is interesting, the authors focus on a specific bias facet (formal texts in a single country), which hardly qualifies as “generalized evaluation of LLM bias detection”. The submission ignores the line of work investigating legal NLP, which has produced datasets including formal texts and studied LLM biases for them. Among others, the authors should cite https://www.dbateyko.info/media/hidden-in-plain-text.pdf, https://aclanthology.org/2024.naacl-long.472, https://aclanthology.org/2025.acl-long.341.pdf, https://aclanthology.org/2025.acl-long.1188.pdf. The central claim in the introduction, i.e., that existing public datasets lack formal texts, is weakened by this line of work, and the authors should position their contribution more carefully. Equally, the authors should analyze their proposed metrics in light of those already used in legal NLP literature, and ground the metrics presented in Section 3 in existing fairness literature. For instance, while GRDC indeed extends equalized odds to a multi-class setting, literature on this very topic already exists and should be cited, including https://proceedings.mlr.press/v267/sabato25a.html and https://proceedings.mlr.press/v306/zhang26cj.html.

Evaluation: The evaluation has several important shortcomings. First, the choice of the tested models should be better motivated. While the closed-source ones are indeed recent, the others are outdated and hard to qualify as “frontier.” Second, LLM performance is known to depend heavily on the prompt formulation, and the authors should have studied result sensitivity by optimizing zero- and few-shot prompts for the task. Third, Section 6 is very dense. It reads more like an enumeration of numeric results in a technical report than an analysis worthy of a top-tier AI conference paper. Fourth, while the evaluation shows differences between formal and informal registers and the discussion briefly touches on the causes, it should go a step further and implement at least one of the mitigation strategies listed in Section H to assess how well LLMs would perform in practice. Such a test would mimic a realistic deployment of LLMs for the task and, if results were better, would provide good practice recommendations to organizations that handle such texts.

Questions for authors

-Q1: Why should the proposed contribution be considered as a generalized evaluation of LLM biases, and how is it positioned with respect to existing legal NLP works and benchmarks?

-Q2: Why choose multiple sources for formal texts included in the dataset but only one for informal ones? Also, since so many informal benchmarks are available, why not balance the two parts of the benchmark?

-Q3: Why generate samples with a small and outdated LLM, given that generation-related literature shows that producing high-quality synthetic samples remains an open issue (https://www.pnas.org/doi/10.1073/pnas.2531748123)? This matters because low-plausibility generated content might confound the evaluation.

-Q4: How would results change if the models were adapted as suggested in Annex H?

3 comments

We would like to thank you for the thoughtful and constructive review. We appreciate the comments recognizing the importance of evaluating bias detection on institutional text. We address the concerns below regarding the scope and positioning of BRIDGE, model selection, prompt sensitivity, presentation, and mitigation.

Weaknesses

1. Scope and Related Work

We agree that the term "generalized evaluation" in our title may suggest broader empirical coverage than BRIDGE currently provides. We do not intend to claim that the seven sources we used provide representative or “sufficient” coverage of every domain in which formal discriminatory language may occur. It would be more precise to say that BRIDGE provides sufficient coverage to demonstrate and study a register-associated evaluation failure within this institutional scope, not to establish universal performance across all formal professional domains. We agree with the reviewer that the manuscript should better motivate the source-selection criteria and have now included that information and changed any broad “generalized” language in the revised version of the manuscript.

Our central claim is not that existing datasets never include formal text, but that the widely used LLM bias-evaluation benchmarks considered in our study do not systematically compare formal and informal registers under a unified bias-detection taxonomy. We believe BRIDGE's contribution is supported by the systematic nature of the failure it identifies, rather than by a claim that formal-text bias detection has never been studied.

We thank the reviewer for identifying the relevant legal NLP literature and agree that these studies should be incorporated into our related work. In particular, Wang et al. (2025) and Spangher et al. (2024) provide important discussions about the context-dependent judgment of fairness and the importance of formal legal-text understanding. These studies provide important foundations for our work, and we will explicitly distinguish their tasks, evaluation dimensions, and contributions from ours. We maintain that BRIDGE offers a distinct benchmark and empirical contribution within its stated scope, and we agree to better communicate the boundaries of that contribution.

2. GRDC Framework

The GRDC framework is intended to be a standardized diagnostic organization of class conditional performance, rather than five entirely independent statistical inventions. We argue that the contribution of the GRDC is to bring together the five components and use them to characterize a failure pattern that aggregate performance can obscure. This organization matters because aggregate performance can hide deployment-critical errors. Mean macro-F1 differs by only 0.024 between formal and informal text, yet harmful recall changes from 0.85 on informal text to 0.40 on formal text, and harmless recall moves from 0.52 on informal text to 0.82 on formal text. The small aggregate difference therefore conceals two large and operationally distinct failures.

We agree that the foundation for the GRDC framework should be more explicitly grounded in existing fairness literature and we should mention prior work on multi-class equalized odds and group-conditional error distributions. We thank the reviewer for highlighting Sabato et al. (2025) and Zhang et al. (2026), and will incorporate both studies into Section 3.

We view GRDC's contribution as a diagnostic organization and application, and not as a new theoretical foundation for multi-class fairness. Unlike previous work that mostly studies fairness across protected populations, we apply group-conditional diagnostics to linguistic registers as a distributional attribute. In the revised version of the manuscript, we now have clarified this distinction and added all the relevant theoretical references mentioned above.

3. Evaluation: Model selection and prompt sensitivity

We agree that describing all eight evaluated models as "frontier" is an imprecise claim to make. In the evaluations, we intentionally included three recent closed-weight models alongside five open-weight models that spanned different model families and capacities. This allows us to examine whether the identified behavior is limited to specific proprietary systems or also appears in established, accessible models. In the revised version of the manuscript, we have now detailed this selection criteria and replaced "eight frontier LLMs" with "eight evaluated LLMs".

In terms of the prompt sensitivity, we further agree that prompt formulation can influence the LLM classification performance. We conducted an additional development-set prompt-selection analysis using Mistral 7B v0.3.

The experiments compared seven candidate configurations with four zero-shot prompts and three three-shot conditions. We selected the prompt on development data by minimizing mean absolute PRE among candidates whose macro-F1 was no more than 0.02 below the original prompt, and we evaluated this prompt on held-out data. The selected formal three-shot prompt improved natural-test macro-F1 from 0.4773 to 0.5730 (+0.0958; 95% CI [+0.0775, +0.1135]), with accuracy increasing from 0.5396 to 0.5900. However, these improvements did not correspond to reduced disparity in terms of the register. On the matched-pair evaluation, the mean absolute PRE value increased only from 0.0938 to 0.125 from the original prompt to the selected prompt.

Thus, we show that even optimizing the prompts can substantially improve aggregate classification performance, it does not necessarily improve consistency across registers. We agree that a broader prompt-sensitivity analysis across models would strengthen this conclusion. We have now included the pilot experiment above with full analysis, results, and prompt selection methodology in the revised version of the manuscript

4. Evaluation: Practical mitigation and deployment

We would like to clarify that BRIDGE is primarily an evaluation framework that we designed to identify reliability gaps, rather than a training method intended to resolve them. Therefore, Appendix H previously only included a discussion of the proposed methods. However, we have now revised the manuscript to include experimental details and results from both pilot experiments. We believe that the ability to expose these failures remains valuable independently of whether a particular adaptation approach successfully eliminates them. Nevertheless, we agree that evaluating mitigation strategies would strengthen the paper's practical relevance.

We conducted an additional QLoRA adaptation pilot on a Mistral 7B v0.3 model, using the register-matched training examples and held-out validation and test data.

On the held-out natural test set, accuracy increased from 0.5429 to 0.5698, while macro-F1 changed from 0.4770 to 0.4725. The fine-tuned adapter's balanced-validation macro-F1 was 0.4270, compared with 0.4441 for the base model. More importantly, the results showed that fine-tuning did not reduce the register disparity that we initially observed on the held out matched-pair evaluation. The harmful PRE increased from 0.000 to +0.3125 (95% CI [+0.125, +0.5625]), while harmless PRE changed from −0.125 to −0.1875.

We acknowledge that these mitigation results are limited to one model and a relatively small matched-pair evaluation. However, they still provide support to demonstrate why aggregate accuracy alone is insufficient for assessing mitigation effectiveness. In this pilot, adaptation improved accuracy modestly but did not improve cross-register consistency. We have now included these findings as a preliminary adaptation analysis in the revised version of the manuscript. We argue this additional experiment actually reinforces the importance of BRIDGE's evaluation framework because register-sensitive diagnostics remain necessary to determine whether the underlying disparity has actually improved even for fine-tuned models.

Questions

Q1.

This response about the positioning of BRIDGE as a benchmark and its scope is already addressed under Weakness 1 above.

Q2.

The sources were selected for our benchmark and register analysis because they are openly licensed, have easily extractable text, are relevant to institutional discrimination or neutral policy contrast, and contain harmful, harmless, and protective language across both historical and contemporary settings. We found for the purposes of creating this benchmark that the above combination provides substantial variation in period, source type, linguistic formality, and class composition within the stated scope. Expanding this coverage especially for the formal sources was a central motivation for developing BRIDGE, given the existing availability of informal bias detection benchmarks.

We chose Social Bias Frames as it provided a large informal corpus that could be mapped to the shared three-way taxonomy and used for comparison purposes. We acknowledge that relying on one just informal source does limit our source diversity and can introduce confounding. However, the observed polarity direction flip is not confined to just the Social Bias Frames data. The independent analyses conducted on the BABE + Measuring Hate Speech evaluation and register-stratified ToxiGen analysis provide supporting evidence for the findings we have.

For the dataset, we prioritized preserving the source-derived examples and their original composition rather than enforcing equal register counts. We focused on balance and augmentation only for the three way bias taxonomy. Since the principal recall comparisons that we conduct are class-conditional, unequal class frequencies do not mechanically produce the observed recall gaps. However, we do acknowledge that more balanced and source diverse evaluations would provide additional useful information. We now further clarify this design rationale in the revised version of the manuscript and have acknowledged further in the limitations that broader informal-source coverage would strengthen the external validity.

Q3.

The overall distribution in the BRIDGE benchmark remains primarily source-derived. Counterfactual generation was used limitedly and only to increase support for the antibias diagnostic class. The naturally occurring examples were too sparse compared to other classes, so augmentation was primarily done with the goal of somewhat reducing this class imbalance. We agree that synthetic-text plausibility is an important concern even after validation. However, counterfactual generation was not intended to reproduce the full real-world distribution of antibias language, provide comprehensive domain coverage, or balance the harmful/harmless register comparison.

We audited the final 19,420 total-row release to check for synthetic data rows: 18,015 rows (92.8% of BRIDGE Benchmark) are natural source examples. 1,405 rows (7.2% of BRIDGE) are generated using counterfactual pairing. More importantly, we actually ran further experiments concerning the main harmful/harmless analysis after excluding all the generated examples.

The result is effectively unchanged by removing synthetic data :

Harmful recall gap: +0.451 with all rows and +0.451 with natural rows only.

Harmless recall gap: −0.297 with all rows and−0.299 with natural rows only.

Thus, the primary Register Polarity Flip (RPF) metrics does not depend on the generated antibias examples at all. Furthermore, the directional consistency remains 8/8 models for both classes regardless of whether the full dataset is being used.

Q4.

This response about the proposed mitigation strategies and model adaptation is already addressed under Weakness 4 above.

In terms of Section 6, we report the complementary analyses corresponding to the questions addressed by GRDC. Together these analyses show why aggregate performance can obscure different, operationally important errors, and thus we mention all analyses we have accordingly. We appreciate the reviewer’s suggestion to have a better structure and narrative connecting the results. The revised version of the manuscript now has better organization for this section that is focused around the central findings and their implications..

We thank the reviewer again for raising these important concerns. We believe BRIDGE’s main value lies in providing a large and openly available benchmark that exposes a deployment-relevant failure mode that is usually overlooked by informal bias evaluations. In the revised version of the paper we have now made sure to further clarify the points stated above, highlight preliminary mitigation results, and discuss the limitations transparently.

Reviewer_9oKbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BRIDGE, a benchmark of 19,420 sentences from seven corpora covering formal institutional text and informal social media text under a shared harmful, harmless, and antibias taxonomy. It proposes GRDC, a suite of five diagnostic metrics, and identifies a Register Polarity Flip across eight LLMs. Harmful recall averages 0.40 on formal text and 0.85 on informal text. Bidirectional register rewrites preserve the pooled error direction, with effects of +0.136 for harmful content and −0.094 for harmless content.

Critical strengths

1.BRIDGE addresses a relevant gap by combining 19,420 formal and informal sentences under a shared taxonomy. Its coverage of statutes, housing documents, and policy language makes it useful for evaluating institutional bias detection.

2.BRIDGE reveals systematic bias detection failures across formal and informal text and helps assess model reliability in different settings. BRIDGE further demonstrates its evaluation value through controlled comparisons and robustness checks.

Critical weaknesses
  1. BRIDGE’s natural register comparison suffers from source confounding, with formal text drawn from six sources and all informal text drawn from Social Bias Frames (lines 291 to 292), limiting attribution of the observed performance gaps specifically to register, given the additional differences in era, topic, and genre acknowledged in lines 362 to 363.

  2. The matched pair control relies on LLM rewrites with only approximate content preservation (lines 376 to 377), providing incomplete control over changes in discriminatory meaning and limiting the causal interpretation of the reported register effect.

  3. The significance calculation derives a probability of approximately 0.00006 under independent shift directions (lines 183 to 185). It uses this probability to substantiate the observed pattern (lines 350 to 351), without justifying independence across models and between class outcomes.

  4. CCD uses BRIDGE macro F1 as its capability measure (lines 322 to 323) and links a nonsignificant correlation of 0.643 (p = 0.086) to model scaling (lines 402 to 405), without controlled comparisons of model size and training procedure to support that interpretation.

Questions for authors
  1. How much of the register gap remains after controlling for topic and era, and how consistent is the gap across the six formal sources?

  2. How did the verification process establish that register rewrites preserve the original discriminatory meaning and explicitness?

  3. What justifies the independence assumptions across models and class outcomes, and does the reported significance persist under an analysis accounting for these dependencies?

  4. How do the current results support the scaling interpretation of CCD without a statistically significant correlation or controlled comparisons of model size and training procedure?

4 comments

We would like to thank you for the thoughtful and constructive review. We appreciate the reviewer’s comments about BRIDGE’s relevance, source coverage, and use of controlled comparisons. We address the remaining concerns below regarding source-register confounding, matched rewrites, statistical dependence, and the interpretation of CCD.

Weaknesses

1. Source Confounding and Register Gap

We thank the reviewer for pointing out that our original wording should be more precise and acknowledge the need for moderating the language for causal claims. We will make sure to highlight that register is not the sole cause of the larger natural-corpus gap, and that source, topic, era, and genre may still amplify the effect. In the revised version of the manuscript, we have now made sure to clarify that this result we found is register-associated shift in the natural benchmark, rather than as an estimate of the causal effect of register alone.

We conducted an additional analysis experiment to clarify the scope of the results and support the main findings. In terms of the sources themselves, this analysis acts as a source-sensitivity check to ensure that there is no major source confounding. The natural harmful and harmless directions found in the results still remain when each formal source is removed in turn. The harmful informal-formal gap ranges from [+0.188 to +0.552] across these analyses where one formal source is left out each turn. Similarly, the harmless gap remains negative, from [−0.290 to −0.319]. These results show that no single formal source drives the pooled natural result in our main findings.

Formal source excludedHarmful gapHarmless gap
Comparative Agendas Project+0.441−0.304
GovReport+0.442−0.319
LSC Eviction Laws+0.441−0.300
Mapping Inequality+0.499−0.312
On the Books: Jim Crow Laws+0.552−0.290
Pile of Law+0.188−0.299

2. Matched-Rewrite Verification

We agree that an LLM rewrite prompt cannot guarantee semantic equivalence. However, during this whole rewrite process, each rewrite was generated from an identified seed and was manually screened by the authors against the intended pair-retention criteria. The goal for the rewrite design is to preserve the proposition, target group, gold label, factual content, and stance of the actual statement while only changing the register. Any rewrite pairs that changed the label or failed to achieve the register change under given criteria were excluded. We do not claim to guarantee that every pragmatic nuance is identical and we acknowledge that limitation of the rewrites. The human validation provides an important safeguard for the matched-pair analysis. Thus, we interpret the PRE results as supporting evidence that register contributes to the polarity flip after semantic content and source-related attributes are held fixed.

In the revised version of the paper, we have now added additional details for the experiment such as the generation prompt, pair-retention criteria, and the instructions for each rewrite direction in the appendix.

3. Significance Analysis

We thank the reviewer for pointing out that our original p=0.00006p=0.00006 calculation for the significance relied on treating the model directions and class outcomes as independent. We acknowledge this oversight in the paper and have removed the calculation from the revised version of the paper where DC will be presented as a descriptive measure of directional consistency.

We did run further analysis using the existing matched-pair predictions to assess the finding while accounting for dependence across the models. For each class, we averaged the effects across the eight evaluated models, then conducted pair-level tests with the sign flipped, and then adjusted for the two register class comparisons using Holm’s method. The harmful PRE value remains +0.136+0.136 (adjusted p=2.0×10−5p=2.0\times10^{-5}; 95% pair-cluster bootstrap CI [0.081,0.192][0.081, 0.192]). The harmless value PRE remains −0.094-0.094 (adjusted p=9.0×10−5p=9.0\times10^{-5}; 95% CI [−0.139,−0.050][-0.139, -0.050]). Thus, the matched-pair analysis results still support the same pooled directional finding under this dependence-aware test. We have now reported this method as well in the revised version of the paper and have directly presented the scope of the analysis.

4. Interpretation of CCD

We acknowledge the distinction between the CCD result and a causal scaling claim. The paper reports that the association is not statistically significant (rs=0.643r_s=0.643, p=0.086p=0.086, n=8n=8), but we agree that our discussion focuses too strongly on the results interpretation. We will instead look at the CCD result and the positive estimate as an exploratory pattern for the scope of our paper where among the eight evaluated models, higher overall macro-F1 ranks tend to co-occur with larger harmful-class register gaps. We see this as a potentially important question for future evaluation as well, since higher aggregate performance alone might not indicate smaller register disparities. We will remove any causal speculation about scaling and safety training from the final version of the paper while preserving CCD as an important exploratory analysis result.

Questions

Q1.

The available natural data unfortunately does not permit a complete topic and source controlled estimate for us to analyze since register is genuinely confounded with the source and era. However, restricting to contemporary examples and class-balancing the natural distribution of the sample still does preserve the direction of the register pattern (harmful gap +0.694; harmless gap −0.271). We acknowledge that this does not remove source differences but provides supporting evidence for the claim. As discussed under Weakness 1 above, excluding each of the formal sources at a time also preserves the natural direction of the polarity flip (harmful gaps range from +0.188 to +0.552 and harmless gaps from −0.290 to −0.319). Thus, we have support for the conclusion that the natural split of the data shows a broad register-associated pattern, and the matched experiment supports a focused register contribution under its controlled pair design.

Q2.

This response about the rewrite preservation is already addressed under Weakness 2 above.

Q3.

The significance does persist when dependence is accounted for between the evaluated models. Further details about the significance under model dependence and the experiment are already addressed under Weakness 3 above.

Q4.

The CCD result provides preliminary evidence that is relevant to the scaling hypothesis. As discussed under Weakness 4 above, we will interpret the CCD results as exploratory evidence and have made sure to remove any causal claims from the revised version of the paper.

We thank the reviewer again for raising these important concerns. We believe BRIDGE’s main value lies in providing a large and openly available benchmark that exposes a deployment-relevant failure mode that is usually overlooked by informal bias evaluations. In the now revised version of the paper we have made sure to further clarify the points stated above, add further experimental analysis as needed, and discuss all limitations transparently.

Reviewer_3J5VOpenReview ↗

Rebuttal acknowledgment

I thank the authors for their detailed, albeit rather late, response. While it addresses some initial concerns, those related to positioning and originality remain. After reading the other reviews and the authors' reply, I still believe the submission is not ready for publication at ICLR.

Reviewer_9oKbOpenReview ↗

Review Updating

Thank you for the responses and additional analyses. The updated statistical analysis and the more cautious interpretation of CCD help address some of my concerns. I still have reservations about source–register confounding and whether the rewrites consistently preserve discriminatory meaning. While I appreciate the improvements, I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.