acceptodds
Under review as a conference paper at ICLR 2027

The Anatomy of Socio-Demographics in Pluralistic Offensiveness Modeling: When and Where It Helps, and the Role of Reasoning

Abstract

Subjective judgments such as offensiveness often vary substantially across individuals, motivating pluralistic modeling approaches that incorporate annotators' socio-demographic information to capture this variation. However, prior findings on the benefits of socio-demographic information for pluralistic modeling are mixed, calling for a systematic analysis of both its effectiveness and the mechanisms through which it improves prediction. In this work, we study and socio-demographics benefit pluralistic offensiveness modeling, as well as in leveraging this information. Across three multi-annotator datasets with prompted and trained LLMs, we first show that demographic gains are small and inconsistent for individual predictions but become clearer at the group and population levels, because aggregation attenuates the idiosyncratic variation that masks small but systematic demographic signals. Second, using a reasoning and prediction format that explicitly decomposes individual judgment into an overall baseline and a demographic-specific deviation, we show that the gains arise from both components: demographic information improves baseline estimation and enables meaningful demographic-specific shifts, and interventions show that better component estimates improve final predictions. Finally, comparing direct demographic conditioning with explicit reasoning about how the text interacts with annotator demographics, we find that reasoning yields more accurate baseline and shift estimates as well as final predictions, whereas conditioning alone rarely produces informative shifts. These results suggest that the mixed evidence in prior work may partly stem from individual-level evaluation, and that the value of socio-demographic information depends on both how it is evaluated and how models are guided to use it.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Pluralistic modeling, Socio-demographic modeling, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_y3F7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to understand the relationship between pluralistic offensiveness modeling and the socio-demographic information of the people whose annotations are being modeled. Specifically, the authors study (1) when and where socio-demographics improve pluralistic modeling, and (2) the role of model reasoning.

The work uses three multi-annotation datasets (DICES, D3CODE, and POPQUORN) and studies the RQs on models that are either:

  • prompted: Llama and Qwen models (3B–122B), GPT-4o-mini, and Haiku-4.5; or
  • trained: 4 open-weight Llama and Qwen models (3B–9B).

In this setting, the authors raise a number of RQs and make several empirical observations to answer them:

  1. Socio-demographic information offers gains in pluralistic offensiveness modeling at the population level, more than at the finer-grained group or individual levels.
  2. The authors treat a judgment (an offensiveness label plus the relevant demographic-specific shift to that label) as a decomposition into two parts: reasoning about the baseline behavior appropriate for the general population (i.e., what most people would think) and the demographic-specific deviation. They find that this decomposition improves the accuracy of both the baseline estimate and the demographic-specific shift.
  3. Model reasoning that uses demographics performs better than demographic conditioning alone.
Critical strengths

Relevance and impact. This work is important and timely. It is potentially interesting to the ICLR community because it shows how socio-demographic information can be used to improve pluralistic modeling, which enables context-specific improvements to user experience.

Critical weaknesses
  1. (Primary) Empirical validation. Some key claims are not corroborated sufficiently.

    • Lines 222–224 claim: "Compared with simple demographic conditioning, explicit demographic reasoning generally yields larger gains, particularly at the group and population levels." In Table 1, the group and population levels together report 16 metrics (8 each for prompted and trained models). Only 3 of the 16 are marked as statistically significant for DemoReason over DemoCond. The statistics reported in Table 1 therefore only weakly corroborate the authors' claim.
    • Similarly, in lines 302–307, Table 2 does not consistently show an improvement of DemoReason over DemoCond. "DemoReason matches DemoCond" may be a more appropriate claim than "DemoReason improves over DemoCond," because the improvement is not statistically significant in any case except one (Trained; W1). The claim "Explicit demographic reasoning thus strengthens baseline estimation beyond simple conditioning" is therefore under-powered.
  2. (Less important than the empirical point above) Under-definition. Some important elements of the setup are incompletely defined, which slightly hinders interpretation and assessment of the paper. See the questions marked "Clarity" below.

Questions for authors

Clarity

  1. Some elements of Fig. 1 are unclear, and the description of "Variants of reasoning & prediction format" is incomplete. What is "crowd," and how should its magnitude be interpreted? Similarly, what is "target," and what labels/values can it take?
  2. Where and how can the numbers "0.006/0.009" and "0.007/0.017" be seen in Table 1?
  3. The authors define a judgment as a decomposition but do not motivate this enough. Why is it desirable or optimal to think of a judgment as an (additive) decomposition into a baseline and a demographic-specific deviation, rather than as a single quantity?
3 comments
AuthorsOpenReview ↗

Response to Reviewer y3F7 (1/N)

Thank you for your constructive feedback! Below, we address your concerns in detail, with a brief TL;DR for each question. We are still incorporating new results and discussions into the revised manuscript, which we plan to upload by November 14. We will then post a general response to all reviewers summarizing the revisions.

TL;DR

[W1: Empirical validation] We report win rates across model–dataset pairs to show the consistency of our findings and explain why some trends are robust across pairs but do not reach statistical significance.

[W2/Q1: Clarification of reasoning and prediction format] We provide a more detailed explanation of each field in the reasoning and prediction format.

[W2/Q2: Clarification of reported values] 0.006/0.009 and 0.007/0.017 are DemoCond/DemoReason - NoDemo in Acc at the group- and population-level in Table 1.

[W2/Q3: Motivation for decomposition] The additive decomposition is motivated by findings in human psychology and is also widely used in prior NLP research.

Below we provide detailed responses to your concerns and questions.

W1: Empirical validation

We acknowledge that some improvements do not reach statistical significance, particularly after multiple-testing correction. However, we believe the empirical results still provide evidence of broadly consistent patterns that generalize across models and datasets. Below, we will first explain why some trends are consistent across model-dataset pairs but do not reach statistical significance. We will then report win rates across model–dataset pairs to support the consistency of our findings.

Why some improvements consistent across model-dataset pairs, but still do not reach statistical significance: First, the limited number of model–dataset pairs may reduce statistical power. For example, trained DemoCond improves group-level MSE over NoDemo by 0.019 (a 9.6% relative reduction from 0.198), with improvements in 11/12 model-dataset pairs, yet the difference is not significant. Second, our analyses of baseline and shift estimation show that demographic benefits are heterogeneous, being more pronounced for high-disagreement items and those with stronger text–demographic interactions. Such effects may be diluted when averaged across all examples.

Below, we revisit the key findings in the paper and report their consistency with win rates across model-dataset pairs.

Explicit demographic reasoning generally provides larger gains than direct conditioning, particularly for prompted models and population-level predictions. As shown in Table 1 below, for prompted models, DemoReason outperforms DemoCond on average in 8 of the 10 reported metrics, with larger and more consistent gains at the population level. For trained models, DemoReason also outperforms DemoCond on all four population-level metrics, with improvements observed in 9–10 of the 12 model–dataset pairs. The mean improvements and win counts provide consistent directional evidence of DemoReason's advantage in prompted models and population-level predictions.

Table 1: Paired comparison of individual-, group-, and population-level performance of DemoReason vs. DemoCond. For clarity, we report the mean paired improvement, the number of model–dataset pairs showing improvement, and the corrected significance results for each comparison. Positive values indicate better performance and are highlighted in bold. ∗* and †\dagger denote statistical significance (padj<0.05p_{\mathrm{adj}} < 0.05) under the paired t-test and paired bootstrap test, respectively, after Holm–Bonferroni correction.

SettingComparisonInd. F1Ind. AccGroup F1Group AccGroup MSEGroup ρPop. F1Pop. AccPop. MSEPop. ρ
PromptedDemoReason vs. DemoCond+0.012(17/27)−0.001(12/27)+0.004(17/27)+0.003(17/27)−0.005(16/27)+0.006(17/27)+0.005(19/27)+0.011(19/27)+0.004(20/27)+0.017(19/27)
TrainedDemoReason vs. DemoCond−0.004(7/12)−0.016(5/12)0.000(7/12)−0.001(8/12)−0.003(4/12)−0.005(6/12)+0.010(10/12)+0.014(9/12)+0.020(9/12)†^{\dagger}+0.007(10/12)
AuthorsOpenReview ↗

Response to Reviewer y3F7 (2/N)

DemoReason provides an additional advantage over DemoCond in baseline estimation, particularly for prompted models. Table 2 shows that, for prompted models, DemoReason outperforms DemoCond on average in seven of the eight metrics, with improvements in 15–19 of the 27 model–dataset pairs. For trained models, DemoReason also achieves better average scores in six of the eight metrics, although the gains are smaller and less robust across model–dataset pairs, particularly for correlation metrics.

Table 2: Paired comparison of baseline estimation performance of DemoReason vs. DemoCond. For clarity, we report the mean paired improvement, the number of model–dataset pairs showing improvement, and the corrected significance results for each comparison. Positive values indicate better performance and are highlighted in bold. ∗* and †\dagger denote statistical significance (padj<0.05p_{\mathrm{adj}} < 0.05) under the paired t-test and paired bootstrap test, respectively, after Holm–Bonferroni correction.

SettingComparisonAnn. MSEAnn. MAEAnn. ρAnn. W1Item MSEItem MAEItem ρItem W1
PromptedDemoReason vs. DemoCond+0.002(19/27)+0.003(18/27)+0.009(19/27)+0.005(18/27)+0.002(16/27)+0.002(15/27)−0.003(16/27)+0.004(19/27)
TrainedDemoReason vs. DemoCond+0.000(7/12)+0.001(6/12)−0.009(5/12)+0.008(9/12)+0.001(8/12)+0.003(9/12)−0.004(6/12)+0.006(9/12)

In the revised paper, we will retain our main empirical findings while adopting a more cautious interpretation of their statistical support and the conditions under which each finding holds. We will also include the win rates as complementary metrics, as they provide evidence of directional consistency across model–dataset pairs.

W2/Q1: Clarification of reasoning and prediction format

crowd∈[0,1]\texttt{crowd} \in [0, 1] represents the model's estimated proportion of a broad population that would consider the text offensive. For example, crowd=0.4\texttt{crowd}=0.4 in Figure 1 indicates that approximately 40% of the population is expected to find the text offensive, meaning that the majority would consider it non-offensive. target\texttt{target} identifies the target of the potentially offensive content. It can take any of the target categories defined in the corresponding dataset, or "none" if no specific group or identity is targeted. We will clarify the meaning and possible values of each field in the revised paper.

W2/Q2: Clarification of reported values

0.006/0.009 and 0.007/0.017 are DemoCond/DemoReason - NoDemo in Acc at the group- and population-level in Table 1. We will report the mean improvements over NoDemo alongside raw scores in the revised paper, such that the improvements and overall trends are more visible.

W2/Q3: Motivation for decomposition

We adopt the baseline–deviation decomposition because it is supported by human pscychological studies and consistent with prior NLP work on pluralistic modeling.

First, psychological studies suggest that humans often predict others' perspectives by anchoring on an initial baseline and then adjusting for differences in social perspectives (Epley et al., 2004; Ames et al., 2004).

Moreover, similar decompositions are common in prior pluralistic modeling approaches, either explicitly or implicitly. Probabilistic and structural models explicitly distinguish shared effects from annotator- or group-specific variations, such as annotator biases and demographic effects (Paun et al., 2018; Chu et al., 2021; Liu et al., 2022). Recent NLP approaches implicitly capture this distinction in their model design through shared representations and annotator-specific heads (Mostafazadeh Davani et al., 2022), demographic representations (Zhang and Çöltekin, 2026), or demographic-aware experts (Xu et al., 2026).

Building on these approaches, we make the baseline–deviation decomposition explicit, allowing LLMs to reason about each component and enabling us to evaluate how socio-demographic modeling affects them separately.

References

  • Epley et al. (2004). Perspective Taking as Egocentric Anchoring and Adjustment. Journal of Personality and Social Psychology.
  • Ames (2004). Inside the Mind Reader's Tool Kit: Projection and Stereotyping in Mental State Inference. Journal of Personality and Social Psychology.
  • Paun et al. (2018). Comparing Bayesian Models of Annotation. TACL.
  • Chu et al. (2021). Learning from Crowds by Modeling Common Confusions. AAAI.
  • Liu et al. (2022). Toward Annotator Group Bias in Crowdsourcing. ACL.
  • Mostafazadeh Davani et al. (2022). Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. TACL.
  • Zhang and Çöltekin (2026). Modeling Human Perspectives with Socio-Demographic Representations. Findings of ACL.
  • Xu et al. (2026). Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives. ACL.

Dear Reviewer y3F7,

As the discussion phase will close in less than 48 hours, we would greatly appreciate it if you could take a moment to review our response and let us know whether our clarifications and additional experimental results have addressed your concerns. We would be more than happy to provide further clarification or address any remaining questions.

Reviewer_ypDJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the effect of including socio-demographic information of the audience in offensiveness labeling, considering variables of demographic information inclusion, use of model reasoning capabilities, and using training or prompting as the method to add information. Additionally, the paper evaluates the results across various levels of granularity: item-level, demographic group-level, and population-level. They find that the inclusion of demographic information paired with a prompted reasoning model leads to the biggest gains, both for group- and individual-level accuracy, but, perhaps surprisingly, this setting, and demographic information inclusion more broadly, leads to major improvements in the baseline labeling capabilities (i.e., the model becomes more accurate at predicting the label that the "crowd" would make). The authors provide follow-up experiments to help explain why this surprisingly baseline improvement exists, leading to two possibles sources: (1) inclusion of demographic information pushes the model away from its consensus prediction and toward modeling the plurality of possible perspectives, and (2) the reasoning mechanism in this setting can separate audience-specific and general "crowd"-aligned aspects of the input.

Critical strengths
  1. This paper's experiments cover a wide range of relevant interaction variables in the offensiveness setting and use multiple LLMs and datasets, which help support the broad claims about LLM behaviors that the paper makes. The use of 4 or 9 models (depending on the setting) and three datasets make the experimental results not conditional on a narrow setting. Breaking down the results according to multiple levels of demographic aggregation, testing both prompting and training, and studying both model capabilities of the crowd inference as well as demographic-specific shifts provide interesting lenses of LLM sensitivities that are sometimes surprising. For example, prompting generally performed better than training and reasoning has a major effect on the models' ability to use provided demographic information)

  2. The authors perform follow-up experiments and analysis for two key results to help explain the causal mechanism for them. The notable improvement of the baseline and the small but relatively consistent demographic-specific gains are studied after the broad results with reasonable interpretations of factors that lead to these results.

Critical weaknesses
  1. The discussion of related work is relegated to an appendix and even so is limited to less than a page. Given the breadth of prior work in this direction, clearly articulating where this paper's results stand in relation to the existing literature is critical for proper evaluation of its scientific contributions. The appendix, in fact, admits that prior work has investigated individual-, group-, and population-level demographic modeling. The appendix distinguishes this paper as "systematically characterizing when the benefits of demographic information become more visible across levels of aggregation", but does not have enough detail the discussion of prior to make clear why such differences across aggregation cannot be inferred from the results of existing literature. With this level of engagement with prior literature in the manuscript, the exact scope of its scientific contributions is muddled and best, and misleading at worst.

  2. The writing does not seem to always match the data that is presented. Section 3.1 states "at the individual level, demographic information provides little consistent advantage", but the only statistically significant improvement is the F1 score for "DemoReason". If anything, this result seems to warrant a follow-up of why accuracy and F1 results may diverge so much. Instead, the paper ignores this F1 result and makes an interpretation based on every other number associated with the prompted individual level setting. In Section 4.4, the discussion on interventional evidence relates results from Table 4 against unpresented info in Appendix G. The synthesis of those is needed to understand the discussion.

Questions for authors
  1. Are there any dataset- or model-level results that do not align with the general trends and claims being made? It's quite rare for there not to be exceptions in models and datasets in studies like these. The general claims being made here may mislead readers into a more universal pattern than is justified from the actual details of the experimental results.
4 comments
AuthorsOpenReview ↗

Response to Reviewer ypDJ (1/N)

Thank you for your constructive feedback! Below, we address your concerns in detail, with a brief TL;DR for each question. We are still incorporating new results and discussions into the revised manuscript, which we plan to upload by November 14.

TL;DR

[W1: Expanded discussion of Related Work] We will expand the related work discussion in the revised paper and include a table summarizing the experimental setups and findings of prior work. Prior work has reported mixed and inconsistent findings, motivating our systematic study of both the effects and underlying mechanisms of socio-demographic modeling.

[W2: More comprehensive result analysis] The current manuscript focuses mainly on the overall findings due to space constraints. We will revise the discussion to provide a more comprehensive and self-contained interpretation of the results.

[Q1: Dataset- and model-level exceptions] The increasing benefits of socio-demographic modeling with aggregation are broadly consistent across datasets and models. The effect of explicit reasoning is less stable for some smaller models.

Below we provide detailed responses to your concerns and questions.

W1: Expanded discussion of Related Work

We agree that a more detailed discussion of prior work would help better position our contributions. We will add the following expanded discussion of related work in the revised paper to better motivate our three research questions. We will also include Table 1 in the appendix to summarize and compare the evaluation settings and findings of prior studies.

Effect of socio-demographic modeling across multiple levels

Prior work has reported mixed findings on the effectiveness of socio-demographic information in pluralistic modeling. While some studies find that incorporating annotator demographics improves prediction performance (Fleisig et al., 2023; Zhang & Çöltekin, 2026; Xu et al., 2026), others report limited or inconsistent benefits (Orlikowski et al., 2023, 2025; Beck et al., 2024; Jaggi et al., 2024; Sun et al., 2025; Cai & Zafarani, 2026; Kamruzzaman et al., 2026; Piot & Parapar, 2026). One possible explanation is that demographic effects may be obscured by individual-level variation and become more apparent after aggregation. However, most prior studies evaluate demographic-aware modeling at only one or a subset of the individual, group, and population levels. Among the few studies considering all levels, Parappan and Henao (2025) report results using different metrics and prediction settings, limiting direct comparisons of demographic gains across levels. Fleisig et al. (2023) evaluate both individual and aggregated judgments and find positive demographic effects at both levels, but do not observe an increase in their magnitude with aggregation. Thus, whether aggregation systematically reveals demographic benefits that are otherwise difficult to detect at the individual level remains unclear. Our work addresses this gap through a controlled comparison of demographic-aware and demographic-blind models across all three levels, examining how demographic gains change with aggregation.

Decomposing pluralistic modeling into baseline and deviation estimates

Many recent NLP studies on pluralistic modeling implicitly capture shared and annotator- or group-specific factors in subjective judgments, for example, through shared representations with annotator-specific prediction heads (Mostafazadeh Davani et al., 2022), demographic group-specific layers (Orlikowski et al., 2023), and demographic-aware mixture-of-experts architectures (Xu et al., 2026). However, these approaches generally do not explicitly distinguish a shared population baseline from demographic-specific deviations, making it difficult to understand how models leverage socio-demographic information and to audit whether their predictions reflect potentially biased demographic associations. To address this limitation, we adopt a reasoning and prediction format that explicitly exposes the population baseline and demographic-specific deviation as separate estimates, enabling us to examine how socio-demographic information affects each component and how these components contribute to final predictions.

Reasoning for pluralistic modeling

LLM reasoning has achieved substantial gains in mathematics and coding (Jaech et al., 2024; Guo et al., 2025) and has recently been extended to subjective tasks such as reward modeling and bias detection (Chen et al., 2026; Fan et al., 2025). These approaches often distill reasoning traces from stronger models, followed by optimization using DPO or GRPO. While prior pluralistic modeling primarily uses direct socio-demographic conditioning (Orlikowski et al., 2023; Sun et al., 2025; Xu et al., 2026), our work systematically compares direct conditioning with explicit demographic reasoning, examining their effects on baseline estimation, demographic-specific shifts, and final predictions.
AuthorsOpenReview ↗

Response to Reviewer ypDJ (2/N)

Table 1: Comparison of prior findings on the effectiveness of socio-demographic information across evaluation levels. ✓ indicates positive effects of socio-demographic information; × indicates negative or inconsistent effects; – indicates that the corresponding evaluation level was not examined.

WorkIndividualGroupPopulationMain Findings
Fleisig et al. (2023)✓✓✓Demographics improve individual, target-group, and aggregated predictions, with comparable relative gains across levels.
Orlikowski et al. (2023)×––Demographic-specific layers do not significantly improve individual annotation prediction.
Beck et al. (2024)××–Demographic prompting yields inconsistent gains across tasks, models, and prompt formulations.
Orlikowski et al. (2025)×––Gains largely reflect learned annotator-specific behavior rather than generalizable demographic patterns.
Sun et al. (2025)–×–Demographic prompting yields inconsistent improvements and sometimes worsens subgroup alignment.
Parappan & Henao (2025)✓✓✓Sociocultural representations improve predictions across levels, but differing metrics limit direct comparisons of gains.
Sorensen et al. (2025)✓×–Demographics contain predictive information, but explain less variation than value profiles and demonstrations.
Zhang & Çöltekin (2026)✓––Socio-contrastive representations improve individual perspective prediction over simple demographic conditioning.
Cai & Zafarani (2026)×––Demographic benefits depend on disagreement, data availability, and modeling conditions.
Xu et al. (2026)✓×–Demographic-aware experts improve prediction, particularly on high-disagreement datasets, but group-level effects are inconsistent.
Kamruzzaman et al. (2026)×––Demographic prompting yields inconsistent gains across model–task pairs.

W2: More comprehensive result analysis

Due to space constraints, the current manuscript focuses mostly on overall trends, leaving some details to the appendix. We will expand the discussion in the revised paper to better address these observations.

In Section 3.1, our main focus is not whether demographic-aware variants outperform NoDemo at each evaluation level, but how their relative advantages change with aggregation. We will revise the discussion accordingly to make this comparison more clearly while explicitly addressing important observations and exceptions.

The relative performance of demographic-aware variants generally grows with aggregation. 
For prompted models, although DemoReason improves individual-level F1 over NoDemo, its Acc remains comparable, as does DemoCond's performance on both metrics. At the group and population levels, both variants show more consistent advantages over NoDemo, with larger Acc gains and more frequent improvements across model–dataset pairs. The aggregation effect is even clearer for continuous metrics (MSE and $\rho$), where both the magnitude and consistency of improvements generally increase from the group to the population level.
Trained models exhibit a similar overall trend. DemoReason outperforms NoDemo on population-level discrete metrics on average, while showing little or no advantage at lower levels. Continuous metrics again reveal clearer improvements with aggregation.

(Following the suggestion of Reviewer oKHc, we have applied multiple-testing correction to our significance tests. The improvements of DemoReason over NoDemo and DemoCond no longer reach statistical significance now. We kindly refer you to our response to Reviewer oKHc if you are interested in the corrected results and a detailed discussion of their implications.)

Regarding Section 4.4, we agree that presenting the intervention results for baseline and shift estimates separately would help clarify their respective effects on final predictions. We will include this discussion in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer ypDJ (3/N)

Q1: Dataset- and model-level exceptions

Our claims describe overall trends across model–dataset pairs rather than universal patterns without exceptions.

The finding that demographic benefits increase with aggregation is broadly consistent across models and datasets. While DemoCond and DemoReason may outperform or underperform NoDemo at any single level, their relative performance generally improves with aggregation. There are some modest exceptions, such as prompted Llama-8B and Haiku-4.5, where the relative advantages remain stable or fluctuate slightly across aggregation levels for certain metrics.

The advantage of DemoReason over DemoCond shows some systematic exceptions. This advantage is generally more robust in larger models, while smaller models show less stable improvements, possibly due to their limited ability to benefit from explicit demographic reasoning.

Across the three datasets, the overall trends remain largely consistent.

We will discuss these exceptions more explicitly in the revised paper, clarifying when each finding holds and where it is less consistent. We will also provide raw model–dataset results in the appendix so readers can examine individual cases.

References

  • Mostafazadeh Davani et al. (2022). Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. TACL.
  • Fleisig et al. (2023). When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks. EMNLP.
  • Orlikowski et al. (2023). The Ecological Fallacy in Annotation: Modeling Human Label Variation Goes Beyond Sociodemographics. ACL.
  • Beck et al. (2024). Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting. EACL.
  • Jaech et al. (2024). OpenAI o1 System Card. arXiv:2412.16720.
  • Jaggi et al. (2024). Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree. EMNLP.
  • Fan et al. (2025). BiasGuard: A Reasoning-Enhanced Bias Detection Tool for Large Language Models. Findings of ACL.
  • Guo et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature.
  • Orlikowski et al. (2025). Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals’ Subjective Text Perceptions. ACL.
  • Parappan and Henao (2025). Learning Subjective Label Distributions via Sociocultural Descriptors. EMNLP.
  • Sorensen et al. (2025). Value Profiles for Encoding Human Variation. EMNLP.
  • Sun et al. (2025). Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMs. NAACL.
  • Cai and Zafarani (2026). When Does Demographic Information Help? Data and Modeling Regimes for Perspective-Aware Hate Speech Detection. arXiv:2605.27313.
  • Chen et al. (2026). RM-R1: Reward Modeling as Reasoning. ICLR.
  • Kamruzzaman et al. (2026). Demographic Prompting at Scale: When More Attributes Hurt LLM–Human Agreement. EMNLP.
  • Xu et al. (2026). Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives. ACL.
  • Zhang and Çöltekin (2026). Modeling Human Perspectives with Socio-Demographic Representations. Findings of ACL.

Dear Reviewer ypDJ,

As the discussion phase will close in less than 48 hours, we would greatly appreciate it if you could take a moment to review our response and let us know whether our clarifications and additional experimental results have addressed your concerns. We would be more than happy to provide further clarification or address any remaining questions.

Reviewer_oKHcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the conditions under which socio-demographic prompting benefits LLMs' pluralistic offensiveness judgement. They find that socio-demographic prompting predicts pluralistic offensiveness better at the group and population level compared to predicting individual responses. The paper also investigate the role of explicit reasoning and find that it can further improve the pluralistic alignment.

Critical strengths

This paper builds on prior mixed evidence on the effects of socio-demographic prompting and further distinguishes the conditions under which such information can improve subjective judgments from different perspectives. It deepens the understanding of this area through a comprehensive evaluation.

Critical weaknesses

My main concern is the paper’s methodological validity and whether addressing these issues would change its main empirical findings. In Tables 1 and 2, the authors test statistical significance using paired t-tests with a cutoff of p < 0.05. I do not find any justification for their decision not to apply multiple-testing corrections, which has important implications for interpreting the results.

I am also confused about why the authors evaluate a set of models on three different datasets but aggregate results across models and datasets when reporting their findings and conducting statistical analysis. This approach requires a clear justification and explicit assumptions. I am concerned that reporting results separately could lead to different conclusions.

Questions for authors

In DemoCond, why are baseline scores scaled to [0, 1], while demographic shifts are assigned values in {-1, 0, 1}? This seems to be a coarse representation of demographic deviation. Could it confuse LLMs about how to assign offensiveness labels based on the numerical sum of the baseline score and demographic shift? I would be curious to know how LLMs assign offensiveness labels when the resulting score can range from −1 to 2.

8 comments
AuthorsOpenReview ↗

Response to Reviewer oKHc (1/N)

Thank you for your constructive feedback! Below, we will address your concerns in detail, with clarifications and new experimental results. To save you time, we also provide a TL;DR with a short response to each of your questions.

We are still incorporating the new results and discussions into the revised manuscript, which we plan to upload by November 14. Once it is available, we will post a general response to all reviewers summarizing the revisions.

TL;DR

[W1: Multiple-test correction] We apply the Holm–Bonferroni correction to our significance tests. While some results are no longer statistically significant, our main empirical findings remain consistent. We also report win rates across model–dataset pairs to further demonstrate the consistency of our findings.

[W2: Raw results without aggregation] We will include the raw model–dataset results in the revised paper. The win rates show that our findings are consistent across model–dataset pairs, rather than being driven by a few outliers.

[Q1: Directional demographic shift] shift∈\texttt{shift} \in {-1, 0, +1} represents only the direction, not the magnitude, of demographic deviations. We also provide supporting statistics showing that LLMs can reliably use these directional predictions, while forcing them to predict magnitudes leads to worse performance.

Below, we provide a more detailed response to each of your question.

W1: Multiple-test correction

We agree that the statistical comparisons in Tables 1 and 2 should account for multiple testing. We therefore re-evaluate our results using both paired t-tests and paired bootstrap tests, applying the Holm–Bonferroni correction for multiple comparisons. After correction, some previously significant improvements no longer reach the 0.05 significance threshold. We acknowledge this change and will revise the corresponding claims accordingly. However, we believe that the corrected results remain consistent with our main findings and do not undermine their validity.

Below, we revisit each table and their main findings, and show how the findings are still supported by the corrected results and comparisons across model–dataset pairs. We then discuss why some findings show consistent trends across these pairs but do not reach statistical significance.

Table 1 after correction

The following table reports the improvements of DemoCond and DemoReason over NoDemo at each level.

Table 1: Paired comparisons of individual-, group-, and population-level performance of NoDemo, DemoCond, and DemoReason. For clarity, we report the mean paired improvement, the number of model–dataset pairs showing improvement, and the corrected significance results for each comparison. Positive values indicate better performance and are highlighted in bold. ∗* and †\dagger denote statistical significance (padj<0.05p_{\mathrm{adj}} < 0.05) under the paired t-test and paired bootstrap test, respectively, after Holm–Bonferroni correction.

SettingComparisonInd. F1Ind. AccGroup F1Group AccGroup MSEGroup ρPop. F1Pop. AccPop. MSEPop. ρ
PromptedDemoCond vs. NoDemo+0.004 (14/27)+0.002 (16/27)+0.008 (16/27)+0.006 (16/27)+0.032 (20/27) ∗†^{*\dagger}+0.026 (20/27)†^{\dagger}+0.004 (17/27)+0.007 (16/27)+0.060 (27/27)∗†^{*\dagger}+0.092 (26/27)∗†^{*\dagger}
DemoReason vs. NoDemo+0.016 (16/27)+0.001 (14/27)+0.012 (18/27)+0.009 (17/27)+0.027 (20/27)+0.032 (22/27)†^{\dagger}+0.010 (19/27)+0.017 (19/27)+0.063 (25/27)∗†^{*\dagger}+0.109 (27/27)∗†^{*\dagger}
DemoReason vs. DemoCond+0.012 (17/27)−0.001 (12/27)+0.004 (17/27)+0.003 (17/27)−0.005 (16/27)+0.006 (17/27)+0.005 (19/27)+0.011 (19/27)+0.004 (20/27)+0.017 (19/27)
TrainedDemoCond vs. NoDemo−0.002 (5/12)−0.002 (4/12)0.000 (5/12)−0.002 (6/12)+0.019 (11/12)+0.012 (6/12)−0.001 (9/12)−0.011 (7/12)+0.025 (12/12)∗†^{*\dagger}+0.039 (9/12)
DemoReason vs. NoDemo−0.006 (7/12)−0.018 (6/12)0.000 (8/12)−0.003 (8/12)+0.016 (9/12)+0.007 (8/12)+0.009 (9/12)+0.003 (8/12)+0.045 (12/12)∗†^{*\dagger}+0.046 (10/12)
DemoReason vs. DemoCond−0.004 (7/12)−0.016 (5/12)0.000 (7/12)−0.001 (8/12)−0.003 (4/12)−0.005 (6/12)+0.010 (10/12)+0.014 (9/12)+0.020 (9/12)†^{\dagger}+0.007 (10/12)
AuthorsOpenReview ↗

Response to Reviewer oKHc (2/N)

Some improvements over NoDemo or DemoCond no longer reach statistical significance after correction in Table 1, such as individual-level F1 for prompted DemoReason. However, our main findings remain consistent with the updated results:

(1) The benefits of socio-demographic modeling become more visible with aggregation, especially on continuous metrics. This trend does not depend on whether DemoCond and DemoReason outperform NoDemo at any single level, but but how their relative performance changes with aggregation. Therefore, the conclusion is not affected by the corrections. Table 1 shows that the relative performance of DemoCond and DemoReason compared to NoDemo improves with aggregation, reflected in both the growing magnitude and consistency of improvements from lower to higher aggregation levels across model-dataset pairs. This aggregation effect is clearer for continuous metrics.

(2) Explicit demographic reasoning generally provides larger gains than direct conditioning, particularly for prompted models and population-level predictions. The statistically significant advantages of prompted DemoReason over DemoCond in individual-level F1 and population-level Acc and MSE no longer hold after correction. Despite that, the benefits of explicit reasoning remain robust across metrics and model–dataset pairs. For prompted models, DemoReason outperforms DemoCond on average in 8 of the 10 reported metrics. For trained models, DemoReason also outperforms DemoCond on all four population-level metrics, with improvements in 9–10 of the 12 model–dataset pairs. The average improvements and win counts provide consistent directional evidence of DemoReason's advantages in prompted models and population-level predictions.

We therefore retain the conclusion about Table 1, while providing a more rigorous description of the results and significance tests.

Table 2 after correction

Table 2: Paired comparisons of baseline estimation performance of NoDemo, DemoCond, and DemoReason. We report the mean paired improvement, the number of model–dataset pairs showing improvement, and the corrected significance results for each comparison. Positive values indicate better performance and are highlighted in bold. ∗* and †\dagger denote statistical significance (padj<0.05p_{\mathrm{adj}} < 0.05) under the paired t-test and paired bootstrap test, respectively, after Holm–Bonferroni correction.

SettingComparisonAnn. MSEAnn. MAEAnn. ρAnn. W1Item MSEItem MAEItem ρItem W1
PromptedDemoCond vs. NoDemo+0.006 (22/27)+0.009 (18/27)+0.006 (15/27)+0.012 (17/27)+0.013 (27/27)∗†^{*\dagger}+0.022 (24/27)∗†^{*\dagger}+0.062 (22/27)∗†^{*\dagger}+0.014 (17/27)
DemoReason vs. NoDemo+0.008 (21/27)+0.012 (19/27)+0.015 (16/27)+0.016 (19/27)+0.015 (24/27)∗†^{*\dagger}+0.024 (21/27)∗†^{*\dagger}+0.059 (23/27)∗†^{*\dagger}+0.018 (19/27)
DemoReason vs. DemoCond+0.002 (19/27)+0.003 (18/27)+0.009 (19/27)+0.005 (18/27)+0.002 (16/27)+0.002 (15/27)−0.003 (16/27)+0.004 (19/27)
TrainedDemoCond vs. NoDemo+0.003 (7/12)+0.005 (8/12)+0.005 (7/12)+0.009 (7/12)+0.007 (10/12)+0.012 (10/12)†^{\dagger}+0.037 (9/12)+0.008 (6/12)
DemoReason vs. NoDemo+0.003 (10/12)+0.006 (9/12)−0.004 (5/12)+0.016 (11/12)+0.008 (11/12)+0.015 (10/12)+0.033 (9/12)+0.014 (8/12)
DemoReason vs. DemoCond+0.000 (7/12)+0.001 (6/12)−0.009 (5/12)+0.008 (9/12)+0.001 (8/12)+0.003 (9/12)−0.004 (6/12)+0.006 (9/12)

Incorporating socio-demographics improves baseline estimation, with additional benefits from explicit reasoning. The baseline estimation results still provide particularly consistent evidence of the benefits of demographic-aware modeling and explicit reasoning. Both DemoCond and DemoReason generally outperform NoDemo across evaluation metrics and settings. Explicit reasoning provides an additional advantage over demographic conditioning, particularly for prompted models, where DemoReason outperforms DemoCond on average in seven of eight metrics, with improvements observed in 15–19 of the 27 model–dataset pairs. For trained models, this additional advantage is smaller and less consistent, particularly for correlation metrics.

AuthorsOpenReview ↗

Response to Reviewer oKHc (3/N)

Interpretation of the corrected significance results

We acknowledge that several improvements no longer reach statistical significance after correction, which warrants a more cautious interpretation rather than invalidating the overall empirical patterns. First, the limited number of model–dataset pairs may reduce statistical power. For example, trained DemoCond improves group-level MSE over NoDemo by 0.019 (a 9.6% relative reduction from 0.198), with improvements in 11/12 pairs, yet the difference is not significant after correction. Second, our analyses of baseline and shift estimation show that demographic benefits are heterogeneous, being more pronounced for high-disagreement items and those with stronger text–demographic interactions. Such effects may be diluted when averaged across all examples. Finally, although win counts do not establish statistical significance, they provide evidence of directional consistency across model–dataset pairs.

In the revised paper, we will therefore retain our main empirical findings while interpreting the results and their statistical significance more cautiously. We will also include the improvement comparisons, win rate statistics, and revised discussions presented above to make the observed trends clearer.

W2: Raw results without aggregation

In the revised paper, we will include the results for each model–dataset pair and the baseline estimates in the appendix, alongside the aggregated results in the main paper. The win counts reported above further support that the observed trends are broadly consistent across model–dataset pairs, rather than being driven by a few outliers.

Q1: Directional demographic shift

The shift values {-1, 0, +1} represent directions rather than magnitudes, as explained in Footnote 2 and explicitly specified in the prompts (Appendix B). The final offensiveness label is predicted by the LLM rather than computed by summing the baseline and shift. In practice, LLMs can reliably use these directional signals, with inconsistent predictions occurring in only 1.3% of trained and 3.0% of prompted cases (e.g., predicting offensive when baseline < 0.5 and shift = −1, or predicting a label inconsistent with the baseline when shift = 0).

We also experimented with predicting continuous shift magnitudes and deriving final predictions from baseline + shift, but found that this consistently degraded performance. For prompted Llama-3B and Qwen-9B, F1 decreased by 0.031 and 0.071, respectively, and balanced accuracy by 0.074 and 0.030. Similar degradation was observed across other models. We hypothesize that estimating precise demographic effect sizes is more challenging than predicting their directions, especially without explicit supervision. Besides, the required shift magnitude depends on the baseline score and whether the shift is sufficient to change the final label, which further complicates the task.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank all reviewers for their constructive feedback. As promised in our individual responses, we have uploaded a revised paper that integrates the new statistics and expanded discussions. Below we first summarize the major revisions and then map each reviewer's concerns to the corresponding changes or clarifications.

Summary of major revisions

  1. Comparison results with corrected significance and win counts (Tables 1 and 2). Below the raw scores of NoDemo, DemoCond, and DemoReason, we now report the mean paired improvement for each comparison together with the number of improvements across model–dataset pairs. Cells are shaded by the direction and consistency of the improvement, which makes the aggregation effect and the effect of explicit reasoning directly visible. Significance is assessed with paired t-tests and paired bootstrap tests under Holm–Bonferroni correction.
  2. Related work in the main text (Section 2; Table 5 in Appendix A). A new Related Work section positions our study with respect to prior findings at different evaluation levels, baseline–deviation decompositions, and reasoning for pluralistic modeling. Table 5 in Appendix A summarizes the setups and findings of prior work.
  3. Revised descriptions of results (Sections 4.1, 5.1; Limitations). We emphasize the main trends while still describing important details, and align all claims with the corrected statistics and win counts. A new paragraph in Limitations discusses statistical power and why some consistent improvements are not significant.
  4. Self-contained intervention analysis (Section 5.4, Table 4; Appendix G). A new summary table reports joint, baseline-only, and shift-only interventions in the main text.
  5. Raw per model–dataset results and exceptions (Appendix D, Tables 9–14). All unaggregated performance and baseline-estimation results are reported, with a discussion of the consistency of the observed trends.

How we addressed each reviewer's concerns

Reviewer oKHc

ConcernRevision / Clarification
Multiple-testing correctionAll tests in Tables 1/2 and Figure 3 are now Holm–Bonferroni corrected. Claims throughout the paper are revised to match the corrected results. Limitations section discusses the statistical power.
Aggregation across models and datasetsThe newly added win counts and Appendix D (Tables 4-14) with unaggregated model-dataset pair results confirm that the findings hold across models and datasets rather than being driven by outliers.
Directional shift values in {−1, 0, 1}Section 3 (footnote) clarifies that shift encodes only the direction of the demographic effect. We provide supporting statistics in the response to show that such directional shifts can be leveraged effectively by LLMs while forcing a magnitude estimate worsens performance.

Reviewer ypDJ

ConcernRevision / Clarification
Related work limited and relegated to the appendixNew main-text Related Work section (Section 2) now motivates our three research questions and positions our contributions. Table 4 (Appendix A) summarizes the setups and findings of 11 prior studies and contrasts them with ours.
Writing does not match the dataSection 4.1 is revised to report both the overall trends and important detailed statistics; Section 5.4 now includes Table 4, which summarizes the joint, baseline-only, and shift-only interventions, along with a self-contained discussion of the results.
Dataset- and model-level exceptionsWe discuss the consistency of our findings and their exceptions in the response and Appendix D, alongside raw model–dataset results for readers to examine individual cases.

Reviewer y3F7

ConcernRevision / Clarification
Weak empirical support for DemoReason > DemoCondSections 4.1 and 5.2 now provide win rates across model–dataset pairs to support the consistency of the finding, with revised discussions that interpret the results more rigorously and explicitly clarify the scope.
Definition of crowd and targetSection 3 now explains crowd and target and what values they take.
Where to find reported valuesTable 1 now reports mean paired improvements over NoDemo directly below the raw scores, so these values (Acc gains of DemoCond/DemoReason at the group and population levels) can be read off the table.
Motivation for the decompositionSection 2 grounds the baseline–deviation decomposition in psychological findings and in prior NLP models that separate shared and annotator-/group-specific effects.

We hope these revisions adn clarifications address the reviewers' concerns, and we are happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.