- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to understand the relationship between pluralistic offensiveness modeling and the socio-demographic information of the people whose annotations are being modeled. Specifically, the authors study (1) when and where socio-demographics improve pluralistic modeling, and (2) the role of model reasoning.
The work uses three multi-annotation datasets (DICES, D3CODE, and POPQUORN) and studies the RQs on models that are either:
- prompted: Llama and Qwen models (3B–122B), GPT-4o-mini, and Haiku-4.5; or
- trained: 4 open-weight Llama and Qwen models (3B–9B).
In this setting, the authors raise a number of RQs and make several empirical observations to answer them:
- Socio-demographic information offers gains in pluralistic offensiveness modeling at the population level, more than at the finer-grained group or individual levels.
- The authors treat a judgment (an offensiveness label plus the relevant demographic-specific shift to that label) as a decomposition into two parts: reasoning about the baseline behavior appropriate for the general population (i.e., what most people would think) and the demographic-specific deviation. They find that this decomposition improves the accuracy of both the baseline estimate and the demographic-specific shift.
- Model reasoning that uses demographics performs better than demographic conditioning alone.
Critical strengths
Relevance and impact. This work is important and timely. It is potentially interesting to the ICLR community because it shows how socio-demographic information can be used to improve pluralistic modeling, which enables context-specific improvements to user experience.
Critical weaknesses
-
(Primary) Empirical validation. Some key claims are not corroborated sufficiently.
- Lines 222–224 claim: "Compared with simple demographic conditioning, explicit demographic reasoning generally yields larger gains, particularly at the group and population levels." In Table 1, the group and population levels together report 16 metrics (8 each for prompted and trained models). Only 3 of the 16 are marked as statistically significant for DemoReason over DemoCond. The statistics reported in Table 1 therefore only weakly corroborate the authors' claim.
- Similarly, in lines 302–307, Table 2 does not consistently show an improvement of DemoReason over DemoCond. "DemoReason matches DemoCond" may be a more appropriate claim than "DemoReason improves over DemoCond," because the improvement is not statistically significant in any case except one (Trained; W1). The claim "Explicit demographic reasoning thus strengthens baseline estimation beyond simple conditioning" is therefore under-powered.
-
(Less important than the empirical point above) Under-definition. Some important elements of the setup are incompletely defined, which slightly hinders interpretation and assessment of the paper. See the questions marked "Clarity" below.
Questions for authors
Clarity
- Some elements of Fig. 1 are unclear, and the description of "Variants of reasoning & prediction format" is incomplete. What is "crowd," and how should its magnitude be interpreted? Similarly, what is "target," and what labels/values can it take?
- Where and how can the numbers "0.006/0.009" and "0.007/0.017" be seen in Table 1?
- The authors define a judgment as a decomposition but do not motivate this enough. Why is it desirable or optimal to think of a judgment as an (additive) decomposition into a baseline and a demographic-specific deviation, rather than as a single quantity?
3 comments
Response to Reviewer y3F7 (2/N)
DemoReason provides an additional advantage over DemoCond in baseline estimation, particularly for prompted models. Table 2 shows that, for prompted models, DemoReason outperforms DemoCond on average in seven of the eight metrics, with improvements in 15–19 of the 27 model–dataset pairs. For trained models, DemoReason also achieves better average scores in six of the eight metrics, although the gains are smaller and less robust across model–dataset pairs, particularly for correlation metrics.
Table 2: Paired comparison of baseline estimation performance of DemoReason vs. DemoCond. For clarity, we report the mean paired improvement, the number of model–dataset pairs showing improvement, and the corrected significance results for each comparison. Positive values indicate better performance and are highlighted in bold. and denote statistical significance () under the paired t-test and paired bootstrap test, respectively, after Holm–Bonferroni correction.
| Setting | Comparison | Ann. MSE | Ann. MAE | Ann. ρ | Ann. W1 | Item MSE | Item MAE | Item ρ | Item W1 |
|---|---|---|---|---|---|---|---|---|---|
| Prompted | DemoReason vs. DemoCond | +0.002(19/27) | +0.003(18/27) | +0.009(19/27) | +0.005(18/27) | +0.002(16/27) | +0.002(15/27) | −0.003(16/27) | +0.004(19/27) |
| Trained | DemoReason vs. DemoCond | +0.000(7/12) | +0.001(6/12) | −0.009(5/12) | +0.008(9/12) | +0.001(8/12) | +0.003(9/12) | −0.004(6/12) | +0.006(9/12) |
In the revised paper, we will retain our main empirical findings while adopting a more cautious interpretation of their statistical support and the conditions under which each finding holds. We will also include the win rates as complementary metrics, as they provide evidence of directional consistency across model–dataset pairs.
W2/Q1: Clarification of reasoning and prediction format
represents the model's estimated proportion of a broad population that would consider the text offensive. For example, in Figure 1 indicates that approximately 40% of the population is expected to find the text offensive, meaning that the majority would consider it non-offensive. identifies the target of the potentially offensive content. It can take any of the target categories defined in the corresponding dataset, or "none" if no specific group or identity is targeted. We will clarify the meaning and possible values of each field in the revised paper.
W2/Q2: Clarification of reported values
0.006/0.009 and 0.007/0.017 are DemoCond/DemoReason - NoDemo in Acc at the group- and population-level in Table 1. We will report the mean improvements over NoDemo alongside raw scores in the revised paper, such that the improvements and overall trends are more visible.
W2/Q3: Motivation for decomposition
We adopt the baseline–deviation decomposition because it is supported by human pscychological studies and consistent with prior NLP work on pluralistic modeling.
First, psychological studies suggest that humans often predict others' perspectives by anchoring on an initial baseline and then adjusting for differences in social perspectives (Epley et al., 2004; Ames et al., 2004).
Moreover, similar decompositions are common in prior pluralistic modeling approaches, either explicitly or implicitly. Probabilistic and structural models explicitly distinguish shared effects from annotator- or group-specific variations, such as annotator biases and demographic effects (Paun et al., 2018; Chu et al., 2021; Liu et al., 2022). Recent NLP approaches implicitly capture this distinction in their model design through shared representations and annotator-specific heads (Mostafazadeh Davani et al., 2022), demographic representations (Zhang and Çöltekin, 2026), or demographic-aware experts (Xu et al., 2026).
Building on these approaches, we make the baseline–deviation decomposition explicit, allowing LLMs to reason about each component and enabling us to evaluate how socio-demographic modeling affects them separately.
References
- Epley et al. (2004). Perspective Taking as Egocentric Anchoring and Adjustment. Journal of Personality and Social Psychology.
- Ames (2004). Inside the Mind Reader's Tool Kit: Projection and Stereotyping in Mental State Inference. Journal of Personality and Social Psychology.
- Paun et al. (2018). Comparing Bayesian Models of Annotation. TACL.
- Chu et al. (2021). Learning from Crowds by Modeling Common Confusions. AAAI.
- Liu et al. (2022). Toward Annotator Group Bias in Crowdsourcing. ACL.
- Mostafazadeh Davani et al. (2022). Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. TACL.
- Zhang and Çöltekin (2026). Modeling Human Perspectives with Socio-Demographic Representations. Findings of ACL.
- Xu et al. (2026). Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives. ACL.
Dear Reviewer y3F7,
As the discussion phase will close in less than 48 hours, we would greatly appreciate it if you could take a moment to review our response and let us know whether our clarifications and additional experimental results have addressed your concerns. We would be more than happy to provide further clarification or address any remaining questions.
Response to Reviewer y3F7 (1/N)
Thank you for your constructive feedback! Below, we address your concerns in detail, with a brief TL;DR for each question. We are still incorporating new results and discussions into the revised manuscript, which we plan to upload by November 14. We will then post a general response to all reviewers summarizing the revisions.
TL;DR
[W1: Empirical validation] We report win rates across model–dataset pairs to show the consistency of our findings and explain why some trends are robust across pairs but do not reach statistical significance.
[W2/Q1: Clarification of reasoning and prediction format] We provide a more detailed explanation of each field in the reasoning and prediction format.
[W2/Q2: Clarification of reported values] 0.006/0.009 and 0.007/0.017 are DemoCond/DemoReason - NoDemo in Acc at the group- and population-level in Table 1.
[W2/Q3: Motivation for decomposition] The additive decomposition is motivated by findings in human psychology and is also widely used in prior NLP research.
Below we provide detailed responses to your concerns and questions.
W1: Empirical validation
We acknowledge that some improvements do not reach statistical significance, particularly after multiple-testing correction. However, we believe the empirical results still provide evidence of broadly consistent patterns that generalize across models and datasets. Below, we will first explain why some trends are consistent across model-dataset pairs but do not reach statistical significance. We will then report win rates across model–dataset pairs to support the consistency of our findings.
Why some improvements consistent across model-dataset pairs, but still do not reach statistical significance: First, the limited number of model–dataset pairs may reduce statistical power. For example, trained DemoCond improves group-level MSE over NoDemo by 0.019 (a 9.6% relative reduction from 0.198), with improvements in 11/12 model-dataset pairs, yet the difference is not significant. Second, our analyses of baseline and shift estimation show that demographic benefits are heterogeneous, being more pronounced for high-disagreement items and those with stronger text–demographic interactions. Such effects may be diluted when averaged across all examples.
Below, we revisit the key findings in the paper and report their consistency with win rates across model-dataset pairs.
Explicit demographic reasoning generally provides larger gains than direct conditioning, particularly for prompted models and population-level predictions. As shown in Table 1 below, for prompted models, DemoReason outperforms DemoCond on average in 8 of the 10 reported metrics, with larger and more consistent gains at the population level. For trained models, DemoReason also outperforms DemoCond on all four population-level metrics, with improvements observed in 9–10 of the 12 model–dataset pairs. The mean improvements and win counts provide consistent directional evidence of DemoReason's advantage in prompted models and population-level predictions.
Table 1: Paired comparison of individual-, group-, and population-level performance of DemoReason vs. DemoCond. For clarity, we report the mean paired improvement, the number of model–dataset pairs showing improvement, and the corrected significance results for each comparison. Positive values indicate better performance and are highlighted in bold. ∗ and † denote statistical significance (padj<0.05) under the paired t-test and paired bootstrap test, respectively, after Holm–Bonferroni correction.