- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors' primary research focuses on how the same social stereotype manifests in fairness evaluations across different tasks and modalities; the paper is essentially a benchmarking study. Specifically, the authors first clarify that the research scope focuses on "stereotype semantics." To enable cross-task and cross-modality evaluation, they designed the Bias Semantic Unit (BSU), which encompasses contexts, protected attributes, target groups, bias expressions, and harm categories. Crucially, the BSU decouples stereotype semantics from specific evaluation tasks and input modalities, allowing a single BSU to generate corresponding evaluation instances across various settings. After organizing these units, the authors constructed an evaluation dataset derived from two sources: real-world news and existing academic fairness benchmarks. They built "BiasAtlas" through LLM-assisted extraction, normalization, and deduplication, selecting 3,075 representative BSUs to generate the "OmniFair Benchmark." For each BSU, the authors generated test cases covering five task paradigms and three input modalities (resulting in 15 evaluation instances per BSU), yielding a total of 46,125 evaluation instances (3,075 × 15). Subsequently, 15 models were tested on their supported tasks and modalities, with results converted into binary classifications (biased vs. unbiased) based on task-specific rules; this process yielded the paper's core evaluation results. Additionally, the authors validated the benchmark's construction quality and the reliability of the evaluation results through human annotation and consistency checks involving LLM judges.
Based on the experimental results, the paper presents 12 findings across six key areas: differences in bias rates across models, tasks, and modalities; changes in bias when a unified BSU is applied to different tasks; near-zero cross-task correlations but higher cross-modal correlations in category-level bias-rate rankings (context, protected attribute, and harm category); the relationship between bias and the direction of the stereotype; differences in bias between real-world news and academic datasets; and differences in observed bias between explicitly spoken group identities and identities conveyed solely through voice timbre.
Critical strengths
-
Cross-task and cross-modal semantic-source alignment: The proposed BSU (Bias Semantic Unit) and its associated cross-task, cross-modality projection scheme offer both novelty and practical value. The authors attempt to abstract stereotype-related semantics from fairness evaluations into a unified representation—independent of specific tasks or modalities—enabling the generation of corresponding test instances for the same stereotype across different evaluation settings. This design establishes a unified foundation for tracking model performance regarding a specific bias semantic under varying evaluation conditions and offers significant potential for scalability.
-
Comprehensive data construction and quality validation: The paper involves extensive data engineering, encompassing 143 academic benchmarks, approximately 11.4 million news articles, 416K unique BSUs, and 46,125 final test instances, thereby providing substantial data support to the fairness research community. Furthermore, for components involving LLM assistance, the authors employed human annotation, verified "Judge" consistency, and conducted robustness analyses on specific metrics, providing supporting evidence for the quality of the constructed benchmark.
Critical weaknesses
-
Insufficient justification regarding the comparability and scientific significance of cross-task fairness evaluation results: One of the paper's core findings is that the same BSU yields inconsistent bias assessments across different tasks and modalities. However, while the authors align the semantic source of the test content (the same BSU), the specific fairness-related behaviors being measured remain task-specific. For instance, consider a BSU concerning the technical competence of older employees: the OG task measures whether the model generates stereotypical content under subtle elicitation prompts; the MCQ task measures whether the model relies on group identity to make judgments when information is insufficient; the DT task measures the model's ability to recognize stereotypes; and the PC task measures whether the model assigns different toxicity, offensiveness, or bias scores to otherwise identical statements referring to different groups. These tasks not only require distinct model behaviors and capabilities but also rely on markedly different operational definitions of bias. Consequently, even though these tasks share the same BSU, inconsistent results do not necessarily indicate inconsistency in the model's underlying fairness performance, nor do they necessarily constitute a novel scientific discovery. I believe a primary issue with the paper is that, while it possesses a solid core proposition and framework (BSU), it attempts to address too many questions; the interpretation of cross-modal and cross-task differences remains insufficiently clear, and the cross-task consistency analysis reduces task-specific outcomes to binary bias labels—a metric far too coarse for the scope of the investigation. The authors need to better justify the rationale behind these cross-task comparisons and articulate what new insights regarding fairness research can be gleaned from the observed inconsistencies.
-
The depth and novelty of the experimental findings are relatively limited: Although the paper reports 12 findings covering tasks, modalities, model scales, "Thinking" configurations, and data sources, most of these remain descriptive—focusing on differences in bias rates and consistency across conditions. The paper lacks in-depth analysis regarding the underlying causes of these differences or the broader principles of fairness they might reveal. For instance, variations in cross-modal bias rates could stem from a combination of model perception capabilities, refusal behaviors, and actual bias manifestations. Although Finding 12 discusses perception failure and Appendix I.4 analyzes refusal rates, I find their implications for the broader cross-task and cross-modal results insufficiently developed. Consequently, I feel the extensive experimental coverage has not yet translated into sufficiently profound scientific insights regarding AI fairness.
-
There is room for improvement in the paper's presentation: In terms of structure, the paper reads more like a technical report; the authors devote considerable space to listing the 12 findings individually, yet the connections between them are weak, causing the core narrative and key scientific conclusions to lose prominence. Regarding the figures, the text in some is too small, and certain elements lack clarity even when zoomed in, which somewhat detracts from the reading experience.
Questions for authors
-
Regarding the comparability of cross-task evaluations: Could the authors further clarify what semantic-source alignment makes comparable across tasks and what conclusions about model fairness can be drawn from the observed differences? Given the differing operational definitions of bias across OG, MCQ, DT, PC, and CT, how do the authors distinguish between inconsistencies arising from differences in the tasks' measurement objectives and genuine, scientifically significant differences in model fairness? This distinction seems crucial for understanding the paper's core conclusions.
-
Interpretation of key findings: What concrete implications do the observed cross-task and cross-modal differences have for the design and interpretation of future fairness evaluations? In particular, how should practitioners use these findings when choosing evaluation paradigms or interpreting aggregated bias rates?
Reviewer–Author Discussion Period Will End Soon
Dear Reviewer wWgv,
The reviewer–author discussion period will end soon. We would be grateful to hear whether our responses and revised manuscript address your concerns, and we remain happy to clarify any unresolved questions.
Your feedback prompted us to sharpen the paper substantially. We now frame OmniFair as a semantically aligned benchmark for testing whether task-local fairness conclusions serve as empirical proxies across evaluation forms, rather than as interchangeable measurements of a latent fairness capability. We also added the paired Direct OG and neutral grounding controls requested in your questions, incorporated controlled task comparisons and extended consistency analyses, and reorganized the manuscript around this focused claim.
At its core, OmniFair asks whether a favorable fairness result in one task or modality provides evidence about how the same discrimination semantics will manifest in another. We believe this question matters because models serve people across many forms of interaction, while harm does not cease to matter when its technical form changes.
We sincerely appreciate your time and consideration.
Warm regards,
The Authors