acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking and Evaluating Implicit Biases in Single- and Multi-Agent Language Models

Abstract

As large language models (LLMs) are increasingly used for decision support, planning and multi-agent societal simulation, evaluating bias in isolated outputs is not sufficient. We also need to measure how bias signals implicitly affect model decisions. Existing bias benchmarks primarily focus explicitly on single-agent settings, leaving limited tools for studying whether multi-agent settings may mitigate, preserve, or amplify such signals. We introduce SIFT, the Surfacing Implicit bias in Foundation models Toolkit, a unified benchmark and evaluation framework for measuring implicit stereotype signals in both single- and multi-agent LLMs. SIFT-Bench contains theory-grounded decision-making scenarios spanning five task families, eighteen domains, eleven social axes and six evidence conditions, yielding 26.4K queries. SIFT-Eval evaluates models across persona and interaction conditions using metrics for stereotype rate, counterfactual and style inconsistency, demographic reasoning and interaction shifts. Across model families, we find that persona conditioning can sharply increase stereotype-consistent decisions. Additionally, multi-agent settings show +12.4 percentage points higher stereotype rates than matched single-agent evaluations in participant-centered tasks like collaborative planning and task allocation. However, initial-to-final analyses reveal both amplification and attenuation depending on the interaction condition and domain. Together, SIFT enables controlled assessment of implicit bias in model decisions, its evolution through interaction, and agentic systems before deployment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Multi-Agent Systems, Social Bias, Stereotypes, Implicit Bias, Fairness Evaluation, Bias Benchmarking, Counterfactual Evaluation, Style Robustness, Agentic AI, Decision-Making, Persona Prompting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rnsKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SIFT, a benchmark and evaluation framework for studying implicit stereotype signals in single- and multi-agent LLM decision-making. Its main contribution is to combine diverse decision tasks with controlled evidence strengths, demographic-label swaps, linguistic-style variants, and interaction conditions. Across multiple model families, the evaluation reports substantial persona effects and condition-dependent changes following interaction, including both amplification and attenuation of stereotype-consistent decisions.

Critical strengths

The research question is well motivated. Evaluating whether demographic and linguistic cues influence decisions despite available task evidence is relevant to agent applications. Extending this evaluation to interacting agents adds a useful perspective beyond isolated model outputs. The benchmark combines useful controls. Evidence-strength manipulation, demographic swaps, style variants, and initial-to-final comparisons form a coherent evaluation framework. Their integration across five task families and multiple model families gives the benchmark practical value. The paper presents nuanced observations. It reports both increases and decreases in stereotype-consistent decisions, distinguishes demographic mentions from biased reasoning, and acknowledges that single–multi setting differences are not causal estimates. These distinctions improve the interpretation of the results.

Critical weaknesses

The connection between the main metrics and the claimed bias effects needs stronger analysis. Pooled SR combines stereotype-consistent choices under equal evidence with choices that override stronger evidence, although these represent different behaviors. The evidence-stratified analyses and EA are helpful, but the main findings do not clearly establish which behavior drives the reported persona and interaction effects. A joint analysis of evidence alignment and counterfactual choice direction would better distinguish demographic preference from general decision errors and more directly demonstrate the value of SIFT’s evidence controls. The interaction findings remain difficult to attribute to specific factors. The paper appropriately qualifies the +12.4-point comparison as a setting-level difference. However, participant framing is tied to planning and allocation, while other tasks use committee framing, making task and role effects difficult to separate. Initial-to-final comparisons are useful, but do not distinguish peer discussion from additional reconsideration. More targeted controls would strengthen the conclusions about when and why interaction increases stereotype risk. Perturbation sensitivity needs clearer validity and stability controls. Choice-flip rates lack a repeated-unchanged-prompt baseline, so ordinary model variability may contribute to the reported sensitivity. Moreover, confidence or hedging changes can alter decision-relevant credibility in evidence-ranking tasks even when factual content is preserved. The unchanged style pairs acknowledged in Appendix B.6.1 further motivate evaluating genuinely applied, task-valid perturbations. These checks matter directly to the claim that SIFT separates demographic-label and linguistic-style influences.

Questions for authors

On matched items, how do the main persona and interaction effects differ between equal-evidence and uniquely stronger-evidence scenarios when SR, EA, and counterfactual choice direction are examined jointly? Do the task-specific interaction patterns persist when role framing is held fixed, and do initial-to-final shifts persist against a comparable no-discussion or self-revision control? What are the choice-flip rates when prompts are repeated unchanged, and do the sensitivity conclusions hold on genuinely modified pairs validated for preserving task-relevant information?

6 comments
AuthorsOpenReview ↗

Official Comment by Authors (1)

Thank you to the reviewer for appreciating the motivation of our work, the demographic and linguistic probing, single and multi-agent interaction settings and the comprehensive controls and settings in the paper.

CW1: The connection between the main metrics and the claimed bias effects needs stronger analysis. Pooled SR combines stereotype-consistent choices under equal evidence with choices that override stronger evidence, although these represent different behaviors. The evidence-stratified analyses and EA are helpful, but the main findings do not clearly establish which behavior drives the reported persona and interaction effects.

AR: Thank you for raising this. The pooled SR definition is intentional, not a mixture of arbitrary errors in outputs. SR includes only cases where selecting the stereotype-favored option is not evidence-justified - (1) equal and near-equal cases measure demographic preferences when evidence leaves discretion. (2) target/minority-stronger evidence cases measure stricter cases in which stereotype-consistent choice overrides countering evidence. We also perform position balancing for robustness. In the paper, we already report these strata and a separate evidence-alignment metric. We will move the decomposition into the main results and make it more prominent.

Evidence subsetNo persona SRTarget/minority persona SRComparator/majority persona SR
Equal evidence13.9%13.4%33.2%
Near-equal evidence39.3%36.9%43.1%
Target/minority stronger1.3%1.4%2.3%

Note that these numbers are averaged across all tasks, decision conditions, stereotype axes and so on. However, this shows that clear counter-evidence can constrain overall stereotype-consistent choice, although specific task/axis differences remain.

CW2 A joint analysis of evidence alignment and counterfactual choice direction would better distinguish demographic preference from general decision errors and more directly demonstrate the value of SIFT’s evidence controls.

AR: We performed the above joint analysis. Using base/counterfactual matched multi-agent pairs, we use scenario-model intersections and compute paired SR, demographic following (whether the selected option changes after the label swap), and evidence alignment in both variants.

We also perform persona contrasts, which is a separate comparison. After calculating the above base/counterfactual metrics within each condition, we compare the final-private rate under a homogeneous persona panel with the rate under no persona on exact scenario--model intersections.

Final-stage persona comparisonEvidencePair-averaged SR: no persona -> persona (difference, 95% CI)Demographic following: no persona -> persona (difference, 95% CI)Both-variant EA: no persona -> persona (difference, 95% CI)
Comparator/majority versus no personaEqual12.6% -> 44.9% (+32.4 pp [+24.3,+41.2])10.6% -> 26.6% (+15.9 pp [+5.2,+28.0])n/a: no unique evidence winner
Comparator/majority versus no personaTarget/minority stronger17.2% -> 39.4% (+22.1 pp [+15.0,+28.6])5.3% -> 8.2% (+2.9 pp [-0.7,+6.7])77.6% -> 74.8% (-2.9 pp [-8.6,+2.2])
Target/minority versus no personaEqual15.1% -> 7.5% (-7.5 pp [-12.6,-3.4])10.0% -> 28.3% (+18.3 pp[+6.1,+30.0])n/a: no unique evidence winner
Target/minority versus no personaTarget/minority stronger17.7% -> 12.4% (-5.3 pp [-9.8,-0.9])5.1% -> 13.9% (+8.8 pp [+3.2,+15.1])77.8% -> 69.6% (-8.2 pp [-15.3,-2.0])

These results show why the three metrics should not be collapsed. Majority personas increase paired SR across both evidence settings, while demographic following is concentrated under equal evidence (this is intuitive). Minority personas lower SR but also increase target-group following, showing that demographic sensitivity can move independently of stereotype direction. Under stronger evidence, the EA results differentiate demographic following from evidence-based choice. Together, this analysis demonstrates the value of our controls: SR captures stereotype-consistent risk, counterfactual direction captures demographic-label following, and EA captures whether those choices are supported by evidence.

AuthorsOpenReview ↗

Official Comment by Authors (2)

CW3: The interaction findings remain difficult to attribute to specific factors. The paper appropriately qualifies the +12.4-point comparison as a setting-level difference. However, participant framing is tied to planning and allocation, while other tasks use committee framing, making task and role effects difficult to separate. Initial-to-final comparisons are useful, but do not distinguish peer discussion from additional reconsideration. More targeted controls would strengthen the conclusions about when and why interaction increases stereotype risk.

AR: Thank you. We already agree that the cross-task comparison does not identify a causal role-framing effect. The paper states that Figure 5 contains setting-level differences rather than a causal effect, and the initial-to-final analysis holds scenario, model, agent role, and condition fixed. Therefore, the current evidence supports specific interaction shifts, not a universal causal claim.

Additionally, we do matched no-persona comparison removes the role-framing variation. Under this fixed frame, multi-agent final minus single-agent SR is +3.7 pp for planning, +3.0 for allocation, +2.3 for hiring, +0.8 for credit, and +0.8 for evidence ranking. The confidence intervals for the last two include zero.

Finally, we also compute final-minus-initial change within the same fixed external-committee multi-agent protocol. The task-level changes are +1.5 pp for planning, +0.8 for allocation, +1.4 for hiring, +0.1 for credit, and +0.4 for evidence ranking. So, the clearest change is in planning and hiring and the last two include zero in CIs. So, in the updated version, we will remove wording that attributes the task pattern to participant framing or to a universal peer-discussion penalty.

CW4: Perturbation sensitivity needs clearer validity and stability controls. Choice-flip rates lack a repeated-unchanged-prompt baseline, so ordinary model variability may contribute to the reported sensitivity. Moreover, confidence or hedging changes can alter decision-relevant credibility in evidence-ranking tasks even when factual content is preserved. The unchanged style pairs acknowledged in Appendix B.6.1 further motivate evaluating genuinely applied, task-valid perturbations. These checks matter directly to the claim that SIFT separates demographic-label and linguistic-style influences.

AR: Thank you. We agree that a same prompt baseline improves calibration. But we want to clarify that the perturbation metrics are behavioral sensitivity measures rather than causal estimates and existing stereotype direction analysis separates demographic and presentation perturbations.

We now evaluate style pairs in the stratified single/multi panel to identical text variants as the pipeline noise baseline. These are independently evaluated text pairs, and we bootstrap complete source families.

SettingOutcomeIdentical textVerified changedChanged minus identical, 95% CI
Single-agentChoice flip2.9%6.0%+3.2 pp [+2.2,+4.1]
Single-agentStereotype-direction flip1.6%3.5%+1.8 pp [+1.2,+2.5]
Multi-agent finalChoice flip26.9%29.5%3.4 pp [+2.7,+4.1]
Multi-agent finalStereotype-direction flip9.9%15.5%5.4 pp [-1.7,+7.8]

The single-agent and multi-agent choice flip effects are certain under calibration. The multi-agent stereotype direction result does not. We therefore report demographic counterfactual direction changes separately.

We will make three changes in the revised version: (1) Use independently evaluated identical-text variants as the pipeline-noise baseline; (2) Restrict primary presentation sensitivity to verified-applied transformations; (3) Separate confidence/hedging from neutral/register/dialect transformations. The text audit also confirms that 36.8% of generated style pairs changed text. So, confidence/hedging will now be labeled a more pragmatic intervention and reported separately from other transformations.

Q1: On matched items, how do the main persona and interaction effects differ between equal-evidence and uniquely stronger-evidence scenarios when SR, EA, and counterfactual choice direction are examined jointly?

AR: We perform experiments for this in the previous weaknesses section on joint analysis and share that all SR, EA and counterfactual directions capture distinct behaviors.

AuthorsOpenReview ↗

Official Comment by Authors (3)

Q2: Do the task-specific interaction patterns persist when role framing is held fixed, and do initial-to-final shifts persist against a comparable no-discussion or self-revision control?

AR: Regarding the fixed-role part, all no-persona tasks use external-committee framing. Under this common frame, matched multi-final minus single-agent estimates remain positive for three tasks: planning (+3.7 pp), allocation (+3.0), and hiring (+2.3), and uncertain for two: credit (+0.8) and evidence ranking (+0.8). Within the same multi-agent runs, final-minus-initial changes are +1.5 pp for planning, +0.8 for allocation, +1.4 for hiring, +0.1 for credit, and +0.4 for evidence ranking. These show that single/multi-agent gap and within-deliberation shift are different estimands.

We also perform self-revision experiments with Qwen-3-4B and Mistral-7B. Here, each agent reconsidered the same scenario using only its own initial response, without peer messages, and we compared this revised answer with its matched post-discussion answer. On 300 matched prompts, peer final minus self-revision SR is -2.9 pp [-3.0,-1.2] for Qwen3 4B and -1.7 pp [-2.5,-1.0] for Mistral 7B. Both intervals exclude zero. And self-revision changes SR by +2.8 pp [+1.0,+2.9] for Qwen and +2.3 pp [+0.6,+3.3] for Mistral. So, the observed SR shifts do establish an effect of peer discussion. DR differences are model-specific rather than general: peer minus self-revision is +4.9 pp for Qwen and -4.7 pp for Mistral.

Q3: What are the choice-flip rates when prompts are repeated unchanged, and do the sensitivity conclusions hold on genuinely modified pairs validated for preserving task-relevant information?

AR: We shared these results in the perturbation sensitivity section above. Overall, the single-agent and multi-agent choice flip effect survives calibration. The multi-agent stereotype direction result does not. We will describe that as general repeated-pipeline/deliberation instability. Thank you!

Reviewer_hCsvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a benchmark and an evaluation framework for measuring implicit stereotype signals in LLMs decision-making, considering both single-agent and multi-agent settings. Specifically, it investigates whether demographic attributes can affect the model decisions even when task-relevant evidence should guide the decision-making process. The main contributions are SIFT-Bench and SIFT-Eval. The first is a benchmark consisting of ~26K examples spanning different tasks, domains, social axes and evidence conditions. SIFT-Eval provides a set of metrics to assess stereotype-consistent decisions, evidence alignment, etc. For the evaluation, different LLMs have been used and tested on the provided benchmark. The experiments consider both single-agent and multi-agent settings, with and without demographic personas.

Critical strengths
  1. The paper is well-written and easy to follow. It addresses a relevant and timely problem, by providing a framework for measuring how demographic attributes can affect LLMs decision even when the decisions should be guided by task-relevant evidence.

  2. In the preparation of the benchmark, the authors introduce two sensitivity probes related to counterfactual label swaps and linguistic style rewrites. These probes are relevant as they allow to investigate whether changes in demographic attributes or linguistic presentation can alter the decisions while preserving the task-relevant evidence and the underlying semantics, respectively.

  3. The benchmark is comprehensive, spanning different tasks, social axes and evidence conditions. The paper also extends the evaluation to multi-agent settings, investigating whether interactions among agents may amplify or attenuate stereotype-based decisions.

Critical weaknesses

Although SIFT-Benck comprises ~26K examples, the evaluation of the multi-agent setting, and the related comparison with the single-agent setting, is conducted on a stratified subset of only 4.5K examples.

Questions for authors
  1. In the multi-agent setting, it is not clear how the final decision is made. Is it based on majority voting over the final individual decisions? Or have other aggregation approaches been used? How are ties handled when an even number of agents is used?

  2. In the committee setting, the number of agents is fixed to three, and the interaction consists of two rounds. How might the results be affected by increasing the number of agents that interact? Are two discussion rounds sufficient with a larger number of agents? Or also the number of rounds needs to be increased?

2 comments
AuthorsOpenReview ↗

Official Response by Authors

We thank the reviewer for their positive assessment of the benchmark probes, comprehensiveness, and readability, and for appreciating the relevance and timeliness of the problem.

CW1: Although SIFT-Benck comprises ~26K examples, the evaluation of the multi-agent setting, and the related comparison with the single-agent setting, is conducted on a stratified subset of only 4.5K examples.

AR: We appreciate the concern about using 4.5k prompts for multi-agent evaluation. However, this is substantial for multi-agent evaluation. It has 2743 unique source scenarios and was sampled using joint task--axis--evidence stratification. So, it retains all variations from the main dataset: 5 task families, 18 domains, 10 axes, 6 evidence conditions, 6 decision formats and 4 option counts. The maximum distribution deviation from the full pool are: Task (2.74 p), Social axis (2.22 pp), Evidence condition (1.08 pp) and Domain (0.79 pp).

All single-versus-multi comparisons are restricted to exact-matched items within this subset, and bootstrap intervals are used to quantify uncertainty. Therefore, we do not run a full 26.4K multi-agent rerun. Additionally, it also adds very high overall computation.

We also added an independently sampled seed-29 no-persona panel containing 500 prompts from 123 distinct scenarios. Comparisons use matched single-agent and final-private multi-agent responses.

ModelMetricMatched nSingle agentMulti-agent finalMulti minus single, 95% CI
Qwen3 4BSR49930.5%31.3%+0.8 pp [+0.7,+2.0]
Qwen3 4BDR5004.0%15.0%+11.0 pp [+8.1,+14.2]
Mistral 7BSR49521.8%22.5%+0.7 pp [+0.2,+2.2]
Mistral 7BDR50058.0%51.8%-6.2 pp [-12.1,-0.1]

For SR, models show a statistically distinguishable single-to-multi SR change on this independent seed. However, DR moves in opposite directions across the two models. So it does not support a universal effect. So, these findings reinforce the importance of reporting model-specific estimates.

Q1: In the multi-agent setting, it is not clear how the final decision is made. Is it based on majority voting over the final individual decisions? Or have other aggregation approaches been used? How are ties handled when an even number of agents is used?

AR: Thank you for your question. The primary outcome is not a majority vote. In multi-agent settings: the agents make an initial private decision, participate in two discussion rounds and submit a final private decision without seeing other agents’ final answers. Primary SR and evidence-alignment metrics use final-private agent outcomes. Agents are averaged within an item for matched single-vs-multi comparisons. So, an even number of agents would not create a tie in the primary outcome. We also save a consensus summary but it is not considered in primary results.

Q2: In the committee setting, the number of agents is fixed to three, and the interaction consists of two rounds. How might the results be affected by increasing the number of agents that interact? Are two discussion rounds sufficient with a larger number of agents? Or also the number of rounds needs to be increased?

AR: Thank you for this question. Yes, the committee settings use three agents. But the participant-centered persona settings use 2-5 agents according to the number of options in a specific scenario. So, the benchmark includes group-size variation. Two rounds were selected as a controlled protocol and also based on previous works on interaction analysis. But, we do not claim that two rounds are universally sufficient or optimal.

To test whether the main results depend on the chosen interaction settings, we conducted a fixed-item ablation on Qwen3-4B and Mistral with 100 prompts per setting. We only varied group size and round count, keeping everything else fixed.

ModelAgentsRoundsFinal SRFinal DRFinal-minus-initial SR, 95% CIUnanimous panels
Qwen3 4B2221.0%7.5%+4.0 pp [0.0,+9.1]100.0%
Qwen3 4B3118.3%11.0%+2.0 pp [0.0,+5.1]99.0%
Qwen3 4B3217.1%10.3%+0.7 pp [-2.2,+4.0]99.0%
Qwen3 4B3318.0%13.7%+2.0 pp [-2.0,+7.2]100.0%
Qwen3 4B5218.6%13.5%+2.2 pp [-1.1,+6.0]98.0%
Mistral 7B2212.6%56.5%-2.5 pp [-7.1,+1.8]99.0%
Mistral 7B3114.7%60.3%-0.7 pp [-4.1,+3.5]95.0%
Mistral 7B3215.5%59.7%-2.0 pp [-7.6,+4.5]94.9%
Mistral 7B338.0%56.0%-5.0 pp [-10.0,+0.4]98.0%
Mistral 7B528.7%58.4%-7.7 pp [-12.9,-2.7]95.0%

These results do not show monotonic increase or decrease in SR with more agents or more rounds. The panel agreement is also near the ceiling for most cases. So, rounds/number of agents are not uniquely responsible for the result. Thank you!

Thank you for clarifying my concerns. I will keep my score.

Reviewer_5HNDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem that existing LLM bias benchmarks mostly measure explicit stereotype associations in single-model outputs, leaving unclear whether demographic cues implicitly shift actual decisions — and whether multi-agent interaction mitigates, preserves, or amplifies such shifts. The authors introduce SIFT (Surfacing Implicit bias in Foundation models Toolkit), comprising SIFT-Bench and SIFT-Eval.

Critical strengths

CS1. Benchmark design that separates stereotype-consistent choices from evidence-justified ones, with validated dual sensitivity probes. Every scenario is crossed with six controlled evidence conditions, SR is computed only on the subset where selecting the stereotype-favored option is NOT evidence-justified (Eq. 1, Sec. 6; evidence-stratified results in Table 14 / Appendix F.4), and evidence alignment is measured separately (Eq. 2, Appendix F.5). Counterfactual label swaps and six directional style rewrites are position-balanced, bidirectional, and audited with structural preservation checks (99.96% numeric-token/option-label preservation; 99.82% within-option) and bidirectional NLI (95.3% pass overall; 92.7%/87.4% on the 207 pairs with actual textual changes; Appendix B.6.1). Per Table 1, no prior benchmark (BOLD, BBQ, HolisticBias, MALIBU, FairMT-Bench, McBE, ImplicitBBQ, Neutral Is Not Unbiased) combines implicit-bias measurement, evidence-controlled decision formats, counterfactual AND style probes, and both single- and multi-agent protocols. This matters for acceptance because the label-vs-style-vs-evidence confound is exactly what limits prior benchmarks, and the design makes SIFT immediately usable for pre-deployment auditing of agentic systems.

CS2. Analytical honesty and inferential care beyond the norm for benchmark papers. The authors explicitly disclaim causal readings of setting-level contrasts (Fig. 5 caption; Finding 2: "setting-level differences rather [than] causal effect"), provide source-family-clustered paired bootstrap 95% CIs for all matched comparisons (Appendix I.1, Tables 22–24), manually validate the DR checker (119/120 agreement, κ=0.94; Appendix E.3), report chance-adjusted human agreement for all three validation studies (Gwet 0.85–0.89; κ 0.80–0.93; Table 7), and present interaction effects as heterogeneous (68 positive / 19 negative / 1 unchanged across 88 model-condition cells; Table 18, F.9) instead of a uniform amplification narrative. The trace-level taxonomy (Appendix H, Fig. 12: evidence-first correction, fairness disclaimer, identity-as-proxy reasoning, deliberative reinforcement, persona-aligned counterstereotypical selection, masked neutrality with 28.1%→35.2% stereotype-consistent tie-breaks) adds mechanism-level value that aggregate rates alone cannot provide. This matters because it makes the paper's directional findings credible and reusable even where exact magnitudes need correction.

Critical weaknesses

CW1. The headline quantitative claim is contradicted by the paper's own rigorous matched analysis, and identical aggregates take irreconcilable values across sections (fixable, but it currently undermines trust in every reported number — for a benchmark paper, the numbers are the product). (a) The abstract, Finding 2, Fig. 5A, and Appendix J (Finding 3) all state that multi-agent SR is +12.4 pp higher than matched single-agent evaluations for collaborative planning and task allocation and "lower for the other three tasks" (Fig. 5A: hiring −3.2, credit −3.5, evidence −2.7). However, the matched no-persona source-family bootstrap — the paper's most careful estimator — reports +2.2 pp overall [95% CI +1.6,+2.7] (Table 22) and per-task SR differences of +3.7 [+2.9,+4.6] (collaborative), +3.0 [+1.7,+4.3] (task allocation), +2.3 [+1.0,+3.7] (hiring), +0.8 [−0.3,+1.9] (credit), +0.8 [−0.0,+1.7] (evidence) (Table 24): all five tasks are positive, and hiring's CI excludes zero, directly contradicting "lower for the other three tasks"; the +12.4 magnitude is 3× the matched estimate, and the aggregation behind Fig. 5A is never stated. (b) Condition-level multi-agent aggregates conflict across sections: no-persona SR = 18.5% (F.2) vs 12.4% (H) vs 18.9% (G); homogeneous-minority = 11.3% (F.2) vs 12.9% (H) vs 12.8% (I); homogeneous-majority = 42.8% (F.2) vs 38.2% (H) vs 38.8% (I); homog-minority DR = 67.1% (H) vs 68.8% (I). My recomputation shows F.2's values equal the unweighted model-level means of Tables 15/16 (18.56, 42.76), so different sections silently use different weightings/denominators, and Fig. 4's annotated deltas (−7.1/+24.3) inherit the F.2 aggregation while Appendix H's narrative uses another. (c) Table 18 reports impossible negative Initial SRs (Gemma 4 31B Hom.-minority: −0.4; Qwen3 4B Dem.+pers.: −0.9, whose Δ=+18.1 is computed from the negative value, indicating upstream corruption rather than a typo). (d) Further unreconciled pairs: Finding 1's FDR "29.5%→40.6%" vs Table 10's 32.1→44.5 (F.1: "+12.4 pp"); Table 3 vs Table 14 single-agent SRs differ by up to 1.6 pp for identical cells (Llama 3.1 8B majority 41.1 vs 39.7; Qwen3 4B majority 31.1 vs 29.5; Gemma 3 4B none 10.2 vs 11.1) with no stated subset difference; F.7's religion "36.7% overall" cannot coexist with task-wise religion SRs of 66.1/54.9/48.1/48.0/44.7% (all ≥44.7) under any weighting of the same population. Fix: adopt one canonical panel and aggregation with explicit denominators, re-issue Figs. 4–6 and Tables 3/5/10/14/18 consistently, repair the negative-rate bug, and re-base the abstract on the matched estimates (or justify Fig. 5A's aggregation with CIs). Until this is done, a reader cannot determine which numbers are authoritative, which caps my rating.

CW2. The sensitivity (flip) metrics underlying Findings 3–4 lack any noise calibration, and the style-variant set appears to contain a majority of no-ops (fixable from existing data, but material). Every reported number comes from a single stochastic run per item (temperature 0.3; Appendix L: "We did not set a separate inference-time sampling seed"); there is no test-retest baseline anywhere in the paper, so counterfactual/style flip rates conflate perturbation sensitivity with decoding noise — noise that the multi-agent pipeline (initial decision + two discussion rounds + final decision, with randomized communication order) compounds. Moreover, Appendix B.6.1 discloses that of 600 sampled style pairs only 207 had "actual textual changes," implying ~65% of sampled style rewrites did not modify the text; flips on unchanged pairs are pure noise, and neither Table 13 nor F.3 restricts flip rates to verified-changed pairs. The paper's own numbers illustrate the risk: in multi-agent settings, label-swap choice flips (24.7%) and style-rewrite choice flips (24.3%) are nearly identical, which is equally consistent with generic deliberation-induced instability; no neutral-paraphrase control condition is run to separate identity-specific sensitivity from baseline instability. Fix: report flip rates restricted to pairs with verified textual change, add a same-prompt re-run (or unchanged-pair) flip baseline at temperature 0.3, ideally add a neutral-rewrite control arm, and recompute Findings 3–4 accordingly; without this, "multi-agent settings substantially increase both sensitivities" is not established beyond plausible noise inflation.

Questions for authors
  1. (material) Fig. 5A / the abstract report +12.4 pp multi-minus-single SR for collaborative planning and task allocation and negative differences for hiring (−3.2), credit (−3.5), and evidence ranking (−2.7), while the matched no-persona bootstrap (Tables 22/24) gives +2.2 pp overall and positive per-task differences for all five tasks (hiring +2.3, CI [+1.0,+3.7]). Please state exactly which data and aggregation Fig. 5A uses (which persona conditions, agent-level vs group-level outcomes, which subset), reconcile it with the matched analysis, and clarify which number the abstract claim should rest on.

  2. (material) Table 18 reports negative Initial SR values (Gemma 4 31B homogeneous-minority: −0.4; Qwen3 4B demographics+persona: −0.9), which are impossible for a rate, and the Δ column is computed from them (e.g., +18.1). Can you identify the pipeline error, provide corrected Init/Final/Δ values for these cells, and confirm whether the F.9 count (68 positive / 19 negative / 1 unchanged of 88) changes after correction?

  3. (material) Condition-level multi-agent aggregates differ across sections: no-persona SR 18.5% (F.2) vs 12.4% (H) vs 18.9% (G); homogeneous-majority 42.8% vs 38.2% vs 38.8%; homogeneous-minority 11.3% vs 12.9% vs 12.8%; homog-minority DR 67.1% (H) vs 68.8% (I); Finding 5's "+6.2 pp SR / +36.1 pp DR" is not reproducible from model-level means of Tables 15/16. Please state the denominator/weighting used in each location (unweighted model mean vs pooled scenario-level; which models and subsets) and provide one canonical aggregation table that all sections reference.

  4. (material) Appendix B.6.1 indicates only 207 of 600 sampled style pairs had actual textual changes. What fraction of the released style variants differ textually from their base in the targeted option? Can you report CFS/SS flip rates (Table 13, F.3) restricted to verified-changed pairs, together with a decoding-noise baseline (flip rate on unchanged pairs and/or a same-prompt re-run at temperature 0.3), so readers can calibrate the 24.7%/24.3% multi-agent flip rates?

  5. (clarification) The benchmark is validated by GPT-5.4-nano and Qwen-3-4B-Instruct (Sec. 4 Stage 4; B.2), while gpt-5.4-nano and Qwen/Qwen3-4B-Instruct-2507 are among the evaluated models (Tables 3/25) and generation used GPT-5.4-mini from the same family. How did you guard against circularity — e.g., did row-quality filtering rates differ systematically by evaluated model family, and could a held-out validator re-check a sample?

  6. (clarification) Sec. 7 states that you "fit grouped binomial logistic regressions ... adding stage terms for multi-agent runs" and that "stage has no significant adjusted main effect, but condition-by-stage effects are significant," but Appendix I reports only univariate diagnostics (Table 21 fits each factor independently against a null model). Where are the adjusted model with stage terms and the condition×stage interaction reported (coefficients, LRT statistics, sample)?

  7. (clarification) Please clarify run completeness per model×condition: (a) Llama 3.2 3B appears in Tables 14/19 and Fig. 9 but not in Table 3 or the backend list (Table 25); (b) Table 14 omits GPT-5.4 nano and Grok 4 Fast entirely and OLMo 2 7B's majority row; (c) F.3 says Grok 4 Fast's multi-agent file "does not yet contain matched benchmark pairs" and calls GPT-5.4 nano's data a "current partial multi-agent run," yet Tables 15/16/18 report complete-looking multi-agent numbers for both; (d) F.3's single-agent paired baselines (GPT-OSS 6.4%, Gemma 4 31B 6.6%) do not match Table 3's CFS None values (7.2%, 7.3%). Which cells are complete, how were partial runs handled in aggregates, and why do Tables 3 and 14 differ for identical cells (e.g., Llama 3.1 8B majority SR 41.1 vs 39.7)?

  8. (clarification) In F.7, religion SR is "36.7% overall" yet the task-wise religion SRs are 66.1% (hiring), 54.9% (evidence), 48.1% (credit), 48.0% (task allocation), and 44.7% (collaborative) — all above the overall value, which cannot hold for the same population; please specify the setting (e.g., homogeneous-majority panels?) and denominator for each figure. Relatedly, Finding 1 quotes FDR rising "from 29.5% to 40.6%" while Table 10 reports 32.1→44.5 (+12.4 pp): which aggregation is canonical, and can main text and appendix be unified?

6 comments
AuthorsOpenReview ↗

Official Comment by Authors (1)

Thank you to the reviewer for the detailed feedback. We appreciate your recognition SIFT’s evidence-controlled design and the validation of our counterfactual and style probes. Thank you for your attention to the trace-level analysis, which can help explain some variations in interaction shifts.

CW1: The headline quantitative claim is contradicted by the paper's own rigorous matched analysis, and identical aggregates take irreconcilable values across sections (fixable, but it currently undermines trust in every reported number — for a benchmark paper, the numbers are the product).

AR: Thank you for this comment. We agree that the current draft did not label some aggregation choices clearly. The findings, however, are reliable. The benchmark construction, validation, evidence controls, and exact-matched outputs are unaffected. We also reproduced the discrepancies, which arose during late-stage revision.

Our paper contains multiple estimands:

  1. A broad descriptive comparison across available single- and multi-agent conditions.
  2. A response-weighted pooled rate.
  3. An equal-weight model-macro rate.
  4. A strict scenario--model--variant matched no-persona comparison.

Therefore, the above may answer different questions and can differ. We will clarify this in the revised version. The corrected analyses retain the central trends--persona composition changes SR, interaction effects depend on condition, and removing evidence increases demographic fallback.

CW1 (a): The abstract, Finding 2, Fig. 5A, and Appendix J (Finding 3) all state that multi-agent SR is +12.4 pp higher than matched single-agent evaluations for collaborative planning and task allocation and "lower for the other three tasks" (Fig. 5A: hiring −3.2, credit −3.5, evidence −2.7). However, the matched no-persona source-family bootstrap — the paper's most careful estimator — reports +2.2 pp overall [95% CI +1.6,+2.7] (Table 22) and per-task SR differences of +3.7 [+2.9,+4.6] (collaborative), +3.0 [+1.7,+4.3] (task allocation), +2.3 [+1.0,+3.7] (hiring), +0.8 [−0.3,+1.9] (credit), +0.8 [−0.0,+1.7] (evidence) (Table 24): all five tasks are positive, and hiring's CI excludes zero, directly contradicting "lower for the other three tasks"; the +12.4 magnitude is 3× the matched estimate, and the aggregation behind Fig. 5A is never stated.

AR: We agree that Figure 5A should have been described precisely. It used a broad row-level aggregation over available single- and final-private multi-agent settings, without fixing the persona condition or model panel. So it describes contextual reweighting, not the effect of changing an identical run.

So, we will use the strict exact-matched no-persona estimate for the abstract: +2.2 pp [95% CI +1.6,+2.7] and remove the +12.4 pp value from the revised version.

Tables 22/24 answer the question about the matched condition: scenario, model, and variant are exactly matched under no-persona conditions, and each multi-agent item is represented by the mean of its final-private agent outcomes. We use this estimate for the abstract:

TaskMatched multi - single SR95% CI
Collaborative planning+3.7 pp[+2.9,+4.6]
Task allocation+3.0 pp[+1.7,+4.3]
Hiring/admission+2.3 pp[+1.0,+3.7]
Credit attribution+0.8 pp[-0.3,+1.9]
Evidence ranking+0.8 ppapproximately [-0.0,+1.7]

Using the above, we will remove the earlier statement about the decrease in the other three tasks. Matched percentage points are higher for multi-agents, but uncertain for two tasks. We will also update Figure 5 accordingly in the revised version.

CW1 (b): Condition-level multi-agent aggregates conflict across sections: no-persona SR = 18.5% (F.2) vs 12.4% (H) vs 18.9% (G); homogeneous-minority = 11.3% (F.2) vs 12.9% (H) vs 12.8% (I); homogeneous-majority = 42.8% (F.2) vs 38.2% (H) vs 38.8% (I); homog-minority DR = 67.1% (H) vs 68.8% (I). My recomputation shows F.2's values equal the unweighted model-level means of Tables 15/16 (18.56, 42.76), so different sections silently use different weightings/denominators, and Fig. 4's annotated deltas (−7.1/+24.3) inherit the F.2 aggregation while Appendix H's narrative uses another.

AR: Thank you. We regenerated the stage comparison, applied the paper's SR-eligible evidence filter, and required exact scenario--model--variant--agent intersections.

ConditionInitial SRFinal SRFinal - initial, 95% CI
No persona12.113.0+0.9 pp [+0.4,+1.3]
Hom. majority32.433.5+1.1 pp [+0.3,+1.8]
Hom. minority9.19.7+0.6 pp [+0.2,+1.0]
Persona/goal23.219.0−4.1 pp [−4.7,−3.5]
Persona/no-goal19.518.4−1.1 pp [−1.7,−0.5]

Figures 4-6 will use this table for initial/final claims, which still show certain differences between final and initial multi-agent settings. We will reconcile the differences between the Main and Appendix.

AuthorsOpenReview ↗

Official Comment by Authors (2)

CW1 (c): Table 18 reports impossible negative Initial SRs (Gemma 4 31B Hom.-minority: −0.4; Qwen3 4B Dem.+pers.: −0.9, whose Δ=+18.1 is computed from the negative value, indicating upstream corruption rather than a typo).

AR: This was a localized table-construction error, not a negative value in the raw outputs or metric calculation. The corrected Qwen3 cell is: Initial SR: 35.3% Final SR: 17.2% Final-minus-initial: -18.1 pp

For Gemma 4 31B, the correct numbers are: Initial SR: 0.4% Final SR: 0.0% Final-minus-initial: -0.4 pp

CW1 (d): Further unreconciled pairs: Finding 1's FDR "29.5%→40.6%" vs Table 10's 32.1→44.5 (F.1: "+12.4 pp"); Table 3 vs Table 14 single-agent SRs differ by up to 1.6 pp for identical cells (Llama 3.1 8B majority 41.1 vs 39.7; Qwen3 4B majority 31.1 vs 29.5; Gemma 3 4B none 10.2 vs 11.1) with no stated subset difference; F.7's religion "36.7% overall" cannot coexist with task-wise religion SRs of 66.1/54.9/48.1/48.0/44.7% (all ≥44.7) under any weighting of the same population.

AR: For FDR, the current versioned full-ranking output supports 29.5% → 40.6%, +11.1 pp. We will update Table 3/14 acc.

Regarding religion, we rebuild the decomposition on one explicit panel: no-persona, final-private responses, and the evidence conditions. The response-weighted overall religion SR is 42.2% (95% source-family bootstrap CI [34.4,50.0]); the same-panel task rates are 60.6% hiring/admission, 43.9% evidence ranking, 39.9% credit attribution, 35.1% task allocation, and 32.0% collaborative planning. We will update this.

Thank you for pointing these out: we agree that the above aggregations require clarifications, however, our overall trends still hold, showing the importance of the benchmark and findings.

CW2. The sensitivity (flip) metrics underlying Findings 3–4 lack any noise calibration, and the style-variant set appears to contain a majority of no-ops (fixable from existing data, but material). Every reported number comes from a single stochastic run per item (temperature 0.3; Appendix L: "We did not set a separate inference-time sampling seed"); there is no test-retest baseline anywhere in the paper, so counterfactual/style flip rates conflate perturbation sensitivity with decoding noise — noise that the multi-agent pipeline (initial decision + two discussion rounds + final decision, with randomized communication order) compounds. Moreover, Appendix B.6.1 discloses that of 600 sampled style pairs only 207 had "actual textual changes," implying ~65% of sampled style rewrites did not modify the text; flips on unchanged pairs are pure noise, and neither Table 13 nor F.3 restricts flip rates to verified-changed pairs. The paper's own numbers illustrate the risk: in multi-agent settings, label-swap choice flips (24.7%) and style-rewrite choice flips (24.3%) are nearly identical, which is equally consistent with generic deliberation-induced instability; no neutral-paraphrase control condition is run to separate identity-specific sensitivity from baseline instability. Fix: report flip rates restricted to pairs with verified textual change, add a same-prompt re-run (or unchanged-pair) flip baseline at temperature 0.3, ideally add a neutral-rewrite control arm, and recompute Findings 3–4 accordingly; without this, "multi-agent settings substantially increase both sensitivities" is not established beyond plausible noise inflation.

AR: The style-flip rate was not calibrated against ordinary pipeline baseline. But, it does not invalidate the counterfactual-label result: demographic counterfactual and presentation perturbations are separate, and the paper reports stereotype-direction changes in addition to generic choice movement. We now evaluate style pair in the stratified single/multi panel to identical text variants that are independently evaluated as a baseline.

SettingOutcomeIdentical textVerified changedChanged - identical, 95% CI
Single-agentChoice flip2.9%6.0%+3.2 pp [+2.2,+4.1]
Single-agentStereotype-direction flip1.6%3.5%+1.8 pp [+1.2,+2.5]
Multi-agent finalChoice flip26.9%29.5%3.4 pp [+2.7,+4.1]
Multi-agent finalStereotype-direction flip9.9%15.5%5.4 pp [-1.7,+7.8]

The single-agent and multi-agent choice flip effect survives calibration. The multi-agent stereotype direction result does not. We therefore report demographic counterfactual direction changes separately. The text audit also confirms that 36.8% generated style pairs changed text. So, confidence/hedging will be now labeled a pragmatic intervention and reported separately from register/dialect transformations. We will update this in the revision.

AuthorsOpenReview ↗

Official Comment by Authors (3)

Q1: (material) Fig. 5A / the abstract report +12.4 pp multi-minus-single SR for collaborative planning and task allocation and negative differences for hiring (−3.2), credit (−3.5), and evidence ranking (−2.7), while the matched no-persona bootstrap (Tables 22/24) gives +2.2 pp overall and positive per-task differences for all five tasks (hiring +2.3, CI [+1.0,+3.7]). Please state exactly which data and aggregation Fig. 5A uses (which persona conditions, agent-level vs group-level outcomes, which subset), reconcile it with the matched analysis, and clarify which number the abstract claim should rest on.

AR: We responded to this in CW1 above.

Q2: (material) Table 18 reports negative Initial SR values (Gemma 4 31B homogeneous-minority: −0.4; Qwen3 4B demographics+persona: −0.9), which are impossible for a rate, and the Δ column is computed from them (e.g., +18.1). Can you identify the pipeline error, provide corrected Init/Final/Δ values for these cells, and confirm whether the F.9 count (68 positive / 19 negative / 1 unchanged of 88) changes after correction?

AR: We also respond to this in CW1c) above (negative SR and reconciliation)

Q3: (material) Condition-level multi-agent aggregates differ across sections: no-persona SR 18.5% (F.2) vs 12.4% (H) vs 18.9% (G); homogeneous-majority 42.8% vs 38.2% vs 38.8%; homogeneous-minority 11.3% vs 12.9% vs 12.8%; homog-minority DR 67.1% (H) vs 68.8% (I); Finding 5's "+6.2 pp SR / +36.1 pp DR" is not reproducible from model-level means of Tables 15/16. Please state the denominator/weighting used in each location (unweighted model mean vs pooled scenario-level; which models and subsets) and provide one canonical aggregation table that all sections reference.

AR: This is answered under CW1 above. We agree that the manuscript should have labeled weighting and coverage in each location. Several conflicting pairs already have a direct explanation:

  • No-persona 12.4% is a response-weighted pooled rate over the available final-private rows.
  • No-persona 18.9% is approximately the equal-weight model-macro mean.
  • Homogeneous-majority 38.2% is response-weighted, while 38.8% is the model-macro mean.
  • Homogeneous-minority 12.9% and 12.8% are the corresponding response-weighted and model-macro estimates.
  • Homogeneous-minority DR 67.1% and 68.8% similarly reflect different weighting/coverage.

These are different estimands. The F.2 values came from an unweighted model panel and should not be mixed with the later panel. We will also reconstruct Finding 5 on the exact intersection between evidence-only and demographics-only runs. On this matched panel:

  • SR increases from 20.6% to 24.2%, +3.5 pp (95% CI [+1.8,+5.2]).
  • DR increases from 51.2% to 60.5%, +9.3 pp (95% CI [+7.2,+11.6]).

Thus, the qualitative conclusion that removing evidence increases demographic fallback is retained.

Q4: (material) Appendix B.6.1 indicates only 207 of 600 sampled style pairs had actual textual changes. What fraction of the released style variants differ textually from their base in the targeted option? Can you report CFS/SS flip rates (Table 13, F.3) restricted to verified-changed pairs, together with a decoding-noise baseline (flip rate on unchanged pairs and/or a same-prompt re-run at temperature 0.3), so readers can calibrate the 24.7%/24.3% multi-agent flip rates?

AR: The 207/600 value describes the sampled NLI audit, not by itself the released-dataset fraction. The full 3650 audit contains 8060/21900 changed pairs (36.8%) and 63.2% identical pairs. In the actually evaluated stratified response panel, all 19,371 style pairs joined the audit: 7079 were verified changed, 12262 were identical, and 30 failed structural checks. The calibrated rates are reported in CW2 above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment for All Reviewers

Thank you to the reviewers for their constructive comments. We revised the draft and incorporated the following changes:

Matched comparisons and consistency (Reviewer 5HND) We rebuilt the primary results from explicitly defined panels and now separate different estimands, such as full-dataset descriptive rates, response-weighted estimates, model macros, and exact-matched comparisons.

  • Abstract, Finding 2, and Figure 5 is now updated to use the exact-matched no-persona comparison (multi-agent final SR is +2.2 pp [1.6,2.7] above matched single-agent SR and task estimates are also updated)
  • In Finding 4, Figure 6, and Appendix F.2/F.10 (Tables 13--14), we use 215995 exact scenario--model--variant--agent matches.
  • In Appendix F.1 (Tables 10-12), we build full-ranking FDR
  • Appendix F.8 (Table 21) reports updated religion decomposition
  • Appendix I (Tables 27-30) adds adjusted regressions, condition-by-stage tests, matched contrasts, and source-family bootstrap CIs.

Evidence, SR, and counterfactual direction (Reviewer rnsK)

We made the distinction among metrics explicit in the draft:

  • In Section 6 and Appendix F.4, we explain that equal/near-equal SR measures stereotype-consistent preference under discretion, whereas target/minority-stronger SR measures choices which overrides counter-evidence. Also, EA is reported only when a unique evidence supported option exists (not equal cases)
  • In Appendix F.6 (Table 18), we add the joint evidence x counterfactual-direction analysis on exact final-stage intersections. It reports paired SR, physical-option persistence, demographic following, and EA. Results show that demographic following, stereotype direction, and evidence alignment are complementary metrics.

Interaction and protocol controls (Reviewers hCsv, rnsK)

  • We update Finding 2 and Appendix I.1 (Table 30), we keep role framing fixed using the common no-persona external-committee condition. These are setting-level comparisons and not causal estimates of interaction.
  • In Appendix K (Table 32), we add a 300-prompt no-peer self-revision control for Qwen3 4B and Mistral 7B. The findings show some impact of interactions, respectively. DR effects are more model-specific.
  • In Appendix K (Table 31), we add an independent seed-29 replication for both models.
  • In Appendix K (Table 33), we vary group size and discussion rounds across five configurations. Neither model shows a monotonic SR trend, and final agreement remains 94.9-100%.

The conclusion is also updated with main trends consistent with the previous version: persona conditioning can sharply alter decisions, while multi-agent interaction can amplify or attenuate stereotype behaviors depending on model, condition, evidence, and task. These findings are important as LLMs are increasingly used in consequential settings and in multi-agent systems, thus showing the need for controlled evaluation before deployment. We appreciate the reviewers’ suggestions, which improved the draft. We hope you will consider these revisions when making the final assessment.

Thank you,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.