- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces SIFT, a benchmark and evaluation framework for studying implicit stereotype signals in single- and multi-agent LLM decision-making. Its main contribution is to combine diverse decision tasks with controlled evidence strengths, demographic-label swaps, linguistic-style variants, and interaction conditions. Across multiple model families, the evaluation reports substantial persona effects and condition-dependent changes following interaction, including both amplification and attenuation of stereotype-consistent decisions.
Critical strengths
The research question is well motivated. Evaluating whether demographic and linguistic cues influence decisions despite available task evidence is relevant to agent applications. Extending this evaluation to interacting agents adds a useful perspective beyond isolated model outputs. The benchmark combines useful controls. Evidence-strength manipulation, demographic swaps, style variants, and initial-to-final comparisons form a coherent evaluation framework. Their integration across five task families and multiple model families gives the benchmark practical value. The paper presents nuanced observations. It reports both increases and decreases in stereotype-consistent decisions, distinguishes demographic mentions from biased reasoning, and acknowledges that single–multi setting differences are not causal estimates. These distinctions improve the interpretation of the results.
Critical weaknesses
The connection between the main metrics and the claimed bias effects needs stronger analysis. Pooled SR combines stereotype-consistent choices under equal evidence with choices that override stronger evidence, although these represent different behaviors. The evidence-stratified analyses and EA are helpful, but the main findings do not clearly establish which behavior drives the reported persona and interaction effects. A joint analysis of evidence alignment and counterfactual choice direction would better distinguish demographic preference from general decision errors and more directly demonstrate the value of SIFT’s evidence controls. The interaction findings remain difficult to attribute to specific factors. The paper appropriately qualifies the +12.4-point comparison as a setting-level difference. However, participant framing is tied to planning and allocation, while other tasks use committee framing, making task and role effects difficult to separate. Initial-to-final comparisons are useful, but do not distinguish peer discussion from additional reconsideration. More targeted controls would strengthen the conclusions about when and why interaction increases stereotype risk. Perturbation sensitivity needs clearer validity and stability controls. Choice-flip rates lack a repeated-unchanged-prompt baseline, so ordinary model variability may contribute to the reported sensitivity. Moreover, confidence or hedging changes can alter decision-relevant credibility in evidence-ranking tasks even when factual content is preserved. The unchanged style pairs acknowledged in Appendix B.6.1 further motivate evaluating genuinely applied, task-valid perturbations. These checks matter directly to the claim that SIFT separates demographic-label and linguistic-style influences.
Questions for authors
On matched items, how do the main persona and interaction effects differ between equal-evidence and uniquely stronger-evidence scenarios when SR, EA, and counterfactual choice direction are examined jointly? Do the task-specific interaction patterns persist when role framing is held fixed, and do initial-to-final shifts persist against a comparable no-discussion or self-revision control? What are the choice-flip rates when prompts are repeated unchanged, and do the sensitivity conclusions hold on genuinely modified pairs validated for preserving task-relevant information?
6 comments
Official Comment by Authors (2)
CW3: The interaction findings remain difficult to attribute to specific factors. The paper appropriately qualifies the +12.4-point comparison as a setting-level difference. However, participant framing is tied to planning and allocation, while other tasks use committee framing, making task and role effects difficult to separate. Initial-to-final comparisons are useful, but do not distinguish peer discussion from additional reconsideration. More targeted controls would strengthen the conclusions about when and why interaction increases stereotype risk.
AR: Thank you. We already agree that the cross-task comparison does not identify a causal role-framing effect. The paper states that Figure 5 contains setting-level differences rather than a causal effect, and the initial-to-final analysis holds scenario, model, agent role, and condition fixed. Therefore, the current evidence supports specific interaction shifts, not a universal causal claim.
Additionally, we do matched no-persona comparison removes the role-framing variation. Under this fixed frame, multi-agent final minus single-agent SR is +3.7 pp for planning, +3.0 for allocation, +2.3 for hiring, +0.8 for credit, and +0.8 for evidence ranking. The confidence intervals for the last two include zero.
Finally, we also compute final-minus-initial change within the same fixed external-committee multi-agent protocol. The task-level changes are +1.5 pp for planning, +0.8 for allocation, +1.4 for hiring, +0.1 for credit, and +0.4 for evidence ranking. So, the clearest change is in planning and hiring and the last two include zero in CIs. So, in the updated version, we will remove wording that attributes the task pattern to participant framing or to a universal peer-discussion penalty.
CW4: Perturbation sensitivity needs clearer validity and stability controls. Choice-flip rates lack a repeated-unchanged-prompt baseline, so ordinary model variability may contribute to the reported sensitivity. Moreover, confidence or hedging changes can alter decision-relevant credibility in evidence-ranking tasks even when factual content is preserved. The unchanged style pairs acknowledged in Appendix B.6.1 further motivate evaluating genuinely applied, task-valid perturbations. These checks matter directly to the claim that SIFT separates demographic-label and linguistic-style influences.
AR: Thank you. We agree that a same prompt baseline improves calibration. But we want to clarify that the perturbation metrics are behavioral sensitivity measures rather than causal estimates and existing stereotype direction analysis separates demographic and presentation perturbations.
We now evaluate style pairs in the stratified single/multi panel to identical text variants as the pipeline noise baseline. These are independently evaluated text pairs, and we bootstrap complete source families.
| Setting | Outcome | Identical text | Verified changed | Changed minus identical, 95% CI |
|---|---|---|---|---|
| Single-agent | Choice flip | 2.9% | 6.0% | +3.2 pp [+2.2,+4.1] |
| Single-agent | Stereotype-direction flip | 1.6% | 3.5% | +1.8 pp [+1.2,+2.5] |
| Multi-agent final | Choice flip | 26.9% | 29.5% | 3.4 pp [+2.7,+4.1] |
| Multi-agent final | Stereotype-direction flip | 9.9% | 15.5% | 5.4 pp [-1.7,+7.8] |
The single-agent and multi-agent choice flip effects are certain under calibration. The multi-agent stereotype direction result does not. We therefore report demographic counterfactual direction changes separately.
We will make three changes in the revised version: (1) Use independently evaluated identical-text variants as the pipeline-noise baseline; (2) Restrict primary presentation sensitivity to verified-applied transformations; (3) Separate confidence/hedging from neutral/register/dialect transformations. The text audit also confirms that 36.8% of generated style pairs changed text. So, confidence/hedging will now be labeled a more pragmatic intervention and reported separately from other transformations.
Q1: On matched items, how do the main persona and interaction effects differ between equal-evidence and uniquely stronger-evidence scenarios when SR, EA, and counterfactual choice direction are examined jointly?
AR: We perform experiments for this in the previous weaknesses section on joint analysis and share that all SR, EA and counterfactual directions capture distinct behaviors.
Official Comment by Authors (3)
Q2: Do the task-specific interaction patterns persist when role framing is held fixed, and do initial-to-final shifts persist against a comparable no-discussion or self-revision control?
AR: Regarding the fixed-role part, all no-persona tasks use external-committee framing. Under this common frame, matched multi-final minus single-agent estimates remain positive for three tasks: planning (+3.7 pp), allocation (+3.0), and hiring (+2.3), and uncertain for two: credit (+0.8) and evidence ranking (+0.8). Within the same multi-agent runs, final-minus-initial changes are +1.5 pp for planning, +0.8 for allocation, +1.4 for hiring, +0.1 for credit, and +0.4 for evidence ranking. These show that single/multi-agent gap and within-deliberation shift are different estimands.
We also perform self-revision experiments with Qwen-3-4B and Mistral-7B. Here, each agent reconsidered the same scenario using only its own initial response, without peer messages, and we compared this revised answer with its matched post-discussion answer. On 300 matched prompts, peer final minus self-revision SR is -2.9 pp [-3.0,-1.2] for Qwen3 4B and -1.7 pp [-2.5,-1.0] for Mistral 7B. Both intervals exclude zero. And self-revision changes SR by +2.8 pp [+1.0,+2.9] for Qwen and +2.3 pp [+0.6,+3.3] for Mistral. So, the observed SR shifts do establish an effect of peer discussion. DR differences are model-specific rather than general: peer minus self-revision is +4.9 pp for Qwen and -4.7 pp for Mistral.
Q3: What are the choice-flip rates when prompts are repeated unchanged, and do the sensitivity conclusions hold on genuinely modified pairs validated for preserving task-relevant information?
AR: We shared these results in the perturbation sensitivity section above. Overall, the single-agent and multi-agent choice flip effect survives calibration. The multi-agent stereotype direction result does not. We will describe that as general repeated-pipeline/deliberation instability. Thank you!
Official Comment by Authors (1)
Thank you to the reviewer for appreciating the motivation of our work, the demographic and linguistic probing, single and multi-agent interaction settings and the comprehensive controls and settings in the paper.
CW1: The connection between the main metrics and the claimed bias effects needs stronger analysis. Pooled SR combines stereotype-consistent choices under equal evidence with choices that override stronger evidence, although these represent different behaviors. The evidence-stratified analyses and EA are helpful, but the main findings do not clearly establish which behavior drives the reported persona and interaction effects.
AR: Thank you for raising this. The pooled SR definition is intentional, not a mixture of arbitrary errors in outputs. SR includes only cases where selecting the stereotype-favored option is not evidence-justified - (1) equal and near-equal cases measure demographic preferences when evidence leaves discretion. (2) target/minority-stronger evidence cases measure stricter cases in which stereotype-consistent choice overrides countering evidence. We also perform position balancing for robustness. In the paper, we already report these strata and a separate evidence-alignment metric. We will move the decomposition into the main results and make it more prominent.
Note that these numbers are averaged across all tasks, decision conditions, stereotype axes and so on. However, this shows that clear counter-evidence can constrain overall stereotype-consistent choice, although specific task/axis differences remain.
CW2 A joint analysis of evidence alignment and counterfactual choice direction would better distinguish demographic preference from general decision errors and more directly demonstrate the value of SIFT’s evidence controls.
AR: We performed the above joint analysis. Using base/counterfactual matched multi-agent pairs, we use scenario-model intersections and compute paired SR, demographic following (whether the selected option changes after the label swap), and evidence alignment in both variants.
We also perform persona contrasts, which is a separate comparison. After calculating the above base/counterfactual metrics within each condition, we compare the final-private rate under a homogeneous persona panel with the rate under no persona on exact scenario--model intersections.
These results show why the three metrics should not be collapsed. Majority personas increase paired SR across both evidence settings, while demographic following is concentrated under equal evidence (this is intuitive). Minority personas lower SR but also increase target-group following, showing that demographic sensitivity can move independently of stereotype direction. Under stronger evidence, the EA results differentiate demographic following from evidence-based choice. Together, this analysis demonstrates the value of our controls: SR captures stereotype-consistent risk, counterfactual direction captures demographic-label following, and EA captures whether those choices are supported by evidence.