- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper asks whether instructing an LLM agent not to use race removes racial bias from its decisions or merely relocates it. Agents with 16 personas, defined by race, education, religion and occupation, decide whether to leave a neighborhood given the share of neighbors who share each attribute. The four shares are sampled independently. Four OpenAI models make 864,000 decisions under three regimes: race visible, race visible but forbidden, and race removed. The authors find that agents almost never report using race under the instruction, yet a race effect persists in two models and reverses in two. The other attributes gain weight, which the authors call displacement, measured by a race retention ratio r and a displacement ratio D. Having agents rate the permitted factors before deciding does not remove the effect, because race leaks into the ratings. Overall, the design is careful and the compliance gap is interesting, but the headline overstates the results: the instruction removes most of the race weight, and the displaced weight cannot be considered racial disparity by construction, which contradicts the paper’s title framing.
Critical strengths
- The independent sampling of the shares is a clean design that rules out confounders.
- The comparison of self-reports with decisions is a useful lens for auditing LLM agents.
- The scale is large, with three replicates and several robustness checks.
- The finding that biases leak through ratings is interesting.
Critical weaknesses
- By construction the proxies are independent of race, so heavier reliance on them creates no racial disparity, and calling this “displacement” overstates the finding. The authors concede that what moves is “weight on similarity,” not race, and that a common factor reproduces 92–113% of D. The model may simply lean harder on the remaining attributes because it must decide.
- The title overstates the result. The instruction removes 91–98% of the race weight in gpt-4o-mini and gpt-4.1-mini (r = 0.015–0.093), about as much as it cuts self-reported race use. The residual effects are statistically detectable but small (1.5–9% of the unrestricted race weight). The reversals in gpt-5.4-nano and gpt-5.6-luna look like over-correction rather than persistent bias. It would be helpful to report the effect sizes.
- Minor: Equation 2: "gain" assumes negative proxy slopes. means more weight only if . For example, and give , so falls from 3 to 2 (less weight), yet counts it as a gain. Defining the gain as avoids this.
- Minor: Figure 2 pools three replicates. Showing per-replicate lines would show whether small reversals (e.g., gpt-5.6-luna, 22.1% → 24.2%) are stable.
Questions for authors
- What is D between two unrestricted replicates, i.e., how much “displacement” appears from noise alone?
- Does the displacement produce any measurable difference in outcomes between Black and White personas?
- Do the findings hold for non-OpenAI models?
4 comments
Follow-up Response
Thank you for your careful review of the content we submitted. Given that the discussion window will close today, we would like to confirm whether our revisions and responses have addressed the key issues you raised.
Thank you for the detailed response and the new experiments. The deferral experiment is helpful and somewhat rules out the model being forced to choose as an explanation. However, displacement is increased sensitivity to the remaining attributes rather than implicit racial bias, so the results still do not support the paper’s framing. While the results are mostly sound, they are not very surprising in my opinion. In addition, the placebo analysis shows that D is unreliable: D between unrestricted replicates is often larger than under the instruction, which is why it had to be replaced by G. As a result, I have raised my soundness score to 2 but will keep my overall rating.
Thank you very much for your careful reading.
We understand your concern that the displacement might stem from an increased sensitivity to the remaining attributes, and we have adjusted the expression in the revised version to more clearly highlight this alternative explanation. Now we consider implicit racial bias as a reasonable mechanism rather than the final explanation for the observed changes.
Regarding your viewpoint that the D indicator is unstable in unrestricted repeated experiments, we have changed to G and have more clearly documented this placebo rationality check and its impact in the revised version. We have also supplemented the discussion on the expectedness of these behavioral changes in the limitations section.
Thank you for raising these points.
W1. Our design samples the retained cues independently of the displayed race share. With identical cue distributions and a common race-blind decision rule, greater sensitivity to those cues alone would preserve equal group mean outcomes. We will clarify that “displacement” refers to increased sensitivity to retained cues and that D is a descriptive ratio of logit-slope changes. A common multiplier reproduces 92–113% of aggregate D. Exact proportionality is rejected in 30 of 32 fits, but the logit estimates leave open whether the common component reflects stronger preferences or lower decision noise.
We tested the forced-choice explanation by crossing the race instruction with the option to defer. With deferral available and used, the instruction increased the low-minus-high education-similarity move-rate gap by +12.04 [8.98, 15.17] pp in GPT-4o-mini, +18.20 [13.35, 23.18] in GPT-5.6-luna, +3.58 [0.16, 7.06] in Haiku 4.5, and +3.29 [2.05, 4.56] in Haiku 5.5. Deferrals remain in the denominator. General cue substitution remains plausible. For example, banning quiet hours increased occupation sensitivity in Haiku 4.5 by 8.69 pp [3.32, 14.16]. Race and quiet hours differed in baseline strength, and the bare J/K tags affected choices despite random assignment. These comparisons identify effects of specific prompts.
W2. We appreciate the point about the title. The instruction substantially attenuates the race effect in the two GPT-4 models (logit r = 0.015–0.093), and the GPT-5 sign reversals are consistent with overcorrection. We would like to retain the title and clarify its scope in the abstract and introduction. After the instruction, at most 2.13% of responses across the six models listed race as a primary factor, while choices still responded to the race cue. We will report the magnitude of attenuation and the signed effects alongside self-reports, on their respective scales. In the original-prompt follow-up (12,288 decisions per model), we measured the low-minus-high same-race move-rate gap (pp). Brackets give 95% neighborhood-block bootstrap intervals throughout.
Absolute race gaps fell by 91.5% and 97.9% for the two GPT-4 models and also shrank for both GPT-5 models.
W3W4 Thank you for these suggestions. We will revise the paper accordingly.
Q1. We compared all three pairs of unrestricted replicates per model using the rounded appendix coefficients. To calculate placebo D, we used the replicate with the larger absolute race slope as the reference. Under a no-change null, D is undefined (0/0), so small estimated changes in the race slope can produce large placebo ratios. Appendix Table S4 defines G and reports its values under the race instruction. We include those values alongside the new placebo results for comparison.
For every model, the maximum placebo G was below the range observed under the race instruction.
Q2. We paired Black and White personas at the same neighborhood composition and with identical education, religion, and occupation. We averaged the low and high Black-neighbor-share conditions equally. The Black-minus-White move-rate gaps are in pp.
These estimates measure the total effect of the instruction on the simulated group gap. Attributing part of that change to retained cues requires a separate intervention or structural assumptions.
Q3. The revision adds four Anthropic models (Haiku 4.5, Sonnet 4.5, Sonnet 4.6, Opus 4.5) on the same prompts and neighbourhoods. All four report zero race use under the instruction, three reverse the race effect, and all four displace weight (D = 0.38–1.49 forbidden, 1.09–3.21 removed). As in the OpenAI models, a common multiplier reproduces most of D (97–109%). In the follow-up experiments, Haiku 4.5 and Haiku 5.5 also reversed the race effect, while self-reported race use fell to 0.28% and zero.