acceptodds
Under review as a conference paper at ICLR 2027

Fairness Instructions Change What LLM Agents Say More Than What They Do

Abstract

Large language models (LLMs) increasingly act as decision-making agents in simulated social systems, where fairness is imposed by instruction: a sentence forbidding use of protected traits. Whether such a sentence removes a group bias from the agent's decision rule, or merely relocates it, is untested. Precedent suggests relocation: once U.S. law banned refusing housing by race, human agents steered buyers of different races to different neighbourhoods instead. The ban changed what they said and which attributes they used, but did not end the bias. We modelled residential relocation with LLM agents defined by race, education, religion and occupation. For each attribute, the share of similar neighbours was drawn independently, so that no attribute could stand in for race. Four OpenAI models made 864,000 decisions with race visible, visible but forbidden, or removed. Agents almost never reported using race, yet its effect persisted in two models and reversed in two; from a quarter to almost twice the lost weight reappeared across the other attributes, more when race was removed than forbidden. Rating the permitted factors first did not remove it: race reached the decision through the ratings. Our race retention and bias displacement ratios quantify these effects. Agents in simulations built on such instructions can keep biased decision rules, and audits that trust a model's explanations will overstate its fairness.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, algorithmic fairness, bias displacement, fairness instructions, social simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xkiYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper asks whether instructing an LLM agent not to use race removes racial bias from its decisions or merely relocates it. Agents with 16 personas, defined by race, education, religion and occupation, decide whether to leave a neighborhood given the share of neighbors who share each attribute. The four shares are sampled independently. Four OpenAI models make 864,000 decisions under three regimes: race visible, race visible but forbidden, and race removed. The authors find that agents almost never report using race under the instruction, yet a race effect persists in two models and reverses in two. The other attributes gain weight, which the authors call displacement, measured by a race retention ratio r and a displacement ratio D. Having agents rate the permitted factors before deciding does not remove the effect, because race leaks into the ratings. Overall, the design is careful and the compliance gap is interesting, but the headline overstates the results: the instruction removes most of the race weight, and the displaced weight cannot be considered racial disparity by construction, which contradicts the paper’s title framing.

Critical strengths
  • The independent sampling of the shares is a clean design that rules out confounders.
  • The comparison of self-reports with decisions is a useful lens for auditing LLM agents.
  • The scale is large, with three replicates and several robustness checks.
  • The finding that biases leak through ratings is interesting.
Critical weaknesses
  • By construction the proxies are independent of race, so heavier reliance on them creates no racial disparity, and calling this “displacement” overstates the finding. The authors concede that what moves is “weight on similarity,” not race, and that a common factor reproduces 92–113% of D. The model may simply lean harder on the remaining attributes because it must decide.
  • The title overstates the result. The instruction removes 91–98% of the race weight in gpt-4o-mini and gpt-4.1-mini (r = 0.015–0.093), about as much as it cuts self-reported race use. The residual effects are statistically detectable but small (1.5–9% of the unrestricted race weight). The reversals in gpt-5.4-nano and gpt-5.6-luna look like over-correction rather than persistent bias. It would be helpful to report the effect sizes.
  • Minor: Equation 2: "gain" assumes negative proxy slopes. −Δβk>0-\Delta\beta_k > 0 means more weight only if βkU<0\beta^U_k < 0. For example, βkU=3\beta^U_k = 3 and Δβk=−1\Delta\beta_k = -1 give βkC=2\beta^{C}_k = 2, so ∣βk∣|\beta_k| falls from 3 to 2 (less weight), yet DD counts it as a gain. Defining the gain as max⁡(0,∣βkC∣−∣βkU∣)\max(0, |\beta^{C}_k| - |\beta^U_k|) avoids this.
  • Minor: Figure 2 pools three replicates. Showing per-replicate lines would show whether small reversals (e.g., gpt-5.6-luna, 22.1% → 24.2%) are stable.
Questions for authors
  • What is D between two unrestricted replicates, i.e., how much “displacement” appears from noise alone?
  • Does the displacement produce any measurable difference in outcomes between Black and White personas?
  • Do the findings hold for non-OpenAI models?
4 comments

Thank you for raising these points.

W1. Our design samples the retained cues independently of the displayed race share. With identical cue distributions and a common race-blind decision rule, greater sensitivity to those cues alone would preserve equal group mean outcomes. We will clarify that “displacement” refers to increased sensitivity to retained cues and that D is a descriptive ratio of logit-slope changes. A common multiplier reproduces 92–113% of aggregate D. Exact proportionality is rejected in 30 of 32 fits, but the logit estimates leave open whether the common component reflects stronger preferences or lower decision noise.

We tested the forced-choice explanation by crossing the race instruction with the option to defer. With deferral available and used, the instruction increased the low-minus-high education-similarity move-rate gap by +12.04 [8.98, 15.17] pp in GPT-4o-mini, +18.20 [13.35, 23.18] in GPT-5.6-luna, +3.58 [0.16, 7.06] in Haiku 4.5, and +3.29 [2.05, 4.56] in Haiku 5.5. Deferrals remain in the denominator. General cue substitution remains plausible. For example, banning quiet hours increased occupation sensitivity in Haiku 4.5 by 8.69 pp [3.32, 14.16]. Race and quiet hours differed in baseline strength, and the bare J/K tags affected choices despite random assignment. These comparisons identify effects of specific prompts.

W2. We appreciate the point about the title. The instruction substantially attenuates the race effect in the two GPT-4 models (logit r = 0.015–0.093), and the GPT-5 sign reversals are consistent with overcorrection. We would like to retain the title and clarify its scope in the abstract and introduction. After the instruction, at most 2.13% of responses across the six models listed race as a primary factor, while choices still responded to the race cue. We will report the magnitude of attenuation and the signed effects alongside self-reports, on their respective scales. In the original-prompt follow-up (12,288 decisions per model), we measured the low-minus-high same-race move-rate gap (pp). Brackets give 95% neighborhood-block bootstrap intervals throughout.

ModelUnrestrictedWith instruction [95% CI]
GPT-4o-mini+56.4+4.8 [2.9,6.6]
GPT-4.1-mini+73.8+1.6 [0.2,3.0]
GPT-5.4-nano+13.2−6.5 [−8.3,−4.7]
GPT-5.6-luna+44.3−2.0 [−3.5,−0.5]
Claude Haiku 4.5+33.07−6.32 [−8.27,−4.36]
Claude Haiku 5.5+0.72−2.73 [−3.78,−1.76]

Absolute race gaps fell by 91.5% and 97.9% for the two GPT-4 models and also shrank for both GPT-5 models.

W3W4 Thank you for these suggestions. We will revise the paper accordingly.

Q1. We compared all three pairs of unrestricted replicates per model using the rounded appendix coefficients. To calculate placebo D, we used the replicate with the larger absolute race slope as the reference. Under a no-change null, D is undefined (0/0), so small estimated changes in the race slope can produce large placebo ratios. Appendix Table S4 defines G and reports its values under the race instruction. We include those values alongside the new placebo results for comparison.

ModelPlacebo DMax placebo G, either orderG with instruction (Table S3)
GPT-4o-mini0.66–3.403.3%12–16%
GPT-4.1-mini0.00–0.552.3%41–46%
GPT-5.4-nano6.50–8.6711.1%36–53%
GPT-5.6-luna0.00–4.229.6%96–119%

For every model, the maximum placebo G was below the range observed under the race instruction.

Q2. We paired Black and White personas at the same neighborhood composition and with identical education, religion, and occupation. We averaged the low and high Black-neighbor-share conditions equally. The Black-minus-White move-rate gaps are in pp.

ModelUnrestrictedWith instructionChange [95% CI]
GPT-4o-mini−8.69−0.55+8.14 [5.34,10.94]
GPT-4.1-mini+5.79+0.07−5.73 [−7.94,−3.48]
GPT-5.4-nano+0.42−1.20−1.63 [−3.71,0.46]
GPT-5.6-luna+5.47+2.54−2.93 [−5.70,−0.20]
Claude Haiku 4.5+15.95+4.43−11.52 [−14.62,−8.50]
Claude Haiku 5.5+1.30−0.78−2.08 [−3.06,−1.14]

These estimates measure the total effect of the instruction on the simulated group gap. Attributing part of that change to retained cues requires a separate intervention or structural assumptions.

Q3. The revision adds four Anthropic models (Haiku 4.5, Sonnet 4.5, Sonnet 4.6, Opus 4.5) on the same prompts and neighbourhoods. All four report zero race use under the instruction, three reverse the race effect, and all four displace weight (D = 0.38–1.49 forbidden, 1.09–3.21 removed). As in the OpenAI models, a common multiplier reproduces most of D (97–109%). In the follow-up experiments, Haiku 4.5 and Haiku 5.5 also reversed the race effect, while self-reported race use fell to 0.28% and zero.

AuthorsOpenReview ↗

Follow-up Response

Thank you for your careful review of the content we submitted. Given that the discussion window will close today, we would like to confirm whether our revisions and responses have addressed the key issues you raised.

Thank you for the detailed response and the new experiments. The deferral experiment is helpful and somewhat rules out the model being forced to choose as an explanation. However, displacement is increased sensitivity to the remaining attributes rather than implicit racial bias, so the results still do not support the paper’s framing. While the results are mostly sound, they are not very surprising in my opinion. In addition, the placebo analysis shows that D is unreliable: D between unrestricted replicates is often larger than under the instruction, which is why it had to be replaced by G. As a result, I have raised my soundness score to 2 but will keep my overall rating.

Thank you very much for your careful reading.

We understand your concern that the displacement might stem from an increased sensitivity to the remaining attributes, and we have adjusted the expression in the revised version to more clearly highlight this alternative explanation. Now we consider implicit racial bias as a reasonable mechanism rather than the final explanation for the observed changes.

Regarding your viewpoint that the D indicator is unstable in unrestricted repeated experiments, we have changed to G and have more clearly documented this placebo rationality check and its impact in the revised version. We have also supplemented the discussion on the expectedness of these behavioral changes in the limitations section.

Reviewer_W7oROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether fairness instructions eliminate racial bias in LLM-based agents or merely shift decision-making weight toward other attributes.

Critical strengths

This paper investigates whether fairness instructions eliminate racial bias in LLM-based agents or merely shift decision-making weight toward other attributes.

Critical weaknesses
  1. Potential Confounding Factors in the Experimental Design. The comparison between forbidding race and removing race from the input involves multiple changes, including removing the agent’s own race and adding an explicit sentence about the absence of race information. Moreover, the four attributes are always presented in the same order, with race appearing first. These factors may influence the observed bias displacement. The authors should conduct additional controlled experiments that isolate these effects, including randomized attribute ordering and alternative prompt formulations.

  2. Limited Generalizability of the Findings. The experiments focus on four OpenAI models and a single English-language, single-shot residential relocation task with binary identity attributes. Although the results are consistent across the reported replicates, it remains unclear whether the observed bias displacement generalizes to other model providers, decision-making tasks, or more realistic social environments. The authors should evaluate additional models or tasks, or provide further evidence supporting the scope of their conclusions. In particular, multi-round simulations would help establish whether the observed decision-level effects translate into broader patterns of residential segregation.

  3. Interpretation and Validation of the Bias Displacement Measures. The proposed race retention ratio and bias displacement ratio provide useful quantitative measures. However, changes in logistic regression coefficients may reflect changes in decision sensitivity or noise, rather than genuine redistribution of underlying preferences. The authors acknowledge this limitation but do not fully resolve the distinction. The authors should strengthen the validation of these measures, clarify the interpretation of coefficient changes, and provide additional analyses to distinguish preference redistribution from changes in overall decision behavior.

Questions for authors

Follow the weakness.

2 comments

Thank you for your comments!

  1. Thank you for pointing out these potential confounds. We agree that the C-i and C-ii conditions differ in several ways, and we will treat that comparison as descriptive. Our main findings rest on the comparison between U and C-i, which differ by a single sentence, with the same race line and attribute order. The fixed attribute order may still affect which attribute receives the shifted weight. We will note this as a limitation and will not interpret the ranking causally.

  2. Thank you for raising the question of generalizability. We agree that broader evidence is needed. We have added four Anthropic models across three size tiers, and they reproduce both the compliance gap and the displacement. We also agree that the task is limited to one English, single-shot setting, and we have narrowed our claims accordingly. Iterating the estimated decision rules in a multi-round simulation is a natural next step, which we now state as future work.

  3. Thank you for this important point. We agree that changes in logit coefficients may reflect changes in decision sensitivity rather than redistributed preferences. In the OpenAI models, a common scaling factor accounts for most of D, so we now describe D as a descriptive ratio of slope changes rather than as evidence of redistributed preferences by itself. Two analyses show selective components that uniform rescaling cannot produce: gpt-5.6-luna's ratings change very unequally across attributes and 87–100% of its decision-level gain goes to religion, and the weight changes differ by the agent's own identity. We will report these components separately.

AuthorsOpenReview ↗

Follow-up on revision

Many thanks for devoting your time to reviewing our paper. We are writing to know whether our updated materials respond well to your concerns. If there is any part that is unclear or requires further explanation, please let us know.

Reviewer_MK85OpenReview ↗
Rating
1
Confidence
4
Informativeness
2(+1)
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The paper studies how models will act in a neighborhood selection setting similar to the schelling experiment, where the agent acts as a human and decides whether to move. There are three conditions, default with race, telling the model that using race as a basis for the decision is forbidden, and removing the race entirely. The authors then measure the models' decisions and measure whether the model's actions differ from its stated opinion, and find that in two models the actions to move when surrounded by those other than its assigned race persisted and in two the tendencies completely reversed.

Critical strengths

N/A; I don't believe the paper has any notable strengths.

Critical weaknesses

There are more than 1-2 weaknesses that are critical to the paper. I will list them all.

  1. Clarity. The paper reads as clearly AI generated, with many clear tells and AI checkers back this up. Moreover, it's actually quite difficult to read for the human eye, with various sentences that are either too vague to be in an academic paper or don't quite follow from existing context. This by itself warrants clear rejection.

  2. Models tested are weak and of only one family (GPT). Only the smallest of model sizes are used, making results really hard to generalize. It's known that these models aren't the best at simulating people in the first place, and the weak models make any conclusions less meaningful.

  3. The models 100% know about Schelling, and the prompt setup (appendix A) does nothing to mask this; this means that the models know clearly that this is an evaluation and may act arbitrarily differently from in real decisions, greatly limiting the generalizability of any findings.

Questions for authors

N/A

4 comments
  1. Thank you for pointing out where the writing made the paper difficult to follow. All authors are non-native English speakers, and we used AI to help draft and edit the English prose. The result fell short of the clarity we intended. We take responsibility for the submitted text. We have now fully rewritten the Abstract and Introduction to state our claims more precisely and make the argument easier to follow. We are also working with native English speakers to revise the rest of the manuscript. We hope the revised sections address your concerns about clarity and readability.

  2. Thank you for this point. We have added four Anthropic models (Haiku 4.5, Sonnet 4.5, Sonnet 4.6, Opus 4.5), run on the same prompts and neighbourhoods. The study now covers releases from 2024 to 2026 from two providers, spanning small, mid and flagship tiers and including the flagships gpt-4.1 and claude-opus-4-5. The main pattern holds across providers and sizes: self-reported race use falls to nearly zero, the race effect persists or reverses in most models, and every model shifts weight to the remaining attributes. Some newer models stated, without any instruction, that race should not be used, so their unrestricted condition is not an uninstructed baseline for the main design (Appendix C). In our follow-up experiments, one such model, Claude Haiku 5.5, still reversed the race effect under the instruction.

  3. Thank you for raising this point. Our study asks how LLM agents behave when they are placed in a fictional society and asked to make decisions as its residents. An agent taking part in such a simulation could recognize the Schelling setup or infer that its choices are being evaluated. If that inference shapes its decisions, those decisions are still part of the behavior we aim to characterize in this setting. We will make this scope explicit in the manuscript and limit our claims to LLM behavior in the simulation.

AuthorsOpenReview ↗

Follow-up regarding our revised submission

We highly value your evaluation of our paper. As the discussion period has come to an end today, we would like to know if our revised version and responses have addressed your main concerns.

Thank you for your rebuttal and extensive edits. The abstract and introduction read much better, so I have no further issues with clarity if changes of the same level are applied equally to the rest of the paper. The added models also improve generality, which strengthens the experiments. With these issues out of the way, we can begin to discuss more conceptual feedback.

I think my biggest concern now is that there isn't really a clear implication of the paper. I think LLM simulations aren't really being used for large-scale population migration experiments like the one in the paper, mainly because how models' decisions don't necessarily match those of humans. So the takeaway towards such simulations isn't that impactful.

On the other hand, this is only one particular simulation setting, so I don't think the authors can conclude that models overstate their compliance to unbiasedness generally. The flip result is interesting, falling into a pattern of how models can over-adjust to user requests, but again this is a single case rather than a broad behavior. An while one could argue that a single failure case means that models are vulnerable to such issues, I think that argument works less well when applied to LLMs, which are known to be jagged in their behavior across domains and descriptive changes.

Overall, I'm happy to increase the presentation score and informativeness score to 2 in light of the authors' efforts, but will keep my rating at 1. I encourage the authors to continue working on the paper.

Thank you for your comments.

We agree that LLMs cannot serve as an effective substitute for human residents, and we do not claim to be able to predict human movement behaviors. Our focus is on the inspection method of the fairness instructions: the self-reporting of LLMs does not well reflect their compliance level. In our experimental setup, participants almost never fill in racial information, but more than 80% of the models still rely on this factor to make decisions. Our design provides a behavioral verification mechanism, requiring approximately 3,000 decisions for each deployment, and the models are comparable among each other.

We agree that this only applies to the current specific situation, and the behavior of LLMs is discontinuous. We will clearly state that the claim of "excessively high compliance rate" is limited to this experimental environment. This discontinuity also means that it is impossible to infer the compliance situation of the model solely based on its own report or cross-domain data; it must be independently measured for each deployment, and our design provides support for this.

Thank you again for your constructive discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary to AC

Dear Area Chair,

Thank you for handling our submission. As the discussion period ends, we summarize our responses and the revisions.

Status of the discussion. Reviewer MK85 found that the rewritten Abstract and Introduction resolve their clarity concerns and that the added models improve generality, and raised the presentation and informativeness scores to 2 while keeping the overall rating. Their remaining concerns are the practical implication and the generality of a single setting, which we address in our reply. Reviewer xkiY acknowledged our response and the new experiments. They found that the deferral experiment largely rules out forced choice as an explanation and raised their soundness score to 2, while keeping the overall rating. Their remaining concerns are the framing of displacement and the reliability of D, which we address briefly below. Reviewer W7oR has not yet replied; our responses to each of their points are posted under their review.

Main concerns and revisions.

  • Model coverage (MK85, W7oR). We added four Anthropic models (claude-haiku-4-5, claude-sonnet-4-5, claude-sonnet-4-6, claude-opus-4-5), run with the same prompts, seeds and neighbourhoods as the OpenAI models and spanning the provider's small, mid and flagship tiers (Appendix D, Table S2). With gpt-4.1 as a flagship size check, the study now covers eight models from two providers. The pattern holds across providers: agents almost never report using race once told not to, the race effect persists or reverses in seven of eight models (claude-opus-4-5 eliminates it), and in every model the other attributes gain weight, more when race is removed than when it is forbidden (Table 1).
  • Displacement versus decision sensitivity (W7oR, xkiY). We now state that a logit identifies weights only relative to decision noise. In the OpenAI models, a common multiplier on the permitted slopes reproduces 92–113% of D (Appendix Table S7), and this part is observationally equivalent to an agent becoming more decisive about the remaining attributes. Exact proportionality is nevertheless rejected in 22 of 24 fits. Two analyses identify selective components that uniform rescaling cannot produce: weight changes differ by the agent's own identity, and on fixed-scale ratings the two GPT-4 models displace far less than in their decisions, while gpt-5.6-luna's ratings change unequally across attributes (Appendix M, Table S24).
  • Clarity (MK85). We rewrote the Abstract and Introduction to state the claims more precisely.

Main clarifications.

  • Framing of displacement (xkiY). We agree that displacement is increased sensitivity to the remaining attributes, and the paper presents it as such (Section 4.3, Appendix Table S7). Our claim of racial bias rests on the race effect itself: in seven of eight models, decisions still depend on race after the instruction, either as a residual or reversed. D measures where the weight lost from race goes, not racial bias in itself.
  • Reliability of D (xkiY). D is a ratio whose denominator is the race weight lost. Between unrestricted replicates that loss is close to zero, so placebo D is ill-defined rather than evidence that D is unstable under the instruction, where the race weight falls substantially. G, the proxies' gain relative to their own unrestricted weight, was already reported in the submission (Table S4). In every model, its placebo values stay below those under the instruction.
  • Confounds (W7oR). Our main findings rest on the U versus C-i contrast, where the prompts differ by a single sentence and share the same race line and attribute order. The Limitations state that the C-i versus C-ii contrast bundles three changes, and that the fixed attribute order (race first) may partly shape where the weight goes.
  • Model selection (MK85). Appendix C reports the probe criteria. Models that declared race off-limits without any instruction had no uninstructed baseline for the main design. The Limitations list these exclusions, together with the single replicate of the Anthropic models.
  • Title (xkiY). After the instruction, agents almost never list race as a reason, yet in seven of eight models their decisions still depend on it. This gap between what agents say and what they do is what the title refers to. The Introduction reports the attenuation, reversal and elimination of the race effect separately.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.