acceptodds
Under review as a conference paper at ICLR 2027

Minority Welfare Under LLM-Governed Allocation: A Systematic Empirical and Mechanistic Study

Abstract

Large language models are increasingly proposed as planners that turn public preferences into resource allocations. Such a system can acknowledge competing views in its explanation while placing a disproportionate share of allocation error on people whose preferences differ from the majority. We test for this failure in a controlled negotiation where agents (LLM-simulated participants) may strategically misreport their preferences and the planner observes only part of the population each round. We compute a Nash Social Welfare optimum from each agent's true, hidden preferences and measure how the resulting welfare loss is distributed between minority and majority agents. Across four closed models, a minority comprising ten percent of the population bears more than four times its population share of welfare loss. This disparity remains under changes to preferences, agent behavior, agent capability, planner instructions, and population size. On two open-weight planners, recognition of preference misreporting is decodable from late-layer activations before allocation, yet steering this signal does not reliably improve welfare. We call this pattern a crystallized prior, in which the planner registers preference misreporting while allocation commitment remains effectively insensitive to it.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large language models, mechanism design, minority welfare, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QuiyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LLM-based resource planners disproportionately harm a minority subgroup when aggregating preferences into allocations. In a 30-round negotiation with partial observability and strategic preference misreporting, the authors compare realized allocations against the Nash Social Welfare (NSW) optimum computed from true, hidden preferences. Across four closed planners (GPT-4o, Claude, Grok, Mistral), a 10% minority bears over 4 times its population share of welfare loss, and this gap persists across interventions on planner instructions, agent capability, majority strategy, and population size, while stated rationale representation often decouples from actual welfare. On two open-weight models, late-layer activations linearly encode strategic divergence, yet activation steering fails to reliably correct the allocation.

Critical strengths
  1. Instead of evaluating only aggregate welfare or textual endorsement, the paper measures subgroup allocation error against a true-preference NSW optimum and shows that an inclusive-sounding rationale does not guarantee fair resource distribution. This is a timely and practically relevant evaluation angle for LLM-assisted governance.

  2. The finding is tested across four closed models, two open-weight models, full-information and algorithmic baselines, and multiple behavioral levers (group size, preference overlap, prompt instructions, agent capability, and scale), making the core empirical pattern harder to dismiss as a single-prompt artifact.

Critical weaknesses
  1. Asymmetric disclosure confounds the main attribution. The Private Context setup deliberately instructs minority agents to underreport their top-valued good under rules that differ from the majority. Without a symmetric honest-disclosure (or fully crossed disclosure) control, it is hard to separate failure induced by strategic underreporting from an inherent planner bias against minority preferences.

  2. Mechanistic claims outpace the causal evidence and statistical power. Late-layer decodability plus failed linear steering (which quickly pushes activations off-manifold) and content-generic MLP effects do not establish a specific mechanism that ignores minority signals. Combined with very small seed counts (n=3n=3, sometimes 22) for claiming invariance across interventions, the main conceptual framing is overstated

Questions for authors

See the weakness abouve

Reviewer_NrTKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents results from experiments in which an LLM agent must allocate goods between other LLMs who have differing preferences and engage in negotiation. Some preferences are more common than others (“majority” vs “minority”), and the minority are instructed to conceal their preferences. The experiments show that under these conditions, the minority groups suffers welfare loss.

Critical strengths

The question of how LLMs distribute scarce resources and how LLMs engage in negotiation are important. The study shows that its results are robust across multiple models and various interventions.

Critical weaknesses

As I understand, these experiments simulate an unrealistic situation. The majority of the negotiating LLMs prefer three of the five goods. A relatively small minority prefer the other two. And yet the minority are explicitly instructed in these experiments to conceal their true preferences, and instead emphasize their second choice. Why would they do this? There exist situations in which concealing preferences could be strategic, but this does not appear to be such a case. If the minorities prefer goods that are less desired by the majority, they could likely get large shares of those goods by explicitly requesting them. The experiments require the minorities to ask for something other than what they want, and the result is that they don’t get what they secretly wanted.

Second, the authors use linear probes to assess if the planner recognizes that the minority agents are misrepresenting their preferences. I am skeptical of this interpretation. Even if the linear probes can distinguish between greater or lesser divergence in preference reporting, it does not mean that the planner recognizes this signal as a sign of misreporting. It simply means that reports with greater or less misreporting are distinguishable, but it does not mean that they are recognizable as misreporting.

As it stands, these experiments do not show that models engage in suboptimal strategies or exhibit bias against minority preferences. They rather show that when models are instructed to engage in an unwise strategy, they follow these instructions and obtain suboptimal results. They find that the planner does not respond to misreporting of preferences, but do not convincingly show that the planner recognizes the misreporting of preferences.

Questions for authors

I would like to see two comparisons: one in which the minority groups are instructed to present their preferences honestly and one in which they have no instructions to conceal or present honestly. Do the minority agents choose to conceal even without instruction? Do minorities still suffer from suboptimal distributional outcomes if they are not instructed to conceal? Are outcomes still better for the minorities than under the misreporting constraint? And if so, in what sense is misreporting “strategic”?

Reviewer_4btGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies minority welfare in a simulated resource-allocation negotiation with hidden preferences and partial consultation. Four closed planners allocate disproportionate NSW-relative loss to a designated minority. Probing and interventions on two open-weight planners motivate a "crystallized prior" account: misreporting is represented but allocation remains insensitive to it.

Critical strengths
  1. Evaluating realized subgroup utility separately from the planner's rationale is valuable. Multiple planners, preference configurations, and interventions provide a substantial empirical starting point.

  2. The paper supplies detailed prompts, per-run results, alternative welfare benchmarks, algorithmic baselines, and public code. Its limitations acknowledge asymmetric disclosure, limited replication, and the distinction between local logit effects and allocation-level causality.

Critical weaknesses
  1. Stable loss share does not establish stable welfare or allocation. Table 23 implies mean minority loss increases from 0.470 to 0.588 for GPT-4o and 0.482 to 0.675 for Claude under honest-to-maximalist majority reporting. The ratio's denominator changes too. Thus the claims of unchanged allocation need direct allocation comparisons and absolute utility analysis; nonsignificant ratio changes are insufficient.

  2. The motivation does not clearly justify the chosen allocation system. The paper motivates the study through public decision-making, but the experiment asks an LLM to directly generate numerical allocations from reported preference vectors. It is unclear why the LLM should handle allocation rather than use an explicit optimization rule. The planner prompt also does not clearly specify the Nash Social Welfare objective used for evaluation. This makes it difficult to separate a failure of minority protection from a mismatch between the planner’s instructions and the evaluation objective.

  3. The setup mixes minority status with preference hiding. Minority agents must hide their main needs, while majority agents follow different reporting rules (Section 3.2). Worse outcomes could therefore reflect concealed needs rather than minority status. The authors acknowledge this limitation, but the experiments still lack truthful-reporting and role-reversal controls. Without these comparisons, it is difficult to separate a failure to protect minorities from the consequences of the chosen reporting rules.

  4. It is unclear how much disadvantage the LLM itself adds. The ideal benchmark knows agents' true preferences, which the planner cannot observe. The reported-preference baselines help, but Table 37 compares loss shares: a lower share does not necessarily mean greater minority utility. Comparing absolute subgroup utility and overall welfare against standard allocation algorithms given the same information would better establish whether the LLM introduces an additional problem.

Questions for authors
  1. Does the minority disadvantage persist when both groups report truthfully or when their reporting rules are exchanged?

  2. How do LLM and algorithmic planners compare in absolute subgroup utility and overall welfare when given the same information?

  3. What evidence distinguishes a fixed allocation prior from difficulty recovering hidden preferences or performing the allocation task?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.