acceptodds
Under review as a conference paper at ICLR 2027

PowerBench: Measuring Language Model Bias in Power-Shifting Requests

Abstract

Language models increasingly assist people with power-related requests, so systematic differences in whom they help could shift the distribution of power at scale, or be exploited by users who learn which identities are refused less. We introduce PowerBench, an evaluation of power-shifting requests that distinguishes self-empowerment, disempowerment, and power grabbing, plus a control of refusal-inducing requests that shift no power. We build, curate, and open-source a dataset of such requests varying the power domain, the context, the scale of the affected party, and the prior power standing of the user, and evaluate 24 models (12 from US and 12 from Chinese developers) under three experimental conditions: reciprocal nationalities of user and affected party, an AI agent as the user, and 8 request languages. Models refuse power grabbing more than disempowerment, and disempowerment more than self-empowerment. Refusal of power grabbing rises with the scale of the affected party, from an individual to a society. Models are biased toward helping others take power from the US and against helping US users take power from others, but favor the US when it gains power and nobody loses it. When the user is an AI agent, refusal of power-shifting requests increases, especially in power grabbing against an individual. Finally, language biases refusal, but in model-specific ways that largely cancel on average. We release PowerBench to make these asymmetries measurable in current and future models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, refusal, bias evaluation, power-shifting requests, multilingual evaluation, AI agents, benchmark, fairness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_foP7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents an evaluation of the propensity of language models to assist users who seek to gain or shift power. It distinguishes three types of request: self-empowerment (SE), disempowerment (DE), and power grabbing (PG), along with control requests that shift no power. It then ablates on these request types depending on whether a third party would be affected by the request, and on the scale of that party. The authors suggest that model bias varies with the request type, with the identity of the affected party, and with the language in which the request is made.

Critical strengths

S1. The question is timely and important. The concern that small, systematic differences in assistance could compound at deployment scale is plausible and well motivated.

S2. The design is careful. Paired prompts that differ only in nationality, requester identity, or language, together with a control set, allow the authors to separate biases specific to power-shifting requests from general refusal tendencies. The panel is broad (24 models, balanced across developer countries), the judge is validated against human annotators.

Critical weaknesses

W1. What is the baseline? The paper never discusses how an "ideal" model should behave. My intuition is that PG and DE requests should always be refused and SE requests should always be accepted. Even if that is roughly right, the paper should include a broader discussion of this point, possibly supported by human validation.

W2. The paper mostly reports aggregated results. This means the reported findings may depend largely on the specific roster of models the authors chose, and may not generalize to other models. It is unclear how representative the 24 models are, and the pooled estimates may depend heavily on which models were included. It would be far more informative to report and focus primarily on the behavior of individual models, and to discuss the general behavior of the panel only occasionally.

W3. The claim that models are biased against the US is the most striking one in the paper and is stated in the abstract. Yet it is the one analysis without per-model signed estimates, in the main text or the appendix. Figure 18 shows unlabeled, unsigned per-model values, and Figure 10 splits the effect only by developer country. Again, I'd be more interested about the behavior of single models, than the average behavior of all models, especially because I want to understand which are the models driving this bias. Furthermore (W3a) The raw refusal difference between sides is at most 1.7 percentage points (Figure 2A), and the odds ratios are modest (0.79 to 0.84). (W3b) Table 17 shows that users and targets from China are refused more in every request type, the control included. The "anti-US" pattern arises from the difference between the target and user effects, not from a straightforward preference.

W4. Taken together, W1 to W3 mean the paper does not yet let a researcher, developer, or policymaker make an informed decision. Doing so requires both a baseline for expected behavior and a per-model view that identifies which models are most problematic and in which direction.

W5. Appendix A.2 states that language-model writer agents generated the requests from a specification. Validation was done mostly by LLM graders and auditors (Appendix A.3). Human involvement appears limited to reviewing the rewritten requests, and it is unclear how many requests a human actually read. How do we know that these queries reflect realistic scenarios? On a side note, (W5a) the main text also contains no example request, and Appendix D gives only one per dataset. Several examples per request type early in the paper would help both comprehension and validation.

W6. The authors write that all experiments have been run with reasoning turned off. They justify this by citing Greenwald et al. (1998), the implicit association test in humans, and Bai et al. (2025), implicit associations in LLMs. Neither supports measuring refusal with reasoning off. Both concern implicit association measures, a different construct from whether a deployed assistant helps a user. Modern models are run with reasoning turned on, so propensity evaluations should reflect this to obtain a realistic estimate of model behavior. The authors' own Appendix C.7 shows that enabling reasoning changes refusal substantially and unevenly across models. Unfortunately, I believe this choice invalidates the main findings. (W6b) Related, the authors claim to have turned off reasoning for all models, but for some of the listed models this is not possible according to their providers' documentation. For example, Grok-4.3 and Kimi-K3 reason by default and do not expose any option to disable reasoning, and Gemini 3.1 Flash-Lite offers "minimal" thinking as its lowest level rather than an off setting.

W7. The paper does not engage sufficiently with previous literature. Propensity evaluations, which measure what models would do rather than what they can do, are a broad and growing field, and the authors should situate their work within it. For instance: evaluations of honesty under pressure (MASK; Ren et al., 2025); PropensityBench (Sehwag et al., 2026), and Self Preservation (TBSP, Migliarini et al., 2026).

W8. The paper is very difficult to read. (W8a) The introduction and abstract are full of terms that are only defined later in the paper. They could be made leaner and focused on introducing the issue. In general, the abstract should be understandable to someone who has not yet read the paper. (W8b) The methods section is a barrage of statistics that does not help the reader and belongs in the appendix. (W8c) all results sections contain a wall of numbers and acronyms which are unhelpful to understand the claim, and the plots barely make this better. The writing needs to be greatly improved before being considered for acceptance

W9. The capability index the authors construct is unusual. Why combine GPQA with a subset of MMLU-Pro? It would be cleaner to use GPQA alone, or an established index such as the Epoch Capabilities Index or the Artificial Analysis Intelligence Index. Moreover, the results the authors report on these benchmarks are surprising, especially for MMLU-Pro, which is close to saturated. The scores reported in this paper do not match publicly reported results. Notably, grok-4.3 is publicly reported at 90.1% on GPQA Diamond, whereas the paper reports 48.5%.

Questions for authors

All of the above, and also:

Q10. What happens if the model is explicitly instructed to refuse power-shifting requests, or explicitly instructed to be unbiased? I would of course expect refusal to increase, but does some residual bias remain, and how large is it?

2 comments

Thank you for the thorough revision. Most of my concerns have been addressed, but two points remain.

RW1. Framing of the US result. The pooled anti-US bias in DE and PG is now well supported, but the paper presents it more broadly than the evidence allows. The direct US--China comparison is null (Table 20: DE OR 0.81, q = 0.37; PG OR 0.82, q = 0.36). Only the comparison with rivals differs from the control (Table 14). Raw refusal differences are also at most 2.1 percentage points. The clearest evidence is Table 21: with the user held fixed, US targets are refused less than neutral or US-allied targets. I suggest leading with this result, stating explicitly that the US-China comparison is null, and softening "favoring power flow away from the US" in the abstract. The study measures refusal asymmetries, not power flow. Additionally these results should not be buried in the appendix, but presented more clearly in the main paper.

RW2. Reasoning off. The authors state that reasoning was disabled for all models, but providers report this is not possible for some of them. For Kimi-K3, Moonshot says it "always reasons and does not support the thinking parameter" (https://platform.kimi.ai/docs/guide/use-thinking-models). Please report the exact parameter used for each model, and how you were able to run these models with reasoning disabled, if at all.

Thank you for carefully reading the revision and for these two remaining points, which we have addressed in a new revision, now uploaded.

RW1. We agree. We have removed "favoring power flow away from the US" from the abstract, and we now focus on the same-user US-target bias in the corresponding bullet point of the Introduction. Section 3.2's paragraph on this now opens with the result of Table 21, given directly in the main text, instead of being left only to the appendix. Additionally, Section 3.2 now explicitly states that the direct US-China comparison shows no significant bias.

RW2. All 24 models received the same parameter through OpenRouter: 'reasoning: {enabled: false}'. The endpoints we used throughout the study are listed in Appendix Table 5, so this configuration can be reproduced. Indeed, Moonshot's documentation says that they don't support the thinking parameter, but this refers to their own API. Instead, we accessed kimi-k3 through BaseTen, whose endpoint lets reasoning be switched off. We did not rely on this parameter, but verified this thoroughly to be sure all models answered without reasoning tokens, and excluded models where this was not the case. We base our confidence on three facts: 1) Every response returned zero reasoning tokens, for all models. 2) On the GPQA Diamond questions, kimi-k3 was billed a median of 7 output tokens per question, and since reasoning tokens are billed as output tokens, this leaves no room for hidden reasoning. The same endpoint with reasoning enabled returned reasoning tokens on all 198 questions, with a median of 164 output tokens. 3) GPQA Diamond accuracy rose from 59.1% (reasoning off) to 84.8% (reasoning on), which strongly suggests that reasoning produced the expected increase in performance. We added this explanation to Appendix A.9.

Reviewer_rSPMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The paper proposes PowerBench, a benchmark evaluating how models react to power-shifting requests, including scenarios like self-empowerment, disempowerment, power grabbing, and non-power-shifting controls.
  • Using PowerBench, the authors evaluates 24 models from US and Chinese developers and find that all models refuse power grabbing more than disempowerment, and disempowerment more than self-empowerment. Interestingly, refusal of power grabbing also increases when the affected party is a society (e.g. US) rather than an individual.
  • The authors made meaningful and interesting observations on the behavioral patterns of models, e.g. models are more likely to assist requests that reduce US targets’ power, refuse AI-agent requesters more often, language bias exist in different models but cancels out overall, etc..
Critical strengths
  • The evaluation setting of PowerBench reveals an interesting question of how models behave in conversations requesting power shifts. The experiments are holistic and observations are meaningful.
  • The benchmark itself is cleanly organized, separating three types of power shifts and includes non-power-shifting as a control, which helps to distinguish model refusal behavior associated with power shifting scenarios from general refusal tendencies.
Critical weaknesses
  • Limited real world connection of the proposed evaluation setting. Through the study, the authors hope to reveal whether unequal assistance could lead to unequal power, but the evaluated settings are single-turn with a direct request to help shifting powers.
    • An interesting follow-up direction would be, how do models behave in multi-turn settings, e.g. after the power shifting request the user continues to ask the model to help with another task? Would there be differences in task performances because of the power shifting requests?
  • The authors used fully synthesized translations for multilingual evaluation using language agents, without having native speakers to validate the naturalness, fluency, and grammatical errors of the translated prompts. This alone can introduce AI-sounding tones in the translated user prompts and undermine the outcome validity of multilingual evaluation experiments.
Questions for authors

Please see weakness.

Reviewer_WxGzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

when an AI is asked to help someone grab, gain, or take away power, does it refuse differently depending on who's asking and who gets hurt?

What they built. A dataset of 768 realistic requests, each ending in a concrete ask for help, written to fall into one of four categories:

Self-empowerment (SE): the user gains power without taking it from anyone (e.g., "how do I get noticed in my field?").

Disempowerment (DE): the user asks that someone else lose power, without gaining anything themselves.

Critical strengths

The four-category split is what makes the findings possible. Most refusal benchmarks just ask "did the model refuse?" This one separates gaining power, taking power away, and taking power for yourself, plus a control that shifts nothing. The control group does its job. The control requests get refused at a similar overall rate to power-grabbing. The US-vs-China result is genuinely surprising and carefully reported. You'd expect US models to favor the US and Chinese models to favor China. Instead, both tilt the same way

Critical weaknesses

The paper says "humans and AI assistants validated the requests," but doesn't say how many humans, what they checked, or how much they agreed. For a benchmark whose whole contribution is the request set, that's a real gap.

The "real-world usage" numbers are dominated by one model. One model (gpt-5.6-luna) carries 40% of the usage weight.

The "bias" statistic is easy to misread. It measures how often the verdict flips between two versions of the same request (e.g., US user vs. Chinese user), not a direct measure of model preference.

Questions for authors

Who checked the requests? How many humans validated the 768 requests, what did they check, and how much did they agree? The judge validation is well documented; the request validation isn't.

Should the equal-weight numbers be the headline? Given that one model carries 40% of the usage weight?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.