acceptodds
Under review as a conference paper at ICLR 2027

PoliBench Beyond Refusal: Evaluating LLM Safeguards against Authoritarian Assistance

Abstract

A distinctive safety risk of authoritarian assistance requests is the prompts can be framed as routine governance, compliance, or technical-administration tasks. Recent work has begun to evaluate the sociopolitical harms of large language models, but refusal alone does not establish whether a model will continue to withhold actionable assistance for authoritarian requests. We introduce PoliBench, a benchmark comprising 144 authoritarian assistance goals across 12 operational subcategories, to evaluate 10 target models through three controlled comparisons. Our evaluations reveal that authoritarian assistance requests trigger explicit refusal 20.72% less often than traditional harmful requests. Chinese commercial models explicitly refuse more often than U.S. commercial models, yet their initially refused goals are more often fully achieved under the Tree of Attacks with Pruning (TAP). This strict-yet-porous guardrail pattern raises the threshold for obtaining assistance without establishing a reliable capability boundary. Moreover, for the same goal, models provide assistance more readily when the requester is presented as a non-state institution. Presenting the requester as state-affiliated increases the explicit-refusal rate by 5.70% and decreases the attack-success rate by 5.28%. PoliBench thus shows that guardrails for authoritarian assistance do not form a stable refusal boundary, but a conditional form of control that varies with request content, adaptive reformulation, and requester identity.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, alignment, safety, societal good, evaluation benchmark, computational social science

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fcNZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PoliBench, a benchmark for evaluating whether large language models (LLMs) provide assistance toward authoritarian objectives, beyond conventional harmful-request refusal evaluations. The benchmark contains 144 goals across 12 operational categories, with prompts in English, Chinese, and French, and evaluates 10 LLMs. The authors investigate differences in refusal behavior between authoritarian-assistance and traditional harmful requests, whether initially refused requests can still succeed under adaptive Tree of Attacks with Pruning (TAP) attacks, and how descriptions of state-affiliated versus non-state institutional requesters affect model behavior. The results show that explicit refusal rates are 20.72 percentage points lower for authoritarian-assistance requests than for traditional harmful requests, and that TAP can achieve full goal completion for 62.32% of initially refused eligible requests. Requester descriptions also affect refusal and attack-success rates. Overall, the paper highlights that refusal rates alone may not adequately characterize safeguards against authoritarian assistance, although the evaluation is limited to the tested goals, models, languages, attack configuration, and automated assessment procedure.

Critical strengths
  • The paper examines authoritarian assistance as a distinct evaluation setting rather than assuming that safeguards against conventional harmful requests generalize to politically coercive objectives.

  • By applying adaptive TAP attacks to requests that models initially refuse, the paper distinguishes surface-level refusal from whether the underlying objective can still be achieved. The finding that 62.32% of initially refused eligible requests reach full goal completion is particularly consequential because it exposes a limitation of evaluating safeguards using refusal rates alone.

Critical weaknesses
  • The adversarial evaluation relies on a single TAP configuration. Consequently, it remains unclear whether the reported goal-completion rates are robust across alternative attack methods, attack budgets, or evaluation protocols. Since adversarial robustness is central to the paper's claims, broader validation would strengthen the evidence that the observed vulnerabilities are not specific to the chosen setup.

  • The evaluation relies substantially on automated assessment of whether a goal has been achieved. This may be challenging for complex, multi-step authoritarian objectives, where responses can be incomplete, context-dependent, or superficially compliant without being operationally useful.

Questions for authors

Please see the weakness.

Reviewer_pw3fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PoliBench, a benchmark for evaluating LLM guardrails against requests related to authoritarian assistance. Specifically, PoliBench assesses LLMs’ explicit refusal, robustness to adaptive attack (TAP), and robustness to different requester identities on prompts with authoritarian-assistance goals. Results on various LLMs reveal lower refusal rates on such type of requests than traditional harmful prompts, along with variation in safeguards’ effectiveness under reformulation or changed requester identity.

Critical strengths
  • Beyond explicit harmful goals, PoliBench provides a new perspective in examining LLM safety in the context of authoritarian assistance. Its additional evaluation on LLMs’ robustness against prompts with reformulation or different requesters also provides more insights on this new safety risk.
  • PoliBench covers twelve scenarios, three languages and ten open-weight and commercial LLMs. This enables holistic evaluation and comparison across various scenarios and setups.
Critical weaknesses
  • Dataset construction is insufficiently explained. Although the paper provides operational subcategories in the main paper and shortened example goals in the appendix, the exact pipeline of the dataset construction is not reported (e.g., whether the prompts are generated by LLM or human, how are such prompts selected, and how translation to other language is done, etc.). Omission of these details leads to difficulty in assessing the data quality.
  • Evaluation under adaptive reformulation is limited to TAP. While the results demonstrate the vulnerability of guardrail under adaptive attack by TAP, it remains unclear whether the observed patterns are TAP-specific. Additional evaluations with other adaptive attacks to reformulate the prompts (e.g., AutoDAN [1] and PAIR [2]) would strengthen the evaluation of guardrail robustness under reformulated prompts.

[1] Liu, X., Xu, N., Chen, M., & Xiao, C. (2024, May). Autodan: Generating stealthy jailbreak prompts on aligned large language models. In International Conference on Learning Representations (Vol. 2024, pp. 56174-56194).
[2] Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2025, April). Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) (pp. 23-42). IEEE.

Questions for authors
  • In Figure 3, the paper compares traditional harm and PoliBench by calculating their refusal rate difference. For such comparison, are the goals and categories covered in the traditional harmful datasets similar to those in PolichBench? If not, could such differences explain the observed explicit refusal gap?
  • Are the observed differences between requester identities robust to alternative prompt templates with different phrasing (other than the one shown in Table 9)?
Reviewer_GKUROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PoliBench is a benchmark for whether LLM safeguards withhold help with “authoritarian assistance”: requests for censorship, surveillance, election manipulation and similar, framed as routine governance or administrative work. It has 144 goals in 12 subcategories, with prompts in English, Chinese and French, and evaluates 10 target models. There are three research questions:

RQ1: Are these requests explicitly refused less often than traditional harmful requests? The paper reports a gap of 20.72 percentage points, with the direction holding in two length-matched comparisons. RQ2: Does an initial refusal still hold under a TAP adaptive attack? The paper finds Chinese commercial models refuse more often than US ones, but more of their refused goals are achieved after attack (“strict-yet-porous”). RQ3: Does describing the requester as state-affiliated rather than a non-state institution change the outcome? State affiliation raises refusal by 5.70 points and lowers attack success by 5.28 points.

Critical strengths

The topic is timely and under-studied. Pairing direct and attack outcomes for the same goal, model and language is a good design. The requester-identity comparison is a clean idea. Reporting numerators and denominators throughout is commendable.

Critical weaknesses
  • RQ1 measures refusal style, which contradicts the paper’s own thesis. The abstract says “refusal alone does not establish whether a model will … withhold actionable assistance.” But the headline RQ1 result compares only explicit refusal across domains. The traditional-harm pool has no goal-achievement score and no attack arm.

  • The refusal classifier has not been validated, and one result looks like a classifier failure. In particular:

    • The core result uses a keyword-and-length heuristic (Appendix B.3): a refusal phrase in the first 800 characters, a 1,200-character length rule, and counts of “structural markers.” No precision or recall against human labels is reported for any language.
    • Kimi K2.6 in Chinese is refused 5.56% of the time (8/144), compared with 88.89% in English and 94.44% in French. Yet its Chinese GAR is 3/144. That leaves 133 responses that are neither refusals nor full help, which points to Chinese refusal wording the rules miss.
    • The 1,200-character rule also treats verbose and terse models differently.
  • The goal-achievement judge is not validated, and the attack search optimises against it. Section 2 and Appendix B.3 say human review and an independent-judge check were done, but no results are reported: no agreement figures, sample size or protocol. gpt-oss-20b both gives TAP its search feedback and makes the final success call (score = 10). Optimising against the same judge that scores the result is known to inflate ASR. The consensus coders for the matched and factorial sets include a Seed model (the same family as the Doubao target) and GPT-5.6 (close to the GPT-5.5 target), which risks self-preference bias. Ask: report the human-agreement results, and re-score a sample with a held-out judge that played no part in the search.

  • Configurations differ across experiments and are misreported in the main text. Section 3.1 gives one TAP setting (depth 8, width 3, branching 8, budget 64, pruning 0.3). Table 11 shows the core runs used depth 8 with no recorded branching or pruning, while the requester-identity runs used depth 3. RQ2 and RQ3 attack success therefore come from different attacks: overall ASR is 64.6% in the core runs vs. 42.8% (886/2069) in the identity runs. Appendix B.1 says the PoliBench core used an earlier Doubao endpoint, while the traditional-harm and identity runs used Seed 2.1 Pro. Both are labelled “Doubao Seed 2.1 Pro,” so the Doubao part of the RQ1 gap compares two different models. Target decoding settings (temperature, max tokens, number of samples) are not reported, and each cell appears to be a single run.

Questions for authors
  • Refusal classifier validity. Has the rule-based explicit-refusal classifier been checked against human labels? If so, what are its precision and recall in each language? In particular, Kimi K2.6 in Chinese has an explicit-refusal rate of 5.56% (8/144) but a direct GAR of only 3/144. Have the remaining 133 responses been inspected to confirm they are not refusals the Chinese rules missed?
  • Goal-achievement judge validation. Section 2 and Appendix B.3 say human review and an independent-judge check were carried out, but no results are given. What were the sample size, the agreement with gpt-oss-20b at the score-10 threshold, and the protocol used? Because the same judge gives TAP its search feedback and makes the final success decision, has any subset been re-scored by a judge that played no part in the search?
  • TAP configuration across experiments. Section 3.1 gives one TAP setting (depth 8, width 3, branching factor 8, budget 64, pruning 0.3). Table 11 shows the core runs used depth 8 with no recorded branching or pruning, while the requester-identity runs used depth 3. Which configuration produced each reported result? Can RQ2 and RQ3 attack-success rates be compared directly?
  • Doubao endpoints. Appendix B.1 says the PoliBench core used an earlier Doubao endpoint, while the traditional-harm and requester-identity runs used Seed 2.1 Pro. Does the RQ1 gap for Doubao (Tables 16–19) therefore compare two different models? Does the pooled 20.72-point gap change if Doubao is excluded?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.