- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PoliBench, a benchmark for evaluating whether large language models (LLMs) provide assistance toward authoritarian objectives, beyond conventional harmful-request refusal evaluations. The benchmark contains 144 goals across 12 operational categories, with prompts in English, Chinese, and French, and evaluates 10 LLMs. The authors investigate differences in refusal behavior between authoritarian-assistance and traditional harmful requests, whether initially refused requests can still succeed under adaptive Tree of Attacks with Pruning (TAP) attacks, and how descriptions of state-affiliated versus non-state institutional requesters affect model behavior. The results show that explicit refusal rates are 20.72 percentage points lower for authoritarian-assistance requests than for traditional harmful requests, and that TAP can achieve full goal completion for 62.32% of initially refused eligible requests. Requester descriptions also affect refusal and attack-success rates. Overall, the paper highlights that refusal rates alone may not adequately characterize safeguards against authoritarian assistance, although the evaluation is limited to the tested goals, models, languages, attack configuration, and automated assessment procedure.
Critical strengths
-
The paper examines authoritarian assistance as a distinct evaluation setting rather than assuming that safeguards against conventional harmful requests generalize to politically coercive objectives.
-
By applying adaptive TAP attacks to requests that models initially refuse, the paper distinguishes surface-level refusal from whether the underlying objective can still be achieved. The finding that 62.32% of initially refused eligible requests reach full goal completion is particularly consequential because it exposes a limitation of evaluating safeguards using refusal rates alone.
Critical weaknesses
-
The adversarial evaluation relies on a single TAP configuration. Consequently, it remains unclear whether the reported goal-completion rates are robust across alternative attack methods, attack budgets, or evaluation protocols. Since adversarial robustness is central to the paper's claims, broader validation would strengthen the evidence that the observed vulnerabilities are not specific to the chosen setup.
-
The evaluation relies substantially on automated assessment of whether a goal has been achieved. This may be challenging for complex, multi-step authoritarian objectives, where responses can be incomplete, context-dependent, or superficially compliant without being operationally useful.
Questions for authors
Please see the weakness.