- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how refusal changes with the user’s stated chance of being in a position to act while the underlying request is held fixed. They test this by appending one sentence to each request stating this chance, varying it from 0.1% to 80%, across 16 models on 84 requests from OR-Bench, and measure refusal using two independent classifiers. Refusal rates increase systematically as the stated chance rises, growing by 7.50 percentage points under an LLM judge and 4.80 percentage points under a rule-based classifier. They conclude that while models are sensitive to stated likelihood information, this sensitivity does not make refusal decisions better targeted toward harmful requests.
Critical strengths
- The experiment is designed cleanly by holding the base prompt constant and properly isolating the effect of stated likelihood. Only one appended clause changes across conditions so any shift in refusal can be traced to that clause rather than to differences between requests. This is a real improvement over the closest prior work (Chen et al. 2026) which varied execution cost and request content together and so could not separate a likelihood signal from the request itself
- Testing is thorough across diverse models (both open-weight and closed-APIs). Evaluation is also done well using two independent classifiers and a targeted human adjudication of responses on which the classifiers disagree
Critical weaknesses
- This is an empirical study of prompt perturbations making the innovation very thin if any. Existing safety literature already demonstrates that small prompt variations change refusal behavior. The authors cite prior work showing that past-tense reformulations, persuasive reframing, and linguistic mutations can change refusal behavior. Adding an appended percentage clause simply checks another specific phrasing on a list of known triggers.
- Result is overclaimed. The experiment uses only 84 requests in total which are 42 toxic and 42 Hard-1K requests, having just 14 prompts per domain. The authors emphasize that there is no evidence that refusal increases more on harmful requests than on benign ones. But their own analysis shows the test can barely detect modest effects and it needs an interaction of 11.6 to 19.5 points (judge) just for 50% to 80% detection, and the confidence intervals are wide and span both directions. Presenting this as a main result confuses low power in a small sample with evidence that the two pools respond equally.
- Refusal does not scale with the stated probability, which suggests the model reacts to the rough level of the clause rather than the number it states. Under the judge classifier, pooled refusal rises in nearly equal steps across the four conditions of +2.80 points from 0.1% to 5%, +2.60 from 5% to 30%, and +2.90 from 30% to 80%. These steps cover very different probability ratios since 0.1% to 5% is a fifty-fold jump while 30% to 80% is under three-fold, yet the response is almost flat across them. Much of the movement is also present at the lowest clause since adding the 0.1% clause already raises refusal by +3.70 points over the no-clause baseline which is about half of the full 7.50-point effect, though that baseline jump has a confidence interval that includes zero. It looks like refusal responds to the presence and rough strength of an acting-related clause, not to the likelihood number it states, which is weaker than the paper's framing that refusal tracks the stated chance.
- The evaluation relies on a small 3.8B model and text heuristics to judge complex outputs from frontier models like GPT-4o and Claude Sonnet 4.6, creating a measurement risk. Phi-3.5-mini already shows basic reliability issues, reversing its label on 13.6% of all responses simply by swapping the prompt choice order. The authors do mitigate this by dropping order-unstable labels, reporting judge-human agreement of kappa 0.70 and showing the endpoint effect stays positive even when every unstable response is forced to one label. But this does not remove the concern since the disagreements concentrate at the partial-refusal boundary exactly where the full vs partial cut determines the reported refusal rate.
Questions for authors
Questions that materially affect assessment:
- Beyond showing that refusal responds to one more cue, what is the contribution relative to existing work on past-tense, persuasion, and linguistic mutations? A direct statement of the delta over those papers would help, since the empirical finding currently reads as incremental.
- Do you have any evidence that the model actually reads the number itself rather than just picking up on whether the clause sounds low or high? Two things suggest the latter. The jumps in refusal are about the same size between each level (+2.80, +2.60, +2.90) even though the probability gaps are very different, and only 7.60% of responses ever repeat the percentage they were given.