- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces "SafetySurface", a two-stage black-box jailbreaking framework that builds reusable behavioural profiles of target model before optimising attack prompts. Experiments on AdvSub and HarmBench, alongside commercial-model evaluations and profile ablations, report improved attack success and query efficiency.
Critical strengths
The cross-profile transfer experiments provide evidence that the profiles capture useful information about the target: using each model’s own profile achieves 91.5% average ASR on AdvSub, compared with 72.3% when using profiles from other models.
Critical weaknesses
- PGA’s methodological contribution appears incremental, as it largely retains the standard iterative attack framework and introduces a reusable pre-attack profile as its main addition.
- Key baseline configurations, including TAP’s search parameters and budget enforcement, PAIR’s stream allocation, and AutoDAN-Turbo’s warm-up and strategy-library setup, remain insufficiently specified. The datasets and sample sizes used for commercial-model evaluations are also unclear. These omissions limit reproducibility and fair assessment of the claimed improvements.
Questions for authors
-
Which dataset, sample size, query budget, and number of runs were used for the results in Figure 3b? Were any baseline results obtained under the same conditions?
-
The main experiments count near-operational responses as successful, whereas the original TAP and PAIR evaluators require comprehensive fulfillment of the harmful task. Please clarify how the baseline evaluation and stopping rules were adapted, and whether the reported gains persist under a stricter, consistently applied success criterion.
-
Table 4 describes Q = 8.1 as the amortized total query cost, whereas Appendices C.3 and C.5 describe the same value as the Phase 2 query cost.
Typo: Figure 2 reports authority_compliance as both 0.00 (in text) and 0.74 (in plot).