- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose an evaluation framework that measures effectiveness of adversarial attacks against LLMs not just by the success of the attack but also by the computational costs of the attacks, which allows to better compare adversarial attacks empirically.
Critical strengths
Writing quality. The paper is generally well-written and easy to follow. The problem statement and experimental settings are also clearly described in the paper. There is some usage of unpolished, AI-generated words and phrases, but overall the text is easy to follow.
Many experiments and insights. The experiments are thorough, in particular, the authors evaluate the safety of a lot of different models, and provide many insights in their experiment section (with a few limitations, see weaknesses). Plots are generally clear and easy to understand. The authors provide additional results in the Appendix as well. Overall, the experimental evaluation is quite thorough.
Critical weaknesses
Weak contributions. The main contribution of this paper (secction 2.1) is to introduce computational costs as an additional axis when evaluating adversarial attacks against LLMs. Although this leads to a few empirical insights, the technical contribution is quite limited in novelty, in particular since evaluating and comparing LLM attacks using FLOPs to approximate compute costs is not new (see also next point). Key evaluation metrics are further not sufficiently motivated or justified (see sections 2.2 on risk-compute curves and 2.3 on summary metrics), for example, it remains unclear when to use which metric and whether they are sufficient or just represent instantiations of the general framework. In general, the contributions of this paper are weak.
Missing comparison to related work. The main claim that evaluating attacks under computational costs has not been done before is wrong, see e.g. [1], which also compares attacks under computational cost budgets. A direct comparison and discussion of this work is missing. In general, the positioning in the literature in the related work section (section 5) could be more thorough.
Insufficient amount of adversarial attack algorithms in the evaluation. The paper mainly claims that introducing computational costs of attacks as another evaluation axis is critical, but they only compare three different attacks (JailBroken, GCG, PAIR). While they certainly evaluate sufficiently many and different models under their evaluation framework, the attack algorithms they use to instantiate their framework is limited to three, which is not sufficient for the general claims of this paper.
Inconsistent wording. The many different words for computational cost (measured by FLOPs) can be misleading, this already starts in the title of the paper. Words such as computational or adversarial 'pressure', 'effort' or 'burden' (L315) all just mean computational cost, which should be used consistently throughout the paper to improve clarity.
To conclude, weaknesses 1), 2) and 3) are the main arguments for recommending weak rejection.
[1] Beyer et al., Sampling-aware Adversarial Attacks Against Large Language Models, ICLR 2026.
Questions for authors
Do you also sample responses from the LLMs as in [1] and consider the costs for sampling, or do you just consider greedy decoding? How does your framework relate to [1] in general? Does your evaluation framework reduce to a special case of [1] without sampling responses from the model's output distribution? Does your framework provide additional insights not already described in [1]?
5 comments
Rebuttal Response - Responses to the Questions
Questions
We use stochastic sampling, not greedy decoding: target responses are generated with temperature 0.7 and top-p=0.9, and we charge the generation FLOPs for each response, as well as the judge FLOPs for evaluating it. We run 10 seeds per configuration to capture variability across attack trials. In [1]’s (T ,n) notation, our setup uses one sampled target response per attack iterate, n=(1,...,1); unlike [1], we do not vary the number of target responses sampled from the same prompt or optimize when to sample them.
Our work and [1] both examine adversarial attacks through a compute-aware lens, but they tell different stories. Reference [1] focuses on how an attacker can allocate a fixed compute budget between prompt optimization and repeated sampling to make an attack more effective. Our paper uses computational cost to examine how conclusions about risk change across attack strategies, model families and sizes, training stages, harm categories, and evaluation choices. We sample target responses and charge their generation cost; our RL attacker also samples candidate prompts across rollout sessions. However, we do not repeatedly sample multiple target responses from each prompt or optimize that sampling schedule, which is the sampling strategy studied in [1].
Our framework is therefore not simply a special case of [1]. Although our current attack setup uses one target response per query, our central contribution is a systematic evaluation across scenarios, with risk–compute curves and complementary metrics, rather than an optimization of the attacker’s sampling schedule. We also examine the sensitivity of conclusions to judge choice and cost accounting, including evaluations with and without judge compute, and alternative cost measures such as wall-clock time and estimated API cost. We go beyond binary ASR with graded harmfulness and category-level analyses.
The broader message of our findings is that the choice of metric can misrepresent how vulnerable a model is and how much computational pressure an attack requires. For example, the RL-based adaptive attack achieves high endpoint ASR, yet the cheaper JailBroken attack reaches the 50% risk threshold with less compute across the evaluated models on both benchmarks. Our results also show that robustness patterns vary across training stages, model scales, harm categories, judges, and cost conventions. These findings complement [1]’s attack-design results: they show why safety evaluations should report multiple, complementary views of risk and cost. If jailbreaks cannot be eliminated entirely, evaluations should help determine whether eliciting harmful behavior is sufficiently costly to make an attack impractical for the relevant adversary.
[2] Nasr, M., Carlini, N., Sitawarin, C., Schulhoff, S. V., Hayes, J., Ilie, M., ... & Tramèr, F. (2026). The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against {LLM} Jailbreaks and Prompt Injections. In 35th USENIX Security Symposium (USENIX Security 26) (pp. 1467-1486).
Thank you for the discussion. I still have to read discussions with the other reviewers.
Regarding W1, from my perspective it is still not entirely clear so far whether your proposed framework provides the right perspective on adversarial cost and is general enough to correctly account for the cost of different sampling strategies. For example, a setting your framework would not correctly cover are algorithms that require fewer attack iterations but are only effective when combined with sampling more than one target response.
I also have a follow-up question, could you elaborate on how and in which scenarios practitioners might adopt your framework?
Rebuttal Response - Discussing Weakness Areas
We thank the reviewer for a careful and useful review. Below, we respond to each weakness and question in turn:
W1,2
We thank the reviewer for identifying this closely related work. To clarify, we do not claim that ours is the first work to evaluate LLM attacks under computational cost budgets. We will make this explicit and cite and discuss Beyer et al., Sampling-aware Adversarial Attacks Against Large Language Models (ICLR 2026). Their work studies how to allocate a fixed FLOP budget between prompt optimization and repeated sampling to improve attack effectiveness. Our work has a complementary focus: we use cumulative FLOPs to characterize how risk changes across compute budgets and provide risk–compute curves and summary metrics for comparing diverse attack strategies, models, training stages, and benchmarks. In this sense, Beyer et al. focus on designing more compute-efficient sampling-aware attacks, whereas we focus on a common evaluation framework for measuring compute-dependent adversarial robustness. We will cite this paper and clarify this distinction and expand the related-work discussion to better situate our contribution.
The risk-compute curves show how attack success changes as computational pressure increases, while C@τ and average efficiency (AE) summarize the curves to support ranking and comparison across settings. We use 50\% as a representative threshold, not a universal operational cutoff. These metrics are complementary rather than alternative summaries: C@τ measures the compute required to reach a specified risk threshold, while AE summarizes risk per unit of compute at the evaluated budget. For example, on Tulu3-SFT, neither GCG nor PAIR reaches τ=0.5, so by [email protected] only, they cannot be distinguished. AE remains a distinguishing metric and shows a sevenfold difference in efficiency. Together with the curves, these metrics help characterize attack behavior and offer a view relevant to attackers with limited budgets. We agree that other summary metrics may be appropriate for specific evaluation or deployment needs.
W3
To clarify, the submission evaluates four attack families (we will reword Section 3.1 of the paper to make it clearer): static template-based JailBroken, black-box iterative refinement (PAIR), white-box gradient optimization (GCG), and the RL-based adaptive attack of Nasr et al. (2025) [2]. The RL attack was reported in Appendix I but not included in the main results table, which may have made it easy to miss. We will make its presence clearer in the paper.
Nasr et al. [2] identify their RL-based adaptive attack as their strongest automated attack, using feedback from the target to iteratively refine prompts and update the attacker with GRPO. In App. I, the RL attacker attains the highest or tied-highest endpoint ASR on 6-7 of 9 models on both HarmBench and JailbreakBench, missing only on Tulu3-Base and Qwen2.5-0.5B, where every attack is at or above 0.98. The compute metrics order the attacks differently: JailBroken is the most efficient attack on all 9 models in both benchmarks (highest AE and lowest [email protected]), and RL is never the most efficient on either. ASR alone therefore ranks RL first while obscuring that it buys its success at 1.1–13× the FLOP cost per unit of risk. This illustrates how attack success alone can obscure differences in computational cost and efficiency.
We selected attacks to represent distinct families and demonstrate how accounting for compute can change the interpretation of results. Our goal was not to exhaustively evaluate every algorithm. Our framework is modular, with attack-specific cost accounting that allows new methods to be added, and we will add a sampling-based attack from Beyer et al. (2026) to broaden the evaluation. We also plan to maintain and expand the framework over time as additional state-of-the-art attacks emerge.
W4
We agree that the submission used several near-synonyms for a single quantity, and we will standardize the terminology. "Computational pressure" will be the only term for the measured quantity: the cumulative FLOPs an attack incurs over refinement steps, averaged across prompts, as defined in Section 2.1. Where we need to name the unit explicitly, we will write "cumulative attack FLOPs." We will remove the near-synonyms currently used for this quantity, including "compute pressure," "adversarial compute cost," "adversarial effort," "computational effort," "computational burden," and "adversarial investment." "Work factor" will appear only where we invoke the classical-security concept. The quantity will be defined once in Section 2.1, with no synonym introduced after that.
We responded to the questions in a follow-up comment.