acceptodds
Under review as a conference paper at ICLR 2027

Benign Safety Is Not a Robustness Certificate: Evidence of a Structural Evaluation Gap Across Attack Conditions

Abstract

We study whether safety evaluation under benign conditions can substitute for direct adversarial robustness evaluation. We operationalize benign evaluation using a specified, equally weighted composite of helpfulness, harmlessness, and honesty across six selected instruction-tuned checkpoints. Exact-token assistant-prefill trajectories reveal the first level of measurement non-interchangeability: stronger scores on this composite do not consistently imply stronger prefix robustness. Among the original four checkpoints, robustness ordering under the prefix condition does not fully carry over to a structurally different transferred adversarial condition, establishing a second finite failure of substitution. Repeated interaction exposes additional frozen-evaluator failures missed by first-response assessment. Within the prefix protocol, model ordering remains stable across the tested grids even though robustness magnitudes depend materially on grid resolution. Human calibration with independent primary author annotation and third-author adjudication provides moderate support for the harm evaluator, with weaker agreement on adaptive examples. We characterize these findings as an empirical structural evaluation gap, not a defect in alignment-training mechanisms. They motivate distinct, condition-explicit benign and adversarial measurements, without implying that every benign evaluation or adversarial condition shares the observed relationships.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, LLM safety, safety alignment, adversarial robustness, jailbreak evaluation, robustness evaluation, language model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_in4COpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a safety evaluation under benign conditions can stand in for a direct adversarial robustness evaluation. Six open instruction-tuned models (2B-8B) are scored with a benign composite (AHHH, the mean of helpfulness, harmlessness and honesty scores from an LLM judge) and with an assistant-prefill attack on 315 harmful requests, summarized by a robustness AUC over prefill lengths of 0-20 tokens. For four of the models the authors also run a three-round retry test and a transferred GCG suffix (100 prompts). The model orderings under these measurements do not coincide, e.g. Qwen3 has a higher AHHH than Qwen2.5 but lower prefix robustness, and Qwen2.5 drops from second to fourth under GCG. The prefix ordering is stable across seven grids, and the GPT-4o harm judge is checked against 240 human labels (accuracy 0.87).

Critical strengths

The paper is careful about what it measures. The claims are restricted to the tested models and conditions, missing judgments are tracked and not imputed, and the judge is calibrated against human labels, with the weaker agreement on the multi-round examples reported openly. I also found the grid analysis useful: the ordering under the prefix attack does not change across the seven grids while the AUC value changes by up to 50%, which is a good reminder not to compare such numbers across papers.

Critical weaknesses
  1. The main message is already well accepted, and the evidence here is narrow. That a model which behaves well on benign prompts can still be jailbroken, and that robustness depends on the attack, is the starting point of the jailbreak literature (HarmBench, JailbreakBench, etc.). What this paper adds is a rank comparison among six small models (four for the GCG and retry parts), with one prefill attack and one transferred suffix whose success rate is only 1-18% on 100 prompts. The authors themselves describe the results as "finite counterexamples". I did not learn much beyond what I would have expected. There is also no method here. A large part of the paper is about bookkeeping (pinned revisions, missingness bounds, complete-case vs available-case sets, seven grids, hashes), and two of the four listed contributions are of this kind. This is good practice, but it is not a research contribution by itself.

  2. I am not convinced that the benign composite is a meaningful "benign safety" measure. Harmlessness is at or near the ceiling for all six models (4.80-5.00 in Table 2), so it cannot order them. The key counterexample (Qwen3 vs Qwen2.5) comes from helpfulness (4.28 vs 1.94) while both have harmlessness 5.00. So what is shown is that a helpfulness-driven score does not predict prefill robustness, and I don't think anyone would expect it to. It is also not clear where the benign prompts and the prefill targets come from; the paper says their source and authorship cannot be recovered.

And, by the way, the paper is a little bit hard to follow with such amount of self-clarification. It seems the author had relied too much on large language models while writing.

Questions for authors
  1. Does the conclusion hold if the benign measure is harmlessness only, or a standard benign safety benchmark, instead of the AHHH composite?
  2. Why is the helpfulness of Qwen2.5-7B so low (1.94 out of 5)? Is this a judge or truncation issue?
  3. For the prefill attack the judge sees the prefill together with the continuation. How much of the measured success comes from the supplied target tokens themselves?
Reviewer_igGyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether a benign safety evaluation can stand in for direct adversarial robustness measurement, and answers with "finite counterexamples" on six open instruction-tuned checkpoints (Llama-3.1-8B, Llama-3.2-3B, Qwen2.5-7B, Gemma-2-2B, plus Qwen3-8B and Phi-4-mini for part of the analysis). Benign behaviour is operationalised as AHHH, an equally weighted composite of helpfulness, harmlessness and honesty scored by a frozen GPT-4o judge on 350 items of undocumented origin. Adversarial robustness is measured three ways: (i) an assistant-prefill attack that inserts the first k ∈ {0,2,5,10,15,20} tokens of a paired harmful target from a locally held 330-row HEx-PHI derivative and summarises the ASR curve as a trapezoidal "Robustness-AUC"; (ii) a three-round persistence-retry interaction under three continuation policies (four models only); (iii) 100 JailbreakBench GCG suffixes optimised on Vicuna-13B and transferred unchanged (four models only). The reported findings are that the benign composite ordering does not match the prefix-RAUC ordering (Qwen3 > Qwen2.5 on AHHH but the reverse on RAUC), that the prefix ordering only partly matches the GCG-transfer ordering (4/6 pairwise agreements, Spearman 0.40 on n = 4), that any-round ASR exceeds first-round ASR, and that prefix ranks are stable across seven k-grids while RAUC magnitudes change by up to 50%. A 240-item author-annotated calibration of the harm judge gives machine–human κ = 0.72 overall and κ = 0.42 on the interaction suite. The authors frame all of this as an "empirical structural evaluation gap" and repeatedly disclaim population, causal or attack-general inference.

Critical strengths
  1. The judge-calibration appendix is done properly. Two blinded author annotators, a third adjudicator, pre-adjudication agreement reported separately from post-adjudication calibration, per-suite breakdown and confusion counts. The honest finding that the judge is weak on the interaction suite, at precision 0.55 and kappa 0.42, is useful, though it mostly undermines Section 4.2.
Critical weaknesses
  1. The headline counterexample rests on a benign measurement that is almost certainly a pipeline failure. Table 2 reports a helpfulness mean of 1.94 out of 5 for Qwen2.5-7B-Instruct against 4.28 for Qwen3-8B, 4.17 for Gemma-2-2B and 3.30 for Phi-4-mini. Qwen2.5-7B-Instruct is a strong general assistant, and that score under a 256-token cap and an empty system prompt points to a template, truncation or judge-parsing fault the paper never investigates. The level-1 ordering mismatch is driven entirely by this value: with Qwen2.5 in the range of every other model its composite would exceed Qwen3's, and its higher prefix robustness would then agree with the benign ordering. The remaining composites, from 4.27 to 4.36, sit inside the reported bootstrap intervals, so no other ordering is resolved either.

  2. The adversarial measurements cannot be interpreted, by the authors' own account. Appendix I.1 states that for the original four checkpoints "the generation/judging implementation is not included in the available manuscript evidence", so the authors cannot say whether the judge scored the continuation alone or the attacker-supplied prefill plus continuation in the runs that produce Figures 1 to 3. For the two models where the payload is known, the judge scores prefill and continuation together, meaning a large part of the measured attack success is the attacker's own text being graded. The prefill targets' origin and authorship are separately described as irrecoverable, and the benign set's sources, selection and authorship are "not established".

  3. The second-level claim rests on a single transferred artifact at floor-level success with four targets. The cross-condition comparison uses one Vicuna-optimised GCG artifact with transfer rates of 1 to 18%, overlapping intervals for the Llama pair, and a rank correlation computed on four points. The three continuation policies carrying Section 4.2 are never defined, since Appendix D records that the source implementation and strings "are not available". Repeated disclaimers that these are descriptive comparisons do not convert undefined measurements into evidence.

Questions for authors
  1. Please show five raw Qwen2.5-7B-Instruct helpfulness outputs with their judge scores. Was the Qwen2.5 chat template applied, and were responses truncated at 256 tokens? If the 1.94 helpfulness score is an artefact, does any benign-vs-prefix ordering mismatch survive after correction, within the bootstrap intervals?

  2. For the original four checkpoints, what text did gpt-4o-2024-11-20 score at each k: the continuation only, or prefill + continuation? If this cannot be established, how can Figures 1–3 and the RAUC values be interpreted at all?

  3. What are the three continuation policies (exact templates for "fixed", "refusal-aware", "paraphrase-retry")? Without them, Section 4.2 is not evaluable.

  4. Where did the 350 benign items and the 330 paired harmful targets come from? Who wrote the targets and under which licence?

  5. At k ≥ 5 all models sit at ASR 0.6–0.95. What does a "successful" response look like once 20 tokens of the harmful target have been force-fed? Is the model adding harmful content, or merely completing an attacker-supplied sentence?

  6. If the goal is to show attack-dependent rankings, running the publicly available HarmBench or JailbreakBench attack suites on the same six checkpoints would give dozens of attacks instead of one transferred artifact, and the comparison would then be against a known reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.