acceptodds
Withdrawn from ICLR 2027

One Feature Label, Many Recipes: Auditing Jailbreak Feature Claims Across Prompt Versions and Environments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) remain vulnerable to jailbreak attacks that elicit harmful responses. Prior work uses interpretable prompt features to explain jailbreak behavior and guide attack and defense design. However, one feature label can correspond to many concrete prompt edits (recipes), making it unclear when evidence from individual recipes supports a broader feature claim. We propose EdgeAudit, a framework that jointly specifies how recipe effects may be aggregated and how the resulting evidence may be interpreted. The key idea is to represent a recipe family by the set of effects induced by its permitted selection policies, and to tie the corresponding confidence bounds to the evidence each interpretation level requires. This formulation distinguishes support for an average over recipes from support for every allowed recipe, and claims about an assigned edit from claims about the intended feature or an edge in a learned feature graph. It thus preserves supported local findings while identifying the additional evidence required for stronger interpretations or extensions to other environments. In paired audits on Mistral at a 200-token output budget, requesting five numbered steps instead of a continuous response raises the HarmBench-judged harmful-compliance rate by 0.462 in plain context but by only 0.115 under jailbreak scaffolds from JailbreakHub. On held-out Qwen behaviors, changing the requested count from one to five has an effect 0.155 larger in a checklist than in a JSON array. Effects attributed to one feature label can therefore depend on context and format.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Safety, Jailbreak Evaluation, Prompt Sensitivity, Robustness Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QiDbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether empirical findings (for example, a certain prompt modification method can increase jailbreak success) about what makes a prompt more likely to jailbreak an LLM still hold when the same idea is expressed in different ways or tested under different settings. It proposes EDGEAUDIT to check how robust such conclusions are. The experiments show that the measured effect can change substantially with prompt format and context, so results from one specific prompt modification should not be generalized too broadly.

Critical strengths

The problem proposed is interesting. Some findings are interesting; for example, a single feature label is related to very different concrete prompt edits, results from one edit should not be generalized.

Critical weaknesses
  • The presentation is very hard to follow. One primary reason is that the paper includes too many new concepts and definitions at the beginning of the paper (in the Introduction section). It introduces these new concepts before giving a simple picture of what problem each of them is solving. The Introduction part uses uncommon terms such as "treatment versions", "selection policies", "local verdicts", and "materiality margin" before they are explained in the paper (even though explained later, it is not clear). Others in the later sections are also not clearly explained in the main content, such as policy support, recipe-to-target bridge, feature realization, and measurement view. Too many symbols and concepts in Section 3 are not clearly defined, e.g. Y_i, X', O, s_v, gamma_M, and U_v. Some of the notation and concepts are unnecessary and add little to the main argument. Introducing them only increases the reading burden and makes the paper harder to follow.

  • The motivation is not clear. The Introduction part says that one feature label may be related to multiple prompt edits, but it doesn't clearly explain why current prompt sensitivity analysis or multiple treatment version analysis are not enough (what the research gap is, how previous works do), and how EDGEAUDIT addresses the issues they have.

  • The experiments part is not clear. The paper uses different experimental settings (different models, datasets, or output budgets) for different conclusions. The main result uses a 200 token cap where many outputs are already near the limit and actual stop reasons were not recorded.

Questions for authors
  • Please give a clear definition of the main concepts and notation, especially for the concepts that are not common knowledge in this track of the ICLR community. Why do you use so many concepts and definitions, is it necessary?

  • What specific research gap does EDGEAUDIT fill beyond existing prompt sensitivity and "treatment" version analysis?

  • Can you state why different conclusions were evaluated with different models, datasets, and output budgets? How should these experiments be combined to validate your proposed framework EDGEAUDIT?

  • Why do you set 200 tokens as the limit in the main context effect result? Since so many outputs are near the cap, and you also do not actually record the stop reasons.

Reviewer_rvT8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper points out that a jailbreak feature can be expressed through different prompt edits, and that the effect of one edit may not apply to others. It proposes EdgeAudit to combine the effects of these edits under predefined selection policies and distinguish three levels of causal claims. The experiments mainly use Mistral-7B, with additional tests on Qwen3-8B and benign tasks to check the decision rules.

Critical strengths

(1) The paper distinguishes average effects across recipes from effects that hold for every recipe. It also separates the effect of a specific prompt edit from that of the intended feature, avoiding unsupported causal claims. Post-hoc analyses and unsuccessful audits are also reported. (2) The main experiments use pre-specified settings. The Qwen carrier difference was confirmed on held-out behaviors, rather than the original test cases. The Mistral context comparison used the same 52 behaviors in both settings, keeping the test cases fixed.

Critical weaknesses

(1) The paper does not explain why this problem matters for jailbreak research. An attack can still be useful if it improves success rates across harmful behaviors and models, even when some prompt edits work better than others. The paper does not show that such differences led to wrong conclusions in previous work, or that applying EdgeAudit would change the attack or defense methods of Pan et al. [1]. (2) The reported differences across prompt formats and contexts could be tested with standard controlled experiments. EdgeAudit mostly uses existing statistical tests and causal inference methods [2–4], and prior work has treated prompt edits as compound treatments [5]. For example, the Specific Opening experiment (Table 4) reports a minimum simultaneous lower bound of 0.10625 across recipes and a separate post-hoc lower bound of 0.20156 for the uniform mixture. (3) The framework defines three levels of causal interpretation, but the experiments cover only Level 1. Levels 2 and 3 are not tested, and the bridge term is never estimated, even in the benign calibration tasks. The main experiments use Mistral and Qwen with 52–160 harmful behaviors per comparison. Positive Character produces only 1–3 harmful responses per 100 behaviors, and Output Encoding is not tested on harmful prompts.

[1] L. Pan et al., “A Causal Perspective for Enhancing Jailbreak Attack and Defense,” NDSS, 2026. [2] R. L. Berger, “Multiparameter Hypothesis Testing and Acceptance Sampling,” Technometrics, 1982. [3] M. A. Hernán and T. J. VanderWeele, “Compound Treatments and Transportability of Causal Inference,” Epidemiology, 2011. [4] T. J. VanderWeele and M. A. Hernán, “Causal Inference Under Multiple Versions of Treatment,” Journal of Causal Inference, 2013. [5] Z. Yang et al., “Compared to What? Baselines and Metrics for Counterfactual Prompting,” COLM, 2026.

Questions for authors

Please see the weaknesses above.

Reviewer_DLreOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper asks: if we know that some specific jailbreak prompt is more likely to work when it's one thing instead of another, does this support the claim that "Number Of Task Steps" (NTS) increases compliance? One feature label can be implemented in many different ways - for example, NTS could mean either "answer five numbered steps", or changing ONE → FIVE inside a checklist, or changing ONE → FIVE in a JSON array. The paper proposes EdgeAudit which answers by thinking of each recipe as an alternate version of the same treatment. A claim about an average over all recipes is a different query than a claim about every possible recipe. The authors also separate claims into three levels of interpretation: the effect of the edit you were actually given, the effect of the intended feature, and the effect of a learned edge between two features. Higher-level claims require more evidence. One jailbreak feature can behave differently across context and format. The authors test eight natural audit questions and find that EdgeAudit returns RETAIN once, NO-RETAIN four times, and ABSTAIN three times.

Critical strengths
  1. The paper shows how an average over recipes and a claim about every recipe can be different
  2. The paper is very carefully done. Most jailbreak papers report a single success rate from one prompt template. The paper's finding of 0.155 difference between checklist and JSON on Qwen3-8B was confirmed by looking at the results of 58 held-out prompts whose outcomes were not seen until after their protocol was fixed, so we have some confidence in its accuracy here. The Background Task and Number Of Background Info nulls are more than just a lack of a found effect; with effects between -0.019 and +0.013, these tests would all have been able to detect an effect size of 0.20 with probability greater than .999. This matters because the recomputed aggregate numbers match theirs to within one part per trillion. If they can be trusted then so can the negatives and there are many negatives
Critical weaknesses

The biggest weakness is that the part the paper claims to be about, moving from a claim about an edit to a claim about a feature or graph edge, isn't actually used anywhere in the paper. The only Level 2 result (as opposed to Level 1) is algebraic, and involves taking 0.21 minus two discrepancies of 0.02 to get 0.17. Every other natural example stays at Level 1. There are some attempts to see if features were realized, e.g. one test on harmless requests with no harmful content finds that "more steps" doesn't work and there's a required increase of 3 but only found an increase of 2. Another test on Output Encoding fails outright: zero out of twenty decodes was correct. What remains at level one is that format and context affect jailbreak effects, but this isn't a novel result.

All three models studied had 7-8 billion parameters. There were between 52 and 160 harmful prompts. For the single RETAIN case, where it's found that requested-count recipes work fine on Mistral, the confidence intervals give lower bounds of .418 and .500 respectively, both well above the threshold of 0.20 needed. But the framework doesn't add anything, you could just calculate it without EdgeAudit and come to the same conclusion. The remaining seven results all decline to make claims about these edits or features. The only value added from using the framework is not to make claims when you shouldn't. I would be interested to see if it ever does something else.

Questions for authors
  1. I'm not sure how to interpret the difference between 0.462 and 0.115 in plain context vs JailbreakHub scaffolds, since the scaffold control rate is already at 0.558. Can we think of this as a ceiling effect? Also, was there some scale you were working with before seeing these results (since your odds-ratio check seems like something that would have been designed after looking at the data)? If it's mostly a ceiling effect then the headline result about "context dependence" becomes weaker.
  2. How do you estimate epsilon for an actual feature (like Number Of Steps) rather than the edit used here? Is there any way to do this? If no, Levels 2 & 3 don't seem very useful.
  3. Did you get humans to label any subset of AI responses?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.