- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies whether empirical findings (for example, a certain prompt modification method can increase jailbreak success) about what makes a prompt more likely to jailbreak an LLM still hold when the same idea is expressed in different ways or tested under different settings. It proposes EDGEAUDIT to check how robust such conclusions are. The experiments show that the measured effect can change substantially with prompt format and context, so results from one specific prompt modification should not be generalized too broadly.
Critical strengths
The problem proposed is interesting. Some findings are interesting; for example, a single feature label is related to very different concrete prompt edits, results from one edit should not be generalized.
Critical weaknesses
-
The presentation is very hard to follow. One primary reason is that the paper includes too many new concepts and definitions at the beginning of the paper (in the Introduction section). It introduces these new concepts before giving a simple picture of what problem each of them is solving. The Introduction part uses uncommon terms such as "treatment versions", "selection policies", "local verdicts", and "materiality margin" before they are explained in the paper (even though explained later, it is not clear). Others in the later sections are also not clearly explained in the main content, such as policy support, recipe-to-target bridge, feature realization, and measurement view. Too many symbols and concepts in Section 3 are not clearly defined, e.g. Y_i, X', O, s_v, gamma_M, and U_v. Some of the notation and concepts are unnecessary and add little to the main argument. Introducing them only increases the reading burden and makes the paper harder to follow.
-
The motivation is not clear. The Introduction part says that one feature label may be related to multiple prompt edits, but it doesn't clearly explain why current prompt sensitivity analysis or multiple treatment version analysis are not enough (what the research gap is, how previous works do), and how EDGEAUDIT addresses the issues they have.
-
The experiments part is not clear. The paper uses different experimental settings (different models, datasets, or output budgets) for different conclusions. The main result uses a 200 token cap where many outputs are already near the limit and actual stop reasons were not recorded.
Questions for authors
-
Please give a clear definition of the main concepts and notation, especially for the concepts that are not common knowledge in this track of the ICLR community. Why do you use so many concepts and definitions, is it necessary?
-
What specific research gap does EDGEAUDIT fill beyond existing prompt sensitivity and "treatment" version analysis?
-
Can you state why different conclusions were evaluated with different models, datasets, and output budgets? How should these experiments be combined to validate your proposed framework EDGEAUDIT?
-
Why do you set 200 tokens as the limit in the main context effect result? Since so many outputs are near the cap, and you also do not actually record the stop reasons.