- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper formalizes the evaluation of LLM safety as a dynamic, two-player game between an evaluator auditing for jailbreaks and an adaptive trainer performing robustness fine-tuning. Moving beyond static benchmarks that fail to differentiate authentic alignment from memorization, the authors model query generation and data augmentation using group actions over a metric prompt space. By formalizing local generalization as an -neighborhood around fine-tuning data, the authors analyze the circle translation game with cyclic groups, uncovering a phase transition at the critical threshold, below which the evaluator sustains a constant miss ratio for linearly many rounds. Empirically, the authors fine-tune three model families (Llama-3.1-8B, Mistral-7B, Qwen2.5-7B) on jailbreak prompts from WildJailBreak, validating that safety refusal transfer decays monotonically with embedding distance to the training set.
Critical strengths
- A game-theoretic framework for adversarial evaluation is proposed.
- The empirical study evaluates that whether fine-tuning on adversarial prompts induces only local generalization in embedding space, providing empirical validation for the foundational -neighborhood modeling assumption.
- This framework redefine the core of adversarial evaluation, i.e., a benchmark is not a static set of prompts but an orbit under the evaluator’s group action, and audit protocols that ignore trainer-side adaptation cannot distinguish a genuine fix from a memorized patch.
Critical weaknesses
- As stated, most of work were conducted by LLMs, causing format errors, such as blank content in Broader impacts, grammar errors, and poor readability of the whole paper. The writing of theoretical analysis is difficult to understand.
- In Section 3.1 (Eq. (1), p. 4), the variable is introduced as an element indexing the dataset (), but is subsequently redefined in Appendix B.1.1 (p. 14) as the integer ratio , creating notational collisions and ambiguity between metric space elements and group indices.
- In Appendix D.7 (Eq. (3), p. 27), the parameter is introduced as a clipping boundary to prevent Binomial likelihood saturation, which overloads the central symbol used throughout Sections 3 and 4 to denote the generalization radius, compromising formal notational consistency.
- The empirical refusal evaluation relies exclusively on a single open-weight judge model (OLMO-3.1-32B-Instruct, Sec. 4, p. 7; Appendix D.6), leaving open the possibility of systematic judge-specific classification biases influencing the observed distance-decay relationships.
- The framework assumes that data augmentations form a group where every operation is invertible and composable (Sec. 1, p. 2; Sec. 3.2, p. 5), whereas realistic text-based adversarial transformations (such as style transfers, obfuscation, or persona framing) frequently lose semantic content and violate invertibility.
Questions for authors
- It is suggested to include cross-validation of refusal labeling using alternative judge models (e.g., more recent models) and report inter-annotator agreement metrics (e.g., Cohen's Kappa) across the three-way rubric.
- It is suggested to expand the active training set beyond 40 prompts to varying scales (e.g., 100 and 500 prompts) to determine whether the boundaries of the Constrained Piecewise-Linear regime shift or smoothly merge into global coverage.
3 comments
Rejudging
We ran experiments by using more recent models as judges. We used Gemma-4 12B and Granite-4.2-8B. For the 3-way grading (Refuse / Comply / Ambiguous), the Cohen's Kappa values for fine-tuned models range from 0.56 to 0.67. When dropping the prompts judged "Ambiguous" by at least one of the models, thus restricting only to Refuse/Comply, the values go from 0.81 to 0.91.
response
Thank you for the author's response. This article still requires extensive revisions and cannot currently meet the acceptance criteria. I will keep my original score.
We thank the reviewer for the detailed reading. We address each point below.
W1. As stated, most of work were conducted by LLMs, causing format errors, such as blank content in Broader impacts, grammar errors, and poor readability of the whole paper. The writing of theoretical analysis is difficult to understand.
Our AI-use statement follows the ICLR policy and is intentionally exhaustive. We did most of the work ourselves and use LLM marginally. The empty “Broader impacts” paragraph is a formatting error: its content is the following paragraph, “Beyond jailbreaks”, which we will merge under the heading.
To improve the readability of the theoretical analysis, we will add a notation table, and we would be grateful if the reviewer could indicate us to the points they found hardest to follow.
W2–W3. Notation collisions.
We thank the reviewer for spotting these collisions, which are both confined to the appendices. We will rename them to avoid notation collisions.
W4 and Q1. It is suggested to include cross-validation of refusal labeling using alternative judge models (e.g., more recent models) and report inter-annotator agreement metrics (e.g., Cohen's Kappa) across the three-way rubric.
We thank the reviewer for this suggestion. We are currently relabeling a subset of the responses with a second, more recent judge model, using the same three-way rubric, and we will report Cohen’s kappa between the two judges in this thread in the coming days. Would the reviewer also find it useful for us to label a subset of responses by hand, so that we can also report the agreement between the judge and human labels? We will add these results to the revision.
Q2. It is suggested to expand the active training set beyond 40 prompts to varying scales (e.g., 100 and 500 prompts) to determine whether the boundaries of the Constrained Piecewise-Linear regime shift or smoothly merge into global coverage.
We thank the reviewer for this question. Expanding the active training set adds new training prompts, each with its own neighborhood, without widening each neighborhood. The boundaries should therefore not shift, and coverage becomes global only once the training prompts are close to every query the evaluator can send.
We used 40 prompts because they already validate the principle we rely on: an audit can be gamed by memorization with as few as 40 prompts. What matters to the evaluator is to know if their test can be gamed, whether this was done by memorizing 40 prompts or with many more.
W5. The framework assumes that data augmentations form a group where every operation is invertible and composable (Sec. 1, p. 2; Sec. 3.2, p. 5), whereas realistic text-based adversarial transformations (such as style transfers, obfuscation, or persona framing) frequently lose semantic content and violate invertibility.
We thank the reviewer for this remark: looking into it more closely, we realized that invertibility is in fact not required, and that the framework extends directly to monoids, i.e. transformations that can be composed but not necessarily undone. For instance, persona framing such as adding a role-play prefix (“You are DAN…”) acts by prepending it to the prompt. Prefixes compose by concatenation, and “no prefix” is the identity, but a prefix cannot be undone, and it need not preserve semantic content. We propose to state the framework this way in the revision. This changes nothing in our paper: none of the framework’s definitions uses inverses, since the players only compose transformations and never undo them, and our theoretical results concern cyclic translation games, whose translations form a monoid that happens to be a group.
We thank the reviewer again for their careful reading and constructive suggestions. In the revision, we will fix the formatting and notation issues, add a notation table, report the cross-validation of refusal labels, and state the framework for monoids. We hope these answers and changes address the reviewer’s concerns, and we would be grateful if they would consider raising their score.