acceptodds
Under review as a conference paper at ICLR 2027

The Evaluation Game: Beyond Static LLM Benchmarking

Abstract

Jailbreaks are adversarially crafted inputs that bypass safety constraints. As they continue to be discovered in Large Language Models, practitioners increasingly rely on fine-tuning as a defensive strategy. Yet the theoretical foundations underlying this robustness fine-tuning remain underexplored. We introduce a game-theoretic model in which the interaction between an evaluator (auditing the LLM for jailbreaks) and a trainer is formalized as a two-player game. A key feature of our approach is the use of group actions to formally represent transformation-based data augmentation. A group action is a mathematical structure that captures symmetries and transformations. The simplest non-trivial instance is the circle with cyclic translation groups, where we exhibit various regimes depending on the trainer's generalization range. Below a critical threshold, the evaluator maintains a constant miss ratio for linearly many rounds, whereas other settings can yield very different behaviors. We further provide empirical evidence supporting locality-dependence of the model: for the three model families we tested (Llama, Qwen and Mistral), we have significant evidence that fine-tuning on adversarial prompts induces only local generalization, with refusal rates on test examples strongly dependent on the distance to the fine-tuning prompts. Our framework recasts the central object of adversarial evaluation: a benchmark is not a static set of prompts but an orbit under the evaluator's group action, and audit protocols that ignore trainer-side adaptation cannot distinguish a genuine fix from a memorized patch.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
mathematics, jailbreaking, game theory, data augmentation, group actions, group theory, fine-tuning, embeddings

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_fH7QOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper formalizes the evaluation of LLM safety as a dynamic, two-player game between an evaluator auditing for jailbreaks and an adaptive trainer performing robustness fine-tuning. Moving beyond static benchmarks that fail to differentiate authentic alignment from memorization, the authors model query generation and data augmentation using group actions over a metric prompt space. By formalizing local generalization as an ϵ\epsilon-neighborhood around fine-tuning data, the authors analyze the circle translation game with cyclic groups, uncovering a phase transition at the critical threshold, below which the evaluator sustains a constant miss ratio for linearly many rounds. Empirically, the authors fine-tune three model families (Llama-3.1-8B, Mistral-7B, Qwen2.5-7B) on jailbreak prompts from WildJailBreak, validating that safety refusal transfer decays monotonically with embedding distance to the training set.

Critical strengths
  1. A game-theoretic framework for adversarial evaluation is proposed.
  2. The empirical study evaluates that whether fine-tuning on adversarial prompts induces only local generalization in embedding space, providing empirical validation for the foundational ϵ\epsilon-neighborhood modeling assumption.
  3. This framework redefine the core of adversarial evaluation, i.e., a benchmark is not a static set of prompts but an orbit under the evaluator’s group action, and audit protocols that ignore trainer-side adaptation cannot distinguish a genuine fix from a memorized patch.
Critical weaknesses
  1. As stated, most of work were conducted by LLMs, causing format errors, such as blank content in Broader impacts, grammar errors, and poor readability of the whole paper. The writing of theoretical analysis is difficult to understand.
  2. In Section 3.1 (Eq. (1), p. 4), the variable ss is introduced as an element indexing the dataset (s∈Dntrains \in D_n^{train}), but ss is subsequently redefined in Appendix B.1.1 (p. 14) as the integer ratio s=p/gs = p/g, creating notational collisions and ambiguity between metric space elements and group indices.
  3. In Appendix D.7 (Eq. (3), p. 27), the parameter ϵ=10−4\epsilon = 10^{-4} is introduced as a clipping boundary to prevent Binomial likelihood saturation, which overloads the central symbol ϵ\epsilon used throughout Sections 3 and 4 to denote the generalization radius, compromising formal notational consistency.
  4. The empirical refusal evaluation relies exclusively on a single open-weight judge model (OLMO-3.1-32B-Instruct, Sec. 4, p. 7; Appendix D.6), leaving open the possibility of systematic judge-specific classification biases influencing the observed distance-decay relationships.
  5. The framework assumes that data augmentations form a group where every operation is invertible and composable (Sec. 1, p. 2; Sec. 3.2, p. 5), whereas realistic text-based adversarial transformations (such as style transfers, obfuscation, or persona framing) frequently lose semantic content and violate invertibility.
Questions for authors
  1. It is suggested to include cross-validation of refusal labeling using alternative judge models (e.g., more recent models) and report inter-annotator agreement metrics (e.g., Cohen's Kappa) across the three-way rubric.
  2. It is suggested to expand the active training set beyond 40 prompts to varying scales (e.g., 100 and 500 prompts) to determine whether the boundaries of the Constrained Piecewise-Linear regime shift or smoothly merge into global coverage.
3 comments

We thank the reviewer for the detailed reading. We address each point below.

W1. As stated, most of work were conducted by LLMs, causing format errors, such as blank content in Broader impacts, grammar errors, and poor readability of the whole paper. The writing of theoretical analysis is difficult to understand.

Our AI-use statement follows the ICLR policy and is intentionally exhaustive. We did most of the work ourselves and use LLM marginally. The empty “Broader impacts” paragraph is a formatting error: its content is the following paragraph, “Beyond jailbreaks”, which we will merge under the heading.

To improve the readability of the theoretical analysis, we will add a notation table, and we would be grateful if the reviewer could indicate us to the points they found hardest to follow.

W2–W3. Notation collisions.

We thank the reviewer for spotting these collisions, which are both confined to the appendices. We will rename them to avoid notation collisions.

W4 and Q1. It is suggested to include cross-validation of refusal labeling using alternative judge models (e.g., more recent models) and report inter-annotator agreement metrics (e.g., Cohen's Kappa) across the three-way rubric.

We thank the reviewer for this suggestion. We are currently relabeling a subset of the responses with a second, more recent judge model, using the same three-way rubric, and we will report Cohen’s kappa between the two judges in this thread in the coming days. Would the reviewer also find it useful for us to label a subset of responses by hand, so that we can also report the agreement between the judge and human labels? We will add these results to the revision.

Q2. It is suggested to expand the active training set beyond 40 prompts to varying scales (e.g., 100 and 500 prompts) to determine whether the boundaries of the Constrained Piecewise-Linear regime shift or smoothly merge into global coverage.

We thank the reviewer for this question. Expanding the active training set adds new training prompts, each with its own neighborhood, without widening each neighborhood. The boundaries should therefore not shift, and coverage becomes global only once the training prompts are close to every query the evaluator can send.

We used 40 prompts because they already validate the principle we rely on: an audit can be gamed by memorization with as few as 40 prompts. What matters to the evaluator is to know if their test can be gamed, whether this was done by memorizing 40 prompts or with many more.

W5. The framework assumes that data augmentations form a group where every operation is invertible and composable (Sec. 1, p. 2; Sec. 3.2, p. 5), whereas realistic text-based adversarial transformations (such as style transfers, obfuscation, or persona framing) frequently lose semantic content and violate invertibility.

We thank the reviewer for this remark: looking into it more closely, we realized that invertibility is in fact not required, and that the framework extends directly to monoids, i.e. transformations that can be composed but not necessarily undone. For instance, persona framing such as adding a role-play prefix (“You are DAN…”) acts by prepending it to the prompt. Prefixes compose by concatenation, and “no prefix” is the identity, but a prefix cannot be undone, and it need not preserve semantic content. We propose to state the framework this way in the revision. This changes nothing in our paper: none of the framework’s definitions uses inverses, since the players only compose transformations and never undo them, and our theoretical results concern cyclic translation games, whose translations form a monoid that happens to be a group.

We thank the reviewer again for their careful reading and constructive suggestions. In the revision, we will fix the formatting and notation issues, add a notation table, report the cross-validation of refusal labels, and state the framework for monoids. We hope these answers and changes address the reviewer’s concerns, and we would be grateful if they would consider raising their score.

We ran experiments by using more recent models as judges. We used Gemma-4 12B and Granite-4.2-8B. For the 3-way grading (Refuse / Comply / Ambiguous), the Cohen's Kappa values for fine-tuned models range from 0.56 to 0.67. When dropping the prompts judged "Ambiguous" by at least one of the models, thus restricting only to Refuse/Comply, the values go from 0.81 to 0.91.

Reviewer_fH7QOpenReview ↗

response

Thank you for the author's response. This article still requires extensive revisions and cannot currently meet the acceptance criteria. I will keep my original score.

Reviewer_MmK2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Evaluation Game, a game-theoretic framework for adaptive LLM safety evaluation. It models data augmentation through group actions and derives coverage thresholds for cyclic translation games. Experiments on Llama, Mistral, and Qwen show that refusal rates after fine-tuning depend on embedding distance from training prompts.

Critical strengths
  1. The framework models interactions between an adaptive evaluator and trainer, highlighting limitations of static safety benchmarks.
  2. The paper derives coverage thresholds and phase behaviors for cyclic translation games. Experiments across three LLM families provide empirical support for distance-dependent refusal behavior.
Critical weaknesses
  1. The analysis relies on invertible group transformations and simplified cyclic translation games. It remains unclear how these assumptions extend to real-world jailbreak transformations.

  2. The experiments demonstrate distance-dependent refusal rates but do not directly validate the predicted coverage thresholds or multi-round game dynamics. The assumed fixed generalization radius is also not established empirically.

Questions for authors
  1. How can common jailbreak transformations be represented as group actions, particularly when they are not invertible?

  2. Can the authors empirically evaluate multi-round evaluator–trainer interactions to validate the predicted game dynamics?

  3. Can the authors quantify the generalization radius and distinguish fine-tuning-induced improvements from the base model's existing refusal behavior?

2 comments

We thank the reviewer for the clear summary and the precise questions, which we answer below.

Q1. How can common jailbreak transformations be represented as group actions, particularly when they are not invertible?

A transformation is represented by its action on prompts, and composing transformations amounts to applying them in sequence.

For instance, a letter-shift cipher acts by shifting every letter of the prompt by k positions. Two shifts compose into a shift by the sum, and each shift can be undone, so the 26 shifts form a group acting on prompts.

Non-invertible transformations are represented in the same way, as monoid actions. For instance, adding a role-play prefix (“You are DAN…”) acts by prepending it to the prompt. Prefixes compose by concatenation, and “no prefix” is the identity, but a prefix cannot be undone.

We thank the reviewer for this question: looking into it more closely, we realized that invertibility is in fact not required, and that the framework extends directly to monoids. We propose to state it this way in the revision. This changes nothing in our paper: none of the framework’s definitions uses inverses, since the players only compose transformations and never undo them.

Q2. Can the authors empirically evaluate multi-round evaluator–trainer interactions to validate the predicted game dynamics?

We are very enthusiastic about this suggestion, which we see as the natural next step for this work. For now, our experiments deliberately validate the framework’s assumptions rather than its predictions. The predicted game dynamics follow by proof once the assumptions hold, this is why the experiment section tests the locality of fine-tuning. We agree that an empirical multi-round evaluation would strengthen the paper. Letter-shift ciphers, whose structure matches the cyclic translation games exactly, offer a natural setting for it. We will make this choice clearer in the article. Unfortunately, due to time constraint, this experiment cannot be completed within the rebuttal period.

Q3. Can the authors quantify the generalization radius and distinguish fine-tuning-induced improvements from the base model's existing refusal behavior?

Yes. For each held-out prompt, we measure its distance to the closest training prompt, and we compare the refusal rates of the base and fine-tuned models on the same prompts (Figure 2). Near the training prompts, fine-tuning raises refusal to at least 90%, well above the base model. This gain over the base model (at most 50%) is what fine-tuning induces.

It shrinks as distance grows and stops shrinking beyond a radius that we estimate for each family: 48%, 44% and 38% of the observed distance range for Llama, Mistral and Qwen (Table 4). This radius is the empirical counterpart of the generalization radius ε\varepsilon.

We thank the reviewer for this question, and we will clarify the analysis of Figure 2 in the revision to make it more explicit.

Weaknesses.

These answers also address the weaknesses. Q1 addresses W1. Q2 addresses the first part of W2, the validation of the predicted coverage thresholds and multi-round game dynamics. Regarding the fixed generalization radius, the radius estimated in Q3 barely changes under a different fine-tuning configuration (LoRA rank 2: 42%, 44% and 37%), which supports treating it as fixed.

We hope these clarifications address the reviewer’s concerns. We would be glad to answer any further question; otherwise, we would kindly ask the reviewer to raise their score.

Reviewer_MmK2OpenReview ↗

Comment

I would like to thank the authors for further clarification on my concerns. I believe that my assessment of this work is fair, and I will raise my rating. However, all the justifications must be incorporated into the revision to make this work more readable and clear.

Clarification: I used an AI tool to organize my review. All technical assessments, including strengths, weaknesses, and scores, are my own judgment.

Comments on the paper

1 comment

We thank the reviewer for the positive assessment and the precise questions, which we address below.

Q1. Mapping to realistic transformations.

Consider an evaluator that hides harmful requests with letter-shift ciphers, shifting every letter by 2, 4, 6… positions, which gives 13 possible encodings (p = 13). A trainer who augments its data with ROT13 reaches only 2 encodings (q = 2), none of which the evaluator uses. As long as fine-tuning on one encoding does not transfer to another (ε\varepsilon below the critical threshold), this trainer can only memorize what it has seen. A trainer who augments with shifts of 4 reaches exactly the evaluator’s 13 encodings (q = 13), so it covers them all in advance. Two similar-looking choices thus lead to opposite audit outcomes, which is the gcd effect of the circle analysis. We would be happy to add this example to the paper if the reviewer finds it helpful, and we thank them for the suggestion.

Q2. Dynamic ε\varepsilon.

We agree that ε\varepsilon may change, and we plan to study this in future work, which requires refining the model. Intuitively, the analysis applies round by round with the current ε\varepsilon. If ε\varepsilon grows, the evaluator keeps its advantage only until ε\varepsilon crosses the critical threshold. If ε\varepsilon shrinks, the evaluator’s advantage lasts longer. In both cases, the outcome lies between those of the games with the smallest and the largest ε\varepsilon reached.

Q3. Non-group transformations.

We thank the reviewer for this question. Looking into it more closely, we realized that invertibility is in fact not required: the framework extends directly to monoid actions, of which group actions are a special case. We propose to state it this way in the revision. This broadens the scope without changing the results, and the circle example is unchanged, since its translations already form a group. For instance, a cipher is a group action, and adding a role-play prefix (“You are DAN…”) is a monoid action: prefixes compose by concatenation and “no prefix” is the identity, but a prefix cannot be undone. Only transformations that reword the request itself remain outside, such as euphemisms, translations, translations, or free-form rewrites that are not restricted to a fixed set of tactics (e.g., PAIR, TAP).

To quantify this scope, we examined the jailbreak tactics that WildJailbreak is built from. Weighted by their frequency, between 71.5% and 91.4% of them are monoid-like (e.g., adding a fictional frame, a role-play, or distractors), and between 8.6% and 28.5% reword the request (e.g., euphemisms, coded keywords), so that whether they fit the framework is less clear. The uncertainty in the estimation comes from the rarer tactics, which account for the remaining 19.9% of occurrences. We labeled by hand only the most frequent tactics, which cover 80.1% of occurrences, and we can label the rarer ones with an LLM judge if the reviewer finds it useful.

We will add an appendix detailing this analysis. In short, WildJailbreak’s adversarial prompts are built from jailbreak tactics that WildTeaming (Jiang et al.,2024) mined from real user–chatbot conversations with an LLM, then grouped into clusters of near-duplicates. The authors release 5,319 clusters with their frequencies. We exclude one cluster that only selects rare requests rather than transforming them, leaving 5,318 clusters and 90,819 occurrences. We take the 17 most frequent clusters, the smallest set covering at least 80% of these occurrences, and label each one, based on its released definitions, according to what its tactic does to the original request: kept with added content (monoid-like) or reworded (outside). The appendix will list the 17 clusters with their definition, count and label.

Q4. Evaluator strategy.

The phase-transition results are not sensitive to the evaluator’s strategy. The critical threshold depends only on the transformations available to the two players, and the cyclic walk is simply the easiest strategy to analyze. In the super-critical regime, a smarter evaluator restricted to the same transformations can at best delay coverage, not prevent it. To sustain high miss ratios, it needs new transformations. This sensitivity to the choice of transformations is precisely our main message: audit results cannot be interpreted without characterizing how queries are generated, and this work is, to our knowledge, a first step in that direction.

Weaknesses.

These answers also address W1: Q1 maps a realistic transformation to the circle game, Q3 lifts assumptions (a) and (b), and Q4 shows that (d) does not drive the results. For W2, we deliberately tested the framework’s assumptions rather than its predictions, which follow by proof once the assumptions hold. We agree that an end-to-end simulation would strengthen the paper, and we will make this choice clearer in the article.

We thank the reviewer again for their constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.