The Evaluation Game: Beyond Static LLM Benchmarking

Jailbreaks are adversarially crafted inputs that bypass safety constraints. As they continue to be discovered in Large Language Models, practitioners increasingly rely on fine-tuning as a defensive strategy. Yet the theoretical foundations underlying this robustness fine-tuning remain underexplored. We introduce a game-theoretic model in which the interaction between an evaluator (auditing the LLM for jailbreaks) and a trainer is formalized as a two-player game. A key feature of our approach is the use of group actions to formally represent transformation-based data augmentation. A group action is a mathematical structure that captures symmetries and transformations. The simplest non-trivial instance is the circle with cyclic translation groups, where we exhibit various regimes depending on the trainer's generalization range. Below a critical threshold, the evaluator maintains a constant miss ratio for linearly many rounds, whereas other settings can yield very different behaviors. We further provide empirical evidence supporting locality-dependence of the model: for the three model families we tested (Llama, Qwen and Mistral), we have significant evidence that fine-tuning on adversarial prompts induces only local generalization, with refusal rates on test examples strongly dependent on the distance to the fine-tuning prompts. Our framework recasts the central object of adversarial evaluation: a benchmark is not a static set of prompts but an orbit under the evaluator's group action, and audit protocols that ignore trainer-side adaptation cannot distinguish a genuine fix from a memorized patch.