acceptodds
Under review as a conference paper at ICLR 2027

Reflections and Fragments: Securing LLMs Against Sequential Mosaic Attacks

Abstract

Self-play red-teaming improves language-model safety by pitting attacker and defender roles against each other in a zero-sum game. However, real adversaries increasingly use mosaic attacks: multi-turn sequences whose individual fragments are innocuous in isolation yet assemble into a harmful payload. We develop a theory of mosaic defense that characterizes what is required to prevent such attacks without sacrificing helpfulness. We first show that no fixed bounded window of recent prompts is sufficient in general: safety-relevant information may occur arbitrarily far back in the interaction. We formalize a watchman, an online state mechanism that carries this information forward, and show that under explicit assumptions it enables zero-failure defense with positive benign helpfulness. Under stronger conditions, it is also optimal among zero-failure defenders. An exact watchman may nevertheless require exponentially many states, while exact maliciousness detection can require exponentially many queries in an unstructured black-box model. These state and query lower bounds do not by themselves imply hard learning: the construction underlying the state lower bound is efficiently learnable from labeled examples, whereas certifying worst-case safety can require substantially more information under restricted access. We also show that self-play equilibrium alone does not certify usefulness, motivating a constrained formulation that maximizes worst-case benign helpfulness among zero-failure defenders. Empirically, training role-specific attacker and defender LoRA adapters over frozen LLMs via multi-turn self-play strengthens both roles: attackers become more effective at eliciting harmful responses, while defenders become more robust to attack, with improvements also observed on unseen attack objectives.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety, Self-play, LLMs, Jailbreaks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UN89OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines self-play with mosaic attack and defense, using theoretical analysis from the perspective of a finite extensive-form game with imperfect information. It describes the mosaic attacks as multi-turn sequences whose individual fragments are innocuous in isolation yet assemble into a harmful payload. To defend against such attacks, it formalizes a watchman, an online state mechanism that carries safety-relevant information forward. Experiments using GRPO and LoRA show that this method strengthens both roles, helping attackers to elicit more harmful responses and defenders to become more robust across harm categories, including on unseen objectives.

Critical strengths
  1. Interesting theoretical analysis from the perspective of a finite extensive-form game.
  2. Transferring self-play to a mosaic attack, concerning the situation of innocuous individual fragments assembling into a harmful payload. This kind of attack can cause a lot of harm in reality.
Critical weaknesses
  1. The work uses LLM-as-a-Judge but lacks human evaluation, so the trustworthiness is not sure. The prompt used by the judge model is also not clear.
  2. The WildJailbreak decomposition study shows that 38.8% pass the equivalence test; how about on WildMosaic? How to guarantee that the decomposed requests still carry the harmful payload?
Questions for authors
  1. The watchman is defined as a discrete finite-state machine, but implemented by using LoRA adapter. How is this fundamental representational gap addressed?
  2. How to guarantee that the decomposed requests still carry the harmful payload?
  3. What’s the reliability of the judge? Why prefer to use the same backbone model as the judge (self-judging)?
3 comments
AuthorsOpenReview ↗

Reviewer UN89 — Comment 1/1

We appreciate your questions about the relationship between the theoretical watchman and the trained models, as well as the reliability of our evaluation. We have revised the paper to clarify both points (the PDF, with changes in blue).

We now address all the reviewers' concerns. We identify them by the number each appears in the review (e.g., W1/Q1 means Weakness, point 1, and Question 1).


Q1: Is the LoRA defender a watchman?

The watchman is a theoretical online state that records information relevant to safety decisions across turns. In the experiments, we train separate attacker and defender LoRA adapters on a frozen backbone. The defender receives the conversation history and can therefore use earlier turns when responding, but we do not train a separate finite-state watchman. The experiments test whether self-play improves the defender's behavior against multi-turn attacks, rather than whether the LoRA implements the theoretical state machine. Section 7 now explains this relationship and the distinction between the theory's information requirements and the observed behavioral improvements.


W2/Q2: Objective preservation and fragment benignness

The 38.8% equivalence acceptance rate reported in Section 7 comes from a separate WildJailbreak decomposition study. We also conducted a curation experiment using fixed decompositions of 262 WildMosaic seeds. For the frozen Qwen model, judged success against the original request increased from 3/262 (1.1%) with direct prompts to 102/262 (38.9%) with the decomposed sequences.

These results provide evidence that decomposing a harmful request can preserve enough of its original objective to elicit a successful response. To examine the attacks generated during self-play, we also assessed whether individual fragments were benign in isolation, whether the sequence preserved the original harmful objective, and whether the defender's final response achieved that objective.

We used Claude Opus 5 as an independent model-based annotator to assess selected successful trained-Qwen attacks (Appendix I.3). On 137 AdvBench successes, 93% have turns judged non-refusable in isolation, 99% preserve the seed objective, and 92% meet both conditions. Among 249 successful WildJailbreak attacks, 72% have turns judged non-refusable in isolation, 98% preserve the seed objective, and 71% meet both conditions. The annotator judged the harmful content to be distributed across turns in 91% of the AdvBench successes and 75% of the WildJailbreak successes (Appendix I.3). The new annotation provides direct evidence about the selected successful attacks. Because the sample was selected for prior judged success, and the benign-in-isolation criterion allows borderline or dual-use turns, it does not estimate strict-mosaic validity among all generated attempts. Defender gains on a strictly valid subset require a separate paired comparison.


W1/Q3: Judge configuration and reliability

Appendix I.2 now includes the judge prompt and configuration. For Qwen, the judge is a separately served copy of the frozen Qwen3.6-35B-A3B weights, using temperature zero, reasoning enabled, a 3,072-token budget, and JSON-constrained output. The same judge prompt is used during training and evaluation.

Gemma uses a judge from its own backbone family, while DeepSeek is evaluated using Qwen.

To assess the consequences of judge choice, we also added a cross-family comparison on fixed AdvBench transcripts at checkpoint 65. Gemma rejudged 4,160 Qwen conversations and Qwen rejudged 4,160 Gemma conversations using the same prompt. Agreement was 88.0% (κ=0.81\kappa=0.81) on Qwen conversations and 77.3% (κ=0.65\kappa=0.65) on Gemma conversations; temperature-zero self-agreement was 93.8% and 91.9%, respectively.

Disagreement matters particularly for attack success: the two judges label 17.1% versus 25.0% of malicious Qwen conversations successful, and 65.0% versus 41.8% of malicious Gemma conversations. Benign-refusal decisions agree on 97.2% and 95.8% of episodes, although κ\kappa is 0.61 and 0.35 in the two directions. This is evidence about consistency, not accuracy; absolute ASR depends on the judge, and paired significance cannot be assumed unchanged without recomputation under the alternative judge. We have not conducted human evaluation.


Ethics

We train and evaluate attackers because a defender trained against a fixed opponent may overfit to that opponent. The observed attacker gains are modest: for Qwen, they are significant at one of six AdvBench checkpoints and none of the three WildJailbreak checkpoints. We include only abbreviated attack excerpts, and the system was not deployed to real users. The revised ethics statement also discusses the dual-use implications of training stronger attackers.

Thank you for the additional information and clarifications. I appreciate the effort to address the reviewers' concerns. However, I still have the following reservations.

  1. The theory and experiments address different questions: the theory is an existence result about an exactly safety-separating watchman, while the experiments train LoRA adapters that, as the authors note, do not implement it. It would be helpful for readers to see this distinction stated explicitly as a scope limitation.
  2. Judge reliability is a boundary on the empirical claim. Given the reported disagreement (17.1% vs. 25.0% on Qwen; 65.0% vs. 41.8% on Gemma) and the authors' own caveat that significance cannot be assumed unchanged under the alternative judge, the robustness of the defender gains across judges remains open.
  3. The equivalence/preservation rates (99%/98%) are computed on already-successful cases, so they do not estimate strict-mosaic validity over all attempts. Since the defender's gains are evaluated on these attacks, the extent to which they reflect mosaic detection rather than rejection of near-direct harmful requests remains unverified.

We appreciate your follow-up and the opportunity to clarify these points. We have conducted additional analyses addressing judge reliability and mosaic validity, and clarify the relationship between our theoretical and empirical contributions.

1. Relationship between theory and experiments.

We agree that the theoretical watchman and the trained LoRA defenders should not be conflated. However, the theory goes beyond establishing the existence of an exactly safety-separating watchman. It characterizes the information needed to defend against sequential mosaic attacks, proves limitations of bounded-history defenders, and establishes state-complexity lower bounds. Our new results in Appendices D.4 and D.5 extend the analysis to weakened assumptions and approximate safety separation.

The experiments address the corresponding behavioral challenge: whether attacker-defender self-play can improve resistance to multi-turn attacks. The defender receives the conversation history and can use information from earlier turns, but we do not claim that its LoRA adapter implements the theoretical watchman or satisfies its formal guarantees. Rather, the theory characterizes the information requirements and limitations of safe defense, while the experiments evaluate whether self-play improves defender behavior against these attacks. We will make this explicit in Section 7.


2. Judge reliability and the source of cross-judge disagreement.

We agree that judge choice affects the reported attack-success rates. To investigate, we performed an additional cross-judge analysis of the vanilla base-attacker/base-defender condition, without either LoRA adapter, on 1,040 conversations per model (520 malicious and 520 benign).

Even without training, cross-judge agreement is 89.4% (Cohen's κ=0.83\kappa=0.83) on Qwen conversations and 74.5% (κ=0.58\kappa=0.58) on Gemma conversations, close to the agreement rates of 88.0% and 77.3% reported across the trained evaluation conditions. The differences in attack-success judgments are also present before training: the two judges report 20.8% versus 29.2% success on vanilla Qwen conversations, and 40.4% versus 70.0% on vanilla Gemma conversations. Same-judge reruns yield 93.8% and 91.9% agreement, respectively, while cross-judge agreement on benign over-refusal is 98.7-99.8%.

These results show that the substantial disagreement in attack-success judgments is already present in the untrained models and before LoRA. They also distinguish disagreement about attack success from the higher agreement on benign over-refusal. We will add this to Appendix I.2.

As a final remark, while this does not establish that the magnitude or statistical significance of the defender's improvement is unchanged under another judge, it provides context for interpreting the disagreement: the judges differ substantially even when evaluating identical conversations generated without self-play training.


3. Mosaic validity beyond successful attacks.

Yes, the previously reported 99%/98% objective-preservation rates were measured on attacks already judged successful. We have extended the analysis to unsuccessful attempts, allowing us to examine how the defining properties of mosaic attacks differ between successful and unsuccessful conversations.

Across 1,169 unsuccessful attempts, 72% still preserved the original harmful objective. Thus, objective preservation is common even when the attack fails, and preserving the payload alone does not explain success.

The differences are more pronounced for mosaic structure. In an annotated sample of 137 successful and 90 unsuccessful attacks, 93% of successes satisfied the benign-in-isolation criterion, compared with 70% of failures. More strikingly, 87% of successes exhibited the full mosaic structure, compared with only 36% of failures.

Among attacks explicitly refused by the defender, 60% satisfied benign-in-isolation and only 31% retained the full mosaic structure. In other words, 40% contained at least one turn judged not benign in isolation, and 69% did not exhibit the full mosaic structure.

These findings provide evidence that payload preservation and mosaic structure are distinct properties. The harmful objective is frequently preserved even in failed attacks, whereas successful attacks are substantially more likely to maintain individually benign-looking fragments and the full mosaic structure. The results are consistent with the defender refusing more often when the decomposition breaks down and harmful intent becomes apparent in individual turns.

Although this comparison does not isolate a mosaic-specific detection mechanism, it addresses the concern that our earlier validity analysis considered only successful attacks. We will incorporate these results into Appendix I.3.


We hope these new analyses clarify the scope of the theoretical contribution and provide a complete picture of judge disagreement and the mosaic structure.

Reviewer_jpi1OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies sequential mosaic attacks, in which individually benign fragments and their responses can jointly enable a harmful objective. It formalizes the problem as a finite extensive-form game and compares defenders with local observations, bounded history windows, and a stateful watchman. Under explicit safety, separability, and helpfulness assumptions, the paper analyzes the limits of these defenses. It also distinguishes state complexity, black-box query complexity, and learnability, and explains why optimizing safety on malicious interactions alone does not guarantee helpfulness on benign requests.

The experiments use multi-turn self-play to train separate attacker and defender LoRA adapters on a shared frozen backbone. Evaluation covers three models, attack success rates, benign refusal rates, and some cross-attack transfer tests. Several settings show improved attack or defense performance, although defense gains come with varying increases in benign refusal, and transfer does not hold uniformly across models.

Critical strengths

1. The theoretical framework identifies meaningful limits of multi-turn safety judgments.

The paper examines the limitations of local prompt judgments, connects retained history to safety decisions, and separates safety from benign helpfulness. This helps explain both how apparently benign requests can form a harmful combination and why optimizing safety alone can cause excessive refusal. Distinguishing state complexity, query complexity, and learnability gives the framework value beyond any particular attack method.

2. The experiments examine both attack and defense performance, including costs to benign users.

The evaluation covers three models and reports benign refusal rates alongside attack success rates, with additional cross-attack tests. This allows readers to assess whether safety gains come at the expense of helpfulness, rather than treating lower attack success alone as a better defense. Despite the remaining validation issues, these choices provide a useful basis for evaluating multi-turn defenses.

Critical weaknesses

1. Training overlap in AdvBench requires the generalization claims to be revised.

The main text on pages 7-8 describes AdvBench as an out-of-distribution evaluation without training overlap. However, Appendix G.1 on page 38 states that 475 of its 520 goals are used for training, with only 45 held out. Improvements on the full benchmark therefore do not directly establish generalization to unseen goals. The appendix discloses this overlap; the concern is the inconsistency between the main-text interpretation and the actual split. This does not imply that all evaluations overlap with training or invalidate the independent evidence from WildJailbreak. Results should be separated for seen and unseen goals, with the generalization claims revised accordingly.

2. The experiments do not sufficiently validate the defining mosaic property or connect the theoretical mechanism to the trained defense.

The central phenomenon is that fragments are benign individually but harmful in combination. Yet the decomposition check mainly tests whether the concatenated fragments preserve the original objective, without independently checking each fragment's benignness. The evaluated trajectories may therefore mix genuine mosaic attacks with other forms of multi-turn jailbreaks. The theoretical watchman relies on particular history states and safety-separation conditions, but there is no direct validation that the trained defender uses a corresponding mechanism. These limitations do not negate the observed training gains, but they weaken the stronger mechanistic interpretation.

Questions for authors

1. Data splits and generalization: clarification and reanalysis of existing results.

Could you report attacker and defender ASR separately for the 475 AdvBench goals included in training and the 45 held-out goals, and clarify whether the held-out goals were used for model or checkpoint selection? Please identify which results support generalization to unseen goals and separately report the corresponding models' refusal rates on the benign evaluation set.

2. Mosaic validity and mechanism: targeted validation of existing trajectories.

On a predefined sample of existing trajectories, could you separately check whether each fragment is benign in isolation, whether the combination preserves the harmful objective, and whether the final output achieves that objective? Do the main defense gains remain on the subset that strictly satisfies the mosaic definition? If the experiments cannot directly establish use of the theoretical watchman mechanism, please clarify which conclusions concern training effects only. These answers would affect my recommendation; I am not requesting a new large-scale benchmark.

2 comments
AuthorsOpenReview ↗

Reviewer jpi1 — Comment 1/2

We appreciate your careful examination of the evaluation protocol. You identified an inconsistency in our description of AdvBench, which we have now corrected throughout the paper (the PDF, with changes in blue).

We now address all the reviewers' concerns. We identify them by the number each appears in the review (e.g., W1/Q1 means Weakness, point 1, and Question 1).


W1: AdvBench overlap

All 520 AdvBench goals appear in the WildMosaic malicious pool: 475 are used as training seeds, while the remaining 45 belong to the 100 held-out seeds. The split is based on exact-string matching, and some training and held-out goals are also near-duplicates. We have corrected the abstract, introduction, Section 7, conclusion, and reproducibility statement accordingly. WildJailbreak test objectives have no exact prompt overlap with training, but we no longer present this as evidence of semantic novelty.


Q1: Checkpoint selection

During training, checkpoints were evaluated every five steps on the 100 held-out malicious seeds, including the 45 held-out AdvBench goals. For Qwen and Gemma, a selection script queued checkpoints for offline evaluation only when their results met preset thresholds. These goals therefore served as development data rather than an untouched test set. The “WildMosaic (held-out)” rows in Table 1 report selection-time evaluations.

The checkpoint-selection procedure described above applies to Qwen and Gemma. The reported DeepSeek results instead come from a separate 90-step training configuration (Table 2).

The Qwen AdvBench row in Table 1 averages three checkpoints with valid offline evaluations, drawn from three separate runs. Appendix I.1 now reports results for all six evaluated checkpoints.


Q1: Seen and held-out results

We recomputed the Qwen AdvBench results from the per-episode records, separating the 475 seen goals from the 45 held-out goals. The recomputation over the full set reproduces the reported results (Table 8).

On seen goals, the trained defender achieves significantly lower ASR than the baseline at all six checkpoints, while attacker gains are significant at only one. On the 45 held-out goals, defender ASR ranges from 0.000 to 0.067, compared with baseline values of 0.156–0.222. It is lower at all six checkpoints, with significant differences at three (p=0.008p=0.008, 0.0080.008, and 0.0390.039). Attacker ASR is higher at four checkpoints, but none of these increases is significant. Benign refusal on 520 evaluation prompts rises from 1.3–2.1% to 5.0–7.9%.

Because the 45 held-out goals were used during checkpoint selection, these comparisons should be interpreted as development-set results. The separate WildJailbreak evaluation provides stronger evidence of transfer beyond exact training prompts: the Qwen defender reduces ASR by 52.6% relative to baseline, significantly at all three checkpoints, while the attacker's relative gain of 15.6% is not significant.

AuthorsOpenReview ↗

Reviewer jpi1 — Comment 2/2

W2/Q2: Strict mosaic validity

To assess the defining property of mosaic attacks, we distinguish three questions: whether each fragment is benign in isolation, whether the sequence preserves the original harmful objective, and whether the defender's final response achieves that objective. We have added an independent model-based assessment using Claude Opus 5 on selected successful trained-Qwen attacks (Appendix I.3).

Among 137 successful AdvBench attacks, 93% have turns judged non-refusable in isolation, 99% preserve the seed objective, and 92% meet both conditions. Among 249 successful WildJailbreak attacks, 72% have turns judged non-refusable in isolation, 98% preserve the seed objective, and 71% meet both conditions. The annotator separately judged whether the harmful content was distributed across turns: 91% of AdvBench successes and 75% of WildJailbreak successes met that criterion. Excluding conversations containing garbled turns or leaked attacker reasoning reduced the share meeting both benign-in-isolation and payload-preservation criteria to 78% on AdvBench and 62% on WildJailbreak.

This additional assessment addresses the reviewer's question about the successful attacks it examines. It also clarifies what the original evaluation measured. In the original evaluation, individual fragments were not judged outside their conversational context. The judge labeled 5.1% of base-attacker and 12.9% of trained-attacker AdvBench episodes as never pursuing a harmful goal. Because that judge did not see the original seed text, its verdict did not independently verify preservation of the seed's specific objective. We have clarified these distinctions in Appendix I.3 and revised the claim about the learned attackers' fragments accordingly.

The new annotation concerns attacks already judged successful, rather than all generated attempts. Its benign-in-isolation criterion means that a turn was judged not to warrant refusal on its own, not that it was wholly innocuous. Measuring the defender's ASR reduction specifically on strictly valid mosaics would require labels for successful and unsuccessful attacks from the paired defender arms; the current success-conditioned sample does not establish that comparison.


W2: What the experiments establish

Our self-play experiments show that the trained defender reduces judged attack success on seen goals, on the smaller held-out development set, and on WildJailbreak, although benign refusals also increase. These results demonstrate the behavioral benefits of self-play against multi-turn attacks and complement our theoretical analysis of the information needed to defend against them.

The trained defender's refusal rate increases from 55% to 86% across malicious turns, but also from 11% to 26% across benign turns. Because the prompts change from one turn to the next, these results do not isolate the effect of earlier conversation history. A matched-history evaluation, in which the current request is held fixed while the preceding conversation changes, would be needed to test this directly. We have revised Section 7 and Appendix I.4 to avoid suggesting that the defender implements a safety-separating state.


Additional Comment: Judge labels outside the taxonomy

We also examined how the results change when the judge produces labels outside its specified categories. Even when these labels are conservatively counted as successes on malicious episodes, the trained defender's reduction in ASR remains significant at every checkpoint (Table 11). Such labels occur on 9.4% of malicious and 3.5% of benign episodes.

The alternative convention also affects the estimated benign-refusal rate: for the trained defender, it is approximately 9–13%, compared with 5–8% under the original convention (Table 11).

Reviewer_MYbWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops a theory of defense against mosaic attacks on language models — multi-turn interactions where each individual fragment is benign in isolation but harmful in combination. The authors formalize the attacker-defender interaction as a finite extensive-form game with imperfect information and study three classes of defenders: stateless, bounded-window, and watchman-based. The main theoretical results establish that (1) no stateless defender can simultaneously guarantee zero failure and positive benign helpfulness when ambiguous fragments exist (Theorem 4.1); (2) no fixed bounded window of recent prompts resolves this (Theorem B.1); (3) an online state mechanism called a watchman — which compresses the interaction history into a finite state sufficient for safety decisions — can achieve both zero failure and positive helpfulness under explicit assumptions about refusal safety and helpfulness compatibility (Theorem 5.1); (4) an exact watchman may require exponentially many states (2m2^m) and exact maliciousness detection may require 2N2^N black-box queries; and (5) equilibrium in the safety-only zero-sum game does not guarantee usefulness, motivating a constrained formulation. Empirically, the authors train role-specific LoRA adapters (attacker and defender) over three frozen mixture-of-experts LLMs via multi-turn self-play on a curated WildMosaic dataset, showing that self-play strengthens both roles across harm categories and unseen objectives.

Critical strengths
  1. Rigorous and well-chosen theoretical framework. The paper asks exactly the right foundational questions for multi-turn LLM safety: what information does a defender need, what is the cost of maintaining it, and what objectives should we optimize? The impossibility of stateless defense (Theorem 4.1) and the insufficiency of bounded windows (Theorem B.1) are clean, surprising results that rule out a large class of defense strategies. The watchman construction (Theorem 5.1) provides a matching positive result. The exponential state and query lower bounds (Section 5) are sobering but important — they characterize the inherent difficulty without suggesting the problem is intractable (the learnability distinction is subtle and well-argued). Together, these results form a coherent theoretical narrative that will be highly influential for future work on sequential LLM safety.

  2. Principled treatment of the safety-helpfulness tension. The separation of safety and helpfulness as distinct objectives, and the formal demonstration that equilibrium in the safety game admits a trivially unhelpful solution (Proposition 6.1), is an important contribution. The constrained formulation (maximize helpfulness subject to zero failure) is the right framing, and the optimality of the watchman defender under this formulation ties the theoretical story together neatly. The analysis of approximate watchmen (bounding failure by NδFNN\delta_{\text{FN}} and helpfulness loss by δFP\delta_{\text{FP}}) provides actionable guidance.

Critical weaknesses
  1. Large gap between theoretical guarantees and empirical instantiation. The theoretical results characterize exact, worst-case guarantees for idealized watchmen. The empirical system trains LoRA adapters via self-play, which is a heuristic approximation. The paper does not attempt to bridge this gap — there is no analysis of whether the trained defenders approximate a watchman, no measurement of the false-negative or false-positive rates in the framework of Section 5's approximate-watchman analysis, and no empirical verification of Assumptions 1 or 2. The experiments primarily show that self-play improves both roles, which is expected and not strongly connected to the theoretical contributions. This weakens the paper's practical impact, though the theoretical contributions stand on their own.

  2. Strong and potentially limiting assumptions. Assumption 1 (refusal safety: refusing always prevents mosaic completion) may not hold when the defender's refusal itself leaks information or when partial responses have already been given. Assumption 2 (helpfulness compatibility: for every safe observation, there exists a substantive response that keeps all benign transcripts helpful) is strong and hard to verify in practice. The paper acknowledges these are modeling choices but does not discuss settings where they might fail or how the results degrade. The theoretical results are contingent on these assumptions, and their practical plausibility for real LLM deployments deserves more discussion.

Questions for authors
  1. Can you provide empirical estimates of the approximate-watchman error rates (δFN\delta_{\text{FN}}, δFP\delta_{\text{FP}}) for your trained LoRA defenders? Even rough estimates would help bridge the theory-practice gap.

  2. The exponential state lower bound applies to exact safety-separating watchmen. Do you have evidence (theoretical or empirical) about whether approximate safety separation with bounded error can be achieved with polynomial states for natural mosaic attack distributions?

  3. Assumption 1 requires that refusing at any turn prevents mosaic completion. In practice, a model may have already provided partial substantive responses in earlier turns. How robust is the theory if refusal safety holds only approximately (i.e., refusing reduces but does not eliminate failure probability)?

  4. The DeepSeek results show inconsistent improvements. Beyond self-judging quality, are there architectural properties (e.g., LoRA rank relative to model capacity) that might explain the discrepancy?

4 comments
AuthorsOpenReview ↗

Reviewer MYbW — Comment 1/4

We appreciate your questions about the practical assumptions and the connection between our theory and experiments. We have revised the paper (the PDF, with changes in blue) to clarify the guarantees provided by the theoretical watchman and how they relate to our empirical evaluation.

We now address all the reviewers' concerns. We identify them by the number each appears in the review (e.g., W1/Q1 means Weakness, point 1, and Question 1).


W1/Q1: Theory and experiments

The theoretical watchman identifies the information a defender needs to retain from earlier turns to distinguish safe requests from those that could complete a mosaic attack. To study the corresponding behavioral challenge, we train attacker and defender LoRA adapters through self-play. They share a frozen language-model backbone but have separate trainable adapters, allowing the attacker to develop multi-turn strategies while the defender learns to respond to them. The defender receives the conversation history, so it can in principle use information from earlier fragments.

Our experiments show that self-play changes judged attack success and benign refusal, providing empirical evidence about the behavioral challenge identified by the theory. These results do not establish that the defender has learned the finite-state safety-separating mechanism itself. Holding the current request fixed while varying the preceding conversation would test whether the defender uses earlier turns, although it would not by itself establish the formal watchman mechanism. We now explain this distinction in Section 7. The new episode-level and per-turn measurements relevant to the reviewer's error-rate question are discussed in our next comment and in Appendix I.4.


W2/Q3: Are the assumptions plausible?

Assumption 1 is plausible in deployments that use a fixed, content-free refusal, perform no external actions (such as tool calls) that could independently cause harm, and define failure in terms of information disclosed by the defender. It may fail when free-form refusals reveal harmful content, when the system has already taken consequential actions, or when harm remains possible after the conversation ends.

Our analysis of the saved evaluations, using the training keyword detector, illustrates the risks of free-form refusals. The evaluated defenders generate responses rather than selecting a fixed refusal message. Responses flagged as refusals by our keyword detector have a median length of about 3,400 characters, and most contain lists, headings, or code. In 36 malicious conversations where every defender response contained a refusal phrase but the judge nevertheless reported success, the judge's rationale describes a refusal followed by the requested harmful content. These appear to be partial answers rather than cases of harm caused by refusal itself. The examples illustrate why a fixed, content-free refusal is preferable to a free-form refusal when relying on Assumption 1. Appendix D.4 discusses the corresponding deployment conditions and introduces new bounds for imperfect refusal safety.

Assumption 2 is plausible when the watchman's state retains the context needed to answer benign requests, so that histories sharing a safe observation admit a common useful response. It may fail when the watchman groups together histories that require different answers. Our experiments train history-conditioned defenders rather than a separate watchman, so they do not directly test this compatibility condition. Only 23 of 12,720 later-turn benign prompts recur after different preceding histories in the saved Qwen evaluations, leaving too few examples for a meaningful matched-context analysis. In Appendix D.4, we introduce a new mixed-response result showing that positive expected helpfulness can still be guaranteed under suitable conditions, even when no single response works for every history.

AuthorsOpenReview ↗

Reviewer MYbW — Comment 2/4

W1/Q1: False-positive and false-negative rates

The theoretical false-positive and false-negative rates concern the watchman's decisions at individual conversation histories, not whether a complete attack succeeds. A false negative occurs when a history at which answering could lead to failure is classified as safe; a false positive occurs when a benign-useful history is classified as risky. Estimating these rates would require independent labels for the histories and a corresponding safety decision by the defender. Our LoRA defenders do not implement an explicit watchman, so the available evaluations provide related behavioral measurements rather than direct estimates of these theoretical rates.

We first measured refusals during the saved Qwen evaluations (Appendix I.4). For each checkpoint, we counted defender turns in conversations generated from 520 benign seeds and applied the training keyword detector, yielding approximately 1,200 observed turns per checkpoint. Across six AdvBench and three WildJailbreak evaluations, 5.8–9.0% of base-defender turns and 14.8–19.9% of trained-defender turns were flagged as refusals. For the trained defender, this fraction increased from 9–13% on the first turn to 23–30% on the third. These frequencies describe behavior on benign-seed conversations; they are not false-positive rates, because later prompts are not necessarily benign-useful and the keyword detector can flag partial answers.

We have also added a more detailed, paired comparison of attack outcomes and benign refusals for Qwen on AdvBench at checkpoint 55 (Appendix I.4, Table 13). On the 520 malicious seeds, the judge marked 119 (22.9%) episodes successful against the base defender, compared with 42 (8.1%) against the trained defender; the first count includes one PARTIAL verdict. The number labeled REFUSAL increased from 333 (64.0%) to 443 (85.2%). On 520 benign seeds, BENIGN_REFUSED increased from 9 (1.7%) to 30 (5.8%). If we also count REFUSAL labels on benign seeds, the counts rise from 19 (3.7%) to 57 (11.0%). All four comparisons are significant in exact McNemar tests pairing episodes by seed (p<10−3p<10^{-3}).

These paired results quantify the empirical safety–helpfulness trade-off. They are not a confusion matrix for the theoretical watchman: the malicious outcomes are judged after the defender responds, and the benign refusal labels do not independently identify benign-useful histories. Estimating the theoretical rates would require independently labeled histories and the corresponding answer-or-refuse decisions, as explained in Appendix I.4.

AuthorsOpenReview ↗

Reviewer MYbW — Comment 3/4

W2/Q3: Weaker safety and helpfulness assumptions

We added new results in Appendix D.4 to show how the safety and helpfulness guarantees change when the original assumptions are weakened. For Assumption 1, Theorem D.4 shows that, against any fixed adaptive attacker, the failure probability is bounded by min⁡{1,ερ+NδFN}\min\lbrace 1,\varepsilon_\rho+N\delta_{\mathrm{FN}}\rbrace, where ερ\varepsilon_\rho is the failure probability under an always-refuse policy, NN is the number of interaction turns, and δFN\delta_{\mathrm{FN}} bounds the probability that the defender answers at a history where answering could lead to failure. Thus, if the defender never answers at such histories, its failure probability is no greater than that of always refusing. Otherwise, the additional failure probability is bounded by NδFNN\delta_{\mathrm{FN}}. In the original model, where harm is determined by the completed transcript, a nontrivial uniform bound against unrestricted attackers requires exact refusal safety. The new Theorem D.5 extends the analysis to settings where harm remains probabilistic even after the transcript is fixed, bounding the overall risk in terms of the risk of always refusing, a threshold on the harm probability of allowed answers, and the probability of incorrectly allowing an unsafe answer.

For Assumption 2, we added Propositions D.1–D.2 to show that positive expected helpfulness does not require a single response that works for every benign history sharing a safe watchman observation. If each such history admits at least one response that remains helpful under every possible continuation, randomizing among responses can still guarantee positive expected helpfulness, provided the relevant sets are finite and the history is reached with positive probability. We also bound the loss of helpfulness when the defender mistakenly refuses a benign request. Thus, the defender can remain helpful even when different benign histories require different answers.


Q2: Approximate separation and state complexity

We added new results in Appendix D.5 to examine whether allowing occasional classification errors can reduce the number of states needed by a watchman. We find that the answer depends on which combinations of fragments are likely to appear in an attack.

We first construct a game in which the attacker presents a subset of mm individually harmless fragments, followed by a request that is unsafe to answer precisely when a particular fragment appeared earlier. Theorem D.6 shows that even with average classification error ϵ<1/2\epsilon<1/2, the watchman still needs at least 2m(1−h2(ϵ))2^{m(1-h_2(\epsilon))} states, where h2h_2 is binary entropy. Thus, allowing a fixed error rate below one half does not eliminate the exponential state requirement.

The new Proposition D.4 extends this result beyond uniformly random combinations of fragments. It allows some combinations to occur more frequently than others and shows that exponentially many states are still necessary when there is enough uncertainty about which fragments appeared. More precisely, if this uncertainty grows linearly with the number of possible fragments, then sufficiently small classification error still requires exponentially many states.

The new Proposition D.5 gives a complementary result. Suppose that most attacks contain one of a relatively small number of possible combinations of fragments, even though each combination may contain many fragments. The watchman can then track whether the fragments observed so far are consistent with one of these common combinations. If they are, it can determine whether answering the final request is safe; otherwise, it conservatively refuses. We prove that when the number of common combinations grows polynomially, this requires only polynomially many states, with bounded average classification error and no false negatives under the specified protocol. Proposition D.3 provides another polynomial-state construction when each attack contains at most a fixed number of distinct relevant fragments.

These results show that allowing errors does not, by itself, eliminate the exponential state requirement. However, a polynomial-state watchman is possible when attacks are concentrated on relatively few combinations of fragments or contain only a limited number of distinct fragments. Together, these results establish both regimes for a specified mosaic-game protocol, including nonuniform distributions over fragment combinations. Applying the criteria to natural attacks would require identifying the safety-relevant fragments in real conversations.

AuthorsOpenReview ↗

Reviewer MYbW — Comment 4/4

Q4: DeepSeek-V2-Lite

We examined why the DeepSeek-V2-Lite results show smaller and less consistent improvements than those for Qwen and Gemma. DeepSeek-V2-Lite-Chat is the smallest backbone in our evaluation and has a different architecture. Its base attacker sometimes generates poorly formatted responses, including visible reasoning tags or repetitions of the seed instead of a usable attack. This matters because the attacker and defender are trained through self-play, so the quality of the attacks generated by the initial model can affect how both improve during training. These differences may therefore help explain the variation in results across backbones.

The evaluation settings also differ. DeepSeek was evaluated on 300300 paired malicious goals, compared with 520520 for Qwen and Gemma, and its baseline attack success rate (ASR) was approximately 6%. This leaves relatively few successful baseline attacks for paired comparisons, making the results more sensitive to individual outcomes. The reported DeepSeek results also come from a single 9090-step training configuration, whereas Qwen and Gemma were evaluated under a broader range of configurations. In addition, DeepSeek was evaluated using Qwen3.6-35B-A3B as the judge rather than a judge sharing its backbone. In preliminary experiments using DeepSeek as the judge, we observed unstable rollouts and generation or formatting difficulties, which motivated our choice of the Qwen judge.

We also examined the adapter configurations. Table 2 reports LoRA rank r=32r=32 and scaling factor α=32\alpha=32 for all three backbones. The adapter parameters account for approximately 0.89% of the parameters active per token for DeepSeek, compared with 0.94% for Qwen and 0.97% for Gemma. These proportions are similar, although DeepSeek's adapters target a different set of modules, including dense and shared-expert MLPs. Thus, the differences in performance are not readily explained by relative adapter size alone.

The exploratory training results provide further insight into DeepSeek's learning dynamics. Reward curves from additional DeepSeek configurations show a change in attacker and defender rewards around step 4040, suggesting a possible transition in training behavior. These runs use different settings from the reported 9090-step experiment, and the observed change may also reflect a run or schedule boundary. We also examined preliminary rank-256256 trials, which showed no improvement over the lower-rank configuration. Together, these observations provide useful insights into how training dynamics and adapter configurations may affect DeepSeek's performance.

Overall, the DeepSeek experiments show that our method can also improve attacker and defender performance on a smaller backbone with a different architecture, although the gains are less consistent than for Qwen and Gemma. Our additional analysis identifies several possible contributing factors, including the initial attacker's generation behavior, training dynamics, and evaluation configuration. Appendix I.5 discusses the differences in evaluation, judge, base-model behavior and adapter size, and their limitations.

Comments on the paper

1 comment

Dear reviewers, as the deadline is approaching, please check the rebuttal and consider participating in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.