acceptodds
Withdrawn from ICLR 2027

REACT: Role-aware Exchange for Attacker–defender Co-evolution Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As adversarial attacks evolve rapidly, online attacker–defender co-evolution has emerged as a promising paradigm for LLM safety alignment. However, existing frameworks typically adopt an adversarial framing, relying on an external judge model to reduce rich interaction trajectories into guided rewards. This compression discards fine-grained contextual signals and creates a strong dependency on the judge's capability. To address this limitation, we propose Role-aware Exchange for Attacker–defender Co-evolution Training(REACT), a framework that preserves adversarial objectives at deployment and complements the co-evolution by enabling a learning-level cooperation during training: the two roles exchange their internal interaction artifacts—the attacker's hidden transformation rationale and the defender's execution feedback—as dense, reciprocal supervision. REACT consists of two complementary mechanisms: (1) Rationale-Grounded Safety Distillation(RGSD), which converts the attacker's hidden transformation intent into response-level supervision for the defender, leveraging multi-view distillation and safety-token weighting to enhance intent sensitivity without causing over-refusal; and (2) Defender-Feedback Interaction Distillation (DFID), which extracts a query-specific interaction residual from the defender's response via a difference-in-differences intervention, guiding the attacker's tactics without altering core intent. Extensive evaluations on standard adversarial benchmarks and benign compliance tasks show that REACT consistently outperforms competitive baselines. Specifically, REACT reduces the strict unsafe rate down to 0.5% on HarmBench while boosting benign compliance on OR-Bench up to 29.9%. Cross-play evaluations further confirm that REACT fosters a genuine co-evolutionary arms race, yielding both a stronger red-teaming adversary and a more resilient defender.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
co-evolution, AI-safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gWQNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes REACT, an online attacker–defender co-evolution framework that replaces judge-mediated scalar training signals with dense, role-specific supervision extracted from the interaction itself. REACT contains two components, RGSD and DFID. Experiments use Qwen3.5 models at 2B, 4B, and 9B scales and evaluate harmful refusal and benign compliance across HarmBench, StrongREJECT, XSTest, WildJailbreak, OR-Bench, and SQuAD. REACT reports strong results at 2B and especially 9B, including a 0.5% strict unsafe rate on HarmBench for the 9B model.

Critical strengths
  • A novel reframing of attacker–defender co-evolution: REACT treats the attacker rationale and defender response as complementary dense supervision rather than compressing the interaction into an external judge score. This is a meaningful direction because it may provide richer token-level credit while reducing dependence on a separate judge during training.
  • Thoughtful task-specific mechanisms: DFID’s query-masked interaction contrast and transformation-token mask directly target plausible shortcuts in feedback-conditioned attacker training.
  • Meaningful evaluation: The evaluation spans three model scales and multiple harmful and benign benchmarks, and cross-play is more informative than evaluating only the final paired checkpoint.
Critical weaknesses
  1. The paper lacks the component-level ablations required to attribute its gains.

    The main results evaluate only the complete REACT system, without comparing no exchange, RGSD-only, DFID-only, and full REACT under a matched training budget. Therefore, the improvements in defender safety, benign compliance, and attacker strength cannot be attributed to either proposed component.

    • For RGSD, the paper does not ablate the Teacher, Self, and None views. Consequently, it remains unclear whether the multi-view construction actually mitigates privilege illusion or whether direct privileged-response distillation would perform similarly. Besides, the safety-token weighting experiment provides some direct evidence, but its effect is small and validates only one subcomponent of RGSD.

    • For DFID, the paper does not compare direct feedback conditioning, the factual first difference $ \ell^{QA}-\ell^{Q0}, $ and the full interaction residual $ (\ell^{QA}-\ell^{Q0})-(\ell^{\bar QA}-\ell^{\bar Q0}). $ Thus, there is no direct evidence that the second difference removes generic response bias. The transformation-mask diagnostics show that copied positions receive zero direct residual weight and that copied tokens otherwise have larger residual magnitudes, but they do not compare the final attack success or intent-preservation performance with and without the mask.

    Overall, the paper demonstrates that the complete REACT system performs well and that several mechanisms operate as implemented, but it does not demonstrate that the specific effects claimed for RGSD and DFID are responsible for the reported gains. This missing causal attribution substantially weakens the paper's central methodological claims.

  2. The strongest empirical and novelty claims need stronger validation.

    • RGSD closely follows DAPD's Teacher/Cross, Self, and None-view structure [1]. Moreover, RGSD reverses the direction of DAPD's inference anchor: DAPD trains Self toward detached None, whereas RGSD trains None toward detached Self. This modification may be reasonable, but the paper provides neither a theoretical justification nor a direct comparison.
    • Experimentally, the reported 9B HarmBench improvement from 20.1% to 0.5% is unusually large, while the 4B gains are much smaller. Analysis on this discrepancy is needed.
    • The high-level idea of bidirectional attacker--defender co-evolution is not itself novel. The more specific distinction of REACT is that it converts role-specific interaction artifacts. However, even this contribution is largely compositional. RGSD substantially builds on DAPD-style privileged multi-view distillation.

[1] DAPD: Dual-Anchored Policy Distillation

Questions for authors
  1. Can the authors provide a module-level comparison of no exchange, RGSD-only, DFID-only, and full REACT under the same rollout and optimization budget? This would materially affect my assessment of the main contribution.

  2. For DFID, can the authors compare direct feedback conditioning, the factual first difference, and the full DiD residual? A shuffled or mismatched-response control would also test whether DFID extracts query-specific feedback rather than generic refusal style.

  3. Figure 4 shows that, from R1 to R3-R5, the fraction of material transformations decreases from approximately 64% to 20-30%, while exact-copy proposals increase from approximately 36% to around 80%. Can this suggest a self-reinforcing degeneration of the attacker?

Reviewer_Zyg4OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This authors propose an attacker-defender co-evolution training framework (REACT) for co-learning both models iteratively through RGSD and DFID. In RGSD, the attacker's rationale r_A conditions a privilege defender teacher to produce the attack-aware safe/refusal response. Three different views distillation are applied to allow the student model learn from the teacher's response with privilege knowledge. In DFID, the defender's response becomes the privilege information for the teacher attacker. A four-view DiD residual cancels query-agnostic response effects, and the correction is applied only to transformation tokens. The multiple-view distillation in both roles are designed so neither models needs the privilege filed at inference time. Through extensive evaluations on adversarial and over-refusal benchmarks, the authors show that REACT outperforms other baselines on both safety and over-refusal metrics.

Critical strengths
  1. This paper considers the most important issue in the distillation style work, which is the privilege information is not available at inference time. The proposed multiple-view conversion makes the privilege info learnable and handle the issue properly. The 'privilege illustion' concern with naive approaches is a valid one.
  2. Most adversarial-training work generates attacks through prompting or a fixed strategy library, the proposed REACT considers both side optimizations. The attacker receives genuine gradient updates from the integration, with the strategy card as optional scaffolding rather than the mechanism itself.
Critical weaknesses
  1. In Figure 4, the exact copy proposal rise and plateau while material transformation fall. The train eligible rollouts drop from 25% to 6% in two rounds. The authors mentioned this as selectively retains only high-quality trajectories. This could also be a signal of the attacker being degenerating toward copying the original query.
  2. This paper mentions using Qwen3Guard-Gen-8B as the evaluator. But other baselines use Qwen3Guard as the training judge. So the baselines are trained against REACT's evaluator. This makes the comparison not fair enough in either direction. And there is no human validation and addition judge introduced in the experiments.
Questions for authors
  1. Why is HarmBench non-monotonic across scale? It seems 9B defender over-refuse on HarmBench.
  2. Can you provide ablation isolating RGSD and DFID and compare different variants? It gives better idea on how the co-evolving framework achieves better performance on both ends comparing to the sole model trained with the same objectives.
Reviewer_WaSGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes REACT, an attacker–defender co-evolution framework for LLM safety that trains both roles by distillation instead of judge-based RL rewards. Rationale-Grounded Safety Distillation (RGSD) uses the attacker's hidden rationale to generate a privileged teacher response for the defender, distilled through three views (teacher, self, none) with safety-token weighting. Defender-Feedback Interaction Distillation (DFID) builds an attacker target from the defender's response via a four-view difference-in-differences logit residual, restricted to transformation tokens on the query path. Experiments train Qwen3.5 at 2B, 4B and 9B with LoRA and compare against Self-RedTeam and MAGIC on HarmBench, StrongREJECT, XSTest, WildJailbreak, OR-Bench and SQuAD, scored by Qwen3Guard-Gen-8B, with training-dynamics and cross-play analyses.

Critical strengths
  1. The judge is kept out of the training loop. Qwen3Guard is used only for post-hoc evaluation and its labels are never sent back to either optimizer (Appendix D.1, E.5), which directly addresses the judge-dependence the paper identifies in prior co-evolution methods.

  2. The implementation is unusually transparent. The appendix gives the prompt contracts, the hyperparameters of the primary run, the strategy-card schema and an end-to-end training trace, and the trace explicitly states that it demonstrates the information routing, not a safety improvement. This makes the method's actual mechanics checkable.

Critical weaknesses
  1. Table 1 does not support the abstract's claim that REACT "consistently outperforms competitive baselines", and the comparison is not controlled. REACT is below MAGIC on XSTest benign at all three scales (e.g., 0.871 vs 0.902 at 9B), contradicting §5.2. At 4B its StrongREJECT unsafe rate (0.008) is worse than the base model (0.003) and both baselines, and HarmBench moves only from 0.234 to 0.222. The paper reports a single seed and no variance, so the stability of the headline 9B HarmBench result (0.005) is unknown. MAGIC is run with different rounds, learning rate and batch size (Table 3 vs Table 2), and Appendix C.2 says the Self-RedTeam artifact uses a different 200+200 WildJailbreak evaluation, yet Table 1 reports Self-RedTeam on all seven benchmarks without explaining where those numbers come from.
Questions for authors
  1. Section 4.1 defines the Teacher View as conditioning on q′, the attacker rationale r_A and a safety system prompt. Appendix B and E.3 instead implement it as the deployment prompt with the teacher completion a_T inserted into the input, with no r_A. Which was used? If the latter, then for harmful records the teacher view scores Y = a_T while a_T is in its context; how does L_ED then differ from supervised fine-tuning on a_T, and can you report an SFT-on-a_T baseline?
  2. How were the Self-RedTeam numbers in Table 1 obtained, given Appendix C.2?
  3. Can you report results over multiple seeds, and a MAGIC baseline with matched rounds, learning rate and batch size?
  4. Table 4 uses α = 0.25, while the main run uses α = 0.5 (Table 2). Its only differences between arms are 0.78 points on 1,024 examples (8 examples) in two cells, from one seed. Is there a larger ablation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.