- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the vulnerability of model immunization, a defense designed to prevent open-weight models from being fine-tuned toward harmful tasks. The authors show that current immunization methods overfit to the defender's simulated attacker setup, creating an overlooked evaluation gap. To expose this fragility, they propose ESCAPE, a framework that intentionally departs from the defender's assumptions across harmful data, model initialization, and adaptation strategies like LoRA configurations. Empirical evaluations across three modalities (diffusion, classification, and language models) demonstrate that ESCAPE effectively bypasses existing defenses and substantially degrades their protection.
Critical strengths
- The paper identifies key limitations in current model immunity/safety evaluation protocols and proposes a comprehensive, multi-dimensional evasion framework.
- The authors conduct extensive experiments across diverse tasks, demonstrating the effectiveness and generalizability of the proposed evasion method.
Critical weaknesses
Lack of adaptive defense evaluation: The paper does not evaluate against an adaptive defender. Specifically, the authors do not report the attack success rate in worst-case scenarios where the defender has full knowledge of the attack algorithm.
Unclear novelty regarding the immunity vs. harmfulness adaptation gap: The paper frames the mismatch between model immunity defense and downstream harmfulness adaptation as a train-test discrepancy. However, it is unclear whether this work is the first to formalize this specific gap. The authors should provide a more thorough discussion of how prior safety benchmarks and fine-tuning protocols approached this problem to clarify the conceptual novelty.
Questions for authors
- Is this work the first to identify this discrepancy in model immunity evaluation, or do existing benchmarks also consider it? Please explicitly contrast your evaluation setup with prior protocols to clarify the core conceptual difference.
- How does the proposed bypass method perform if the defender has full knowledge of your algorithm (white-box/adaptive defense setting)? Could you provide concrete results or a worst-case analysis under this threat model?