- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes ARMOR, a memory-augmented multi-agent framework for automated black-box jailbreaking. The framework consists of a Strategist Agent, an Attacker Agent, a persistent memory module for retrieving previously successful attack strategies, and a multi-judge evaluation system. The authors further introduce Expected Target Queries (ETQ) to account for survivorship bias in query efficiency evaluation. Experiments on LLaMA and Nemotron models demonstrate improvements in attack success rates and target-query efficiency compared with PAIR and TAP. The paper also includes an ablation study to examine the contribution of persistent memory.
Critical strengths
-
Clear presentation and framework design. The paper is well organized, and the overall workflow is clearly explained. The division between strategy planning, attack generation, and evaluation is easy to follow.
-
A useful perspective on attack efficiency. The introduction of ETQ is a valuable attempt to address survivorship bias in conventional query efficiency metrics, which often consider only successful attacks.
Critical weaknesses
1. The practical significance and novelty of ETQ require further justification.
I appreciate the authors' attempt to account for unsuccessful attacks when measuring query efficiency. However, it remains unclear how much additional insight ETQ provides beyond existing evaluation protocols. In adversarial settings, attack success is generally the primary objective, and prior work has already reported success rates under different attack or query budgets.
Moreover, the proposed ETQ measures average target queries per intent, including unsuccessful attempts, rather than the expected number of queries required to obtain a successful jailbreak. A method with a low success rate may achieve a relatively low ETQ simply by terminating early. Therefore, ETQ should be interpreted jointly with ASR. I would encourage the authors to clarify its practical significance and compare it with success-rate-versus-query-budget evaluations.
2. The experimental evaluation is insufficient to establish the effectiveness and generalizability of ARMOR.
The experimental evaluation has several limitations. First, the baseline comparisons are restricted to PAIR and TAP, which do not adequately represent recent developments in automated red-teaming. In particular, since the main contribution of ARMOR is persistent strategy reuse, comparisons with existing experience-based or adaptive attack methods are essential. I recommend referring to OpenRT (https://arxiv.org/abs/2601.01592), which provides a broader collection of recent attack methods.
Second, the evaluation uses only the first 100 queries from AdvBench, which is insufficient to demonstrate generalizability. Given that ARMOR progressively accumulates attack strategies across queries, the results may also be sensitive to the ordering and distribution of the selected samples. Evaluation on additional benchmarks, such as HarmBench, would provide stronger evidence.
Finally, although the evaluated models range from 8B to 550B parameters, they are limited to the LLaMA and Nemotron families. Model size does not necessarily reflect safety alignment strength. Since ARMOR operates in a black-box setting, I would expect evaluations on more recent and strongly aligned frontier models from different providers. OpenRT also provides useful references for broader model coverage.
Overall, the current experimental setup is too limited to support the paper's claims regarding the superiority and generalizability of ARMOR.
Questions for authors
-
ETQ: What additional practical insights does ETQ provide compared with reporting ASR under different query budgets? Could the authors clarify how ETQ should be interpreted when comparing methods with substantially different success rates?
-
Baselines: How does ARMOR compare with more recent automated red-teaming approaches, particularly those incorporating feedback, contextual memory, or adaptive attack strategies? What advantages does ARMOR provide over existing experience-based approaches?
-
Dataset and generalization: Why were only the first 100 AdvBench intents selected? How sensitive are the results to the ordering of these intents, and is there evidence that accumulated attack strategies transfer to held-out queries or different benchmarks?
-
Frontier models: Can the authors clarify why the evaluation is restricted to LLaMA and Nemotron models? Is there evidence that ARMOR remains effective against more recent, strongly aligned frontier models?