- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a MAP-Elites-based framework for mapping where red-teaming failures occur in a behavioral descriptor space, with each occupied cell retaining the highest observed Alignment Deviation. Across three full-budget and five reduced-budget model evaluations, it reports up to 72.32% coverage and 370 vulnerable cells, with different failure patterns for Llama-3-8B, GPT-OSS-20B, and GPT-5-Mini.
Critical strengths
-
The paper addresses a timely and practically relevant problem in LLM safety evaluation.
-
The proposed framework offers an intuitive way to organize and visualize failure patterns beyond aggregate scores.
-
The evaluation across multiple models and defense settings provides a useful starting point for further study.
Critical weaknesses
-
The paper is short and does not reach nine pages, which makes it difficult for me to understand the work fully.
-
There are obvious formatting errors, such as “Appendix ??” on line 93, and the authors should check the manuscript carefully.
-
The paper defines a vulnerable cell as having AD strictly greater than 0.5, and Appendix A.11.2 states that Figure 13 reports the mean AD of these cells. However, the figure reports 0.49 for “information”, 0.47 for “process”, and 0.38 for “create”, which is impossible if all the cells included in each mean have AD greater than 0.5.
Questions for authors
see weaknesses