- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how the two roles of criticism and proposal should be assigned in LLM-based automatic red teaming. The authors decompose the existing iterative red-teaming process into criticism generation and attack proposal, and compare three configurations: External Criticism + External Proposal, Self-Criticism + External Proposal, and Self-Criticism + Self-Proposal. The experiments show that self-criticism based on the target’s complete execution context can produce more effective failure diagnoses, while asking the target itself to further generate attack proposals is more likely to result in refusal or performance degradation. Based on this observation, the authors propose a role-separated red-teaming method in which the target performs self-criticism and an independent attacker performs external proposal. Experiments on DTAP-Bench cover multiple agent frameworks, task domains, and both direct misuse and indirect prompt injection threat models. The results show that this configuration achieves the highest overall ASR, and the paper further analyzes its effectiveness through reasoning-context ablation and refusal analysis.
Critical strengths
-
The paper clearly separates the two roles of criticism and proposal in automatic red teaming, and the problem formulation has a certain degree of novelty. Compared with prior work that mainly optimizes attack prompts, search strategies, history management, or branching/pruning, this paper focuses on the role assignment problem of “who should diagnose the failure, and who should act on that diagnosis?” This perspective is clear and also reveals an evident asymmetry between self-criticism and self-proposal.
-
The experimental evaluation is relatively comprehensive, covering multiple base models, agent frameworks, task domains, and both direct and indirect threat models. In addition to comparing the final ASR, the paper conducts reasoning-context ablation and analyzes explicit refusal versus non-refusal degradation, which provides additional evidence for understanding the observed performance differences.
Critical weaknesses
-
The paper presents criticism source and proposal source as two independent design dimensions, but the experiments do not cover all possible combinations, with External Criticism + Self-Proposal missing. The current experiments can only show that self-criticism or external proposal performs better under specific conditions, but cannot fully disentangle the effects of criticism source, proposal source, and their interaction.
-
There is an evident information asymmetry in the comparison between Self-Criticism and External Criticism. Self-criticism can access the complete live execution context, while external criticism only receives a transferred subset of the information. Therefore, it is currently difficult to distinguish whether the performance improvement comes from the target itself or from access to more complete execution information.
Questions for authors
-
Does Table 3 for the Workflow domain mistakenly reuse the Telecom results? Please confirm and provide the correct table.
-
Why was External Criticism + Self-Proposal not evaluated? If the result of this combination is inconsistent with the current theoretical expectation, would it affect the paper’s conclusion regarding the target’s roles as its own best critic and a poor proposer?
-
What execution information does external criticism actually receive? If its input is less complete than that of self-criticism, how do the authors distinguish whether the advantage of self-criticism comes from target identity or from information advantage?
-
Are the system prompt, role prompt, and context composition the same for Self-Proposal and External Proposal? Can these factors be further controlled so that the effect of session separation itself can be compared?
-
What exactly does H_k in Algorithm 1 contain? Does it include previous attacks, criticisms, judge outputs, or execution trajectories?
2 comments
Thank you for the authors' response. However, I will maintain my original rating, as the authors have not adequately explained why evaluating the External Criticism + Self-Proposal would not be worthwhile.
We thank the reviewer for recognizing the novelty of separating criticism and proposal, as well as the broad evaluation across agent frameworks and threat models.
Q1. Is Workflow Table 3 duplicated?
Yes. Appendix E, Table 3 mistakenly repeats the Telecom category-level results.
The separately reported Workflow findings are:
The duplicated table is a reporting error and cannot support Workflow category-level claims. The underlying category results require verification against the original evaluation records.
Q2. Why is External Criticism + Self-Proposal missing?
The original experiments focus on two controlled comparisons:
External Criticism + Self-Proposal is the missing fourth cell of the 2×2 factorial design. However, given the existing comparison and analysis, we do not think it would be a worthy evaluation for this configuration.
Q3. Target identity versus information advantage
We attribute an important part of the advantage to retained decision-relevant execution information.
In Sec. 3.3, removing reasoning while retaining the remaining interaction history reduces downstream ASR:
A fully information-matched external critic would receive the same serialized trajectory and decision-relevant context, with identical model and criticism instructions.
This separates information availability from session continuity. The existing ablation establishes the importance of execution information, but does not independently establish a target-identity advantage after information matching.
Q4. Are proposal prompts and contexts identical?
Model identity is controlled, but session contexts are intentionally different.
Thus, the original comparison evaluates the complete role-assignment configuration rather than session identity in isolation.
A finer-grained comparison can distinguish identical criticism and proposal instructions, matched serialized context, and original-session continuation.
The observed proposal-stage difference remains substantial:
These results establish the practical advantage of external proposal in the evaluated configurations, while leaving the precise causal contribution of context composition unresolved.
Q5. What does Hk contain in Algorithm 1?
Hk denotes the previous attack-search history, distinct from the target's live execution context.
Specifically,
Hk={(xi,ci,yi)}i=0k−1,
where xi is the previous attack, ci is its criticism, and yi is the judge outcome.
The external proposer receives
xk+1=A(q,xk,ck,Hk).
Therefore, its inputs contain the task objective, current attack, current criticism, and previous attacks, criticisms, and judge outcomes.
Complete target execution trajectories and retained reasoning are not transferred by default. Instead, the target-generated criticism serves as a compact interface between the target and external attacker. This selective context transfer is a central component of the role-separated design.