acceptodds
Under review as a conference paper at ICLR 2027

Unified Hierarchical Red-Teaming: Strategy Search and Evidence-Guided Policy Evolution

Abstract

Large language models (LLMs) remain vulnerable to adversarial attacks after safety alignment, motivating automated red-teaming methods that systematically search for model failures. Existing strategy-based methods are mostly designed for a specific attack task and often use final attack outcomes to improve different attack decisions without distinguishing where the failure occurs. We propose a unified red-teaming framework consisting of a Unified Hierarchical Red-Teaming Policy and Hierarchical Evidence-Guided Policy Evolution (HEPE). The hierarchical policy separates task-specific attack knowledge from a shared decision process: each task maintains its own strategy library, interaction protocol, and evaluator, while the shared policy performs strategy selection, structured program construction, and response-conditioned action generation. A two-level tree search explores multiple strategy branches and executes each strategy through a structured attack program and sibling candidate actions under a shared target-query budget. HEPE improves the shared policy using decision-specific evidence collected during search. Same-instance strategy comparisons revise strategy selection, program and execution records revise program construction, and same-state sibling comparisons revise action generation. Compatible evidence for these shared components is aggregated across tasks, while each task-specific strategy library is revised only from cross-instance evidence collected for that task through structured ADD, REFINE, and MERGE operations. Candidate updates are evaluated against the current version on held-out instances using paired validation before acceptance. We evaluate the framework on jailbreaking (JB), system-prompt extraction (SPE), and direct prompt injection (DPI) across six target models. The results show consistent improvements over recent automated red-teaming baselines across all three tasks under limited target-query budgets.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Red-Teaming, Hierarchical Strategy Search, Evidence-Guided Policy Evolution, Automated Attack Generation, Jailbreaking, Prompt Injection, System-Prompt Extraction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QC2zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Unified Hierarchical Red-Teaming framework for attacking multiple LLM safety settings with one shared decision process. The framework separates task-specific knowledge from a shared policy: each task keeps its own strategy library and evaluator, while the shared policy performs strategy selection, structured attack-program construction, and response-conditioned action generation. A two-level tree search explores multiple strategy branches, and Hierarchical Evidence-Guided Policy Evolution (HEPE) updates different components using decision-specific evidence rather than only final attack outcomes.

Critical strengths
  1. The paper addresses an important practical limitation of prior red-teaming methods by providing one framework that can handle jailbreaking, prompt extraction, and prompt injection with a shared decision process.
  2. The hierarchical decomposition into strategy selection, program construction, and action generation is intuitive and enables more targeted improvement than updating the entire attack trajectory from a single final reward.
  3. HEPE makes good use of fine-grained search evidence, including same-instance strategy comparisons and same-state sibling comparisons, rather than relying only on coarse attack success signals.
Critical weaknesses
  1. Despite being presented as a unified framework, much of the actual attack knowledge remains task-specific through separate strategy libraries, interaction protocols, and evaluators, so the degree of genuine cross-task generalization is limited. 51646_Unified_Hierarchical_Red
  2. The three evaluated tasks are all closely related prompt-level adversarial attacks, making it unclear whether the shared policy would generalize to substantially different red-teaming settings such as tool-use attacks, agentic environments, or multimodal attacks.
  3. The paper does not isolate the benefit of cross-task evidence sharing itself; the reported ablations show that HEPE helps, but do not clearly compare joint multi-task evolution against independently evolving the same policy on each task.
Questions for authors
  1. How much improvement actually comes from sharing evidence across tasks rather than training a separate hierarchical policy for each task?
  2. Have you tested the framework on a red-teaming task that is substantially different from prompt-based attacks, such as tool-use or agentic attacks?
  3. Would the framework still perform well if the task-specific strategy libraries were much smaller or unavailable?
  4. Can you report the total inference cost of your method, including the attacker, verifier, and evaluator models, rather than only target-model query counts?
2 comments

We would like to thank the reviewer for taking the time to review our paper and for the valuable comments. We hope our response has adequately addressed your concerns raised in the review.

Please kindly let us know if anything is unclear. We truly appreciate this opportunity to improve our work and shall be most grateful for any feedback you could give us.

Thank you again for your time and thoughtful feedback. With one days remaining in the discussion period, please let us know if any questions or concerns remain after our responses. We would be happy to provide further clarification.

Reviewer_Hmn2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a unified hierarchical red-teaming framework that organizes automated attacks into three stages: strategy selection, structured attack program construction, and prompt generation. The framework employs a two-level tree search to explore different attack strategies and generate attack prompts, while Hierarchical Evidence-Guided Policy Evolution (HEPE) uses decision-specific evidence to refine the corresponding policy components and task-specific strategy libraries. This design aims to address the issues of insufficient universality and imprecise failure attribution in traditional methods. Tested on three types of attack tasks (JB, SPE, and DPI) and six target models show that the proposed framework outperforms eight traditional red-teaming methods in attack effectiveness while using fewer target model queries.

Critical strengths
  1. The paper proposes a unified red-teaming framework that covers three attack tasks: JB, SPE, and DPI. This general framework addresses the difficulty of directly reusing attack procedures across different tasks. In experiments, the proposed framework maintains consistently high attack success rates across different target models compared with eight automated red-teaming methods.

  2. The paper introduces a hierarchical evidence-guided optimization mechanism, HEPE, which continuously improves attack strategies and execution processes using evidence collected during the search. Compared with traditional approaches that mainly modify prompts based on feedback, this approach provides more targeted optimization of different stages of the overall attack process rather than relying solely on final attack outcomes. Moreover, the validation of updated policies helps ensure that updates improve overall performance and avoid error accumulation or performance degradation.

Critical weaknesses
  1. The paper's contributions are somewhat overstated. Techniques such as dynamic adjustment, feedback-based optimization, and policy evolution have already been explored in prior work. Therefore, these techniques themselves should not be presented as major contributions. The contribution would be more appropriately framed around the specific optimization mechanisms and integration introduced by this work compared with existing approaches.

  2. HEPE is presented as one of the paper's two major contributions, enabling the framework to learn from attack experience, modify policies, and prevent overall performance degradation. However, the paper does not provide sufficient evidence that this evolution process is stable and consistently valuable in practice. In particular, the constructor updates shown in the appendix are all rejected. It therefore remains unclear how often HEPE successfully produces beneficial updates and how much of the final performance improvement actually comes from HEPE. It is possible that Hierarchical Search itself accounts for most of the performance gains, while the additional contribution of HEPE remains insufficiently demonstrated.

Questions for authors

I would like to understand how many effective policy updates HEPE actually produces. Could the authors report the number and proportion of accepted versus rejected candidate policy updates across the complete experiments? In particular, the constructor updates shown in the appendix are all rejected. What proportion of the policy updates generated by HEPE are actually accepted and result in effective improvements in the experiments?

2 comments

We would like to thank the reviewer for taking the time to review our paper and for the valuable comments. We hope our response has adequately addressed your concerns raised in the review.

Please kindly let us know if anything is unclear. We truly appreciate this opportunity to improve our work and shall be most grateful for any feedback you could give us.

Thank you again for your time and thoughtful feedback. With one days remaining in the discussion period, please let us know if any questions or concerns remain after our responses. We would be happy to provide further clarification.

Reviewer_vW3fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Unified Hierarchical Red-Teaming Policy and Hierarchical Evidence-Guided Policy Evolution (HEPE) for jailbreaking (JB), system-prompt extraction (SPE), and direct prompt injection (DPI). The framework separates task-specific attack knowledge from a shared decision process: each task maintains its own strategy library, interaction protocol, and evaluator, while the shared policy performs strategy selection, structured program construction, and response-conditioned action generation. HEPE uses same-instance strategy comparisons, program and execution records, and same-state sibling comparisons to update the strategy selector, program constructor, and action policy, respectively. In the reported experiments, the method generally outperforms the baselines across the JB, SPE, and DPI metrics.

Critical strengths
  1. The paper separates task-specific attack knowledge from a shared decision process, allowing the strategy selector, program constructor, and action policy to be updated using evidence from their respective decision levels while retaining task-specific strategy libraries, interaction protocols, and evaluators.

  2. The proposed method generally outperforms the evaluated baselines across JB, SPE, and DPI on six target models, including both open-weight and API-based models.

Critical weaknesses
  1. The paper reports performance gains across three tasks. However, these results do not establish that aggregating compatible evidence across tasks for shared-policy updates is more effective than independently updating the same policy components using only evidence from each task under matched resources. The paper does not provide this controlled comparison experiment, thus the additional contribution of cross-task evidence aggregation to HEPE’s performance is unclear.

  2. The paper claims that “decision-specific policy updates provide most of the improvement,” but Figure 2b does not sufficiently support this interpretation. If Frozen Strategy Library retains decision-specific policy updates and disables only strategy-library evolution, its improvement over w/o HEPE is smaller than its gap to the full method on all three tasks. These results do not support the claim that policy updates provide most of the improvement. In addition, the components updated and the specific settings used in Generic Outcome Evidence are insufficiently described.

Questions for authors
  1. Do you have results comparing cross-task aggregation of compatible evidence against independent updates of the same policy components using only evidence from each task, with the resources (e.g. amount of evidence, update budget) matched?

  2. In the HEPE ablation, if Frozen Strategy Library disables only strategy-library evolution, how do you explain that its improvement over w/o HEPE is smaller than its gap to the full method on all three tasks in Figure 2b, and what supports the claim that “decision-specific policy updates provide most of the improvement”?

  3. For SPE outputs that fail exact matching but are counted as successful under Approximate Extraction by the Qwen2.5-7B-Instruct evaluator, beyond calibrating the threshold on positive and negative examples, have you verified the judgments on actual test outputs through manual spot checks or by reconstructing the protected access code, and what is the false-positive rate? Where do the 230 extraction-detection examples used to calibrate the evaluator threshold come from?

2 comments

We would like to thank the reviewer for taking the time to review our paper and for the valuable comments. We hope our response has adequately addressed your concerns raised in the review.

Please kindly let us know if anything is unclear. We truly appreciate this opportunity to improve our work and shall be most grateful for any feedback you could give us.

Thank you again for your time and thoughtful feedback. With one days remaining in the discussion period, please let us know if any questions or concerns remain after our responses. We would be happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.