- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a Unified Hierarchical Red-Teaming framework for attacking multiple LLM safety settings with one shared decision process. The framework separates task-specific knowledge from a shared policy: each task keeps its own strategy library and evaluator, while the shared policy performs strategy selection, structured attack-program construction, and response-conditioned action generation. A two-level tree search explores multiple strategy branches, and Hierarchical Evidence-Guided Policy Evolution (HEPE) updates different components using decision-specific evidence rather than only final attack outcomes.
Critical strengths
- The paper addresses an important practical limitation of prior red-teaming methods by providing one framework that can handle jailbreaking, prompt extraction, and prompt injection with a shared decision process.
- The hierarchical decomposition into strategy selection, program construction, and action generation is intuitive and enables more targeted improvement than updating the entire attack trajectory from a single final reward.
- HEPE makes good use of fine-grained search evidence, including same-instance strategy comparisons and same-state sibling comparisons, rather than relying only on coarse attack success signals.
Critical weaknesses
- Despite being presented as a unified framework, much of the actual attack knowledge remains task-specific through separate strategy libraries, interaction protocols, and evaluators, so the degree of genuine cross-task generalization is limited. 51646_Unified_Hierarchical_Red
- The three evaluated tasks are all closely related prompt-level adversarial attacks, making it unclear whether the shared policy would generalize to substantially different red-teaming settings such as tool-use attacks, agentic environments, or multimodal attacks.
- The paper does not isolate the benefit of cross-task evidence sharing itself; the reported ablations show that HEPE helps, but do not clearly compare joint multi-task evolution against independently evolving the same policy on each task.
Questions for authors
- How much improvement actually comes from sharing evidence across tasks rather than training a separate hierarchical policy for each task?
- Have you tested the framework on a red-teaming task that is substantially different from prompt-based attacks, such as tool-use or agentic attacks?
- Would the framework still perform well if the task-specific strategy libraries were much smaller or unavailable?
- Can you report the total inference cost of your method, including the attacker, verifier, and evaluator models, rather than only target-model query counts?
2 comments
Thank you again for your time and thoughtful feedback. With one days remaining in the discussion period, please let us know if any questions or concerns remain after our responses. We would be happy to provide further clarification.
We would like to thank the reviewer for taking the time to review our paper and for the valuable comments. We hope our response has adequately addressed your concerns raised in the review.
Please kindly let us know if anything is unclear. We truly appreciate this opportunity to improve our work and shall be most grateful for any feedback you could give us.