- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CoPair-Bench, a 175-pair benchmark intended to isolate content-dependent harmfulness under fixed instructions and tools. It also proposes HP-Shield, combining content inspection with safety experiences distilled from sandbox simulations. Across three agent backbones, the method improves contrastive pair accuracy by an average of 14.48 percentage points over the strongest existing guardrail for each backbone.
Critical strengths
- Paired harmful/benign evaluation exposes over-refusal that harmful-only blocking rates conceal, making the benchmark question consequential.
- Prompt/experience ablations, cross-model transfer, and repeated-run/judge checks support a practical benefit from content-grounded guidance.
Critical weaknesses
- Evaluation setting: The paper attributes failures to superficial cues, but the cue-free subset consists only of OpenAgentSafety and AgentDojo, whereas other sources are entirely cue-bearing (Table 3). The cue-removal experiment helps, but source difficulty and access to decisive content still need controlled comparisons before concluding that existing guards principally use shortcuts.
- Method contribution: HP-Shield adds an inspection prompt and a library of safety rules. Prompt-only and iteration ablations show improvement, but do not establish why sandbox execution is necessary. A rule library generated directly from the same entity cards/cases, with matched rule length and budget, would distinguish self-exploration from simply supplying more detailed instructions.
- Generalization and comparison: The 108 exploration cases come from 18 manually specified entity cards and run in benchmark environments. Cross-model transfer does not establish transfer to unseen entities, tools, or risks. The paper needs a clear separation of preparation and evaluation, and comparisons where existing guards receive the same inspected evidence; otherwise gains may reflect tailored guidance or information access.
- Benchmark validity: The defining instruction-invariance claim is unclear: Figure 14 changes the requested item from handmade crafts to illegal drugs, while Section 3.2 says 24 native AgentHarm pairs are retained as strictly matched. If such pairs enter the evaluation, they cannot isolate content-only harmfulness. Dataset membership and results on an audited strictly invariant subset are needed.
Questions for authors
- Does HP-Shield outperform a directly generated safety checklist from the same 18 entity cards and 108 cases, with matched prompt length and preparation budget?
- What separates entity-card preparation from evaluation, and do gains persist for held-out risks/tools and when baselines receive the same inspected evidence?
- Is Figure 14 part of the final evaluation? Please report instruction-equality checks and controlled within-source cue-removal results.
Thank you for the response. However, my main concerns remain unaddressed.
The response mischaracterizes my concern about the defense. My review did not object to the general paradigm of distilling experiences into the agent’s context. It specifically compared HP-Shield’s pipeline with one prior pipeline, ToolShield. The response replaces this with a claim I did not make. The paper’s own Figure 5 shows the actual issue. Comparing it with ToolShield’s Figure 9, I cannot identify the novelty of HP-Shield’s pipeline:
The only differences are the input (entity cards instead of tool documentation), the type of synthesized cases (benign/harmful instead of single-/multi-turn), and the content-inspection prompt added during execution. The two pipeline figures are also nearly identical, including the stage names, the Safe/Unsafe branch, and the feedback edge to stage 2. None of [1–6] has this pipeline: those methods learn from attempts on given tasks, not from safety test tasks they synthesize themselves and execute in a sandbox. Describing the pipeline as a “widely used paradigm” is therefore inaccurate, and my concern about novelty remains unaddressed. Moreover, the new results do not isolate the effect of these modifications: of the 7.81-point gain over re-learned ToolShield, about 3.0 points come from benign cases, and the rest is unattributed. I suspect that much of the remainder comes from the entity cards or from the content-inspection prompt. The authors also do not specify how the entity cards are produced, or whether they are derived from the benchmark itself.
The revision is internally inconsistent on the benchmark’s central claim. Section 3.1 now allows pairs whose instructions differ in the tokens naming the content. However, Figure 2 still states that “every retained pair shares one instruction” and is labeled “Same instruction & tools,” and the introduction and conclusion still describe CoPair-Bench as “the first strictly paired benchmark,” built from pairs that preserve task instructions. Under the authors’ own revised definition, 110 of the 175 pairs do not satisfy this. Either the headline claim must be restated to cover only the 65 pairs that do, or the definition has been adjusted after the fact to fit the data. I therefore do not consider the central claim of a strictly paired benchmark to be supported in its current form.
Based on these concerns, I will maintain my rating.