- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes the Guard-to-Action framework, which tests whether guard bypasses propagate into restricted tool calls by downstream agents. It translates 1901 harmful prompts into 98 languages, filters them through 5 guards, and feeds the admitted prompts to 3 agents in a paired experiment across 8K to 32K contexts and beginning, middle, and end positions. Safety proves model-specific: Qwen's beginning-position failure rate rises to 35.24 percent while Llama's falls to 0.72 percent, prompt position reverses model rankings, and low-resource languages fail at roughly twice the rate of high-resource languages.
Critical strengths
-
Deterministic sample ids make all main comparisons paired, and restricted-tool selection avoids LLM-judge distortion. This makes the ranking-reversal result statistically credible as an existence finding.
-
The guard-to-action framing asks the right question: following guard false negatives into downstream tool use is a genuinely useful evaluation unit that prior benchmarks miss. With a specified harness and verified translations, this pipeline could become a solid benchmark.
Critical weaknesses
1 Headline multilingual claim is confounded by construction. All prompts are machine-translated; the translation-quality stratified analysis promised in RQ1/RQ3/Sections 3.1 and 3.5 never appears in the results. The reference set is 77% low-resource and admission itself is tier-dependent (67.2% vs 38.8% bypass), so the 2x low-resource gap cannot be separated from translation artifacts and admission selection.
2 The two headline axes are never integrated. No language/tier by position/length interaction is reported; the multilingual and long-context studies read as two separate works under one title.
3 The primary metric is uninterpretable. The harness is unspecified: no system prompt, policy text, tool list or schemas, benign-tool count, definition of escalate, or decoding parameters. Restricted-tool selection may measure instruction-following rather than safety failure. Model scope is narrow and dated. Three agents only, two from 2024, no frontier-scale or post-2025 architectures; single checkpoint and decoding config per model with no seed variance.
4 The one solid finding is a thin negative result. "No universal positional pattern" is supported, but its actionable content reduces to "test your own model".
5 No benign controls or defense-in-depth. No over-refusal measurement despite citing XSTest; no output guards or execution-time authorization; position effects are tested with a single benign-English filler type.
- The paper suffers from significant language problems; the full text should be carefully checked and revised.
Questions for authors
-
Where are the promised translation-quality stratified results? Does the low-resource gap survive controlling for translation quality?
-
Please provide the full harness: system prompt, policy text, tool list and schemas, benign-tool count, definition of "escalate", decoding parameters, seeds.
-
The paper needs a thorough language check, so the writing throughout should be carefully proofread and revised.
-
A comprehensive comparison against the latest baseline models is needed.