acceptodds
Under review as a conference paper at ICLR 2027

Guard-to-Action: Multilingual Long-Context Safety in Tool-Using Agents

Abstract

Large language model (LLM) agents are increasingly deployed with access to real-world tools, yet most safety evaluations remain English-only and short-context, leaving open whether multilingual translation and long-context placement can jointly weaken input guardrails and downstream agent behavior. We introduce a large-scale multilingual, long-context safety benchmark built from 1,901 unsafe English root prompts spanning 19 harm categories (derived from AEGIS2.0), translated into 98 languages to yield up to 186,298 prompt instances with translation-quality metadata. Each translated prompt is screened by five runtime input guards (AprielGuard, CREST, GuardReasoner, WildGuard, XGuard); guard-admitted prompts are then embedded, unchanged, in benign English conversation history at 8K and 32K token lengths and at the beginning, middle, or end of the context, and forwarded to tool-using downstream agents that must refuse, escalate, or select a restricted (inert) tool. Using matched, sample-level comparisons with paired-bootstrap confidence intervals and exact statistical tests, we find that downstream safety depends jointly on agent identity, context length, and prompt position, with no single condition uniformly safest, and that increasing context length can raise or lower risk depending on this interaction rather than acting uniformly in either direction. We further find that low-resource languages exhibit substantially higher downstream risk than high- or medium-resource languages despite comparable guard admission rates, and that aggregate safety rates across context lengths can mask considerable prompt-level instability, with a nontrivial share of individually matched prompts flipping between safe and unsafe outcomes. Our current evaluation covers two open-weight instruction-tuned agents (Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct); we are extending this evaluation to additional agent families to assess how broadly these interaction effects generalize.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multilingual LLM safety, agentic AI safety, runtime guardrails, long-context safety, tool-using agents, guardrail bypass, prompt position, restricted-tool selection, safety benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_LCvUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the Guard-to-Action framework, which tests whether guard bypasses propagate into restricted tool calls by downstream agents. It translates 1901 harmful prompts into 98 languages, filters them through 5 guards, and feeds the admitted prompts to 3 agents in a paired experiment across 8K to 32K contexts and beginning, middle, and end positions. Safety proves model-specific: Qwen's beginning-position failure rate rises to 35.24 percent while Llama's falls to 0.72 percent, prompt position reverses model rankings, and low-resource languages fail at roughly twice the rate of high-resource languages.

Critical strengths
  1. Deterministic sample ids make all main comparisons paired, and restricted-tool selection avoids LLM-judge distortion. This makes the ranking-reversal result statistically credible as an existence finding.

  2. The guard-to-action framing asks the right question: following guard false negatives into downstream tool use is a genuinely useful evaluation unit that prior benchmarks miss. With a specified harness and verified translations, this pipeline could become a solid benchmark.

Critical weaknesses

1 Headline multilingual claim is confounded by construction. All prompts are machine-translated; the translation-quality stratified analysis promised in RQ1/RQ3/Sections 3.1 and 3.5 never appears in the results. The reference set is 77% low-resource and admission itself is tier-dependent (67.2% vs 38.8% bypass), so the 2x low-resource gap cannot be separated from translation artifacts and admission selection.

2 The two headline axes are never integrated. No language/tier by position/length interaction is reported; the multilingual and long-context studies read as two separate works under one title.

3 The primary metric is uninterpretable. The harness is unspecified: no system prompt, policy text, tool list or schemas, benign-tool count, definition of escalate, or decoding parameters. Restricted-tool selection may measure instruction-following rather than safety failure. Model scope is narrow and dated. Three agents only, two from 2024, no frontier-scale or post-2025 architectures; single checkpoint and decoding config per model with no seed variance.

4 The one solid finding is a thin negative result. "No universal positional pattern" is supported, but its actionable content reduces to "test your own model".

5 No benign controls or defense-in-depth. No over-refusal measurement despite citing XSTest; no output guards or execution-time authorization; position effects are tested with a single benign-English filler type.

  1. The paper suffers from significant language problems; the full text should be carefully checked and revised.
Questions for authors
  1. Where are the promised translation-quality stratified results? Does the low-resource gap survive controlling for translation quality?

  2. Please provide the full harness: system prompt, policy text, tool list and schemas, benign-tool count, definition of "escalate", decoding parameters, seeds.

  3. The paper needs a thorough language check, so the writing throughout should be carefully proofread and revised.

  4. A comprehensive comparison against the latest baseline models is needed.

Reviewer_vE9KOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper discusses safety guardrails in a multilingual setting (98 languages), spanning long contexts (8K to 32K).

The authors attempt to intercept the prompt with five different guards before allowing an LLM to call any tools to solve the tasks.

Across multiple models, the authors study how and where guardrails break down and begin admitting harmful requests, specifically how the risk varies across languages, context length, harm categories, and translation qualities.

Critical strengths

I appreciate that the paper discusses safety across multiple languages and long contexts. This is a much required analysis in real-world scenarios.

Critical weaknesses

The paper is not well-written, and reeks of AI. At this point, it feels less like a research paper that provides well-motivated and coherent arguments, but more like a technical report of an experimental setup mostly written/refined by AI. Even the introduction section fails to properly motivate the issue, but discusses related works.

The RQs and contributions seem quite thin and standard, like a simple analysis of existing artifacts. No new guard-artifacts or mitigation strategies are discussed.

I am also quite wary about the translation of the english prompts. The authors have also not discussed how the translation was done (using which models – any standard translation systems, or LLMs), and if any human verification of the data was conducted. Further, since the benchmark has multiple low-resource languages and requires translating extremely long context documents, it’s highly necessary to validate the quality of the multilingual prompts, and make sure they are as required. As far as I know, no existing model “faithfully” translates such long documents across so many low-resource languages (even high-resource, to some extent). Further, there are also multiple works which discuss how biases, or harms don’t survive or translate accurately across languages, so I do not buy this benchmark as of now.

The metrics used for translation quality (BERTScore-F1, COMET) are also model based and quite old, and have varying performance across languages and are not meant for long-context inputs. In fact, LLM-judges could have been a better stand-in, but they also require human-verification.

The 5 guards discussed in this paper seem to be developed for general safety, but do not claim “good” multilingual, or long-context abilities. In such a case, this study seems very contrived, and failures in safety are expected. So, I am not sure what kind of problem or takeaways I should have as a real-world user/developer, as the issue can be due to undertrained models which do not have the basic abilities or lack of safety.

Questions for authors

In Table-1, Bypass rate is not required. Block rate already gives it. The term “factorial comparison” is quite awkward and immediately shows the AI usage.

What exactly is meant by “downstream study” in Line 211? And why are they gold-labelled unsafe? This seems a bit vague.

Also, can you provide some justification for the chosen guards? Is there any specific reason to use/select them only?

As a control study, it would have been good to discuss the safety/guarding under 8K, at regular context lengths. So, this way, you have an ablation of just context length increase (from regular < 8K to long-context > 10K) causing unsafe behavior. Also, despite the claim of long-context analysis, only 3 main points (8K, 20K, 32K) seem to be studied. I think the claims, contributions should be better phrased to reflect it since points are quite distributed. A few more intermediate points would have helped.

Since the number of languages are so high (98), at least some language-family based analysis should have been provided.

Can you please elaborate on the agentic behaviours studied (what kind of tools)?

Maybe you can try train (SFT) the guardrails for better long-context reception (ex. Delta-Mem), and then evaluate their performance across languages now that they can better/more faithfully accommodate longer inputs.

Reviewer_e3CSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper evaluates whether multilingual harmful requests admitted by an input guard lead a downstream agent to select a restricted tool. It translates 1,901 unsafe English prompts into 98 languages, evaluates five guards, and studies three downstream models under varying context lengths and request positions using inert symbolic tools.

Critical strengths
  1. The study covers a broad range of languages and harm categories, offering a useful resource for multilingual safety analysis.
  2. Following guard-admitted requests into downstream tool selection is a reasonable evaluation direction. The matched comparisons help examine how context length and request position affect the same request.
Critical weaknesses
  1. Limited coverage of the defense landscape. The evaluated pipeline focuses on input filtering followed by an agent decision. It does not adequately represent defenses that combine input and output guards, runtime monitors, and execution-time authorization. The authors acknowledge some of these omissions, but their implications for the scope of the study need more attention. I would encourage the authors to use resources such as Promptfoo's LM Security Database [1] to identify relevant work and better situate the evaluation within broader agent-defense research.

  2. Unclear novelty and insufficient comparative evidence. I am not convinced that combining multilingual prompts, long contexts, and downstream tool selection provides enough new insight over existing safety evaluations. The paper cites Llama Guard, ShieldGemma, and AgentDojo, but does not empirically connect its findings to these guard models or dynamic agent-evaluation settings. The five-guard comparison and three downstream models provide a bounded diagnostic, but leave the additional value over prior approaches insufficiently established. Some downstream models, such as Llama-3.1-8B and Qwen2.5-14B, also limit how representative the findings are of current agent capabilities.

  3. Limited insight into adaptive robustness. My concern is that high detection rates on a fixed prompt collection can give an overly optimistic impression of protection while leaving adaptive and in-the-wild attacks untested. Increasing the number of translated prompts does not by itself address this gap. I would find an evaluation that connects guards to adaptive red teaming, or examines how defenses respond to newly discovered attacks over time, more informative. The current results do not establish how the evaluated guards behave against attackers who adapt to them.

  4. The dataset does not capture important contextual failure modes. The benchmark primarily translates existing unsafe requests and places them within benign English conversation history. I am not convinced that this construction establishes sufficient attack difficulty or realism. Agent failures can also arise through task context, tool outputs, and harness interactions, even when the immediate user request is benign. Internal Safety Collapse [2], for example, studies harmful generation under otherwise benign tasks with particular task and validation conditions. The present evaluation does not capture this class of failure, and selecting an inert restricted tool provides limited evidence about harmful execution in a realistic workflow.

Overall, I find the motivation insufficiently connected to current agent-security challenges, and the contribution too incremental. Despite the breadth of the multilingual data, the evaluation offers limited insight into realistic attacks or effective defenses. In its current form, I do not think the paper advances the field sufficiently to warrant acceptance.

[1] Promptfoo. LM Security Database. https://www.promptfoo.dev/lm-security-db/

[2] Yutao Wu et al. Internal Safety Collapse in Frontier Large Language Models. NeurIPS 2026. https://arxiv.org/abs/2603.23509

Questions for authors

no specific questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.