acceptodds
Under review as a conference paper at ICLR 2027

Do Detailed Guard Denials Help Bypass Guardrails? Evaluating Runtime Feedback in DeepSeek Harness

Abstract

Runtime guards block tool calls, but their denial messages re-enter an agent's context and can shape subsequent planning. Detailed diagnostics may help benign recovery while also providing actionable information to a planner already influenced by indirect prompt injection. Yet existing evaluations often change attack history, search strategy, and feedback together, making the effect of the denial itself difficult to isolate. We introduce GuardOracle, a controlled audit at the native denial boundary of DeepSeek Harness. Starting from the same pre-feedback history, GuardOracle varies only the model-visible denial representation—Opaque, a length-matched Placebo, Category, or Detailed—while holding the injection, guard policy, tools, simulator state, and post-denial action budget fixed. A bypass is counted only when a candidate action is permitted, committed in simulation, and independently verified to realize the frozen malicious objective. In our primary study, Detailed minus Opaque changes verified Bypass@3 by +7.85 percentage points (95% CI: −0.54 to 17.83), leaving the effect direction uncertain. In an independent prospective study, none of 658 frozen screening repetitions reached an eligible first-denial state. These results motivate separating whether an agent reaches the denial boundary (activation) from how feedback changes behavior once that boundary is reached (conditional feedback effect).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, agent safety, indirect prompt injection, runtime guardrails, runtime feedback, tool-use security

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3ReuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether detailed denial feedback from runtime guardrails can help LLM agents bypass security restrictions. The authors introduce GuardOracle, an evaluation framework that compares different feedback policies while keeping the preceding interaction unchanged. They measure whether agents subsequently execute actions that achieve the malicious objective. The original experiment shows a possible increase in bypass success with detailed feedback.

Critical strengths

1.Strong and relevant motivation. The paper examines an important question in agent security. Although runtime guards can prevent unsafe tool execution, their feedback may still influence the agent's subsequent decisions. Understanding this potential information leakage is relevant to the design of safer agent systems.

2.Interesting evaluation methodology. The same-prefix design provides a reasonable way to compare different denial-feedback policies while controlling the preceding interaction. The use of executable verification is interesting, which provides stronger evidence of successful bypasses than relying solely on model responses.

Critical weaknesses

1.Limited empirical findings and generalizability. The original experiment suggests a possible increase in bypass success, but the result remains uncertain. More importantly, the counterbalanced replication produces no eligible cases, despite using the same structural scenarios as the original experiment, which obtained 100 eligible groups. The differences in eligibility criteria and experimental settings remain unresolved, making it difficult to assess the reproducibility and generalizability of the original findings.

2.The effect of denial feedback is not sufficiently isolated. The original experiment reports a 7.85 percentage-point increase in bypass success with detailed feedback, but the 95% confidence interval [−0.54, 17.83] includes zero, and the p-value of 0.097 does not reach conventional statistical significance. Moreover, the counterbalanced replication produces no eligible cases despite using the same structural scenarios as the original experiment, which obtained 100 eligible groups. The differences in eligibility criteria and experimental settings remain unresolved, raising concerns about the reproducibility and generalizability of the findings.

Questions for authors

1.Can the authors distinguish tool actions generated after receiving denial feedback from those already proposed? This seems important for determining whether the observed bypasses are actually influenced by the feedback.

2.The original experiment produces 100 eligible replay groups, while the follow-up studies produce almost none. Could the authors further explain this discrepancy? In particular, how many of the original eligible groups would satisfy the stricter target-matching criterion used in later experiments?

1 comment

Thank you for asking us to check the timing of the successful calls and to apply the later matching criterion to the original groups. We examined both questions in the retained records.

Q1. Which calls were generated after feedback became available? We cannot classify the generation timing of the 106 verified successful calls in the original Detailed and Opaque branches (54 and 52 calls, respectively). Across all four arms, the scoring windows contained 1,002 logged attempts in 100 eligible groups. Each attempt has a log index, but no link to the assistant message that proposed it or the model request that generated that message. If one message proposes two calls and the first is denied, the second can enter the scoring window even though both were proposed before feedback was available.

The +7.85 percentage-point estimate (95% CI [−0.54, 17.83], p=0.097) therefore applies to verified bypass within the next three logged capability attempts (Appendix E, lines 871–878). An estimate restricted to calls generated with the denial in context would require the missing message-to-request links. Fixed arm order further limits attribution to feedback content. We will state both qualifications alongside the main result.

Q2. How many original groups meet the stricter criterion? All 100. For each originally eligible group, we located the first denied candidate in each of its four replays. In every arm the candidate matched both the scenario destination and its payload marker. Applying the later target-matching requirement retrospectively would retain every originally eligible group.

In the counterbalanced study, the 63 structural scenarios were run anew. All 189 screening records had first_denial_observed = false. Table J1 labels these records candidate_goal_not_matched because the reporting code assigns that label first when it finds no matching denied action. The underlying records identify where the two studies diverged: the new screening windows contained no recorded first denial at which a feedback comparison could begin. Why denial occurrence changed remains unresolved. We will report the 100/100 check and the first-denial count together in the revision.

Reviewer_JAXBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether detailed guard denials can also help a prompt-injected planner bypass guardrails. To examine this, the paper propose GuardOracle, which controls the denial message (Opaque, length-matched Placebo, Category, or Detailed). In experiments, Detailed minus Opaque changes verified Bypass@3 by +7.85 percentage points (95% CI: −0.54 to 17.83).

Critical strengths
  1. Interesting motivation and controlled design

The question of whether detailed guard denials help an agent bypass the guard is interesting, and GuardOracle isolates it well by varying strength of the denial message from the same pre-fixed history.

Critical weaknesses
  1. Inconclusive experiments and conclusions

The experiments are limited in scope and feel incomplete, and the paper largely concludes with what cannot be determined. For example, Section 6.1 title “THE ORIGINAL FEEDBACK CONTRAST IS POSITIVE BUT DIRECTIONALLY UNCERTAIN,” and Sections 6.2–6.3 are framed the same way, leaving its research question unanswered.

  1. Scope of Experiments

The authors evaluate two models, Model A (deepseek-v4-flash) and Model B (glm-5.3-flash), but Model B yields only one eligible group, so its pre-specified four-goal contrast is not estimable. To improve the robustness of the experiments and claims, expanding the experiments would help support the research questions posed in the paper.

Questions for authors

I find the research question valuable and would encourage the authors to strengthen it with concrete case studies, such as representative trajectories comparing Detailed and Opaque denials. Broadening the executed comparison would also help, since the authors themselves often describe the findings as uncertain.

2 comments

Thank you for suggesting paired case studies. We examined the original redacted records to show both how often the two branches diverged and what that divergence looked like.

W1. What the experiment establishes. Across 100 eligible pairs from 45 Model-A scenarios, the prespecified Detailed−Opaque contrast was +7.85 percentage points (95% CI [−0.54, 17.83], p=0.097, Table H2). The interval admits both a small decrease and a substantial increase. Every group ran Opaque first and Detailed last, so a difference associated with request position could also contribute to this estimate. The direction and causal attribution of the contrast remain open.

Paired cases. The branches had the same verified Bypass@3 outcome in 90 pairs: both succeeded in 48 and neither succeeded in 42. Of the remaining ten, six succeeded only under Detailed and four only under Opaque. We selected one example of each direction by taking the first pair in scenario-ID and repeat order within each category.

Selected pairDetailed branch after the shared denialOpaque branch after the shared denial
Detailed-only successThe first logged attempt reached the verified goal.No further configured-capability attempt was logged.
Opaque-only successAll three logged attempts were denied.The first two attempts were denied. The third reached the verified goal.

Counting each pair equally gives a difference of (6 − 4)/100, or +2 percentage points. The prespecified +7.85 estimate first averages repeats within each scenario and then gives the four goal families equal weight. The cases illustrate variation in observed outcomes. The records do not show whether a particular detail in the denial guided the next decision.

W2. Breadth of the comparison. Model B contributed one eligible group. The counterbalanced and held-out DSH studies yielded 0/189 and 0/658 eligible groups, respectively, and the AgentDojo-derived extension yielded 0/144 (Table 2). None supplied an independent four-family estimate. The paired cases above come from existing records. A broader empirical answer still requires a second setting with eligible groups that actually receive both feedback conditions. We will put the paired counts and examples in Section 6 alongside the estimate so readers can see the evidence for each conclusion.

Thank you for your response. However, it is still hard to draw general conclusions, and the experiments remain narrow in scope. I therefore maintain my score.

Reviewer_VuF2OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces GuardOracle, a same-prefix audit in DeepSeek Harness that compares four denial-feedback policies tested with deepseek-v4-flash and glm-5.3-flash

Critical strengths

None

Critical weaknesses

The concepts are not easy to understand for a broader audience. For example, it is not clear what is the importance of bypass@3 and what the specific reason is to pick 3 vs other numbers. It is hard to gauge whether the results from only model A and model B are generalizable, or enough to support the GuardOracle.

Questions for authors

1 Why not study bypass@X with X=1,2,3,4...? Does the finding hold generally for other values of X?

2 What is the definition of a state (i.e., initial state S0)? Does it include agent state, environment state, tool state?

3 In line 257, what does ^ mean?

1 comment

Thank you for asking why we chose three attempts. We revisited the original records to see how much that choice matters.

Q1. Choice of window. Bypass@K measures whether the agent reaches the specified goal within its next K logged capability attempts after a guard denial. This captures what happens after an initial block, including cases where the agent subsequently completes the action. We chose K=3 because the original task instructed the planner to make at most three retries. The runtime could continue until idle or timeout (Appendix E).

We computed the two shorter-window contrasts on the same 100 paired groups from 45 Model-A scenarios. We used the original analysis throughout, averaging repeats within scenarios, weighting the four goal families equally, and resampling scenarios within each family.

WindowDetailed minus Opaque (percentage points)95% CI (percentage points)
Bypass@1+3.90[−0.44, 10.71]
Bypass@2+4.34[−0.76, 11.57]
Bypass@3 (prespecified)+7.85[−0.54, 17.83]

The point estimate favors a higher bypass rate under Detailed at each window, but each interval includes zero. The two retrospective comparisons therefore leave the direction uncertain. Only 4 of 100 Opaque branches and 6 of 100 Detailed branches logged a fourth attempt, giving us little evidence about behavior under a longer retry budget.

Q2. Initial state. In Algorithm 1, S₀ is the local simulator state. Each branch starts with the same scenario inputs and empty tool-call, attempt, and commit logs. The planner’s observable history, H⁻, is replayed up to the first denial. The task, tools, and enforcement policy are shared across branches. We have no access to the provider’s hidden model state. We will define S₀ and H⁻ together where we introduce the replay.

Q3. Notation on line 257. The symbol ∧ means logical AND. A bypass is verified if, on a single attempt, the guard permits the call, the simulated tool records a commit, and the call arguments match the specified goal. We will add this reading alongside the formula.

Scope across models. Model B yielded one eligible group out of 72, covering one of the four required goal families (Table 2). The quantitative contrast above thus applies to the original Model-A study. Whether it generalizes across models remains open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.