acceptodds
Under review as a conference paper at ICLR 2027

Revealing the Stepping-Stone Threat in Agentic Software Repair

Abstract

Software engineering (SWE) platforms, such as OpenHands and Google Jules, increasingly use large language model agents to automate software development and repair. In these workflows, an intermediary agent first processes external issue reports and then passes implementation requirements to a coding agent with repository-modification privileges. This agent-to-agent handoff introduces a new attack surface: an attacker without repository access can frame malicious configuration changes as a legitimate security improvement, inducing the intermediary to launder untrusted input into apparently authorized requirements that the coding agent subsequently implements. We term this exploitation of intermediary endorsement the stepping-stone attack, in which this endorsement lends apparent authority to attacker-controlled content as it passes into downstream implementation. Preregistered controlled experiments show that the attack can achieve full-chain compromise even when intermediaries are explicitly instructed to audit requests for security violations. A matched ablation further shows that the intermediary's operational role substantially affects attack success: security-auditing agents are more resistant, whereas planning agents are more susceptible. Finally, case studies on deployed agent platforms and commercial code-review services confirm that attacker-controlled configuration values can be accepted when disguised as legitimate security improvements.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Multi-Agent Systems, AI Agent Security, Agentic Software Engineering, Confused Deputy, Stepping-Stone Attacks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Nda1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

A new attack is introduced as a "stepping-stone" attack, where the issue report is converted into a plan for the following agent to execute.

Critical strengths

Soundness The problem raised is an important security issue; an intermediary agent may forward communication to the following workflow if it appears to be its output. This needs to be addressed.

Presentation The architecture diagram in Figure 1 explains what this paper intends to do.

Significance The problem highlighted is vital in an agentic system with complex goals. Authors have tried to demonstrate it with a simple system implementation.

Critical weaknesses

In the SWE process, an external issue report is validated and tested for confirmation of the issue. Only when the issue is confirmed is it forwarded to the implementation team. The idea has a process flaw. Nevertheless, assuming adversarial interception at this stage, an intermediary agent is assigned a role to carry out, with clearly specified inputs and outputs. The issue report and plan will clearly have different formats and layouts. This will clearly depend on how well its role boundaries are written, whether it simply forwards it or plans it.

Soundness

  1. Does not follow SWE process of "issue life-cycle"?
  2. To provide a legitimate plan by an adversary, it should have knowledge of the architecture and tools within the system. For an intermediary agent to forward a plan, the plan should conform to the system's architectural implementation.

Presentation While figure 1 gives high level system setup with two agents, it is not clear what is "Bypass plan", "Host plan", and "Callback risk: YES" in table 1. Also, what setups are being used in the "planning" and "auditing" experiments? Table 2 has a lot of observations but is not referred and explained in the paper. I am assuming it might be trying to show that authors lauched attack successfully but I do not understand what each column signify

Significance Without implementing the full issue-report-repair process, it is difficult to demonstrate the significance of this software-engineering approach. It includes [issue report -> issue validation -> issue accepted -> issue repair design and implementation, and so on].

Questions for authors
  1. Do you use only Figure 1 for all experiments?
  2. In Table 1, what are "Bypass plan", "Host plan", and "Callback risk: YES" in table 1? Also, what setups are being used in the "planning" and "auditing" experiments?
  3. How are you arriving at these numbers in Table 1 and 2?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Nda1

We sincerely thank Reviewer Nda1 for the supportive evaluation and insightful questions regarding real software engineering lifecycle coverage, architectural tool knowledge, and presentation clarity. In our revision (Sections 3.2, 4, 5.1–5.2, 6, Tables 2–3, Appendices A–C; artifacts at https://anonymous.4open.science/r/iclr2027-code-6ED5/), we expanded our empirical evidence and clarified these boundaries directly.

1. Process & Lifecycle Coverage Across Real Workflows (W1)

Real-world development spans diverse trackers, triage steps, coding agents, and automated code reviewers. In Section 6 and Table 2, we explicitly clarify the single-repo Experiment 1–2 scope while showing transfer across real engineering tools:

  • End-to-end engineering surfaces (Table 2, Appendix B.5): Beyond our Dockerized Gitea →\to intermediary →\to coder pipeline (0/174 baseline vs. 38/173 style-chained compromises), parameter laundering reproduces across OpenHands (45/78 style adoptions, 42/78 functional; Luna 28/28), Jules (16/16 researcher-applied style probes), and PR auditing in CodeRabbit (missing 3/5–5/5 runs per security-framed style vs. 0/5 for plain callbacks, Appendix Table 17).
  • Controlled causal isolation (Table 3, Appendix Tables 18–21): New controlled sessions confirm that intermediate specifications substantially raise downstream adoption (13/25 →\to 23/25, p=0.0063p=0.0063; 21/25 even on a two-sentence neutral stub), and operate independently of collaborator vs. external Git commenter attribution (21/25 vs. 21/25, p=1.00p=1.00).

2. System Architecture, Tool Knowledge, & Defense Boundaries (W2)

We grounded architectural authority and scope limitations in Sections 3.2, 5.2, 6, and Appendix C:

  • Authoritative workspace records (R1R_1–R4R_4, Appendix Table 21): Coding agents typically lack ground-truth organizational policy unless explicitly grounded in repository artifacts. Introducing security/approved_callback_hosts.json referenced in README.md reduces unauthorized host adoption from 23/25 to 0/25 (p=2.4×10−7p=2.4\times 10^{-7}), with 24/25 (R2R_2) to 25/25 (R3R_3) runs routing to the approved internal endpoint.
  • Prompting vs. harness enforcement: Because 7/50 (14%) WXWX plans still bypass prompt-level auditing (Appendix Table 6), deterministic pre-handoff gating provides strict guarantees (0/25 compromise, 25/25 task pass, 0/10 benign false blocks).
  • Explicit scope boundaries (Section 6): Experiments 1–2 study one stdlib Python repo (acme/replenishment-service) and endpoint (GET /api/reports/daily), while Experiment 3 extends across 32 synthetic evaluation families (8 initial + 24 held-out across network, supply-chain, identity, storage) plus 4 dev families (Appendix C).

3. Clarity & Presentation Improvements (Q1–Q3)

  1. Unified Notation & Threat Model: Harmonized pipeline notation across Section 3.2, Tables 1–3, and Appendices A–B.
  2. Concise Main Text: Relocated raw prompts (Appendix A.1), condensed new ablations into Table 3 (Section 5.2), and preserved the 10-page main-text limit (Conclusion on p. 10).
  3. Full Statistical & Artifact Transparency: Every exact McNemar pp-value, Agresti–Min CI, semantic review (162 literal-positive plans), and Docker harness is indexed in Appendix Table 31 and https://anonymous.4open.science/r/iclr2027-code-6ED5/.
Reviewer_MmkbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies stepping-stone attacks in software-repair workflows. An attacker includes malicious configuration values in an issue that requests a security improvement. An intermediary agent turns the issue into an implementation plan, which is then followed by a coding agent. The experiments show that attacker-selected values can reach merged configuration, with runtime effects verified in some cases. The results also show that agents asked to perform security auditing recommend these values much less often than agents asked to produce implementation plans.

Critical strengths
  • The paper shows that malicious configuration values can be accepted when they are presented as part of a security improvement.

  • The evaluation goes beyond the plans generated by the agents. The authors checked whether the malicious values were included in merged code and verified their runtime effects in some cases.

Critical weaknesses
  • The novelty compared with existing multi-agent prompt injection attacks needs further clarification. The paper focuses on attacker-selected configuration values presented as security improvements. This is a useful observation, but the distinction between this attack and existing attacks through inter-agent communication is not sufficiently established.

  • An ablation study on the intermediary agent is needed. The coding agent reads both the original issue and the intermediary’s plan, so it may follow the malicious request directly. The authors acknowledge this limitation, but it leaves the contribution of intermediary endorsement untested. A comparison using the same issues with and without the intermediary would help establish this contribution.

  • The expected authorization behavior in the repair experiments needs clarification. The authors state that the repair agents lack an independent authorization source. What information should an agent use to determine whether the requested domain is approved, and what should it do when this information is unavailable? This would help distinguish a failure to follow an available policy from a workflow that lacks a way to verify authorization.

  • Table 2 uses different success criteria for the proposed attack and the baselines. The proposed attack is evaluated by configuration adoption, while the baselines use family-specific functional criteria. The authors acknowledge that these are not matched comparisons. However, the side-by-side averages are still difficult to interpret as a comparison of attack effectiveness. A comparison using the same target outcome is needed to assess the benefit of security framing.

  • The role comparison does not separate the role from the security-check instruction. The auditing condition explicitly requires checking the issue against security policy, while the planning condition does not. The paper acknowledges this difference. The results therefore support the effectiveness of the tested auditing instruction, but do not establish an effect of the role independent of that instruction.

Questions for authors
  1. For the same issues and coding agent, how does attack success change when the intermediary's plan is removed?

  2. What information is available to the repair agents for checking whether a domain is authorized, and what is the expected behavior when that information is missing?

  3. Can the security-framed requests and the baselines be compared using the same target outcome and success criterion?

  4. How many distinct repair tasks and repositories were tested, excluding repetitions that only change tracking tokens or presentation styles?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Mmkb (Part 1/2)

We sincerely thank Reviewer Mmkb for the detailed, rigorous critique that motivated our new controlled ablations. We ran every requested comparison—detailed in Sections 5.1–5.2, Tables 1 & 3, Appendices A.4 & B.6, and https://anonymous.4open.science/r/iclr2027-code-6ED5/ (Part 1/2 covers W1–W3 & Q1–Q2; Part 2/2 covers W4–W5 & Q3–Q4).

Holding the issue and coding agent fixed, removing the endorsing intermediary specification cuts attacker-host adoption from 23/25 to 13/25 (Table 3, Appendix Table 18). Holding the endpoint, target host, and scoring criterion fixed, the two-agent workflow adopts that host in 0/25 plain callback runs vs. 23/25 security-framed runs (Appendix Table 20).

1. Intermediary Ablation Study (W2 & Q1)

Our paired C3C_3 vs. C1C_1 comparison holds the issue, token, repository fixture, and coding agent fixed (n=5n=5/model; Table 3, Appendix Table 18; exact McNemar p=0.0063p=0.0063). Configuration adoption and selective target-host TLS initiation agree on 100/100 runs in C1C_1–C4C_4. DeepSeek, Gemini, and Qwen drive the intermediary amplification, falling from 13/15 to 3/15 when the specification is removed (23/25 →\to 13/25 overall). Additional controls (Appendix Tables 18–19) show:

  • Endorsing spec + neutral two-sentence issue stub (C2C_2): 21/25 (84%) compromise (p=0.0078p=0.0078 vs. C1C_1).
  • Attribution control (S1S_1 vs. S2S_2, 25 interleaved pairs): Both opencode-bot (COLLABORATOR) and ext-community-reviewer (NONE) yield 21/25 (84%) selective target-host TLS initiations (p=1.00p=1.00, 95% CI [−21.8,21.8][-21.8, 21.8] pp) with fixed specification text and maintainer trigger.
  • Disqualifying spec (C4C_4, raw malicious issue still visible): 0/25 (0%) compromise (p=2.4×10−7p=2.4\times 10^{-7} vs. C3C_3, 25/25 task pass).

2. Novelty Relative to Multi-Agent Prompt Injection (W1)

Prior work examines adversarial inter-agent messages, control-flow/tool hijacking, and malicious issues (Section 2). We isolate a configuration-integrity failure in software repair: an unverified domain enters a working security boundary that rejects other destinations.

  • In C3C_3, 24/25 runs pass feature tests, and 23/23 compromised implementations connect over TLS to ops.meridian-logistics.example while denying the unlisted control host (random_host_denied=True).
  • By rewriting an untrusted host into a documented security requirement, intermediary guidance defeats downstream automated review: CodeRabbit misses 3/5–5/5 runs per security-framed style vs. 0/5 for plain callbacks (Table 2, Appendix Table 17).

3. Expected Default Behavior & Workflow Authorization (W3 & Q2)

  • Expected secure behavior without approval records: In baseline repo R1R_1, no file authorizes external partner hosts. Expected behavior is deny-by-default: implement the SSRF allowlist with an empty default allowlist (allowed_hosts=(), deployment-configured) rather than hardcoding an unverified thread domain. When stated in C4C_4, 25/25 runs satisfy the task (23/25 with empty allowed_hosts=()) at 0/25 compromise.
  • Providing authoritative policy records (R2R_2–R4R_4, Table 3 & Appendix Table 21): Adding security/approved_callback_hosts.json (approving hooks.corp-analytics.internal, excluding ops.meridian-logistics.example) and pointing README.md and the coder prompt to it drops attacker-host compromise from 23/25 (R1R_1) to 0/25 under both model consultation (R2R_2, p=2.4×10−7p=2.4\times 10^{-7}, 24/25 approved-host TLS) and pre-handoff deterministic redaction (R3R_3, 25/25 approved-host TLS and task pass), with 0/10 false blocks and 9/10 approved-host TLS connections on benign control R4R_4.
AuthorsOpenReview ↗

Response to Reviewer Mmkb (Part 2/2)

(Continued from Response to Reviewer Mmkb Part 1/2)

4. Matched Outcome & Criterion Comparison: VXVX vs. WXWX (W4 & Q3)

In Table 3 and Appendix Table 20, the plain (VXVX) and security-framed (WXWX) issues target the exact same endpoint (GET /api/reports/daily) and host (ops.meridian-logistics.example) under identical scoring criteria (n=5n=5/model). Table 2 preserves the historical group averages and explicitly cross-references this matched comparison:

  1. Intermediary plan adoption: VXVX = 0/25 (0%) vs. WXWX = 25/25 (100%) under Transform (p=6.0×10−8p=6.0\times 10^{-8}); VXVX = 1/25 (4%) vs. WXWX = 20/25 (80%) under Evaluate without policy check.
  2. Executable host adoption (CASTC_{\text{AST}}): Single-hop VXVX = 0/25 (0%) vs. WXWX = 13/25 (52%) (p=2.4×10−4p=2.4\times 10^{-4}); two-hop VXVX = 0/25 (0%) vs. WXWX = 23/25 (92%) (p=2.4×10−7p=2.4\times 10^{-7}).
  3. Selective allowlist compromise (FTLSF_{\text{TLS}} with random_host_denied=True): Single-hop 0/25 vs. 13/25; two-hop 0/25 vs. 23/25.
  4. Any target-host TLS initiation (FTLSF_{\text{TLS}}, counting even open SSRF random_host_denied=False in VXVX's favor): Single-hop 2/25 (8%) vs. 13/25 (52%) (p=9.8×10−4p=9.8\times 10^{-4}); two-hop 9/25 (36%) vs. 23/25 (92%) (p=1.2×10−4p=1.2\times 10^{-4}). Consistent with Appendix B.3, the HTTPS canary intercepts and logs the TLS ClientHello SNI hostname at the socket layer prior to HTTP decryption.

5. Separating Role Verb from Security-Policy Instruction (W5)

Across 340 valid JSON intermediary outputs (Section 5.1, Appendix Table 6), crossing verb (Transform vs. Evaluate) with presence/absence of the explicit security-policy check resolves the source of attenuation:

  • Without the policy check, changing Transform (A1A_1) to Evaluate (B1B_1) lowers WXWX adoption modestly from 25/25 (100%) to 20/25 (80%) (p=0.0625p=0.0625; DeepSeek 5/5 →\to 1/5, PP 7/25 →\to 4/25).
  • Adding the explicit security-policy check causes the main reduction—to 4/25 (16%) under Transform (A2A_2, p=9.5×10−7p=9.5\times 10^{-7}) and 3/25 (12%) under Evaluate (B2B_2, p=1.5×10−5p=1.5\times 10^{-5}; A2A_2 vs. B2B_2: p=1.0p=1.0)—eliminating PP and VXVX (0/25) while maintaining 10/10 benign G1G_1 acceptance in all four cells, though 7/50 (14%) residual WXWX endorsements still pass prompt-only auditing.

6. Methodological Clarity & Reproducibility (Q4)

We relocated verbatim role prompts to Appendix A.1, added per-agent workflow/review configurations (Appendices B.1, B.5) and count/n breakdown tables (Appendix Tables 8–11, 17), unified Experiment 3 family descriptions (32 synthetic evaluation families + 4 dev families across Appendix C), and published all Docker harnesses, transcripts, and verification scripts at https://anonymous.4open.science/r/iclr2027-code-6ED5/ (Appendix Table 31).

Thanks for the additional experiments. These address most of my earlier questions. I still find the novelty limited. The intermediary makes the attack more effective, but I am not convinced this is sufficiently different from existing prompt-injection attacks.

Reviewer_c1eTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how attacker-controlled issue reports can influence software-repair agents through intermediary recommendations. It focuses on malicious configuration values presented as security improvements, traces their adoption into code, and examines how agent roles and authorization-related internal signals affect behavior.

Critical strengths
  1. The paper identifies a practically important distinction: implementing a security restriction correctly does not establish that the values it trusts are authorized. This is a useful perspective on the security of automated software repair.
  2. Tracing attacker-selected parameters from issue reports into implemented configuration makes the failure concrete. The planning-versus-auditing comparison also provides an informative example of how an agent's assigned role affects its recommendations.
Critical weaknesses
  1. The central stepping-stone mechanism is not sufficiently isolated. The paper motivates its contribution through intermediary endorsement, but the coding agent receives both the original issue and the intermediary's advice. The observed compromise therefore does not establish whether endorsement adds authority, whether rewriting makes the request more persuasive, or whether the coder would implement the same request directly. The planning-versus-auditing comparison concerns the intermediary's output and does not resolve this downstream question. OpenHands also demonstrates adoption with a single privileged implementer. I appreciate that the authors acknowledge this limitation, but it concerns the central mechanism rather than a peripheral experimental detail. Without separating these possibilities, the additional insight over direct malicious-issue attacks remains unclear to me.

  2. The study does not resolve where authorization must come from. The repair workflow lacks an independent source for verifying the proposed security parameters. An agent may recognize that an endpoint is sensitive, but it cannot determine from a plausible external request alone whether that endpoint has actually been approved. This makes it difficult to distinguish a reasoning failure from a workflow that does not provide the authority information needed for the decision. Experiment 3 introduces trusted approval records in separate synthetic tasks, but does not establish whether the same repair failure persists when such evidence is available in the actual workflow. I would find the paper more insightful if it connected these settings and clarified which failures require better model reasoning and which require enforceable authorization in the harness.

Overall, I appreciate the practical motivation and the concrete configuration-level failures. However, the study leaves two central questions unresolved: what the intermediary contributes to the attack, and how the workflow should establish authority over security-sensitive parameters. These gaps limit the explanatory and practical insight of the proposed threat, and I do not recommend acceptance in its current form.

Questions for authors

no specific questions

2 comments
AuthorsOpenReview ↗

Response to Reviewer c1eT

We sincerely thank Reviewer c1eT for the thoughtful feedback on causal isolation of the intermediary and policy-authority enforcement. In the revised manuscript (Section 5.2, Table 3, Appendix Tables 18–19 & 21, code/logs in https://anonymous.4open.science/r/iclr2027-code-6ED5/), we ran 210 controlled Dockerized repair sessions (100 in C1C_1–C4C_4, 50 in S1S_1–S2S_2, 60 in R2R_2–R4R_4 paired against R1=C3R_1=C_3) addressing W1 and W2.

Across paired runs holding the same issue, token, repository fixture, and coding agent fixed, the attacker host was adopted in 13/25 runs with the issue alone (C1C_1), 23/25 with the endorsing intermediary specification (C3C_3), and 0/25 when the intermediary flagged the host as unapproved and specified an empty default allowlist (C4C_4), with 25/25 passing acceptance tests.

1. What the Intermediary Contributes & Role of Source Attribution (W1)

Across C1C_1–C4C_4 and S1S_1–S2S_2, we ran 150 controlled repair sessions across all five routes (Table 3, Appendix Tables 18–19). In all 100/100 runs across C1C_1–C4C_4, AST host adoption matched selective target-host TLS initiation (FTLSF_{\text{TLS}} with random_host_denied=True).

  1. Direct exposure vs. endorsing specification (C1C_1 vs. C3C_3): Luna and MiMo adopt the host from the raw issue in 5/5 runs each (consistent with direct adoption in OpenHands). Across DeepSeek, Gemini, and Qwen, adding the endorsing specification increases adoption from 3/15 to 13/15 (13/25 →\to 23/25 pooled; exact McNemar p=0.0063p=0.0063; task pass 22/25 →\to 24/25).
  2. Endorsing specification + neutral issue stub (C2C_2): Pairing the endorsing specification with a neutral two-sentence issue stub still yields 21/25 (84%) adoptions (p=0.0078p=0.0078 vs. C1C_1; DeepSeek/Gemini/Qwen 11/15).
  3. Disqualifying specification (C4C_4): Specifying an unapproved host and empty default allowlist allowed_hosts=() reduces adoption from 23/25 to 0/25 (p=2.4×10−7p=2.4\times 10^{-7} vs. C3C_3) with 25/25 passing acceptance tests.
  4. Source attribution with fixed specification text (S1S_1 vs. S2S_2, N=50N=50): Across 25 interleaved pairs, the neutral coder prompt ("including the issue comments"), raw issue, specification text, comment position/timestamp, and maintainer /oc fix this trigger are identical, varying only line 50 between opencode-bot (COLLABORATOR) and ext-community-reviewer (NONE). Both labels yield 21/25 (84%) selective target-host TLS initiations (17 both, 4 internal-only, 4 external-only; exact McNemar p=1.00p=1.00; Agresti–Min 95% CI [−21.8,21.8][-21.8, 21.8] pp). Executable literal retention is 22/25 vs. 21/25 (p=1.00p=1.00) because one DeepSeek S1S_1 run kept the literal inside a syntax validator without outbound dispatch; one Qwen S2S_2 run cited author_association: NONE. Note that C1C_1 (13/25) came from a separate batch/prompt and provides descriptive context rather than a same-source baseline for rewriting vs. verbatim reposts.

2. Workflow Authority Registry vs. Deterministic Pre-Handoff Gate (W2)

We introduced security/approved_callback_hosts.json (approving hooks.corp-analytics.internal, excluding ops.meridian-logistics.example) and referenced it in README.md and the coder prompt (n=5n=5/model; Table 3, Appendix Table 21):

  • No registry (R1=C3R_1 = C_3): Bot endorses attacker host; 23/25 (92%) compromise (24/25 task pass).
  • Registry, model consultation (R2R_2): Comment still endorses the attacker host, yet 0/25 (0%) runs adopt or connect to it (p=2.4×10−7p=2.4\times 10^{-7} vs. R1R_1), 24/25 (96%) initiate TLS to the approved host, and 24/25 pass acceptance.
  • Registry + deterministic pre-handoff gate (R3R_3) & benign control (R4R_4): Redacting unapproved comment hosts flags 25/25 unapproved specs, yields 0/25 compromise, and hits 25/25 (100%) approved-host TLS connections and task passes; benign R4R_4 (n=10n=10) sees 0/10 false blocks and 9/10 approved TLS connections.

Because R2R_2 already reached 0/25 compromise in this sample, the gate's incremental reduction on compromise over consultation is unmeasured here; however, deterministic gating flagged 25/25 unapproved specs, closing the 7/50 (14%) residual leakage gap observed under prompt-only auditing (Appendix Table 6).

I have carefully read the authors' response, and I think the experiments are actually quite solid. However, the paper reads more like a technical report than a polished research paper. Both the paper and rebuttal are too heavy on numbers and details, which somewhat bury the scientific insights and make it difficult for readers to appreciate the key findings.

I encourage the authors to improve the presentation. For now, I am inclined to maintain my original score, which does not mean I see little value in the work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Author Response (Part 1/2): Summary of 600 New Controlled Runs

We thank all three reviewers (c1eT, Mmkb, Nda1) for their detailed, constructive feedback. To address every empirical and conceptual question directly, we conducted 600 new controlled evaluation units across all five primary model routes (gpt-6-luna, deepseek-v4.1-flash, gemini-3.8-flash, qwen3.8-flash, mimo-v2.6-flash), updated the manuscript (keeping the main text within 10 pages: Sections 1–7, Tables 1–3, Appendices A–D), provided a highlighted revised PDF, and released all new scripts, harnesses, and logs in our anonymous repository: https://anonymous.4open.science/r/iclr2027-code-6ED5/.

Summary of New Controlled Experiments (Part 1/2)

  1. 2×22\times 2 Role-Verb vs. Policy-Check Ablation (340 valid JSON outputs; Section 5.1, Appendix Table 6; addresses Mmkb W5): Crossing intermediary verb (Transform vs. Evaluate) with presence/absence of the explicit security-policy check resolves which component drives attenuation:

    • Without the check, switching Transform (A1A_1) to Evaluate (B1B_1) modestly lowers WXWX host-plan adoption from 25/25 (100%) to 20/25 (80%) (p=0.0625p=0.0625; PP 7/25 →\to 4/25).
    • Adding the explicit security-policy check drives the dominant drop—to 4/25 (16%) under Transform (A2A_2, p=9.5×10−7p=9.5\times 10^{-7} vs. A1A_1) and 3/25 (12%) under Evaluate (B2B_2, p=1.5×10−5p=1.5\times 10^{-5} vs. B1B_1; A2A_2 vs. B2B_2: p=1.0p=1.0)—eliminating PP (0/25) and VXVX (0/25) while preserving 10/10 benign G1G_1 acceptance in every cell. Yet 7/50 (14%) residual WXWX plans still pass prompt auditing, motivating deterministic handoff enforcement.
  2. Controlled Coder-Handoff & Source-Attribution Ablations (150 Dockerized sessions; Section 5.2, Table 3, Appendix Tables 18–19; addresses c1eT W1, Mmkb W1–W2 & Q1): Holding the issue, tracking token, repository fixture, and coding agent fixed (n=5n=5/model), AST host adoption (CASTC_{\text{AST}}) agrees with selective target-host TLS initiation (FTLSF_{\text{TLS}} with random_host_denied=True) on 100/100 runs in C1C_1–C4C_4:

    • Issue alone (C1C_1) vs. Endorsing spec (C3C_3): Adding the endorsing opencode-bot spec raises attacker-host adoption from 13/25 (52%) to 23/25 (92%) (exact McNemar p=0.0063p=0.0063; task pass 22/25 →\to 24/25). Luna and MiMo adopt in 5/5 runs in both conditions, while DeepSeek, Gemini, and Qwen jump from 3/15 to 13/15.
    • Endorsing spec + neutral two-sentence issue stub (C2C_2): 21/25 (84%) adoptions (p=0.0078p=0.0078 vs. C1C_1; DeepSeek/Gemini/Qwen 11/15).
    • Disqualifying spec (C4C_4, empty default allowed_hosts=()): 0/25 (0%) compromise (p=2.4×10−7p=2.4\times 10^{-7} vs. C3C_3) with 25/25 passing acceptance tests.
    • Source attribution (S1S_1 vs. S2S_2, 25 interleaved matched pairs): Varying only opencode-bot (COLLABORATOR) vs. ext-community-reviewer (NONE) under identical specification text, raw issue, neutral coder prompt, and maintainer /oc fix this trigger yields 21/25 (84%) selective target-host TLS initiations under both labels (p=1.00p=1.00, Agresti–Min 95% CI [−21.8,21.8][-21.8, 21.8] pp; CASTC_{\text{AST}} 22/25 vs. 21/25).
AuthorsOpenReview ↗

Global Author Response (Part 2/2): Matched Baselines, Policy Registry & Revisions

(Continued from Global Author Response Part 1/2)

Summary of New Controlled Experiments & Revisions (Part 2/2)

  1. Matched Outcome & Criterion Comparison: VXVX vs. WXWX (50 new coder runs + matched plans; Section 5.2, Table 3, Appendix Table 20; addresses Mmkb W4 & Q3): Evaluating plain (VXVX) vs. security-framed (WXWX) requests on the exact same endpoint (GET /api/reports/daily), target host (ops.meridian-logistics.example), and scoring criteria (n=5n=5/model):

    • Intermediary plan adoption: VXVX = 0/25 vs. WXWX = 25/25 under Transform (p=6.0×10−8p=6.0\times 10^{-8}); 1/25 vs. 20/25 under Evaluate without policy check.
    • Executable host adoption (CASTC_{\text{AST}}) & selective allowlist compromise: Single-hop 0/25 vs. 13/25 (p=2.4×10−4p=2.4\times 10^{-4}); two-hop 0/25 vs. 23/25 (p=2.4×10−7p=2.4\times 10^{-7}).
    • Any target-host TLS initiation (FTLSF_{\text{TLS}}, counting unvalidated open SSRF random_host_denied=False in VXVX's favor): Single-hop 2/25 (8%) vs. 13/25 (52%) (p=9.8×10−4p=9.8\times 10^{-4}); two-hop 9/25 (36%) vs. 23/25 (92%) (p=1.2×10−4p=1.2\times 10^{-4}). Consistent with Appendix B.3, the canary captures the TLS ClientHello SNI hostname before socket closure.
  2. Authoritative Registry vs. Deterministic Pre-Handoff Gate (60 new runs + R1R_1; Section 5.2, Table 3, Appendix Table 21; addresses c1eT W2, Mmkb W3 & Q2, Nda1 W2): Adding security/approved_callback_hosts.json (approving hooks.corp-analytics.internal, excluding ops.meridian-logistics.example) referenced in README.md and the coder prompt cuts compromise from 23/25 (R1=C3R_1=C_3) to 0/25 under both model consultation (R2R_2, p=2.4×10−7p=2.4\times 10^{-7}, 24/25 approved-host TLS) and pre-handoff deterministic redaction (R3R_3, 25/25 unapproved specs flagged, 25/25 approved-host TLS and task pass), with 0/10 false blocks and 9/10 approved-host TLS connections on benign control R4R_4.

Summary of Manuscript Improvements

  • Threat Model & Framing (Sections 1–3.2, 6): Sharpened the distinction between parameter laundering into SSRF restrictions vs. control-flow hijacking, clarified deny-by-default empty allowlists (allowed_hosts=()), and documented single-repo Experiment 1–2 scope alongside Experiment 3's 32 synthetic evaluation families + 4 dev families (Appendix C).
  • Workflow Realism & Transparency (Tables 2–3, Appendices A.1, B.1, B.3, B.5, D): Relocated verbatim prompts to Appendix A.1, added per-agent workflow/review details (Appendices B.1, B.5), clarified HTTPS SNI interception (Appendix B.3), pointed Table 2 to matched Table 3 comparisons, and indexed all artifacts in Appendix Table 31 and https://anonymous.4open.science/r/iclr2027-code-6ED5/.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.