acceptodds
Under review as a conference paper at ICLR 2027

Securing Agentic Code Generation with Intent Code Graph Learning

Abstract

Large language model (LLM) agents have shown strong potential in automating real-world software engineering tasks. However, their deployment introduces new security risks, particularly their susceptibility to adversarial attacks that can manipulate agents into generating vulnerable or malicious code. Detecting such adversarially induced behavior remains a significant challenge. Existing detection approaches include guardrail model-based methods and static analysis-based methods. Guardrail models struggle in long-context settings due to degraded reasoning, while static analyzers depend on predefined security checks whose coverage may not extend to obfuscated or task-dependent malicious behavior. To address these limitations, we propose \tool, a graph-based detector for adversarially manipulated prompt-and-patch pairs. Inspired by semantic alignment between natural language and code, \tool first decomposes prompts into structured intent blocks and generated code into code blocks, enabling fine-grained modeling in long-context settings. It then treats these blocks as nodes and constructs an intent-code graph that captures their semantic correspondences and structural relationships. A graph neural network learns from semantic node representations and their connectivity using benign and synthetically augmented graphs, without training on examples from the evaluated attack algorithms. Experiments across diverse benchmarks, attack methods, and agent-LLM configurations show that \tool generally outperforms the evaluated baselines under attacks with code obfuscation, while static checkers' performance varies substantially across settings despite their advantage on non-obfuscated runs. These results support structural misalignment as a promising detection signal beyond predefined vulnerability checks. Our implementation and generated prompt-and-patch data are publicly available.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Secure Code Generation, LLM Agent

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CDwvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SIGIL, a graph-based classifier for detecting adversarially manipulated prompt-and-patch pairs by software-development agents. SIGIL augments benign examples with synthetic adversarial instructions and code, constructs intent-code graphs, and trains a detector on their semantic and structural features. Experiments report accuracy and F1 scores across three agents on SWE-bench Lite and LiveCodeBench.

Critical strengths
  • The paper clearly describes the threat model

  • The main pipeline is easy to follow

Critical weaknesses
  • Incremental methodology

The key insight is that an injected instruction and its implementation may align locally while remaining weakly integrated with the legitimate task. This insight have been widely explored in CHABADA/WHYPER-style app analysis as well as Task Shield. The relationship between SIGIL and these prior work needs to be delineated more clearly.

In addition, the intent-code graph is a potentially useful, but the current evaluation does not isolate its contribution. The auxiliary structural analysis shows associations between graph features and adversarial labels, but it cannot establish that graph learning is responsible for the detector for the detector's performance. More targeted ablations are need, such as a classifier on pooled CodeBERT embeddings, removal of individual edge types, shuffled or random edges, or removal of suspiciousness metadata.

  • Insufficient experimental evidence

First, the paper reports accuracy and F1 without providing the numbers of benign and adversarial examples in each evaluation settings. Therefore, it is hard to interpret these accuracies and F1 scores. For example, accuracy can be dominated by the majority class, while F1 remains sensitive to class prevalence and small numbers of positive examples.

Second, the labeling progress is unclear. Positive labels are assigned manually to patches that ``meaningfully attempt the injected objective'', while the number of annotators, inter-annotator agreement, or whether annotators were blinded to detector outputs, are all unknown.

Third, attack-generated pairs that do not meet the criterion are excluded. Consequently, the evaluation measures detection conditional on a retained implementation attempt rather than performance over the full distribution of attack-generated outputs. The authors should report how any examples are excluded and clarify how this filtering affects the interpretation of the results.

Fourth, the main experiments focus on VWE-78, while the synthetic-data prompts explicitly specify CWE-78 and CWE-95 operations, including subprocess.run, os.system, eval, and exec. Therefore, the detector can receive supervision containing strong priors about the vulnerability patterns evaluated in the main experiments. RQ3 considers three additional CWE types, but the evaluation is tiny. Consequently, the generalization of SIGIL remain unclear.

Fifth, while the threat model states the attacker has no knowledge of the deployed defense, an adaptive evaluation, following the evaluation principles discussed in ``On Adaptive Attacks to Adversarial Example Defense'', should be considered to help assess whether a stronger attack can evade SIGIL.

  • AI-use, ethics and reproducibility statements are inconsistent with the paper.

The statements on page 10 discuss factor generation, program revision, and operator construction,'' public historical stock-market observations,'' backtest performance,'' factor-selection rules,'', ``portfolio settings,'' do not correspond to SIGIL's method or experiments. Even worse, the reproducibility statement refers to sections that do not exist (search procedure, archive rules). These look like copy-paste from another submission.

Questions for authors
  • What is the contribution of the intent-code graph to SIGIL's detection performance?

  • How well does SIGIL generalize to CWE types not represented in its synthetic training data?

  • Why do the AI-use, ethics and reproducibility statements describe research unrelated to SIGIL?

3 comments
AuthorsOpenReview ↗

[1/3] Response to Reviewer CDwv

Thank you for the feedback on SIGIL’s methodology, evaluation, and robustness.

W1. Methodological Novelty and Relationship to Prior Work

SIGIL builds upon existing research in intent–behavior alignment and graph-based vulnerability detection. Its methodological contribution lies in a task-conditioned heterogeneous graph that jointly represents natural-language requirements, generated code, and their semantic and structural relationships to detect adversarially induced implementations.

CHABADA and WHYPER: CHABADA (Gorla et al., 2014) uses topic modeling to group Android applications by their descriptions, then applies a one-class SVM to identify unusual API usage relative to similar applications. WHYPER (Pandita et al., 2013) uses dependency parsing and semantic representations to determine whether application descriptions justify requested permissions. While both establish relationships between intended functionality and program behavior, SIGIL operates at a finer granularity, constructing a graph for each prompt-and-patch pair that connects individual task requirements to CFG-derived code blocks rather than relying on application-level API distributions or predefined permission semantics.

Task Shield: Task Shield (Jia et al., 2025) evaluates whether lower-trust instructions and tool calls contribute to trusted user objectives, using LLM-based alignment judgments to intervene during agent execution. SIGIL shares this task-alignment motivation but differs in its detection mechanism: it analyzes completed generated patches without requiring tool traces, employing a learned graph classifier rather than explicit task-contribution judgments. Unlike Task Shield, SIGIL does not independently authenticate trusted objectives, instead learning patterns from the supplied prompt and implementation.

Graph-based vulnerability detection: Devign (Zhou et al., 2019) uses program-semantic graphs and GNNs to classify vulnerable source code, while Vul-LMGNN combines pretrained code representations with program graphs. SIGIL extends beyond code-only representations by incorporating intent nodes and three relation types: intent dependencies, code control flow, and semantic intent–code grounding. Heterogeneous GraphSAGE integrates these relationships to classify potentially adversarial implementations relative to their associated tasks.

The underlying hypothesis is that malicious instructions may align locally with their implementations while remaining weakly integrated with the broader legitimate task. The additional decomposition, grounding, and architectural ablations investigate this hypothesis, although graph connectivity does not consistently improve performance across all settings.

The revised Related Work section details these differences in representations, alignment mechanisms, and detection objectives.

W2. Contribution of the Intent–Code Graph and Additional Ablations

The original structural-feature analysis establishes associations between graph properties and adversarial labels, but does not independently demonstrate that graph learning drives detection performance. Several additional ablations investigate this distinction.

AblationMain resultInterpretation
Intent decompositionOriginal beats a single prompt node in 5/6 settings, up to +35.4 ppFine-grained intent representation is often useful
Code decompositionCFG-AST beats a single code node in 5/6 settingsCode granularity matters
Semantic groundingEmbedding grounding beats random grounding in 6/6 settings, by 19.8–29.6 ppSemantic correspondence provides useful information
Pooled CodeBERTSIGIL wins 5/7 settings, significantly in three; pooled embeddings significantly win twoGraph classification has mixed benefits over pooled semantics
No-edge GraphSAGEFull wins 4/4 completed seed-42 accuracy comparisons by 0.32–1.42 pp, none statistically significantStrong detection remains possible without connectivity

Note: pp denotes percentage points (absolute differences between two percentages).

The pooled CodeBERT and no-edge experiments use archived attack-family training data rather than the original synthetic-only training procedure. Their findings therefore characterize architectural contributions under these additional training conditions, rather than definitively excluding shortcuts in synthetic training.

RQ4 (Section 4.5) summarizes the parsing, grounding, and pooled CodeBERT ablations. Appendix G.3, Table 14 reports their quantitative comparisons, with protocols and limitations in the accompanying text. The no-edge GraphSAGE results are reported here. RQ4 also describes the structural-feature analysis in Figure 2(b) as associative evidence.

AuthorsOpenReview ↗

[2/3] Response to Reviewer CDwv

W3. Dataset Composition, Class Balance, and Evaluation Metrics

The original presentation of accuracy and F1 without corresponding class distributions limits the interpretability of the results, particularly when the number of retained adversarial examples varies across evaluation settings.

Tables 1 and 2 now report benign and adversarial sample counts alongside accuracy and F1. Appendix D reports per-detector TPR and FPR in Tables 7 and 8. The saved-adaptive and cross-attack-family transfer results appear in Table 3 (RQ1, Section 4.2) and Table 13 (Appendix G.2), respectively, with their class counts and metrics.

Section 4.1 and Appendices D, F.1, F.3, and G.1–G.2 document the splits and distinguish generated candidates from retained evaluation examples.

These additions will make it easier to assess the trade-off between adversarial detection and false positives, independently of class prevalence.

W4. Adversarial Labeling and Annotation Protocol

Appendix F now clarifies the labeling criterion and annotation procedure. An attack-generated patch is labeled adversarial when it contains code that meaningfully implements the injected objective beyond the legitimate task. For example, merely repeating the instruction in comments or importing a module without implementing the requested behavior is insufficient. The positive label therefore identifies an adversarial implementation attempt. Labels were assigned through LLM-assisted annotation and targeted manual review by the authors. During manual review, the authors compared each reviewed patch with its injected requirement against this criterion. Appendix F.3 describes the scope of the procedure.

W5. Excluded Examples and Evaluation Scope

Appendix D, Table 4 reports filtering counts for each attack, agent, and benchmark setting in Tables 1 and 2. Across these settings, the ledger records 22,350 agent completions, of which 2,157 were excluded. The exclusions comprise 199 empty patches, 400 malformed or non-applicable diffs or invalid Python outputs, 43 scanner-error exclusions, 232 patches that did not implement the injection, and 1,283 patches that did not implement the requested obfuscation. The retained evaluation set comprises 12,393 benign and 7,800 adversarial examples. Appendix F.1 describes instance filtering, and Appendix F.3 explains how labels are assigned to retained examples.

The reported detection rates are conditional on this retained population. A positive label requires a meaningful attempt to implement the injected objective. The positive-label criterion does not require demonstrated exploitability. Among retained examples, outputs that do not meet the positive-label criterion are counted as negatives. Excluded outputs are outside the denominators of the detector metrics. These rates therefore describe detection on the retained evaluation set, not end-to-end attack success over all generated outputs. Appendix G.1, Table 12 separately reports the screening outcomes for the saved adaptive candidates.

W6. Generalization to Unseen Vulnerability Types

The synthetic-training procedure emphasizes CWE-78 and CWE-95 operations, raising the possibility that the detector learns vulnerability-specific patterns rather than broader intent–code relationships.

To investigate this concern, an additional cross-vulnerability transferability experiment trained SIGIL using benign examples and SWExploit attacks only, excluding FCV attack-family examples from training and model selection. The source training set contained 508 benign and 506 SWExploit examples (1,014 total), while source validation contained 217 benign and 216 SWExploit examples (433 total).

The frozen detector was evaluated on four held-out FCV families, Base64-obfuscated attacks, and independently held-out SWExploit controls.

DatasetBenignAdv.TotalACCF1TPRFPR
SWExploit (validation)21721643399.08%0.990798.15%0.00%
FCV-7831726658390.22%0.882080.08%1.26%
FCV-8931731363092.54%0.919986.26%1.26%
FCV-9431731963689.62%0.886280.56%1.26%
FCV-50231730362092.74%0.920986.47%1.26%
FCV-78 (Base64)31728460186.69%0.838773.24%1.26%
SWExploit (held-out)31731663395.73%0.956092.72%1.26%
SWExploit (Base64)3179140899.02%0.9785100.00%1.26%

ACC denotes accuracy; Adv. denotes adversarial examples.

AuthorsOpenReview ↗

[3/3] Response to Reviewer CDwv

W6. Generalization to Unseen Vulnerability Types (continued)

Source validation was used for checkpoint and threshold selection. All seven held-out evaluations reuse 317 benign examples. SWExploit controls test unseen tasks from the training family, and FCV outcomes measure transfer across families.

Across the four non-obfuscated FCV families, SIGIL achieves 80.08–86.47% TPR at a 1.26% FPR. Base64 FCV-78 yields 73.24% TPR, illustrating variation across attack conditions. The SWExploit held-out controls additionally demonstrate performance on unseen tasks from the source attack family.

In our static-analysis baseline, Bandit flags a patch when a selected warning code is triggered on an added line (Appendix F.4, Table 10). Its coverage therefore depends on whether the implementation triggers one of the selected rules. SIGIL uses task semantics and code structure without requiring vulnerability-specific warning codes at inference. Table 13 shows that a detector trained on benign and SWExploit examples transfers across all four evaluated plain FCV families without adding FCV-specific rules.

The results support cross-attack-family transfer. Training used archived SWExploit attacks, which may share underlying CWEs with FCV. Generalization to entirely unseen CWEs remains open.

RQ3 (Section 4.4) summarizes transfer. Appendix G.2 explains the protocol and limits. Table 13 reports counts and metrics across all seven held-out settings.

W7. Adaptive Attacks and Robustness

SIGIL's original threat model assumes a defense-unaware attacker. Because the method does not rely on secret detection mechanisms, an informed adversary could potentially exploit knowledge of graph construction and classifier feedback to craft evasive prompts.

Motivated by On Adaptive Attacks to Adversarial Example Defenses (Tramèr et al., 2020) and AutoDojo: Adaptive Attacks Expose Superficial Defenses and User-Underspecification Limits in LLM Agents (Ma et al., 2026), an AutoDojo-style feedback-guided search targeted earlier frozen SIGIL checkpoints with up to three attempts per eligible LiveCodeBench task. Independently retrained family-specific SIGIL detectors were then evaluated on the saved attack candidates without further optimization.

AttackBenignAdversarialAccuracyF1TPRFPR
FCV-7831623794.94%0.938389.87%1.27%
SWExploit31618979.60%0.643649.21%2.22%

LiveCodeBench; mini-SWE-agent. These saved attacks targeted earlier frozen checkpoints, not the independently retrained detectors evaluated here. Confusion counts (TP/FN/FP/TN): FCV-78 213/24/4/312; SWExploit 93/96/7/309. Both evaluations reuse the same 316 benign cases.

The retrained detectors detect 213/237 FCV-78 and 93/189 SWExploit candidates. Valid/invalid/unclassified counts are 237/708/18 and 189/190/14, respectively. We skipped 190 planned SWExploit tasks with unresolved objectives. Under an earlier validity screen, task-level evasion on the original targeted checkpoints was 14.64% (47/321) and 69.47% (91/131), respectively.

RQ1 (Section 4.2), Table 3 reports saved-attack transfer. Appendix G.1 and Table 12 document the protocol, limitations, and validity screening. Fresh adaptive evaluation of the retrained detectors and formal defense-aware robustness remain open.

W8. AI-Use, Ethics, and Reproducibility Statements

The statements concerning financial factor generation, stock-market observations, and portfolio backtesting were mistakenly included during manuscript preparation. They are unrelated to SIGIL and do not describe the research conducted. We apologize for this error.

The Ethics Statement has been replaced with the following SIGIL-specific disclosure:

ETHICS STATEMENT

This work studies defenses against adversarially induced vulnerabilities in LLM coding agents. It involves no human subjects or personal data, and it uses public benchmarks (SWE-bench Lite and LiveCodeBench) and previously published attacks. The synthetic adversarial prompts and vulnerable patches we generate for training and evaluation are dual-use, since they resemble the attack inputs that SIGIL aims to detect. Generated patches were evaluated only against benchmark repositories and were not deployed. SIGIL's main evaluation focuses on fixed, non-adaptive attackers, supplemented by a limited feedback-guided attack study. The evaluated vulnerability types and attack conditions remain limited, so SIGIL should not be treated as a complete safeguard. Deployments should combine it with other defenses and human review.

Reviewer_nrbhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the problem that attackers may issue adversarial prompts for code fixing, which will cause an agent to generate functionally correct code but with vulnerabilities. The idea of the paper is to build the connection between the intents of the prompts and the code segments generated by agents. An intent-code graph is built where edges are formed between intents (based on sequential order the appear in the prompt), between codes (according to CFG), and between intend and code (based on embedding similarity. The rationale is that intent and code nodes will appear less connected with benign/legitimate intent and codes for code fixing. The paper then uses synthetic generated adversarial prompts and vulnerable code along with benign ones to train a GNN to detect malicious intents and codes from the intend-code graph. The paper run experiments over a few benchmarks and compares with static analyzers and LLM-as-judge solutions to show the effectiveness of the proposed technique.

Critical strengths

The idea of building intent-code graphs and using GNNs to isolate adversarial intent and code is interesting. The paper provides some rationale behind the idea though it is based on an attacker with a fixed strategy rather than being adaptive given the knowledge of the proposed detection mechanisms. The paper utilizes a few existing benchmarks to conduct experiments and compared with a few existing solutions with different types of approaches.

Critical weaknesses
  1. The paper assumes that the attacker has no knowledge of the proposed defense, which seems a rather restrictive assumption, especially considering that there is no secret/key involved in the design. This security by obscurity approach is questionable for a real security solution. Given the design of SIGIL, it seems there is no way to prevent an attacker from building the same/surrogate GNN model and use it to check whether their adversarial prompts can be detected or not, and only when they are unlikely to be detected, they will be deployed. Similarly, knowing how the intent-node graph is built, the attacker could manipulate their prompts so that malicious intent-code forms closer connections with other part of the graphs. It is better that the paper could discuss why it is hard for attackers to evade such detections.

  2. The interpretation of the experimental results heavily depends on whether the testing data is balanced or how imbalanced the two classes are. Only reporting accuracy and F1 without such information is hard to understand the real impacts of the proposed solution. From table 1, we see high accuracy (0.8) but very low F1 (.333), suggesting the data is very unbalanced. I would suggest that the paper includes information of the balance of each dataset, or better, also report TPR and FPR to better understand the capability of the proposed technique.

  3. The paper does not do a good job to analyze the experimental results. For example, comparing Table 1 with Table 2, it is surprising that in quite a few settings SIGIL gives much better performance with obfuscated attacks than with unobfuscated attacks. Why obfuscation actually helps SIGIL detects attacks better? I hope the paper could go deeper in its analysis rather than focusing on general interpretation to support their claims.

Questions for authors
  1. class distributions in each dataset
  2. TPR and FPR in each experiment results.
1 comment
AuthorsOpenReview ↗

Response to Reviewer nrbh

Thank you for the feedback on adaptive attackers, dataset composition, and obfuscation.

W1. Threat Model and Adaptive Attacks

The original threat model assumes a defense-unaware attacker. To examine defense-aware attacks, we used an AutoDojo-style evaluation of earlier frozen SIGIL checkpoints on LiveCodeBench, with up to three attempts per eligible task. We then evaluated the saved candidates on family-specific detectors independently trained on nonadaptive examples.

AttackBenignAdversarialAccuracyF1TPRFPR
FCV-7831623794.94%0.938389.87%1.27%
SWExploit31618979.60%0.643649.21%2.22%

Both mini-SWE-agent evaluations reuse the same 316 benign cases. TP/FN/FP/TN: 213/24/4/312 for FCV-78 and 93/96/7/309 for SWExploit. Bandit's TPR/FPR on these cases is 98.31%/2.85% for FCV-78 and 46.03%/2.85% for SWExploit. Under an earlier validity screen, the original targeted checkpoints had task-level evasion rates of 14.64% (47/321) and 69.47% (91/131), respectively.

The saved pools contain 237/708/18 valid/invalid/unclassified FCV-78 attempts and 189/190/14 SWExploit attempts. We excluded 190 SWExploit tasks with unresolved objectives before generation. Screening was model-assisted at a 0.98 confidence threshold. Table 3 (RQ1, Section 4.2) reports saved-attack transfer. Appendix G.1 and Table 12 document screening and scope. Fresh adaptive robustness and formal guarantees against defense-aware attackers remain open. We then evaluated the saved candidates on family-specific detectors independently trained on nonadaptive examples.

W2. Dataset Balance and Classification Metrics

Tables 1–2 add benign and adversarial counts, and Appendix D, Tables 7–8 report per-detector TPR/FPR. Table 3 and Appendix G.2, Table 13 report counts and metrics for saved-attack transfer and cross-family transfer, respectively.

Section 4.1 and Appendices D, F.1, F.3, and G.1–G.2 describe training and evaluation splits, sample-retention criteria, and the distinction between generated candidates and the final evaluated population. In particular, an attack-generated patch is retained as a positive example when it meaningfully attempts the injected objective, rather than merely repeating malicious instructions in comments or failing to implement the attack.

These additions will make it easier to distinguish genuine improvements in detection from effects associated with class imbalance, false positives, or small adversarial populations.

W3. Why Does SIGIL Sometimes Perform Better Under Obfuscation?

Across Tables 1–2, SIGIL's accuracy rises in six settings, falls in five, and ties in one at reported precision. Bandit benefits from recognizable patterns in plain code. Obfuscation can make these patterns less visible, while SIGIL can still draw on code structure and its semantic relationships with the task. This may explain SIGIL's relative advantage over Bandit in some obfuscated settings.

AttackAgentBenchmarkPlain ACCBase64 ACCΔACC (pp)Plain F1Base64 F1
SWExploitSWE-agentSWE-bench Lite97.8%96.5%−1.397.2%88.3%
SWExploitSWE-agentLiveCodeBench80.8%97.2%+16.466.1%88.6%
SWExploitmini-SWE-agentSWE-bench Lite80.0%96.8%+16.876.1%95.3%
SWExploitmini-SWE-agentLiveCodeBench94.3%95.2%+0.990.2%89.8%
SWExploitOpenHandsSWE-bench Lite94.7%94.1%−0.691.4%90.9%
SWExploitOpenHandsLiveCodeBench95.8%95.6%−0.295.7%48.2%
FCVSWE-agentSWE-bench Lite80.1%99.3%+19.233.3%97.6%
FCVSWE-agentLiveCodeBench97.5%97.5%0.097.7%97.8%
FCVmini-SWE-agentSWE-bench Lite76.7%99.5%+22.877.3%99.4%
FCVmini-SWE-agentLiveCodeBench94.0%98.6%+4.693.5%98.6%
FCVOpenHandsSWE-bench Lite93.7%77.8%−15.937.5%50.0%
FCVOpenHandsLiveCodeBench98.4%97.9%−0.598.4%97.2%

ΔACC is Base64 minus plain accuracy in percentage points. For SWExploit/OpenHands on LiveCodeBench, accuracy changes little (95.8% to 95.6%), while F1 drops from 95.7% to 48.2%. Base64 can affect CodeBERT representations and intent–code links. It can also change which candidates pass screening. Because the plain and Base64 patches were generated and filtered separately, these comparisons do not isolate a causal effect of obfuscation.

SIGIL outperforms pooled CodeBERT on accuracy and F1 in five of seven held-out settings, with three significant accuracy gains after adjustment. Pooled CodeBERT wins significantly in two settings, including Base64 FCV-78, showing that semantic features can remain predictive under obfuscation. See RQ2 (Section 4.3) and Appendix G.4 for the obfuscation analysis, and Appendix G.3, Table 14 for the ablation.

Reviewer_Av86OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SIGIL, a graph-based framework for detecting adversarially induced malicious code in LLM coding agents. It constructs intent–code graphs from task prompts and generated patches, and trains a GraphSAGE classifier using benign and synthetically augmented examples. Experiments across two benchmarks, two attack methods, and three agent configurations show promising detection performance, particularly against Base64-obfuscated attacks, where SIGIL outperforms the evaluated baselines in most settings.

Critical strengths

A meaningful task-conditioned formulation of code-generation security. The paper goes beyond identifying potentially dangerous APIs or suspicious code patterns by explicitly modeling relationships between the requested task and the generated implementation. This is a useful direction because the same operation may be legitimate in one task and malicious in another. The intent–code graph provides a concrete representation for studying such relationships.

Critical weaknesses

1. Limited Novelty.
The proposed method primarily combines existing techniques, including LLM-based requirement decomposition, AST/CFG analysis, CodeBERT embeddings, and GraphSAGE. While applying these components to intent–code alignment is interesting, the paper does not sufficiently demonstrate a substantial methodological advance over existing task-alignment or graph-based detection approaches.

2. Inconsistent Statements Raise Concerns About Research Reliability.
The AI Use, Ethics, and Reproducibility Statements on page 10 discuss factor generation, stock-market data, portfolio evaluation, and other topics unrelated to the proposed security framework. These substantial inconsistencies raise concerns about the manuscript's preparation and the reliability of its research disclosures. The authors should clarify these discrepancies and provide accurate, paper-specific statements.

3. Potential Shortcut Learning from Synthetic Data.
SIGIL constructs adversarial training samples by inserting additional malicious intent and code nodes into benign graphs. The classifier may therefore learn synthetic structural artifacts rather than genuine intent–code misalignment. The lack of sufficiently controlled experiments makes it difficult to establish what the model actually learns.

4. Limited Generalization and Robustness Evidence.
The evaluation mainly considers two attack methods and a specific Base64 obfuscation strategy. SIGIL also underperforms Bandit in most non-obfuscated settings. Without stronger evidence on adaptive attacks and more diverse malicious behaviors, its claimed robustness and generalizability remain insufficiently supported.

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer Av86

Thank you for the detailed feedback on methodological novelty, research disclosures, synthetic-data construction, and robustness. The concerns raised have motivated several additional experiments and revisions, detailed below.

W1. Methodological Novelty and Relationship to Prior Work

SIGIL builds upon existing research in intent–behavior alignment and graph-based vulnerability detection. Its methodological contribution lies in a task-conditioned heterogeneous graph that jointly represents natural-language requirements, generated code, and their semantic and structural relationships to detect adversarially induced implementations. The unit of analysis is a completed prompt-and-patch pair, with each task requirement linked to code blocks.

CHABADA and WHYPER: CHABADA (Gorla et al., 2014) uses topic modeling to cluster Android applications by their descriptions, then applies a one-class SVM to identify unusual API usage relative to functionally similar applications. WHYPER (Pandita et al., 2013) uses dependency parsing and semantic representations to determine whether application descriptions justify requested permissions. Both establish connections between described functionality and program behavior. Unlike these approaches, SIGIL constructs a graph for each individual prompt-and-patch pair, linking fine-grained requirements to CFG-derived code blocks rather than relying on application-level API distributions or predefined permission semantics.

Task Shield: Task Shield (Jia et al., 2025) evaluates whether lower-trust instructions and tool calls contribute to trusted user objectives, using LLM-based alignment judgments to intervene during agent execution. SIGIL shares the task-alignment motivation but differs methodologically: it analyzes completed generated patches without requiring tool traces, using a learned graph classifier rather than explicit task-contribution judgments. Importantly, SIGIL does not independently authenticate trusted objectives, and its effectiveness depends on patterns learned from the supplied prompt and implementation.

Graph-based vulnerability detection: Devign (Zhou et al., 2019) models program semantics through graph neural networks, while Vul-LMGNN combines pretrained code representations with program graphs for vulnerability classification. SIGIL extends beyond code-only representations by introducing intent nodes and three relationship types: intent dependencies, code control flow, and semantic intent–code grounding. Heterogeneous GraphSAGE integrates these signals to classify whether a generated implementation exhibits adversarial behavior relative to its task.

The underlying hypothesis is that malicious instructions may align locally with their implementations while remaining weakly integrated with the broader task. Additional decomposition, grounding, and architectural ablations investigate this hypothesis, although the observed benefits of graph connectivity vary across settings.

The Related Work section has been expanded to include these methods, clarifying their representations, alignment mechanisms, and detection objectives relative to SIGIL. This positions the contribution in the joint modeling of task intent and generated-code structure, rather than claiming novelty in task alignment or graph learning individually.

W2. Inconsistent AI-Use, Ethics, and Reproducibility Statements

The statements concerning financial factor generation, stock-market observations, and portfolio backtesting were mistakenly included during manuscript preparation. They are unrelated to SIGIL and do not describe the research conducted. We apologize for this error.

The Ethics Statement has been replaced with the following SIGIL-specific disclosure:

ETHICS STATEMENT

This work studies defenses against adversarially induced vulnerabilities in LLM coding agents. It involves no human subjects or personal data, and it uses public benchmarks (SWE-bench Lite and LiveCodeBench) and previously published attacks. The synthetic adversarial prompts and vulnerable patches we generate for training and evaluation are dual-use, since they resemble the attack inputs that SIGIL aims to detect. Generated patches were evaluated only against benchmark repositories and were not deployed. SIGIL is primarily evaluated against fixed, non-adaptive attackers, supplemented by a limited feedback-guided attack study; it should not be treated as a complete safeguard. Deployments should combine it with other defenses and human review.

The AI-Use and Reproducibility Statements will also be corrected to accurately describe SIGIL's use of LLMs, dataset construction, graph generation, training procedures, and evaluation methodology.

AuthorsOpenReview ↗

[2/2] Response to Reviewer Av86

W3. Potential Shortcut Learning from Synthetic Data

Synthetic node augmentation is restricted to training. We add a malicious intent and corresponding code node to parsed benign graphs and recompute intent–code grounding. Evaluation uses held-out benign examples and independently generated FCV-Attack and SWExploit prompt-and-patch pairs. We build their graphs from actual agent patches without synthetic nodes. This tests transfer beyond synthetic construction, though structural or lexical shortcuts may still contribute.

AblationMain resultInterpretation
Intent decompositionOriginal outperforms a single prompt node in 5/6 settings, up to +35.4 ppFine-grained intent representation is often useful
Code decompositionCFG-AST outperforms a single code node in 5/6 settingsCode granularity can affect detection
Semantic groundingEmbedding grounding beats random grounding in 6/6 settings, by 19.8–29.6 ppMeaningful semantic links outperform arbitrary connections
Pooled CodeBERTSIGIL wins 5/7 settings, significantly in three; pooled embeddings significantly win twoGraph-based classification helps in some, but not all, settings
No-edge GraphSAGEFull wins 4/4 completed seed-42 accuracy comparisons by 0.32–1.42 pp, none statistically significantStrong detection remains possible without connectivity

pp = percentage points.

The last two ablations use archived attack-family supervision and cannot rule out shortcuts in synthetic-only training. RQ4 (Section 4.5) and Appendix G.3, Table 14 cover the parsing, grounding, and pooled CodeBERT ablations. The no-edge results appear here.

W4. Limited Generalization, Adaptive Robustness, and Comparison with Bandit

Generalization across vulnerability types (LiveCodeBench): An additional cross-vulnerability experiment trained SIGIL using benign examples and SWExploit attacks only (508 benign/506 adversarial training; 217/216 validation), excluding FCV attack families from training and model selection.

Held-out attackBenignAdversarialTotalAccuracyF1TPRFPR
FCV-7831726658390.22%0.882080.08%1.26%
FCV-8931731363092.54%0.919986.26%1.26%
FCV-9431731963689.62%0.886280.56%1.26%
FCV-50231730362092.74%0.920986.47%1.26%

The same 317 benign examples are reused across these evaluations. RQ3 (Section 4.4) summarizes the transfer experiment, and Appendix G.2, Table 13 gives the full held-out results. These results support cross-attack-family transfer, although the experiment uses attack-family supervision rather than the original synthetic-only training procedure and does not establish generalization to arbitrary vulnerabilities.

Adaptive attacks. An AutoDojo-style study targeted earlier frozen SIGIL checkpoints with up to three attempts per eligible LiveCodeBench task. Family-specific retrained detectors used nonadaptive data (benign/adversarial: FCV-78 train 651/568, validation 74/59; SWExploit train 651/527, validation 74/68). We tested them on saved candidates without reoptimization.

AttackBenignAdversarialAccuracyF1TPRFPR
FCV-7831623794.94%0.938389.87%1.27%
SWExploit31618979.60%0.643649.21%2.22%

Mini-SWE-agent on LiveCodeBench. TP/FN/FP/TN: FCV-78 213/24/4/312, SWExploit 93/96/7/309. Both reuse 316 benign cases.

The retrained detectors caught 213/237 FCV-78 and 93/189 SWExploit valid candidates. Valid/invalid/unclassified counts were 237/708/18 of 963 FCV-78 attempts and 189/190/14 of 393 SWExploit attempts. For SWExploit, 190 of 321 planned tasks had unresolved objectives. Under an earlier validity screen, the original checkpoints had task-level evasion rates of 14.64% (47/321) and 69.47% (91/131), respectively. RQ1 (Section 4.2), Table 3 reports saved-attack transfer. Appendix G.1 describes the protocol, and Table 12 gives screening counts. Robustness under fresh adaptive evaluation remains open.

Bandit comparison. SIGIL matches or exceeds Bandit on accuracy and F1 in 3 of 12 plain settings (two wins and one tie at reported precision, Table 1). For FCV with mini-SWE-agent on LiveCodeBench, Bandit reaches 100% recall at 0.9% FPR on plain patches. On the separately generated Base64 set, recall is 2.1% for Bandit and 99.2% for SIGIL (Tables 7 and 8). Bandit’s rules detect recognizable code patterns and can also flag benign changes. SIGIL uses code structure and task semantics, which can still provide evidence when those patterns are obscured. See RQ2 (Section 4.3) and Appendix G.4 for the limits of comparing separately generated sets.

Thank you for the response and the additional experiments. However, the rebuttal does not fully address my concerns, particularly regarding whether the intent–code graph is responsible for the detection performance and the robustness against adaptive attacks. Also I consider the inconsistent AI-Use, Ethics, and Reproducibility statements a serious issue rather than a minor editing error. Statements describing an unrelated project, including references to sections that do not exist in this paper, raise concerns about research integrity and the reliability of the manuscript as a whole, which cannot be resolved by replacing the text during the rebuttal. I will therefore maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.