acceptodds
Withdrawn from ICLR 2027

Detecting Contaminated Code-Generation Prompt Batches via Influence Functions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) are increasingly used for code generation, yet they remain vulnerable to prompts that elicit insecure implementations. Existing defenses typically rely on predefined threat models or known vulnerability patterns, limiting their effectiveness against novel attacks. We propose CodeSIFT, a threat-model-agnostic detection method that leverages influence functions to identify batches of prompts that induce anomalous model behavior. Rather than detecting specific vulnerabilities, CodeSIFT measures the parameter-space influence of generated code and uses a statistical test to determine whether a candidate prompt set deviates from a benign reference distribution. To evaluate our approach, we introduce two benchmark datasets covering a variety of vulnerabilities. We evaluate CodeSIFT on three open-weight code LLMs ranging from 3B to 7B parameters, achieving AUROC scores of up to 0.98 at moderate-to-high injection rates, while maintaining well-calibrated false positive rates and substantially outperforming static analysis baselines. These results suggest that influence-function-based detection is a promising direction for identifying malicious code-generation prompts without requiring prior knowledge of the underlying attack class.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
influence functions, LLM security, code generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ruXfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CodeSIFT, a method for detecting whether a batch of code-generation prompts is contaminated with requests for insecure implementations. For each prompt, CodeSIFT samples model completions, computes the norms of preconditioned loss gradients over selected MLP parameters, and averages them into a prompt score. A one-sided Welch test then compares the candidate batch’s scores against a benign reference pool. The preconditioner is motivated by an influence-function (PBRF) interpretation. It is estimated from 100 examples from The Stack v2 using normalized gradient Gram matrices and a one-sided approximation.

For validation, the authors construct two Python prompt datasets, AUTHSEC and INFRACLOUD, with 400 malicious prompts in total, alongside 400 benign prompts. Experiments on three code models report strong batch-level discrimination in several settings, particularly with larger batches and higher contamination rates. Further experiments vary batch size, contamination rate, and number of completions, and assess false-positive calibration. NLL, perplexity, and entropy are introduced as baselines.

Critical strengths
  1. A useful problem formulation. Testing prompt batches against a benign reference combines weak per-prompt evidence without needing a catalogue of vulnerability signatures. The paper limits its claim to batch-level detection and acknowledges that localizing individual malicious prompts remains open. If the signal turns out to be specific to security, this would be a practical use of model-internal statistics for monitoring.

  2. Consistent positive results across settings. The evaluation covers three model families and two application domains, with sweeps over batch size, contamination rate, and completion count. Figures 1–2 and Appendix F show substantial discrimination in multiple settings, and detection generally gets stronger as more contaminated evidence accumulates. The false-positive calibration experiments are also useful, though their construction needs clarification (Q4).

Critical weaknesses
  1. The benchmark does not separate security behavior from ordinary task differences. Balancing the two domains across labels does not make the malicious and benign tasks otherwise comparable. In Figure 3, the malicious examples are Flask endpoint implementations that ask for behaviors such as insecure password hashing or bypassing CSRF validation. The benign examples are shorter standalone functions for password hashing, token creation, and constant-time comparison. Figure 4 similarly contrasts attack-oriented endpoint skeletons with tasks such as returning a greeting or writing lines to a file. These examples differ in program structure, complexity, length, and prompt wording. Any of these could shift gradient norms whether or not the generated code is vulnerable. Length is a particular concern: Section 3 defines a summed token loss, and the paper does not say how the implementation normalizes over tokens. The paper reports confirming the targeted weaknesses in malicious-pool completions. However, it does not give vulnerability rates for each model, and it does not verify that benign-pool completions are secure. As a result, the experiments show more convincingly that the two authored pools can be separated than that CodeSIFT detects security-relevant misalignment. Matched secure/insecure variants of the same skeletons, plus controls for benign task shift, would test this directly. Calibration on batches drawn from the same benign pool does not show specificity when legitimate tasks change. This concern goes to the paper’s central claim and would need new controlled data to resolve, so it is the main basis for my soundness score.
  2. Neither the influence-function interpretation nor the benefit of the implemented preconditioner is established. Appendix B estimates the factors from normalized weight-gradient Gram matrices, E[g~⊤g~]E[\tilde g^\top\tilde g] and E[g~g~⊤]E[\tilde g\tilde g^\top]. These differ from the activation and backpropagated-derivative factors used in standard K-FAC (Martens & Grosse, ICML 2015). The manuscript does not derive how they relate to the claimed Gauss–Newton approximation. The PBRF interpretation of Bae et al., NeurIPS 2022 does not by itself justify this particular preconditioner. A heuristic that works empirically could still be valuable. However, the paper does not compare against unpreconditioned gradient norms or simpler preconditioners. NLL, perplexity, and entropy are introduced as baselines, but Tables 1–3 report correlations rather than detection performance, and weak correlation does not show that these signals are weaker detectors. Section 4 also asserts that using the same approximation for both pools makes the approximation error cancel. This does not hold in general, because an input-dependent error can change the difference between the two score distributions. Without these ablations, the reported gains cannot be attributed to influence-based curvature information.
Questions for authors

Q1 and Q2 address the two weaknesses above, and convincing answers to them could materially change my recommendation. Q3 and Q4 are clarification requests. I am not asking for extensions to larger models or other languages.

  1. Security specificity (additional analysis). Take a matched subset that shares code skeletons and has comparable completion lengths. Does CodeSIFT distinguish secure from insecure variants there? What false-positive rate does it get on secure Flask or infrastructure tasks that differ from the reference pool? Clarification: how were completion vulnerabilities validated for each evaluated model, including completions of benign prompts?
  2. Contribution of each component (additional analysis). What batch AUROC and detection power at α = 0.01 do raw gradient norms, a simple diagonal preconditioner, and the NLL/entropy signals you already compute achieve? Does CodeSIFT’s advantage persist after controlling for completion length?
  3. Implemented quantity (clarification). What derivation supports the normalized gradient-Gram estimator as an approximation to the curvature that the PBRF interpretation requires? Are gradients computed over completion tokens only, or over the full prompt and completion? Is the loss summed or averaged over tokens?
  4. Evaluation protocol (clarification). How are reference, uncontaminated candidate, and contaminated candidate batches sampled? Are reference and candidate prompts disjoint within each test? What are the detection rates at the stated operating threshold, particularly at the smallest contamination rate tested? Please specify:
    • the reference size
    • the fixed settings behind each sweep
    • the number of repetitions
    • which statistic is used to compute AUROC
    • the error-bar definition
    • the decoding parameters
Reviewer_UE7cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CodeSIFT, a batch-level detection method for determining whether a set of code-generation prompts contains malicious prompts that induce insecure implementations. For each prompt, the method samples multiple completions, applies curvature preconditioning to their gradients using a simplified K-FAC approximation, and computes an influence score based on the resulting gradient norms. A one-sided Welch’s t-test then determines whether the candidate batch has a significantly higher mean influence score than a benign reference set.

The evaluation uses two manually constructed Python datasets, AuthSec and InfraCloud, and three open-weight code models ranging from 3B to 7B parameters. Compared with Bandit, Semgrep, and a custom AST-based taint tracker, CodeSIFT provides more consistent batch-level discrimination across the tested settings, achieving an AUROC of up to approximately 0.98 at moderate-to-high contamination rates. The paper also examines false-positive calibration and the sensitivity of detection performance to batch size, contamination rate, and the number of sampled completions.

Critical strengths

This paper addresses a relevant security problem: detecting batches of code-generation requests that contain prompts inducing insecure implementations. This problem is important as coding assistants are increasingly used in software development.

The paper explores model-internal influence scores as an alternative to predefined vulnerability rules. This provides a potentially useful direction for detecting insecure behavior that is difficult to capture through surface-level code patterns.

The evaluation covers three code models and two application domains. The additional experiments examine how contamination rate, batch size, and completion sampling count affect detection performance.

Critical weaknesses

The evaluation does not clearly establish whether CodeSIFT detects security-related behavior or general differences between the malicious and benign task distributions. The examples in Figures 3–4 differ in functionality and task structure, which could affect influence scores independently of security. Controlled comparisons that preserve the task and code skeleton while changing only security-relevant requirements are needed to strengthen the interpretation of the results.

The benefit of curvature preconditioning is not sufficiently demonstrated. The paper lacks a direct comparison with unpreconditioned gradient norms, and the correlation analysis with NLL, perplexity, and entropy does not establish better detection performance. Without these comparisons under the same evaluation protocol, it is difficult to determine whether curvature preconditioning provides meaningful gains over simpler signals.

Although Section 5 states that the malicious prompts were validated by generating completions and confirming the targeted weaknesses, the paper does not report model-specific vulnerability rates or examine how influence scores relate to these outcomes. Malicious prompts may fail to induce vulnerabilities, while benign prompts may produce insecure code. This leaves unclear whether the detection signal reflects actual insecure behavior or other characteristics of the prompt pools.

The robustness of false-positive control to benign distribution shifts is insufficiently evaluated. Calibration on the tested benign distribution does not establish that false-positive rates remain controlled when legitimate workloads change in framework, complexity, or coding style. This is important because the method treats an increase in mean influence as evidence of contamination.

The evaluation does not sufficiently establish the usefulness of batch-level detection under sparse contamination. The headline results emphasize moderate-to-high contamination rates, but it remains unclear what detection rate is achievable at a fixed low false-positive rate when a batch contains only one or a few malicious prompts. This limits the evidence for practical monitoring scenarios in which malicious requests are rare.

Questions for authors

Please address the concerns raised in the Critical Weaknesses section.

Reviewer_54ZEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CodeSIFT, a method that uses influence-function scores and a statistical test to detect batches containing prompts that elicit insecure code. The authors construct 800 benign and malicious Python prompts across two domains and evaluate the method on three code-generation models.

Critical strengths

The paper investigates an important problem and proposes a detection signal that does not require predefined vulnerability rules.

The evaluation covers three model families and investigates the effects of contamination rate, batch size and the number of sampled completions.

Critical weaknesses

It remains unclear whether CodeSIFT detects security weaknesses or other differences between the prompt pools. In Figures 3–4, malicious examples explicitly request insecure behaviour and often contain more elaborate code skeletons than benign examples. The evaluation does not establish robustness to benign changes in task complexity or content.

The contribution of influence-function preconditioning is not sufficiently established. There is no comparison against unpreconditioned gradient norms

Questions for authors

Can you secure and insecure variants of the same coding tasks, matched for length and complexity, and measure false positive rates when the benign task distribution changes?

How does CodeSIFT compare with raw gradient norms, NLL and entropy when all use the same batch-testing procedure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.