acceptodds
Under review as a conference paper at ICLR 2027

VeilPRBench: Can Code Review Agents Catch Vulnerabilities in Pull Requests That Resolve Real Issues?

Abstract

Code review is a critical line of defense for software security. As the volume of pull requests (PRs) grows, automated review agents increasingly support review and merge decisions. However, a PR can genuinely resolve its reported issue while introducing logic vulnerabilities. To quantify this risk in automated code review, we introduce VeilPRBench, a benchmark for evaluating whether code review agents block such PRs while accepting benign repairs. The benchmark contains 192 manually constructed attack PRs across 14 open-source projects and 29 historical issue families, together with 135 benign repairs covering 13 projects and 27 families, across Python, C, and JavaScript. Independent functional tests verify the intended repairs, and executable security checks establish the target consequences of the attacks against applicable reference states. Benign repairs pass the prescribed functional and security checks within their validation scope. Our evaluation of 16 deployed review workflows reveals a substantial security gap: attack blocking averages only 25.0%, and the highest blocking rate is 55.2%. VeilPRBench provides an executable testbed for evaluating and improving the security of automated code review.

Primary area:
datasets and benchmarks
Keywords:
software security, pull request security, code review agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_APF5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark, VeilPRBench, for evaluating the security of automated code agents, specifically they accept/refuse benign or malicious pull requests. The benchmark includes multiple open-source projects, across diverse programming languages. The benchmark was evaluated on 16 publicly available code reviews agents where review output has been analyzed for acceptance, rejection on benign and adversarial PRs.

Critical strengths
  • The paper explores an important security problem in automated code review agents, broadly used for vibe-coding or by developers who fully automate their coding workflow.
  • The paper proposes a benchmark for code review agents evaluation to analyze how they approve/reject benign/adversarial code changes and whether they can even identify potential malicious code updates automatically.
  • The benchmark comprises 14 open-source projects with 135 benign repairs across C, python, and JavaScript.
  • The proposed VeilPRBench was evaluated on 16 automated code review agents (baselines), reporting interesting findings that can make these agents more robust.
  • The paper is well-written and flows nicely.
Critical weaknesses
  • The paper’s scope and problem is very interesting, however, what distinguishes this benchmark from the typical code vulnerability benchmark used to detect malicious code logic in LLM generated code? I understand VeilPRBench is meant for workflow security evaluation but how can this not be done by simply taking one of the available and more covering LLM-generated code security benchmarks and evaluating them on the 16 code review agents evaluated in this paper? As a reader, I find it hard to understand the main novelty and scientific contributions of this paper. The authors should make this clear in the paper, probably by comparing VeilPRBench against existing benchmarks and highlighting what’s missing and what’s new in each.
  • The threat model considered in the paper is somehow very strong and its practicality needs further justification. The attacker is assumed capable of bypassing PR provenance checks in the code review agents, which might not always be the case. Also, it’s unclear if a simple check of the PR provenance or even context can robustify those agents. Overall, the practicality of the threat model should be further justified since the VeilPRBench premise relies on it.
  • The benchmark choices in terms of how many PRs, open-source projects, attacks, programming languages..etc should be also justified. The design choices must be justified to understand the degree of coverage and comprehensiveness, which is important for the benchmark’s significance.
  • The evaluated attack's scalability is also unclear. 192 attacks seems very rigid, I wonder how attacks can be scaled to include sophisticated variants or even unseen attacks that can further solidify the security evaluation for code review agents.
Questions for authors
  • Could you clarify the novelty of VeilPRBench compares to existing LLM-code vulnerability detection benchmarks?
  • Could you justify and discuss of the threat model's practicality?
  • Could you justify the design choices of the benchmark and discuss its extensibility?
  • Could you discuss the attack scalability too? Could you provide further evaluation on strong LLM-generated vulnerable code?
5 comments

Thank you for pointing out the insufficient explanations of our benchmark construction process, validation artifacts, and evaluation settings. We have added the relevant details in the revised manuscript and address each concern below.

Weakness 1 and Question 1

Thank you for pointing out that the positioning of our contribution needs further clarification. The central question of VeilPRBench is: when a PR has already passed the repair checks for a real historical issue and the prescribed regression checks, can a complete code-review workflow still identify and block an execution-validated security problem within it? Our contribution lies in combining this task setting, sample-level evidence, and evaluation design. Specifically, each evaluation instance includes a real historical issue, a fixed repository baseline, and a complete candidate PR. An attack PR must satisfy two requirements simultaneously: first, the pre-repair baseline fails the issue-directed checks, whereas the version with the candidate applied passes the repair and prescribed regression checks; second, under specified inputs and execution preconditions, the candidate exhibits a target security violation that an applicable clean reference does not exhibit under the same conditions, with code inspection confirming the relationship between the violation and the candidate modifications. Benign PRs come from the same repair contexts and pass functional, regression, and applicable security checks. These requirements are described in Sections 3 and 4.1–4.2 and Appendices B.1–B.4.

We agree that existing benchmarks already provide some of these important elements. For example, PrimeVul includes paired vulnerable and patched versions, together with project, commit, and file context; CWEval uses both functional and security tests; and SWExploit studies patches that complete functional repairs while containing vulnerabilities. Thus, we do not characterize these studies as consisting only of isolated code snippets or ignoring functional correctness. The following table distinguishes their primary evaluation targets and the specific problem addressed by VeilPRBench.

Benchmark / StudyMain evaluation focusRelation to VeilPRBench
PrimeVulVulnerability detection, including vulnerable/patched pairs and project, commit, and file metadataSupplies vulnerability-detection data; VeilPRBench evaluates workflow decisions on PRs with verified historical-issue repair and security consequences.
SecurityEvalSecurity of generated code using generation prompts and vulnerability examplesEvaluates code-generation security; VeilPRBench evaluates review of fixed candidate PRs in executable repository repair contexts.
CWEvalJoint functional and security outcomes of LLM-generated codeShares the distinction between functional correctness and security; VeilPRBench studies repository-level historical-issue repairs and complete PR-review workflows.
SWExploit / SWEADVAdversarial issue descriptions that induce unsafe patches during successful program repair; SWEADV also studies detectionStudies attacks on repair generation; VeilPRBench fixes the resulting review object and compares complete review workflows with repair-context-matched benign controls.
SEVRA-BENCHReview-agent responses to reversed historical security fixes wrapped in social-engineering narrativesStudies narrative manipulation around vulnerability-introducing PRs; VeilPRBench requires the candidate to satisfy a specified historical-issue repair task.
PRWeaverExecution-validated attacks under single-PR, decomposed, interleaved, and genuine-change-fused presentationsStudies concealment across review contexts and repository evolution; VeilPRBench centers admission and evaluation on validated issue repairs.
MalPR-BenchMechanism-grounded malicious PRs and paired benign controls, with separate verdict, diagnosis, and evidence scoringStudies attributable security review; VeilPRBench centers the task on validated issue repair and jointly measures attack blocking and benign acceptance.
VeilPRBenchComplete PR-review workflows on validated historical-issue repairs containing executable security violations, with repair-context-matched benign controlsCombines repair validation, target-specific security evidence, clean-reference attribution, and workflow-level outcomes.

Existing datasets can serve as construction materials, but merely feeding their vulnerability samples to the same 16 review agents does not automatically establish the repair task and its verification evidence described above. For data that do not already satisfy these conditions, one would still need to recover a fixed repository version and real issue, organize the sample as an applicable complete PR, verify the target repair and prescribed regression behavior, establish security probes and execution comparisons against an applicable clean reference, and prepare benign controls from the same repair context. For studies that already provide some of these elements, their artifacts can be reused rather than rebuilt from scratch; VeilPRBench's construction work lies in consistently integrating these requirements into comparable review instances. This design also allows us to examine identifying security problems separately from producing a blocking decision: we evaluate complete workflows, including native prompts, contextual file access, tool use, and decision pathways, while reporting both attack blocking and benign acceptance, so a system that rejects every change is not considered an effective reviewer. We have added the relationships to SWExploit, SWEADV, SEVRA-BENCH, PRWeaver, MalPR-Bench, and other studies in Section 2; Sections 3–4 clarify our task and admission criteria.

Weakness 2 and Question 2

Thank you for raising this concern. Our threat model does not grant attackers the ability to bypass PR provenance checks. Section 3 limits attacker capabilities to reading public source code, issues, and tests, and controlling the code changes, title, and description of the candidate PR under review. Appendix A.3 explicitly places the trusted baseline, protected branches, review configuration, and trusted validation artifacts outside the attacker's write permissions. Thus, submitting a candidate does not imply the ability to disable checks, modify protected branches, or merge the PR unilaterally. The applicable setting of our evaluation is that a candidate has satisfied a deployment's upstream admission requirements and reached its declared code-review interface; we then ask whether the review workflow can block the security problem within that candidate. This scenario does not require identity spoofing. For example, in a project that allows external contributions to enter review, a contributor can submit a PR using their own identity; verifying its origin does not directly establish that its changes preserve all security properties. If a deployment's provenance policy already rejects the candidate, that candidate never reaches the review stage studied here.

Provenance and contextual information are indeed valuable for defense. Prior work such as Anomalicious uses commit histories, repository metadata, and author-trust signals to identify anomalous or potentially malicious commits, showing the detection value of this information. However, such metadata-based screening evidence is not equivalent to proving that a provenance check guarantees the semantic security of code. Provenance screening can restrict which candidates enter the workflow, while code review examines whether modifications that enter it are secure. We also directly evaluate the effect of additional review context. The standard review input includes the PR diff, description, issue, and repository context permitted by the protocol, rather than isolated code snippets. Section 6.3 and Table 3 fix PR-Agent and compare three additional-guidance conditions across five backbones using all 192 attacks and 135 benign PRs:

ConditionAdditional guidanceOverall ABROverall BAP
L0No benchmark-specific security instruction29.2%93.0%
L1Generic security guidance50.7%89.3%
L2Joint repository- and weakness-informed guidance, replacing L155.0%85.2%

Generic security guidance increases ABR from 29.2% to 50.7%, a paired gain of 21.6 percentage points with a 95% issue-family bootstrap interval of [15.6, 27.4], while benign acceptance decreases from 93.0% to 89.3%. Under L2, ABR reaches 55.0%, but its additional gain over L1 is 4.3 percentage points, with an interval of [-1.2, 9.9]. These results support the conclusion that security guidance can improve blocking but does not eliminate the gap under the tested conditions. We also record triggering conditions separately from attacker permissions. In the original set, 82 PRs contain environment-variable-controlled branches, indicating explicit environmental preconditions for the vulnerability, not that a remote attacker can modify deployment configuration. Data-plane triggers need not require such permissions: for example, the Werkzeug case in Appendix B.7 uses a client-controlled field name to bypass resource accounting. Appendix B.1 clarifies the interpretation of environmental preconditions.

Weakness 3 and Question 3

Thank you for suggesting that we justify these design choices. VeilPRBench uses purposive sample construction: we select open-source repair tasks with meaningful real-world software relevance, inspectable repair code, and executable validation. Project selection considers both application context and verifiability. Projects such as Git, curl, Redis, SQLite, Requests, and urllib3 involve version control, network communication, data storage, and web infrastructure, providing different repair logic and security requirements. Admission to the construction process also requires a real historical issue, a fixed repository revision, an inspectable reference repair, and an execution environment capable of reproducing the problem and validating candidates. Python, C, and JavaScript provide different language and project contexts in the present collection, but coverage of three languages does not imply exhaustive coverage of language-specific security problems. Appendix B.1 adds the selection criteria; Appendix B.10 and Tables 12–14 report the project, issue-family, and CWE distributions.

The sample counts arise from construction and validation outcomes. We constructed 608 attack candidates; 443 passed issue-directed functional checks and prescribed regression checks; and 192 ultimately passed executable security validation and manual eligibility review. These samples cover 14 projects, 29 historical issue families, and 19 target CWE classes. We did not relax admission criteria to reach a predetermined attack count. For the 27 issue families containing benign samples, we constructed five types of benign PRs per family, yielding 135 benign samples spanning 13 projects. These samples test whether review systems can preserve valid repairs rather than merely block attacks. The overall composition is reported in Sections 4.1–4.3, admission counts in Appendix B.1 and Table 5, and benign construction in Appendix B.4.

The 19 CWE classes represent the target weaknesses covered by the current validated collection. Construction is organized around security properties implicated by repairs, including identity and access control, sensitive-information handling, input validation, path isolation, state maintenance, and resource boundaries. Each admitted attack must specify the violated requirement, triggering preconditions, and observable consequence, supported by an execution comparison with an applicable clean reference. For instance, CWE-200, CWE-22, and CWE-770 cover sensitive-information exposure, path traversal, and missing resource limits, respectively.

For extensibility, we organize construction around the repair context, target security property, and implementation mechanism of the security failure. A single issue can be associated with different security properties, and the same property can be violated through different code changes or triggering conditions. Therefore, 192 is the size of the current collection, not an upper bound imposed by the protocol. The additional experiment below directly examines whether other attack designs can be introduced into existing repair contexts.

Reviewer_gpsdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes VeilPRBench, a benchmark for evaluating whether code review agents block such PRs while accepting benign repairs. The benchmark contains 192 manually constructed attack PRs and 135 benign PRs, across Python, C, and JavaScript. The evaluation on 16 publicly available code review agents show that existing PR agents are not sufficient to classify benign and adversarial PRs.

Critical strengths
  1. Overall, the paper is well written and easy to follow.

  2. The problem studied is important and timely, especially as more agents are being deployed in CI/CD workflows.

  3. The evaluation results are promising and demonstrate limitations in current pull request agents.

Critical weaknesses
  1. The benchmark construction process is confusing and miss some details. For example, how are the benign PRs are collected, how are the vulnerable code injected, the author only mentioned "We manually construct attacks by modifying checks, control flow, state transitions, data propagation, or resource handling while preserving the issue repair." without detailed information.

  2. Some of the information seems to be useless, for example, the benchmark contains the test cases to validate the PR functionality and security inputs for validating the security issues. However, these tests seem to be useless in the current settings, as the PR are fixed, and PR agent will never see these tests.

  3. Some of the setting is confusing. For example, why force the PR agent to return block, signal, or none instead of block and signal? Also, what is LLM adjudicator? Rq.8 defines three metrics, but only 2 in Table 2.

  4. For sec 6.1, what is the difference between baselines and backbones? For backbones, why only 100 PRs are evaluated?

  5. As a dataset and benchmark paper, the author did not provide their benchmark for validation.

  6. Missing some related work

[1] Red teaming program repair agents: When correct patches can hide vulnerabilities

[2] Adversarial bug reports as a security risk in language model-based automated program repair

Questions for authors
  1. Could the authors describe the benchmark construction process in more detail?

  2. What roles do the functionality and security test cases play? Are both required for developers to use the benchmark?

  3. What is the rationale for the return block, signal, and none settings? why not just pass and fail.

  4. Why does the evaluation include only 100 PRs, and how were they selected?

8 comments

Thank you for pointing out the insufficient explanations of our benchmark construction process, validation artifacts, and evaluation settings. We have added the relevant details in the revised manuscript and address each concern below.

Weakness 1 and Question 1

We agree that the original high-level description did not sufficiently explain how a real historical issue is transformed into an attack PR with a verifiable security consequence. We have expanded Sections 4.1–4.2 (pp. 4–5) and detailed the construction and validation protocol in Appendices B.1–B.6, with Algorithm 1 summarizing the complete procedure. Appendix B.7 provides concrete construction examples from Werkzeug and Requests.

Repair-context selection and candidate planning.

We begin with a real historical issue and fix the repository revision preceding its upstream repair. We inspect the upstream fixing commit, the repaired version, and the relevant functional code to identify the reported defect, expected repair behavior, and associated security requirements. We then apply the clean upstream repair to the fixed baseline to establish a clean repair reference. Issue-directed functional tests confirm that the baseline exhibits the reported defect and that the repaired version satisfies the expected behavior, consistent with the upstream fix for the functionality covered by the issue. We prioritize existing repository tests and regression tests introduced by the historical repair. When these are insufficient, we derive supplementary tests from the issue description, maintainer explanations, or project specifications.

Next, we analyze security properties associated with the repair, such as whether inputs are fully validated, whether distinct identities or security configurations remain isolated, and whether resource boundaries are enforced. Each candidate-construction plan records the target security property, relevant program states or data relationships, candidate CWE, triggering preconditions, and expected security consequence. A single issue may involve multiple security properties, and the same property may be weakened through different code modifications. Each repair context can therefore yield multiple candidate plans.

Locating and modifying relevant code.

Attack PRs are constructed on top of the clean repair. We trace the relevant function calls, parameter passing, conditional checks, state maintenance, and sensitive-data processing paths to locate code responsible for the target security property. Modifications may occur within files touched by the original repair or in other code locations related to the repaired functionality or target security property. After determining the target property, candidate CWE, and modification scope, we construct candidates using five nonexclusive classes of code modifications, described below.

Five classes of vulnerability-injection mechanisms

(1) Conditional and control-flow modifications. These modifications alter branch conditions and execution-path selection under different inputs or program states. During construction, we identify the security checks associated with the clean repair and analyze the conditions under which they execute. Based on the candidate weakness and its intended consequence, we may introduce additional branches, modify combinations of existing conditions, or narrow the input range covered by a security check. For example, a check may remain effective for ordinary inputs while an unexpected execution path is taken under a specific input condition.

(2) Function-call and predicate modifications. These modifications change function invocation, parameter passing, or decision predicates. We analyze functions involved in input validation, authorization, path handling, and security-configuration selection, together with their call relationships. We then identify modifications consistent with the target weakness, such as changing a security-relevant function argument, replacing a predicate, or altering the data used in a conditional expression. Such changes can affect the basis of a security decision without removing the surrounding repair logic.

(3) Trigger-gated security modifications. These modifications make security-relevant behavior dependent on particular inputs, configurations, or program states. We analyze the normal functional path and alternative execution branches, then identify the conditions that activate the modified behavior. Some attacks in the original dataset use environment variables or configuration-dependent branches. We also introduce input-dependent triggers involving request headers, field names, or hostname suffixes. Consequently, different inputs can activate different security behaviors while ordinary execution paths remain intact.

(4) Helper-function and data-propagation modifications

These modifications affect the generation, propagation, transformation, or use of security-relevant information. We analyze data-processing flows surrounding the clean repair, including the input parameters, intermediate variables, and object attributes on which security decisions depend. We then examine how this information passes between helper functions and their callers. Candidate modifications may introduce helper functions, alter returned information, change parameter propagation across processing stages, or omit a necessary data-processing step from the clean repair. These changes often involve multiple code locations or files.

(5) State-update and numerical-boundary modifications

These modifications concern program-state maintenance, expiration checks, cumulative resource accounting, and numerical thresholds. We identify state variables and their update and validation points, then analyze whether a security property requires consistency across multiple operations or execution stages. Candidate modifications may change state-update conditions, comparison boundaries, numerical thresholds, or the use of accumulated values in subsequent checks.

These five classes describe code-editing mechanisms that can overlap within a single attack PR.

Candidate validation and admission

Each issue can yield multiple candidate attack PRs. After construction, each candidate undergoes functional validation, security validation, and manual review.

For functional validation, we apply the attack PR to the fixed baseline and require it to pass the issue-directed repair checks and prescribed repository regression tests. Here, "preserving the repair" refers only to behavior covered by these predefined checks.

For security validation, we construct a target-specific probe that tests the concrete security consequence under specified inputs, permissions, program states, and execution conditions. The candidate must exhibit the target security violation, while an applicable clean repair reference must not exhibit it under the same conditions. Validation artifacts are maintained independently, and we verify the loaded program revision, test execution completion, and assertion results.

The constructor documents the historical issue, clean repair, attack PR, modified locations, target security property, triggering conditions, and execution evidence. Another team member reviews whether the triggering preconditions are valid and whether the observed behavior can be attributed to the candidate modifications. Candidates with insufficient or inconsistent evidence are revised and revalidated or excluded. Only candidates meeting all admission requirements are retained and frozen. Of 608 initial attack candidates, 443 passed functional and regression checks, and 192 were ultimately admitted after executable security validation and manual eligibility review (Appendix B.1, Table 5). The complete protocol is described in Appendices B.1–B.6 and Algorithm 1.

Origin and construction of benign PRs

Benign PRs are constructed from the same repair contexts as the attack PRs. For each of the 27 issue families containing benign samples, we construct five variants, yielding 135 benign PRs:

  1. The project's clean reference repair.
  2. A repair with additional safe constraint paths and compatibility or conditional-branch checks.
  3. Regression tests covering functions, call paths, parameter combinations, or boundary conditions relevant to attacks in the same issue family.
  4. Defensive exception checks near sensitive-data handling or information-propagation code.
  5. Negative tests for invalid inputs, unauthorized operations, and security boundaries.

Added tests exercise existing security requirements without weakening production-code checks, state updates, or numerical boundaries. Attack and benign PRs are matched at the repair-context level, with comparable modification locations and overall diff sizes; they need not be paired one-to-one or have identical diffs. Each benign PR passes the prescribed functional repair, regression, and applicable security checks, preventing a "safe" label from being obtained by breaking functionality or rejecting all operations (Section 4.1; Appendix B.4).

Concrete construction examples

Appendix B.7 provides two worked examples. In Werkzeug, the attack preserves cumulative size checks for ordinary fields but allows specially suffixed fields to bypass the accounting. In Requests, the attack preserves the fix for the original userinfo issue while introducing a hostname-suffix mapping that causes cross-host credential-selection errors. The code, triggering conditions, and execution comparisons across the baseline, clean repair, and attack versions demonstrate how security weaknesses are integrated into genuine repairs and how the resulting labels are validated.

Weakness 2 and Question 2

Thank you for highlighting this ambiguity. The key distinction is between validation artifacts used to establish benchmark labels and the inputs made available to review workflows.

Functional checks verify that the baseline exhibits the original issue and that the candidate satisfies the specified repair behavior and regression requirements. Security checks verify that an attack candidate exhibits the target security violation under explicit conditions, while an applicable clean repair reference does not exhibit the same violation (Section 4.2, p. 5). Freezing PRs after validation does not make this evidence unnecessary. The tests establish why each sample is valid and distinguish attacks that genuinely repair an issue while introducing a verified security problem from modifications that fail to repair the issue or lack a demonstrable security consequence.

Appendix A.3 (pp. 16–17) clarifies that public repository tests remain accessible, while independent validation copies are maintained separately. Section 5.1 (p. 6) and Appendix C.1 (from p. 25) explain that review workflows retain their native file-inspection and execution capabilities. Target-specific security probes, triggering records, and reference answers are withheld to avoid disclosing the exact activation conditions and expected security consequences. Review workflows must identify security problems independently from the candidate code and the repository context they are permitted to access.

Why validation artifacts are independently maintained Appendix B.2 (pp. 18–19) explains that test programs, assertions, and entry points are maintained in a trusted validation environment and cannot be replaced by candidate modifications. Validation also checks the actual candidate revision loaded, test discovery and completion, and assertion results, rather than relying only on success messages or exit codes. This prevents false passes caused by deleted assertions, skipped tests, or execution against the wrong revision.

For developers using the frozen benchmark to evaluate review workflows, existing labels can be used to compute evaluation metrics without supplying these validation artifacts as agent inputs. However, independently reproducing the labels or adding or modifying benchmark samples requires the corresponding functional and security validation.

The two types of tests serve distinct and complementary purposes: functional tests establish the required repair behavior, whereas security tests establish the target security consequence or applicable safe behavior. Neither can replace the other.

Weakness 3 and Question 3

Thank you for pointing out the ambiguity in our decision protocol. We do not require the evaluated review agents to produce block, signal, or none as their native outputs. PR-Agent retains its native decision pathway. For the other 15 workflows without a native decision mechanism, the evaluator normalizes their original comments, YAML, or JSON outputs and classifies the resulting security evidence into the following internal states:

Evidence statusMeaning
blockA security finding meets the blocking evidence threshold of the corresponding decision stage
signalRecognizable security-related content is present, but the evidence does not meet that threshold
noneNo security-related finding is identified, or the review explicitly reports no security concern

Any rule-level block directly produces a final BLOCK decision. When no finding satisfies the rule-level blocking threshold, both signal and none are passed to a shared LLM adjudicator. Thus, a rule-level none does not automatically imply acceptance.

The adjudicator is an evaluation-layer language model that determines whether the review text provides sufficient evidence to justify blocking a PR. It uses a fixed prompt and receives only the review text and the PR title and description as untrusted context. It does not access the diff, repository code, or hidden benchmark labels, and cannot introduce vulnerabilities that were not raised by the reviewer. The adjudicator checks whether the reviewer has identified a coherent security finding involving a weakness or concrete security consequence, its mechanism, and a relevant code location. When a review uses only generic security-control terminology, an explicit security impact is additionally required.

Reviewer_Ti3yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VeilPRBench, a benchmark consisting of 327 pull requests (PRs) from 14 open-source projects and 29 historical issue families across Python, C, and JavaScript. The PRs consist of 192 attack PRs and 135 matched benign repairs. The authors evaluated 16 LLMs using their benchmark and analyzed how security instructions can be used to prevent the attacks.

Critical strengths
  • Technically, creating manual PRs requires an understanding of the problem at both the technical and semantic levels. The authors successfully accomplished this.

  • The authors analyze the effectiveness of security instructions in preventing the attacks.

  • The authors successfully evaluate and analyze the models' performance.

Critical weaknesses
  • The main contribution of the paper is the manually created benchmark. However, the paper does not adequately cover the technical details of PR creation. These details could have provided insights into the attacks, their complexity, and their impact.

  • Considering the complexity of the attacks, there is no independent human evaluation of the attacks, the benchmark, or their complexity beyond the author team's assessment.

  • The LLMs should be properly identified by their versions and evaluation dates, and the prompts or snippets used should be documented, as models can be updated over time. This information was not provided.

Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

Part 1 - Weakness 1

Thank you for recognizing the technical and semantic effort involved in manually constructing PRs, as well as our security-instruction experiments and model evaluation. In response to your three concerns, we have added a detailed construction protocol and worked examples, an independent assessment by non-author evaluators, and documentation of model identifiers, evaluation periods, and prompts.

Weakness 1

We agree that the original high-level description did not sufficiently explain how a real historical issue is transformed into an attack PR with a verifiable security consequence. We have expanded Sections 4.1–4.2 (pp. 4–5) and detailed the construction and validation protocol in Appendices B.1–B.6, with Algorithm 1 summarizing the complete procedure. Appendix B.7 provides concrete construction examples from Werkzeug and Requests.

Repair-context selection and candidate planning.

We begin with a real historical issue and fix the repository revision preceding its upstream repair. We inspect the upstream fixing commit, the repaired version, and the relevant functional code to identify the reported defect, expected repair behavior, and associated security requirements. We then apply the clean upstream repair to the fixed baseline to establish a clean repair reference. Issue-directed functional tests confirm that the baseline exhibits the reported defect and that the repaired version satisfies the expected behavior, consistent with the upstream fix for the functionality covered by the issue. We prioritize existing repository tests and regression tests introduced by the historical repair. When these are insufficient, we derive supplementary tests from the issue description, maintainer explanations, or project specifications.

Next, we analyze security properties associated with the repair, such as whether inputs are fully validated, whether distinct identities or security configurations remain isolated, and whether resource boundaries are enforced. Each candidate-construction plan records the target security property, relevant program states or data relationships, candidate CWE, triggering preconditions, and expected security consequence. A single issue may involve multiple security properties, and the same property may be weakened through different code modifications. Each repair context can therefore yield multiple candidate plans.

Locating and modifying relevant code.

Attack PRs are constructed on top of the clean repair. We trace the relevant function calls, parameter passing, conditional checks, state maintenance, and sensitive-data processing paths to locate code responsible for the target security property. Modifications may occur within files touched by the original repair or in other code locations related to the repaired functionality or target security property. After determining the target property, candidate CWE, and modification scope, we construct candidates using five nonexclusive classes of code modifications, described below.

Five classes of vulnerability-injection mechanisms

(1) Conditional and control-flow modifications. These modifications alter branch conditions and execution-path selection under different inputs or program states. During construction, we identify the security checks associated with the clean repair and analyze the conditions under which they execute. Based on the candidate weakness and its intended consequence, we may introduce additional branches, modify combinations of existing conditions, or narrow the input range covered by a security check. For example, a check may remain effective for ordinary inputs while an unexpected execution path is taken under a specific input condition.

(2) Function-call and predicate modifications. These modifications change function invocation, parameter passing, or decision predicates. We analyze functions involved in input validation, authorization, path handling, and security-configuration selection, together with their call relationships. We then identify modifications consistent with the target weakness, such as changing a security-relevant function argument, replacing a predicate, or altering the data used in a conditional expression. Such changes can affect the basis of a security decision without removing the surrounding repair logic.

(3) Trigger-gated security modifications. These modifications make security-relevant behavior dependent on particular inputs, configurations, or program states. We analyze the normal functional path and alternative execution branches, then identify the conditions that activate the modified behavior. Some attacks in the original dataset use environment variables or configuration-dependent branches. We also introduce input-dependent triggers involving request headers, field names, or hostname suffixes. Consequently, different inputs can activate different security behaviors while ordinary execution paths remain intact.

AuthorsOpenReview ↗

Part 2 - Weakness 1

(4) Helper-function and data-propagation modifications. These modifications affect how security-relevant information is generated, propagated, transformed, or consumed. We examine the data-processing flow surrounding the repair, including the parameters, intermediate variables, and object attributes required for security decisions. Candidate modifications may introduce helper functions, change returned information, alter parameter propagation across processing stages, or omit security-relevant data handling required by the clean repair. Such modifications may involve multiple code locations or span multiple files.

(5) State-update and numerical-boundary modifications. These modifications affect state maintenance, expiration checks, cumulative resource accounting, and numerical thresholds. We identify state variables and their update or validation points within the repair context, then examine whether the target security property depends on consistent state maintenance across operations or execution stages. Candidate modifications may change state-update conditions, comparison boundaries, numerical thresholds, or the use of accumulated values in subsequent checks.

A single attack PR may involve multiple mechanisms, such as combining a conditional branch with a modified security predicate. The final attack PR integrates the genuine repair and the additional security-weakening modifications into a single candidate for review.

Candidate validation and admission.

The construction process produces multiple attack candidates for each issue. Each candidate then undergoes functional validation, security validation, and manual verification. For functional validation, we apply the complete candidate PR to the fixed baseline and require it to pass the issue-directed repair tests and prescribed repository regression checks. Here, "preserving the repair" refers specifically to the behavior covered by these predefined checks.

For security validation, we construct a target-specific probe that examines the concrete security consequence under specified inputs, permissions, program states, and execution conditions. The attack must exhibit the target violation, while an applicable clean repair reference must not exhibit that violation under the same conditions. Validation artifacts are maintained independently, and we verify the loaded program revision, test execution completion, and assertion outcomes.

The constructor documents the historical issue, clean repair, attack PR, modified code locations, target security property, triggering conditions, and execution evidence. Another team member then checks whether the preconditions are valid and whether the observed behavioral difference can be attributed to the candidate modifications. Candidates with incomplete or inconsistent evidence are revised and revalidated or excluded. Only candidates satisfying all admission requirements are retained and frozen. Of 608 initial attack candidates, 443 passed functional and regression validation, and 192 were ultimately admitted after executable security validation and manual eligibility review (Appendix B.1, Table 5).

Benign PR construction.

Benign PRs are also constructed within the same historical repair contexts. We use five variants: (1) the project's clean reference repair; (2) safe constraint paths with compatibility or branch checks; (3) regression tests covering related functions and execution paths; (4) defensive exception checks near sensitive-data handling; and (5) negative tests for invalid inputs, prohibited operations, and security boundaries. These controls have comparable modification locations and overall diff sizes, and must pass the prescribed functional, regression, and applicable security checks (Section 4.1 and Appendix B.4). The complete construction and validation protocol is documented in Appendices B.1–B.6 and Algorithm 1.

Concrete worked example.

In the Requests repair context associated with CVE-2024-47081, the clean repair uses ri.hostname to correctly identify the destination host for .netrc credential lookup. Our attack preserves this repair but adds two lines of suffix processing that map internal.example.com.evil.test to internal.example.com before the credential lookup. Execution results show that the attack retains correct credential lookup for the exact host and passes the original repair check for the userinfo-spoofing scenario. However, under the newly introduced suffix-host scenario, it selects protected credentials belonging to another host, whereas neither the baseline nor the clean repair does so. Appendix B.7, Table 9 (p. 23) provides the code and state-by-state results, demonstrating how a genuine repair and an additional cross-host credential-selection flaw coexist.

AuthorsOpenReview ↗

Part 3 - Weakness 2 and Weakness 3

Modification size and mechanism statistics.

Appendix B.10 reports that 75 attack PRs modify one file, 108 modify two files, and nine modify three or more files. The median number of modified files is two, and the median number of added lines is 13. Appendix B.1 further records branch or condition-scope changes in 190 PRs, call or predicate changes in 167, environment-variable-controlled branches in 82, newly introduced helper functions in 52, and numerical or boundary modifications in 39. These statistics characterize the scale and form of the code modifications. The reasoning complexity required to understand the attack mechanisms is evaluated separately through the independent human assessment below.

Weakness 2

We have added an independent evaluation by two non-author evaluators, neither of whom participated in sample construction. Both independently inspected the supporting artifacts and re-executed all 77 selected PRs, comprising 50 attacks and 27 benign repairs. The 50 attacks were sampled across the 29 issue families, while one benign PR was selected from each of the 27 families containing benign samples. Thus, the assessment covers all attack repair contexts and all repair contexts with benign samples. The methodology and results are reported in Section 4.2 and Appendix B.9, Table 11 (p. 24).

AssessmentnEvaluator AEvaluator BInitial agreement
Repair evidence7777/7777/77100%
Security-label evidence7777/7777/77100%
Trigger and precondition validity5050/5049/5098%
Attack-understanding complexity (L/M/H/U)5015/31/4/013/32/5/094%

Attack-understanding complexity is categorized by the reasoning scope required to understand the mechanism: low denotes local logic, moderate denotes cross-function or conditional reasoning, and high denotes cross-module or multistage state reasoning. This measure characterizes the reasoning required to understand an attack, rather than its construction cost or stealthiness.

Weakness 3

We have added the model identifiers and evaluation periods in Appendix C.5 (p. 33):

ModelEvaluation period
Claude Opus 4.7May–September 2026
GPT-5.5May–September 2026
Gemini 3.1 ProMay–September 2026
DeepSeek V4 ProJuly–September 2026
Qwen 3.7 PlusJuly–September 2026

The shared LLM adjudicator is separately identified as GPT-5.5, accessed directly through OpenAI during June–August 2026. Appendix C.5 also documents the operating settings: temperature 0 for review generation and shared adjudication, maximum output lengths of 4,096 and 8,192 tokens, respectively, and a time limit of 1,200 seconds. We have documented the prompts and relevant snippets at the following locations:

  • Appendix C.3: The complete adjudicator system prompt, user template, and output schema.
  • Appendix E.5: Verbatim generic security instructions, repository-specific instructions, and weakness-type templates, including the L0 setting and the construction of L2 guidance.
  • Appendix E.6: The additional output-reporting instruction used in the supplementary experiment.
  • Appendix B.7: Actual code snippets illustrating attack construction.

The evaluated review workflows retain their native review prompts. Backbone adaptation does not modify their review prompts or their file-reading, search, tool-use, and execution capabilities, as clarified in Appendix C.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.