- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a benchmark, VeilPRBench, for evaluating the security of automated code agents, specifically they accept/refuse benign or malicious pull requests. The benchmark includes multiple open-source projects, across diverse programming languages. The benchmark was evaluated on 16 publicly available code reviews agents where review output has been analyzed for acceptance, rejection on benign and adversarial PRs.
Critical strengths
- The paper explores an important security problem in automated code review agents, broadly used for vibe-coding or by developers who fully automate their coding workflow.
- The paper proposes a benchmark for code review agents evaluation to analyze how they approve/reject benign/adversarial code changes and whether they can even identify potential malicious code updates automatically.
- The benchmark comprises 14 open-source projects with 135 benign repairs across C, python, and JavaScript.
- The proposed VeilPRBench was evaluated on 16 automated code review agents (baselines), reporting interesting findings that can make these agents more robust.
- The paper is well-written and flows nicely.
Critical weaknesses
- The paper’s scope and problem is very interesting, however, what distinguishes this benchmark from the typical code vulnerability benchmark used to detect malicious code logic in LLM generated code? I understand VeilPRBench is meant for workflow security evaluation but how can this not be done by simply taking one of the available and more covering LLM-generated code security benchmarks and evaluating them on the 16 code review agents evaluated in this paper? As a reader, I find it hard to understand the main novelty and scientific contributions of this paper. The authors should make this clear in the paper, probably by comparing VeilPRBench against existing benchmarks and highlighting what’s missing and what’s new in each.
- The threat model considered in the paper is somehow very strong and its practicality needs further justification. The attacker is assumed capable of bypassing PR provenance checks in the code review agents, which might not always be the case. Also, it’s unclear if a simple check of the PR provenance or even context can robustify those agents. Overall, the practicality of the threat model should be further justified since the VeilPRBench premise relies on it.
- The benchmark choices in terms of how many PRs, open-source projects, attacks, programming languages..etc should be also justified. The design choices must be justified to understand the degree of coverage and comprehensiveness, which is important for the benchmark’s significance.
- The evaluated attack's scalability is also unclear. 192 attacks seems very rigid, I wonder how attacks can be scaled to include sophisticated variants or even unseen attacks that can further solidify the security evaluation for code review agents.
Questions for authors
- Could you clarify the novelty of VeilPRBench compares to existing LLM-code vulnerability detection benchmarks?
- Could you justify and discuss of the threat model's practicality?
- Could you justify the design choices of the benchmark and discuss its extensibility?
- Could you discuss the attack scalability too? Could you provide further evaluation on strong LLM-generated vulnerable code?
5 comments
Part 2
Existing datasets can serve as construction materials, but merely feeding their vulnerability samples to the same 16 review agents does not automatically establish the repair task and its verification evidence described above. For data that do not already satisfy these conditions, one would still need to recover a fixed repository version and real issue, organize the sample as an applicable complete PR, verify the target repair and prescribed regression behavior, establish security probes and execution comparisons against an applicable clean reference, and prepare benign controls from the same repair context. For studies that already provide some of these elements, their artifacts can be reused rather than rebuilt from scratch; VeilPRBench's construction work lies in consistently integrating these requirements into comparable review instances. This design also allows us to examine identifying security problems separately from producing a blocking decision: we evaluate complete workflows, including native prompts, contextual file access, tool use, and decision pathways, while reporting both attack blocking and benign acceptance, so a system that rejects every change is not considered an effective reviewer. We have added the relationships to SWExploit, SWEADV, SEVRA-BENCH, PRWeaver, MalPR-Bench, and other studies in Section 2; Sections 3–4 clarify our task and admission criteria.
Weakness 2 and Question 2
Thank you for raising this concern. Our threat model does not grant attackers the ability to bypass PR provenance checks. Section 3 limits attacker capabilities to reading public source code, issues, and tests, and controlling the code changes, title, and description of the candidate PR under review. Appendix A.3 explicitly places the trusted baseline, protected branches, review configuration, and trusted validation artifacts outside the attacker's write permissions. Thus, submitting a candidate does not imply the ability to disable checks, modify protected branches, or merge the PR unilaterally. The applicable setting of our evaluation is that a candidate has satisfied a deployment's upstream admission requirements and reached its declared code-review interface; we then ask whether the review workflow can block the security problem within that candidate. This scenario does not require identity spoofing. For example, in a project that allows external contributions to enter review, a contributor can submit a PR using their own identity; verifying its origin does not directly establish that its changes preserve all security properties. If a deployment's provenance policy already rejects the candidate, that candidate never reaches the review stage studied here.
Provenance and contextual information are indeed valuable for defense. Prior work such as Anomalicious uses commit histories, repository metadata, and author-trust signals to identify anomalous or potentially malicious commits, showing the detection value of this information. However, such metadata-based screening evidence is not equivalent to proving that a provenance check guarantees the semantic security of code. Provenance screening can restrict which candidates enter the workflow, while code review examines whether modifications that enter it are secure. We also directly evaluate the effect of additional review context. The standard review input includes the PR diff, description, issue, and repository context permitted by the protocol, rather than isolated code snippets. Section 6.3 and Table 3 fix PR-Agent and compare three additional-guidance conditions across five backbones using all 192 attacks and 135 benign PRs:
| Condition | Additional guidance | Overall ABR | Overall BAP |
|---|---|---|---|
| L0 | No benchmark-specific security instruction | 29.2% | 93.0% |
| L1 | Generic security guidance | 50.7% | 89.3% |
| L2 | Joint repository- and weakness-informed guidance, replacing L1 | 55.0% | 85.2% |
Part 3
Generic security guidance increases ABR from 29.2% to 50.7%, a paired gain of 21.6 percentage points with a 95% issue-family bootstrap interval of [15.6, 27.4], while benign acceptance decreases from 93.0% to 89.3%. Under L2, ABR reaches 55.0%, but its additional gain over L1 is 4.3 percentage points, with an interval of [-1.2, 9.9]. These results support the conclusion that security guidance can improve blocking but does not eliminate the gap under the tested conditions. We also record triggering conditions separately from attacker permissions. In the original set, 82 PRs contain environment-variable-controlled branches, indicating explicit environmental preconditions for the vulnerability, not that a remote attacker can modify deployment configuration. Data-plane triggers need not require such permissions: for example, the Werkzeug case in Appendix B.7 uses a client-controlled field name to bypass resource accounting. Appendix B.1 clarifies the interpretation of environmental preconditions.
Weakness 3 and Question 3
Thank you for suggesting that we justify these design choices. VeilPRBench uses purposive sample construction: we select open-source repair tasks with meaningful real-world software relevance, inspectable repair code, and executable validation. Project selection considers both application context and verifiability. Projects such as Git, curl, Redis, SQLite, Requests, and urllib3 involve version control, network communication, data storage, and web infrastructure, providing different repair logic and security requirements. Admission to the construction process also requires a real historical issue, a fixed repository revision, an inspectable reference repair, and an execution environment capable of reproducing the problem and validating candidates. Python, C, and JavaScript provide different language and project contexts in the present collection, but coverage of three languages does not imply exhaustive coverage of language-specific security problems. Appendix B.1 adds the selection criteria; Appendix B.10 and Tables 12–14 report the project, issue-family, and CWE distributions.
The sample counts arise from construction and validation outcomes. We constructed 608 attack candidates; 443 passed issue-directed functional checks and prescribed regression checks; and 192 ultimately passed executable security validation and manual eligibility review. These samples cover 14 projects, 29 historical issue families, and 19 target CWE classes. We did not relax admission criteria to reach a predetermined attack count. For the 27 issue families containing benign samples, we constructed five types of benign PRs per family, yielding 135 benign samples spanning 13 projects. These samples test whether review systems can preserve valid repairs rather than merely block attacks. The overall composition is reported in Sections 4.1–4.3, admission counts in Appendix B.1 and Table 5, and benign construction in Appendix B.4.
The 19 CWE classes represent the target weaknesses covered by the current validated collection. Construction is organized around security properties implicated by repairs, including identity and access control, sensitive-information handling, input validation, path isolation, state maintenance, and resource boundaries. Each admitted attack must specify the violated requirement, triggering preconditions, and observable consequence, supported by an execution comparison with an applicable clean reference. For instance, CWE-200, CWE-22, and CWE-770 cover sensitive-information exposure, path traversal, and missing resource limits, respectively.
For extensibility, we organize construction around the repair context, target security property, and implementation mechanism of the security failure. A single issue can be associated with different security properties, and the same property can be violated through different code changes or triggering conditions. Therefore, 192 is the size of the current collection, not an upper bound imposed by the protocol. The additional experiment below directly examines whether other attack designs can be introduced into existing repair contexts.
Part 1
Thank you for pointing out the insufficient explanations of our benchmark construction process, validation artifacts, and evaluation settings. We have added the relevant details in the revised manuscript and address each concern below.
Weakness 1 and Question 1
Thank you for pointing out that the positioning of our contribution needs further clarification. The central question of VeilPRBench is: when a PR has already passed the repair checks for a real historical issue and the prescribed regression checks, can a complete code-review workflow still identify and block an execution-validated security problem within it? Our contribution lies in combining this task setting, sample-level evidence, and evaluation design. Specifically, each evaluation instance includes a real historical issue, a fixed repository baseline, and a complete candidate PR. An attack PR must satisfy two requirements simultaneously: first, the pre-repair baseline fails the issue-directed checks, whereas the version with the candidate applied passes the repair and prescribed regression checks; second, under specified inputs and execution preconditions, the candidate exhibits a target security violation that an applicable clean reference does not exhibit under the same conditions, with code inspection confirming the relationship between the violation and the candidate modifications. Benign PRs come from the same repair contexts and pass functional, regression, and applicable security checks. These requirements are described in Sections 3 and 4.1–4.2 and Appendices B.1–B.4.
We agree that existing benchmarks already provide some of these important elements. For example, PrimeVul includes paired vulnerable and patched versions, together with project, commit, and file context; CWEval uses both functional and security tests; and SWExploit studies patches that complete functional repairs while containing vulnerabilities. Thus, we do not characterize these studies as consisting only of isolated code snippets or ignoring functional correctness. The following table distinguishes their primary evaluation targets and the specific problem addressed by VeilPRBench.