- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes GRASP, a framework for secure code generation guided by actionable security principles. It organizes 214 OWASP principles into a directed acyclic graph containing specialization and prerequisite relationships. At inference time, GRASP selects relevant categories and iteratively applies principles through dependency-aware traversal and pruning. Experiments across five models and three benchmarks show improved joint security and correctness over unmodified models, although gains over competing methods vary across benchmarks.
Critical strengths
-
Improving the security of LLM-generated code is a timely and important research problem.
-
The paper is well written and generally easy to follow.
-
The method requires neither model fine-tuning nor an analyzer in the generation loop, making it applicable to both proprietary and open-weight models.
Critical weaknesses
-
The principles are extracted from the OWASP checklist, which already provides organized security guidance. The checklist is largely static and updated manually, which limits the flexibility and scalability of the resulting principle collection.
-
GRASP repeatedly evaluates principles and revises code, introducing additional model calls and token usage. Although Appendix J compares its cost with PromSec, comparisons with other strong baselines under matched inference budgets would help establish whether the improvements justify the additional computation.
-
The ablation studies need clearer explanations to establish the graph's contribution. Table 6 shows limited differences in output quality after removing the graph, suggesting that its main benefit is efficiency. However, the description of the graph-free variant could be more explicit about how principles are selected, ordered, and grouped during refinement. More detailed explanations and comparisons with simpler traversal or retrieval strategies would help clarify the necessity and novelty of the proposed graph design.
Questions for authors
-
The ablation results suggest that providing the selected principles in a single prompt performs worse than applying them iteratively. Could the authors provide concrete examples illustrating why the single-prompt approach fails and how iterative application resolves these failures? This would help distinguish the benefits of structured principle application from those of additional refinement rounds.
-
Which models perform category scoring, principle applicability judgments, and code revision? Are these roles handled by the same evaluated model? Could the authors clarify the total number of calls, including seed generation and category scoring, given the differing descriptions of calls and iterations?
5 comments
Q1. Why one prompt does worse, with an example. Take cwe-022/3-c, a path traversal task. The single-prompt variant gets 0 of 25 samples secure, and GRASP gets 11 of 25.
One trace shows how the graph’s ordering helps. OWASP_194, "validate and sanitize untrusted input and output", is a specialization of OWASP_002 and has OWASP_005 as a prerequisite, so the traversal cannot admit it until both have been visited. Round 1 applied OWASP_002 together with null-termination after copying and resource release. Round 2 considered OWASP_005 and applied nothing. Round 3 then admitted OWASP_194 and applied it, logged as "adding checks to prevent path traversal in the filename". The single prompt holds all of these at once and gets one edit to the seed program.
The table also separates the two effects you ask about. Adding rounds without the graph, moving from one prompt to w/o SCP graph at GRASP's budget, raises SR from 66.2 to 69.7 but lowers sp@1 from 53.6 to 51.4. Adding the graph at the same budget raises SR a further 3.7 points to 73.4 (CI 0.8 to 7.1) and sp@1 to 56.7. Paired over all 54 scenarios, GRASP beats one prompt on 21, loses on 9, and ties on 24, with SR 7.2 points higher (CI 3.2 to 11.5).
Q2. Which models do what, and the total calls. The evaluated model does all three jobs. Seed generation, category scoring, and principle application all go to the same model, and applicability and revision happen in one call per batch. The accounting is
total calls per sample = apply_iterations + 2
That is a seed call, a category-scoring call, and one call per batch of principles. The "refinement rounds" in Table 7 are these apply rounds, which is where the paper's descriptions diverged. For Qwen3, the measured mean is 6.1 apply rounds, so 8.1 calls per sample, and Appendix F's GPT5 mean works out to 9.53. We will state the formula and the per-model means in the revision, and give PromSec's count the same treatment in Appendix J.
Thank you again; your questions will make the ablations and cost accounting much clearer in the revision.
Dear Reviewer 83rc,
As the author-reviewer discussion period draws to a close, we would like to follow up and let you know that we have uploaded a revised manuscript incorporating the changes discussed in our response. For your convenience, all newly added content is highlighted in blue.
-
Claims. The abstract, introduction, contributions and conclusion now present the SCP Graph as the efficiency mechanism that makes applying 214 principles practical, reaching the same security in fewer refinement rounds, rather than as the source of the security gains.
-
New experiment. Appendix F gives the graph-free variant the same number of rounds as GRASP and nothing more. It still trails on every metric while inspecting more principles, separating what the graph contributes from the effect of running longer.
-
Ablations. Appendix F now describes each variant against the implementation.
-
Cost. Section 3.2, the Discussion and Appendix J state the per-sample call count and what it includes.
-
Prompts. Section 3.2 states that no edge is shown to the model, and that the graph acts only in the controller.
-
Post-cutoff CVEs. Section 4.4 is retitled and gives both cutoff dates, the selection criteria, and what the study does and does not measure.
-
Baselines. Section 4.1 specifies the RAG configuration, the RESCUE comparison is stated as a boundary rather than an average, and one-point gaps on a near-saturated CWEval are treated as ties.
-
Graph construction. Section 3.1 and Appendix C describe the pairwise judging accurately and report its cost.
-
Corpus. The Discussion explains that the pipeline does not depend on OWASP and that updates are incremental.
If there are any remaining points you would like us to address, please do not hesitate to let us know. Your feedback has made the paper's claims considerably sharper, and we would be glad to address anything further.
Thank you for the constructive review and the close reading of our ablations and call accounting. The security gains come from the proactiveness of the approach, meaning GRASP is driven by the secure coding obligations a task raises, not by a weakness that something has already found and named. No task is labeled with a weakness class, and nothing selects principles by one, which is why the knowledge source can be swapped.
W1. OWASP is static and hand-maintained. We agree it limits the corpus, and the pipeline does not depend on it. Normalization, edge detection, structural resolution and category attachment are automated and source-agnostic, so any checklist of code-level obligations will do. Updates are incremental too, since a new principle only has to be judged against the existing ones. CERT C or CERT Java is a rerun rather than a redesign, paid once per corpus; building the OWASP graph cost $2.70 in API fees, with the other two judges run locally. We will strengthen the Discussion here, and agree that a second source is the right way to show that the corpus can change while the method stays the same.
W2. The extra model calls. We reran the graph-free variant with a cap on its budget (w/o SCP graph in the table). It sees the same principles and keeps relevance scoring and refinement, and the cap is the number of rounds GRASP uses. This is a fresh run on Qwen3-Coder-30B, 25 samples for each of 50 scenarios, so 1,250 generations per arm, and intervals are 95% from a paired bootstrap.
Held to GRASP's budget, the graph-free arm falls 3.7 points of SR behind (CI 0.8 to 7.1) and 5.3 on the joint measure (CI 0.6 to 11.0). Uncapped, as in Table 7, it matches GRASP (74.9% SR, 57.4% sp@1) but needs 58% more calls and 89% more tokens.
The tokens column answers the cost question. The graph more than halves the bill against the pruning-off arm because it is selective, applying 9 principles per sample where that arm applies 17.7 for 31.5k tokens against GRASP's 14.3k, and it keeps a higher pass@1 than both the graph-free and the pruning-off arm. Pruning is what does this, by keeping irrelevant specializations away from code that already works. The cheapest setup is not the best either, since one prompt uses 6.7k tokens against GRASP's 14.3k and trails it by 7.2 points of SR. We will also add per-sample calls and tokens for the external baselines to the main table.
W3. What the ablations remove. Every variant keeps the same seed, category scoring, relevance threshold, batch size, and refinement prompt, and they differ only in how a batch forms. w/o SCP graph draws batches of up to
max_sizeprinciples at random from the selected set, without consulting prerequisite or specialization edges, so nothing enforces order and nothing prunes a specialization once its parent is judged inapplicable. This is the simplest traversal.w/o graph reasoning keeps every edge and the ordering, and disables only the rule that prunes a principle's children once that principle has been judged inapplicable. It is our most expensive variant at 15.3 calls. Its SR is slightly higher than GRASP’s (74.9 against 73.4), but GRASP is 8.2 points higher on pass@1 (CI 2.3 to 13.7) and 3.7 higher on sp@1, so the graph's effect on correctness comes from pruning rather than from the edge set as a whole. We will describe both at this level of detail in Appendix F.
On simpler strategies, RAG is already in the baselines table as single-shot TF-IDF retrieval over the identical corpus, trailing GRASP on sp@1, and we will add a retrieval arm at GRASP’s budget.