acceptodds
Under review as a conference paper at ICLR 2027

Guided Reasoning with Actionable Security Principles for Secure Code Generation

Abstract

Large Language Models generate code that is frequently insecure, and existing defenses are vulnerability-centric: they teach models what not to do, by fine-tuning on vulnerable and fixed code pairs, by retrieving records of past vulnerabilities, or by revising drafts under static analyzer feedback. Such defenses depend on curated vulnerability data, generalize poorly to weaknesses absent from it, and are often impractical for proprietary models. We take a principle-centric direction instead: rather than describing past vulnerabilities, we condition generation on prescriptive Secure Coding Principles that state how secure code should be written. We present GRASP, which organizes 214 such principles into a directed acyclic graph capturing specialization and prerequisite dependencies, and traverses it at inference time, selecting the principles relevant to a task and applying them in dependency order while pruning branches that do not arise. GRASP requires no training, no weight access and no analyzer in the generation loop, so it applies to proprietary and open-weight models alike. Across five models we raise the Security Rate from 0.46–0.55 at baseline to 0.70–0.84, outperform zero-shot, planning-based, retrieval-augmented and analyzer-guided prompting on secure-pass@k on our dataset, and improve robustness on unseen vulnerabilities, measured on real CVEs disclosed after the evaluated models' training cutoffs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
secure code generation, large language models, secure coding principles, structured reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_83rcOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GRASP, a framework for secure code generation guided by actionable security principles. It organizes 214 OWASP principles into a directed acyclic graph containing specialization and prerequisite relationships. At inference time, GRASP selects relevant categories and iteratively applies principles through dependency-aware traversal and pruning. Experiments across five models and three benchmarks show improved joint security and correctness over unmodified models, although gains over competing methods vary across benchmarks.

Critical strengths
  1. Improving the security of LLM-generated code is a timely and important research problem.

  2. The paper is well written and generally easy to follow.

  3. The method requires neither model fine-tuning nor an analyzer in the generation loop, making it applicable to both proprietary and open-weight models.

Critical weaknesses
  1. The principles are extracted from the OWASP checklist, which already provides organized security guidance. The checklist is largely static and updated manually, which limits the flexibility and scalability of the resulting principle collection.

  2. GRASP repeatedly evaluates principles and revises code, introducing additional model calls and token usage. Although Appendix J compares its cost with PromSec, comparisons with other strong baselines under matched inference budgets would help establish whether the improvements justify the additional computation.

  3. The ablation studies need clearer explanations to establish the graph's contribution. Table 6 shows limited differences in output quality after removing the graph, suggesting that its main benefit is efficiency. However, the description of the graph-free variant could be more explicit about how principles are selected, ordered, and grouped during refinement. More detailed explanations and comparisons with simpler traversal or retrieval strategies would help clarify the necessity and novelty of the proposed graph design.

Questions for authors
  1. The ablation results suggest that providing the selected principles in a single prompt performs worse than applying them iteratively. Could the authors provide concrete examples illustrating why the single-prompt approach fails and how iterative application resolves these failures? This would help distinguish the benefits of structured principle application from those of additional refinement rounds.

  2. Which models perform category scoring, principle applicability judgments, and code revision? Are these roles handled by the same evaluated model? Could the authors clarify the total number of calls, including seed generation and category scoring, given the differing descriptions of calls and iterations?

5 comments

Thank you for the constructive review and the close reading of our ablations and call accounting. The security gains come from the proactiveness of the approach, meaning GRASP is driven by the secure coding obligations a task raises, not by a weakness that something has already found and named. No task is labeled with a weakness class, and nothing selects principles by one, which is why the knowledge source can be swapped.

W1. OWASP is static and hand-maintained. We agree it limits the corpus, and the pipeline does not depend on it. Normalization, edge detection, structural resolution and category attachment are automated and source-agnostic, so any checklist of code-level obligations will do. Updates are incremental too, since a new principle only has to be judged against the existing ones. CERT C or CERT Java is a rerun rather than a redesign, paid once per corpus; building the OWASP graph cost $2.70 in API fees, with the other two judges run locally. We will strengthen the Discussion here, and agree that a second source is the right way to show that the corpus can change while the method stays the same.

W2. The extra model calls. We reran the graph-free variant with a cap on its budget (w/o SCP graph in the table). It sees the same principles and keeps relevance scoring and refinement, and the cap is the number of rounds GRASP uses. This is a fresh run on Qwen3-Coder-30B, 25 samples for each of 50 scenarios, so 1,250 generations per arm, and intervals are 95% from a paired bootstrap.

armcallstokensprinciplesSRp@1sp@1
base1.00.6k051.4%91.0%46.9%
selected principles in one prompt3.06.7k12.066.2%79.4%53.6%
GRASP8.114.3k9.073.4%77.6%56.7%
w/o SCP graph7.813.9k10.569.7%73.8%51.4%
w/o graph reasoning, pruning off15.331.5k17.774.9%69.4%53.0%

Held to GRASP's budget, the graph-free arm falls 3.7 points of SR behind (CI 0.8 to 7.1) and 5.3 on the joint measure (CI 0.6 to 11.0). Uncapped, as in Table 7, it matches GRASP (74.9% SR, 57.4% sp@1) but needs 58% more calls and 89% more tokens.

The tokens column answers the cost question. The graph more than halves the bill against the pruning-off arm because it is selective, applying 9 principles per sample where that arm applies 17.7 for 31.5k tokens against GRASP's 14.3k, and it keeps a higher pass@1 than both the graph-free and the pruning-off arm. Pruning is what does this, by keeping irrelevant specializations away from code that already works. The cheapest setup is not the best either, since one prompt uses 6.7k tokens against GRASP's 14.3k and trails it by 7.2 points of SR. We will also add per-sample calls and tokens for the external baselines to the main table.

W3. What the ablations remove. Every variant keeps the same seed, category scoring, relevance threshold, batch size, and refinement prompt, and they differ only in how a batch forms. w/o SCP graph draws batches of up to max_size principles at random from the selected set, without consulting prerequisite or specialization edges, so nothing enforces order and nothing prunes a specialization once its parent is judged inapplicable. This is the simplest traversal.

w/o graph reasoning keeps every edge and the ordering, and disables only the rule that prunes a principle's children once that principle has been judged inapplicable. It is our most expensive variant at 15.3 calls. Its SR is slightly higher than GRASP’s (74.9 against 73.4), but GRASP is 8.2 points higher on pass@1 (CI 2.3 to 13.7) and 3.7 higher on sp@1, so the graph's effect on correctness comes from pruning rather than from the edge set as a whole. We will describe both at this level of detail in Appendix F.

On simpler strategies, RAG is already in the baselines table as single-shot TF-IDF retrieval over the identical corpus, trailing GRASP on sp@1, and we will add a retrieval arm at GRASP’s budget.

Q1. Why one prompt does worse, with an example. Take cwe-022/3-c, a path traversal task. The single-prompt variant gets 0 of 25 samples secure, and GRASP gets 11 of 25.

One trace shows how the graph’s ordering helps. OWASP_194, "validate and sanitize untrusted input and output", is a specialization of OWASP_002 and has OWASP_005 as a prerequisite, so the traversal cannot admit it until both have been visited. Round 1 applied OWASP_002 together with null-termination after copying and resource release. Round 2 considered OWASP_005 and applied nothing. Round 3 then admitted OWASP_194 and applied it, logged as "adding checks to prevent path traversal in the filename". The single prompt holds all of these at once and gets one edit to the seed program.

The table also separates the two effects you ask about. Adding rounds without the graph, moving from one prompt to w/o SCP graph at GRASP's budget, raises SR from 66.2 to 69.7 but lowers sp@1 from 53.6 to 51.4. Adding the graph at the same budget raises SR a further 3.7 points to 73.4 (CI 0.8 to 7.1) and sp@1 to 56.7. Paired over all 54 scenarios, GRASP beats one prompt on 21, loses on 9, and ties on 24, with SR 7.2 points higher (CI 3.2 to 11.5).

Q2. Which models do what, and the total calls. The evaluated model does all three jobs. Seed generation, category scoring, and principle application all go to the same model, and applicability and revision happen in one call per batch. The accounting is

total calls per sample = apply_iterations + 2

That is a seed call, a category-scoring call, and one call per batch of principles. The "refinement rounds" in Table 7 are these apply rounds, which is where the paper's descriptions diverged. For Qwen3, the measured mean is 6.1 apply rounds, so 8.1 calls per sample, and Appendix F's GPT5 mean works out to 9.53. We will state the formula and the per-model means in the revision, and give PromSec's count the same treatment in Appendix J.

Thank you again; your questions will make the ablations and cost accounting much clearer in the revision.

Dear Reviewer 83rc,

As the author-reviewer discussion period draws to a close, we would like to follow up and let you know that we have uploaded a revised manuscript incorporating the changes discussed in our response. For your convenience, all newly added content is highlighted in blue.

  1. Claims. The abstract, introduction, contributions and conclusion now present the SCP Graph as the efficiency mechanism that makes applying 214 principles practical, reaching the same security in fewer refinement rounds, rather than as the source of the security gains.

  2. New experiment. Appendix F gives the graph-free variant the same number of rounds as GRASP and nothing more. It still trails on every metric while inspecting more principles, separating what the graph contributes from the effect of running longer.

  3. Ablations. Appendix F now describes each variant against the implementation.

  4. Cost. Section 3.2, the Discussion and Appendix J state the per-sample call count and what it includes.

  5. Prompts. Section 3.2 states that no edge is shown to the model, and that the graph acts only in the controller.

  6. Post-cutoff CVEs. Section 4.4 is retitled and gives both cutoff dates, the selection criteria, and what the study does and does not measure.

  7. Baselines. Section 4.1 specifies the RAG configuration, the RESCUE comparison is stated as a boundary rather than an average, and one-point gaps on a near-saturated CWEval are treated as ties.

  8. Graph construction. Section 3.1 and Appendix C describe the pairwise judging accurately and report its cost.

  9. Corpus. The Discussion explains that the pipeline does not depend on OWASP and that updates are incremental.

If there are any remaining points you would like us to address, please do not hesitate to let us know. Your feedback has made the paper's claims considerably sharper, and we would be glad to address anything further.

Reviewer_RX7AOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GRASP encodes 214 secure coding principles into a directed acyclic graph, and traverses the graph by selecting principles relevant to the task and applying them in dependency order. The edges of the graph capture specialization, which specifies a broad principle to a more concrete principle, and prerequisite, which captures ordering constraints. The graph and its edges are constructed by LLMs. Given seed code, the model traverses the graph and applies principles to refine the code. There is also a 54-prompt benchmark with unit tests and vulnerability tests. Across five models, GRASP improves the Security Rate from 0.46-0.55 to 0.70-0.84. On GPT-5, GRASP outperforms zero-shot, planning, RAG, and analyzer-guided baselines on secure-pass@k on the authors' dataset.

Critical strengths
  1. GRASP is model agnostic and requires no training.
  2. GRASP deals with secure code generation in an interesting way by structuring using OWASP principles in a DAG.
Critical weaknesses
  1. The method is hard to distinguish from agentic harnesses, iterative prompting, or RAG. The main contribution is the principle graph but the ablation in Tables 6-7 shows that the graph does not improve the Security Rate. The RAG baseline is unspecified. Section 4.1 describes the baseline as "retrieval-augmented prompting over security documentation", but what the documentation is and how it's stored and retrieved is unclear.
Questions for authors
  1. Since removing the graph does not change secure-pass@1 or Security Rate (Table 7), what evidence shows the graph contributes beyond reducing the number of iterative refinement rounds? How does GRASP compare to a RAG or iterative baseline over the same 214 principles with the same number of LLM calls?
  2. What does the model see in each GRASP prompt? Is any GRASP edge information shown to the model?
  3. How is the RAG baseline configured? Does it retrieve from the same OWASP principles that GRASP is constructed with?
  4. Why are there only two relation types?
  5. What was the graph construction cost in tokens, $?
  6. How much of the GRASP samples that were considered secure (under Security Rate), passed their unit tests? What's the Security Rate for functionally correct samples?
  7. How many of the secure outputs validated manually or with dynamic execution?
  8. What sample size was used for the ablations (Tables 6-9)?
4 comments

Thank you for the detailed reading. You are right that Tables 6-7, as presented, do not show the graph improving security, and we will revise our claims accordingly. The security gains come from the proactiveness of the approach, meaning that GRASP is driven by the secure coding obligations a task raises, not by a weakness that something has already found and named. No task is ever labeled with a weakness class, nothing selects or filters principles by one, and no weakness label reaches any prompt the model sees. There is no detection step, no vulnerability-fix corpus, and no analyzer. The graph is how we make that practical over 214 principles, by ordering and pruning their application.

W1, Q1. What the graph adds, and the matched-call baseline. The w/o SCP graph arm is the iterative baseline you ask for, with the same 214 principles and the same scoring and refinement but no graph. We re-ran the ablation with that arm held to GRASP's call budget, which is the comparison Tables 6-7 do not make. This is a fresh run on Qwen3-Coder-30B, 25 samples on each of 50 scenarios, so 1,250 generations per arm, and intervals are 95% from a paired bootstrap. In the table, the w/o SCP graph is the capped arm.

armcallstokensPrinciples appliedSRp@1sp@1
base1.00.6k051.4%91.0%46.9%
GRASP8.114.3k9.073.4%77.6%56.7%
w/o SCP graph7.813.9k10.569.7%73.8%51.4%
w/o graph reasoning, pruning off15.331.5k17.774.9%69.4%53.0%

At a matched budget, the graph is worth 3.7 points of SR (CI 0.8 to 7.1) and 5.3 on the joint measure (CI 0.6 to 11.0). Uncapped, as in Table 7, the graph-free arm matches GRASP (74.9% SR, 57.4% sp@1) but needs 58% more calls and 89% more tokens, so the graph’s value is reaching the same security at about half the cost rather than a higher ceiling.

The graph is also selective about what gets applied. GRASP applies 9.0 principles per sample, where the arm with pruning switched off applies 17.7 and drops to 69.4% on pass@1 against GRASP's 77.6% (CI 2.3 to 13.7). The graph-free arm applies 10.5, close to GRASP, yet its pass@1 is still lower at 73.8, so which principles are applied matters, not only how many.

On the comparison you ask for, our RAG arm makes a single call, so it is not budget-matched yet, and we will add a retrieval arm at GRASP's budget. The baselines table already shows it trailing GRASP on sp@1. What separates GRASP from an agentic harness is not the loop alone but how the principles are scheduled within it: delivering the same principles in one prompt is 7.2 points of SR worse, paired over all 54 scenarios (CI 3.2 to 11.5), and the same loop without the graph is 3.7 points worse at a matched budget.

Q2. No edge information reaches the model. Each prompt holds the task, the current code, the texts of the principles in the current batch, and the instructions. No edge and no relation between principles ever appears in it, which we checked across all 8,207 apply prompts in this run. The graph acts only in the controller, deciding which principles are considered, in what order, and which are pruned. Keeping edges out of the prompt is deliberate, since it keeps prompts short and the method model-agnostic, and we will state it in Section 3.2.

Q3. RAG configuration. It uses TF-IDF retrieval (scikit-learn, English stop words) with cosine similarity against the task description, takes the top 5 principles, and places them in a fixed eight-step prompt. It makes one call and does not iterate. It retrieves from the same 214 OWASP principles the graph is built from, so the two differ only in how they use the same knowledge. Retrieval supplies the documentation once, where GRASP keeps applying obligations as the program changes. We will add this configuration to the revision.

Q4. Why only two relations. Each relation drives one traversal operation. Specialization narrows scope, so an inapplicable parent prunes its children. Prerequisite fixes order, so a principle is applied only after the ones it depends on. A relation that drove neither operation would add edges to the graph without changing which principles are selected, the order they are applied in, or which ones are pruned, so the traversal would run exactly as it does now. That is why we stopped at two.

Q5. Graph construction cost. Judging all 22,791 principle pairs is almost the whole cost. Each pair goes to three judges, so 68,373 calls, which we measure at 107.2M input and 33.5M output tokens. Only one judge is paid, gpt-5-nano through the Batch API at 35.7M input and 9.0M output tokens, which is $2.70 at current Batch list prices of $0.025 and $0.20 per 1M tokens. The other two run locally, so they add no API cost. This is paid once per corpus, and the graph ships with the artifact. The cost per sample at inference is the tokens column above.

Q6. Secure samples that also work. In the run above, 77.3% of the samples GRASP calls secure also pass their tests, compared with 91.1% for base. Among functionally correct samples, SR is 0.73 for GRASP against 0.52 for base. Asking for both at once is the sp@1 column, where GRASP leads base by 9.8 points (CI 1.1 to 19.1). The pass@1 cost is concentrated in Python tasks whose secure form swaps out an API, which we discuss in our reply to Reviewer 9sd7. We will report SR, pass@1 and sp@1 together in the main tables.

Q7. Validation of secure outputs. On CWEval and BaxBench, security is judged dynamically by executing each benchmark’s own security tests. On our dataset and the zero-day set, it is static analysis, as the paper states, using CodeQL with per-scenario queries plus Bandit and Cppcheck. Functional correctness is always decided by running the code. We did not validate outputs by hand.

Q8. Sample size. Every row of Tables 6-9 is 25 samples on each of 54 scenarios, so 1,350 generations per arm, and the p-values in Table 7 pair arms by scenario, so n is 54 there. The table above uses 50 of those scenarios, excluding four whose functional tests do not respond to the generated code and so cannot report a true result either way.

We hope these results clarify the graph’s role and the baseline setup. We are happy to answer any further questions during the discussion, and would be grateful if you would take these clarifications into account in your assessment.

Thank you for the responses. The new matched-budget experiment, the RAG configuration, the cost breakdown, and the clarifications on prompts, validation and sample sizes answer most of my questions. My main concern is that the graph is the paper’s central contribution, but the evidence shows it improves efficiency rather than security, with only a modest quality gain at a matched budget on one model. Could you revise the paper to frame the graph as an efficiency mechanism rather than as the source of the security gains? The ablations attribute the security gains to supplying and iteratively applying the selected principles, not to the graph itself.

Dear Reviewer RX7A,

As the author-reviewer discussion period draws to a close, we would like to follow up and let you know that we have uploaded a revised manuscript incorporating the changes discussed in our response. For your convenience, all newly added content is highlighted in blue.

  1. Claims. The abstract, introduction, contributions and conclusion now present the SCP Graph as the efficiency mechanism that makes applying 214 principles practical, reaching the same security in fewer refinement rounds, rather than as the source of the security gains.

  2. New experiment. Appendix F gives the graph-free variant the same number of rounds as GRASP and nothing more. It still trails on every metric while inspecting more principles, separating what the graph contributes from the effect of running longer.

  3. Ablations. Appendix F now describes each variant against the implementation.

  4. Cost. Section 3.2, the Discussion and Appendix J state the per-sample call count and what it includes.

  5. Prompts. Section 3.2 states that no edge is shown to the model, and that the graph acts only in the controller.

  6. Post-cutoff CVEs. Section 4.4 is retitled and gives both cutoff dates, the selection criteria, and what the study does and does not measure.

  7. Baselines. Section 4.1 specifies the RAG configuration, the RESCUE comparison is stated as a boundary rather than an average, and one-point gaps on a near-saturated CWEval are treated as ties.

  8. Graph construction. Section 3.1 and Appendix C describe the pairwise judging accurately and report its cost.

  9. Corpus. The Discussion explains that the pipeline does not depend on OWASP and that updates are incremental.

We would sincerely appreciate it if you would consider revisiting your score if these additions have addressed your concerns. If there are any remaining points you would like us to address, please do not hesitate to let us know. Your feedback has made the paper's claims considerably sharper, and we would be glad to address anything further.

Reviewer_9sd7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GRASP, a principle-centric approach to secure code generation. It organizes 214 OWASP secure coding principles into a directed acyclic SCP Graph with specialization and prerequisite edges, and uses a dependency-aware, iterative reasoning procedure to select, apply, and prune task-relevant principles at inference time. The method is evaluated on five LLMs, three benchmarks, five baselines, two fine-tuned checkpoints, and 20 real CVEs.

Critical strengths

1.The principle-centric operationalization is clear, model-agnostic, and interpretable. GRASP turns secure coding principles into actionable obligations, organizes them into a dependency graph, and leaves a per-obligation trace of what was applied, skipped, or pruned. This is consequential because it offers a genuine alternative to vulnerability-centric tuning and retrieval: the reasoning unit is a prescriptive obligation rather than a past exploit pattern, and the method requires no weight access or retraining. The interpretability advantage over opaque tuning and over post-hoc critique methods is real and worth preserving.

2.The empirical package is unusually broad for this problem, and the authors report inconvenient results. Five LLMs, three benchmarks, five baselines, two fine-tuned checkpoints, and 20 CVEs is a wider evaluation than several closely related papers. The post-cutoff CVE study directly addresses the memorization concern, and the fine-tuning complementarity analysis answers a question that retrieval-based and agentic secure-generation work has not systematically treated. Reporting the graph ablation as a null result on quality, rather than hiding it, is a mark of research integrity and should be credited.

Critical weaknesses

1.The core SCP Graph contribution is not empirically supported on quality, which creates a mismatch between the paper’s central claim and its evidence. In Table 7, removing the SCP graph leaves SR essentially unchanged (0.84 vs. 0.82, p = 0.845) and sp@1 unchanged (0.61 vs. 0.59, p = 0.169); the graph mainly reduces refinement rounds from 10.65 to 7.53. Since the graph is one of the paper’s two central components, this result materially weakens the claim that dependency-aware graph structure is necessary for the observed security gains. The paper should either recalibrate the graph’s claimed contribution as primarily efficiency-oriented, or provide stronger evidence that the graph improves quality across models and benchmarks. This is consequential to the acceptance decision because closely related work—RESCUE at ICLR 2026 and SCGAgent (Saul et al., arXiv:2506.07313)—attributes its core gains to a retrieval or agentic component whose quality contribution is directly demonstrated. If GRASP’s graph is a quality-neutral efficiency mechanism, the paper’s contribution should be framed accordingly; if it is claimed as a quality mechanism, the current evidence is insufficient.

2.The external validity and statistical strength of the evaluation are limited relative to the strength of the generalization claims. The primary security oracle is static-analysis-based; the zero-day study covers only six CVEs post-cutoff for both models, has no unit tests, and reports SR only; the self-curated set has 54 prompts; and external benchmark results are mixed: on BaxBench, GRASP’s sp@1 is 0.37 versus 0.40 for zero-shot security prompting and 0.38 for RESCUE, and CWEval is near-saturated (GRASP’s sp@1 of 0.93 sits below PaS/RAG at 0.94). In addition, functional correctness drops in some settings, e.g., GPT4o Python p@1 from 0.83 to 0.70, so the security–functionality trade-off needs more explicit treatment. These are addressable but material concerns for the strength of the generalization claims, especially because the paper positions GRASP as a general principle-driven alternative to retrieval and tuning.

Questions for authors

1.Graph ablation across models and benchmarks. The current ablation covers two of the five models (GPT5 and Qwen3, Table 6). Can you extend the SCP-graph ablation to the remaining models and to the external benchmarks, with confidence intervals? If the graph mainly improves efficiency rather than quality, will you revise the central claims accordingly? This question is material because it determines whether the paper’s two-component framing is supported or whether the graph should be repositioned as an efficiency mechanism.

2.Zero-day statistical rigor. For the zero-day CVE study, can you provide confidence intervals, per-CVE variance, and the exact selection criteria for the 6/20 CVEs? How robust is the conclusion given the small sample and the absence of functional tests? A point estimate of 0.46 versus 0.25 is suggestive, but without intervals it is difficult to assess whether the advantage is stable.

3.Graph error propagation. The graph construction uses LLM judges and a single-rater manual audit (87% edge correctness). Can you provide an error-propagation analysis showing how injected edge errors affect reasoning rounds and final SR? This would help separate the graph’s efficiency value from its correctness value, and would strengthen the case for the graph even if its quality contribution is null.

4.Direct comparison with retrieval and agentic secure-generation baselines. RESCUE reports consistent SecurePass@1 gains across four benchmarks, and SCGAgent (Saul et al., arXiv:2506.07313) reports roughly 25% security improvement while maintaining 98% functionality. Can you clarify under what conditions GRASP is preferable to these approaches, and under what conditions it is not? Aggregate numbers are not sufficient; the empirical boundary matters for assessing the principle-centric claim.

5.Security–functionality trade-off. GPT4o Python p@1 falls from 0.83 to 0.70, and Qwen2.5 Python p@1 falls from 0.71 to 0.46. Can you report functional cost alongside security gains in the main tables and discuss the sampling budget under which this cost is acceptable? This is important because secure generation methods are often criticized for trading away too much correctness.

6.Hyperparameter fairness. The paper uses τ = 3 and max_size = 8 as model-agnostic defaults even though Appendix F shows they are not optimal for every model. How sensitive are the main conclusions to per-model tuning, and do the baselines receive comparable tuning? Fairness in tuning is relevant to the strength of the comparative claims.

4 comments

Thank you for the careful reading. We adopt your framing of the graph. The security gains come from the proactiveness of the approach, meaning GRASP is driven by the secure coding obligations a task raises, not by a weakness that something has already found and named. The graph is how we make that work over 214 principles by ordering and pruning their application, and we will present it that way rather than as a source of higher converged security.

W1, Q1. What the graph is worth. Tables 6-7 do not hold the two arms to the same budget, so we re-ran the ablation with the graph-free arm capped at GRASP's calls (w/o SCP graph in the table). This is a fresh run on Qwen3-Coder-30B, 25 samples for each of 50 scenarios, so 1,250 generations per arm, and intervals are 95% from a paired bootstrap.

armcallstokensprinciplesSRp@1sp@1
base1.00.6k051.4%91.0%46.9%
GRASP8.114.3k9.073.4%77.6%56.7%
w/o SCP graph7.813.9k10.569.7%73.8%51.4%
w/o graph reasoning, pruning off15.331.5k17.774.9%69.4%53.0%

Held to the same budget, the graph-free arm loses 3.7 points of SR (CI 0.8 to 7.1) and 5.3 on the joint measure (CI 0.6 to 11.0). Uncapped, as in Table 7, it matches GRASP (74.9% SR, 57.4% sp@1) but needs 58% more calls and 89% more tokens. So the null in Table 7 is real at convergence, and the graph’s value is reaching that level at about half the cost.

The graph is also selective. The arm with pruning switched off applies 17.7 principles against GRASP's 9.0 and drops to 69.4% on pass@1 against 77.6%. The graph-free arm applies 10.5, close to GRASP, yet stays lower at 73.8, so which principles are applied matters more than how many. Pruning is the part that carries this. Against the variant that keeps every edge and disables only pruning, GRASP uses 47% fewer calls and 55% fewer tokens and is 8.2 points better on pass@1 (CI 2.3 to 13.7).

Yes, we will revise the central claims. In the abstract, introduction, and contributions, we will say that the graph orders and prunes the work, while the principles do the securing, and we will remove any claim that the graph is necessary for the security gains. Two limits remain. The capped arm is Qwen3, while Table 7 reports GPT5, and the other three models and the external benchmarks have not run yet. We will add all of them, with intervals.

W2. External validity. Correctness is decided by running the code. Security is measured with CodeQL, Bandit, and Cppcheck on our dataset and the zero-day set, and on CWEval and BaxBench it is judged dynamically, by executing each benchmark's own security tests. We accept the limitation you raise. With 54 prompts, and with CWEval close to saturated, a gap such as 0.93 against 0.94 is a tie, and we will treat gaps of that size as ties. On BaxBench, GRASP does not lead (0.37 against 0.40 for zero-shot security prompting and 0.38 for RESCUE), and we will report it that way. Its tasks hinge on framework-specific APIs, which is where library-specific knowledge wins (see Q4).

Q2. Zero-day rigor. Each model is tested only on CVEs disclosed after its own cutoff, so 20 CVEs postdate GPT4o's cutoff and 6 of those also postdate GPT5's, which is what Table 3 aggregates. We will state both dates. Resampling CVEs, GRASP's lead over zero-shot holds on both models. Over RESCUE, it holds on GPT5 but not GPT4o, where the interval runs from -20 to +65, so that claim becomes model-specific. We will report these intervals alongside the per-CVE results in Appendix H.

On the selection criteria, you are right. We kept only CVEs that the base model still reproduces, which floors the base by construction. The study measures hardening on CVEs the model is known to reproduce, not detection of unseen ones, and we will label it that way.

Q3. Graph errors. We corrupted 25% of the 646 prerequisite and specialization edges and reran GRASP untouched, leaving category-membership links alone. Every quality interval spans zero, so output quality is robust to construction errors well beyond the 13% error rate in our audit. Call counts move both ways, deletion costing more and rewiring and flipping less, so the edges matter for cost rather than for the final security level, which is the framing above. CWEval agrees across the nine paired tests there, bar one favoring rewiring that would not survive a multiplicity correction.

Q4. Retrieval and agentic baselines. The per-CVE numbers are in Section 4.4 and Appendix H, and the pattern is the answer. GRASP leads RESCUE where the weakness names a transferable obligation such as validating untrusted input, and RESCUE leads where the fix turns on one library's behavior. The same boundary fits BaxBench, whose tasks turn on framework-specific APIs. We will state that boundary rather than an average. We could find no public code or corpus for SCGAgent, so we will cite it and say why. It keys retrieval to predicted CWEs, where GRASP uses none.

Q5. Security against correctness. In the run above, correctness falls 13.4 points against the base (CI 7.8 to 19.7), while SR rises 22.0, and asking for both still favors GRASP by 9.8 points (CI 1.1 to 19.1). The cost is concentrated. Almost all of it is Python tasks where the secure form means swapping out an API, which matches the drops you cite, both on Python (GPT4o 0.83 to 0.70, Qwen2.5 0.71 to 0.46), while on C, in Table 1, it is 5 points at pass@1 and nothing at pass@5. We will add pass@1 to the ablation table, name that failure mode, and note the gap closes by pass@25 while the security gain holds.

Q6. Hyperparameter fairness. GRASP ran one untuned setting, tau = 3 and max_size = 8, for every model and benchmark, and baselines ran at their papers' defaults. The Appendix F sweeps cover GRASP alone, so we claim no parity. They are flat, and on GPT-5 a lower threshold scores higher, so tau = 3 is a balanced default rather than a per-model optimum.

Thank you again; these questions have substantially improved the paper.

Dear Reviewer 9sd7,

As the author-reviewer discussion period draws to a close, we would like to follow up and let you know that we have uploaded a revised manuscript incorporating the changes discussed in our response. For your convenience, all newly added content is highlighted in blue.

  1. Claims. The abstract, introduction, contributions and conclusion now present the SCP Graph as the efficiency mechanism that makes applying 214 principles practical, reaching the same security in fewer refinement rounds, rather than as the source of the security gains.

  2. New experiment. Appendix F gives the graph-free variant the same number of rounds as GRASP and nothing more. It still trails on every metric while inspecting more principles, separating what the graph contributes from the effect of running longer.

  3. Ablations. Appendix F now describes each variant against the implementation.

  4. Cost. Section 3.2, the Discussion and Appendix J state the per-sample call count and what it includes.

  5. Prompts. Section 3.2 states that no edge is shown to the model, and that the graph acts only in the controller.

  6. Post-cutoff CVEs. Section 4.4 is retitled and gives both cutoff dates, the selection criteria, and what the study does and does not measure.

  7. Baselines. Section 4.1 specifies the RAG configuration, the RESCUE comparison is stated as a boundary rather than an average, and one-point gaps on a near-saturated CWEval are treated as ties.

  8. Graph construction. Section 3.1 and Appendix C describe the pairwise judging accurately and report its cost.

  9. Corpus. The Discussion explains that the pipeline does not depend on OWASP and that updates are incremental.

If there are any remaining points you would like us to address, please do not hesitate to let us know. Your feedback has made the paper's claims considerably sharper, and we would be glad to address anything further.

Area_Chair_1Qe2OpenReview ↗

take part in discussion

Dear reviewer 9sd7,.

pleae take part in the discussion. time is running out.

thank you your ac

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.