acceptodds
Withdrawn from ICLR 2027

SecFlow: Learning to Interpret and Use Security Evidence in Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding the surface meaning of security evidence does not ensure that models interpret its security implications or use it correctly. We investigate how reliably large language models (LLMs) understand security facts and use evidence to guide decisions. Using vulnerability information extraction and patch localization as complementary study settings, we identify limitations in both capabilities across seven commercial LLM systems. Our audit attributes 73.28% of incorrect localization runs to dominant reliance on surface cues and documents cases where systems inspect the correct patch but fail to select it. We introduce SecFlow, which represents textual and program evidence as source-linked behavior flows encoding operations, conditions, and effects. It evaluates these flows against the vulnerability under analysis and injects the routed memory into later language-model layers through gated attention to guide answer generation. Built on Qwen2.5-Coder-7B, SecFlow improves both security-fact understanding and evidence-guided decision-making. Across the two study settings, it increases grounding F1 by 16.51 percentage points and achieves 65.24% Top-1 accuracy and 95.92% Hit@5 in candidate selection, surpassing the strongest evaluated commercial baseline by 40.13 percentage points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, cybersecurity, evidence-grounded reasoning, program understanding, structured representations, evidence routing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WQC7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks three questions: How well do LLMs extract security facts from vulnerability reports? How well do they use repository evidence to identify fixing commits, and what causes their errors? Can SecFlow improve these capabilities by representing and routing security evidence? In tests on 15,365 OSV cases, the best of seven commercial models reaches 48.48% grounding F1 and 23.88% patch-localization Top-1; trace audits attribute many localization failures to surface cues. SecFlow combines source-linked behavior representations, event-conditioned routing, and gated attention. On held-out sets, it reports 49.59% grounding F1 and 65.24% localization Top-1; ablations and a same-backbone SFT comparison provide evidence that its components contribute to the gains.

These results support SecFlow’s effectiveness on the paper’s constructed benchmarks, but do not establish readiness for operational security work. The paper does not validate performance in real analyst workflows or show that SecFlow improves security outcomes in deployed repositories. The manual benchmark verification is not described in enough detail for independent audit, and the supplementary archive lacks the paper data and comparison code needed to reproduce the main results.

Critical strengths
  • The paper separates security-fact extraction from patch localization and audits model traces, making the diagnostic goals clearer than a single end-to-end success metric.
  • The proposed method is described through explicit components and training objectives. Ablations show substantial drops when components are removed: localization Top-1 falls from 65.24% to 35.41% without event routing.
  • The 466-instance localization evaluation uses records without explicit patch references and reports manual inspection of repository histories and diffs, addressing a potential source of model knowledge leakage.
  • Results include comparisons with seven commercial systems and a same-backbone SFT baseline: SecFlow reports 65.24% Top-1 versus 54.29% for SFT.
Critical weaknesses
  • The paper gives limited detail about annotation and audit reliability: it does not report annotator qualifications, agreement, adjudication, or enough examples and counts to independently verify the failure taxonomy and the claim that surface cues explain over 90% of incorrect localization runs.
  • The manual verification process and the 466 benchmark labels are not independently auditable from the supplied archive. The ZIP has no paper dataset, annotations, README, or trained checkpoints.
  • The supplementary code does not provide the seven-system evaluation harness or the SFT baseline implementation. Its reproduction script requires external train/dev/test data, so it cannot reproduce the main comparisons from the ZIP alone.
  • The code can silently ignore annotation references that do not match evidence-unit IDs. Its split utility prevents exact normalized-description overlap but does not itself prevent repository or vulnerability-family overlap. The paper should clarify whether the released data and final splits address these risks.
Questions for authors
  • Could you provide the annotation guidelines, annotator and adjudication details, agreement measures, and audit counts supporting the reported failure analysis?
  • Could you release the 466 evaluation instance identifiers, repository versions, fixing labels, and verification criteria, and clarify how multiple valid fixes and overlap with training data are handled?
  • Do the final data splits enforce repository- or vulnerability-family separation beyond exact description matching? Could you also clarify whether mismatched annotation spans or edge references are checked before training, given that the supplied code appears to skip some unmatched references silently?
Reviewer_KTdvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors study whether LLMs can extract security-relevant facts from vulnerability descriptions and use repository evidence for patch localization. In a preliminary study on 15,365 OSV vulnerabilities, seven commercial LLMs run in a common agent framework achieve at most 48.48% grounding F1 and 23.88% Top-1 localization accuracy. A manual audit of failed patch localization finds that most failures to reliance on surface level facts rather than repair behavior. Thus, from the study, the authors develop SecFlow, a supervised adapter, trained on Qwen2.5-Coder-7B. SecFlow adds three modules: 1) a behavior-grounding module that splits the description and code into evidence units, tags each unit’s operation, security effect, and phase, and predicts relations between units to build a source-linked behavior memory, 2) a FlowScorer that compares each memory entry with the vulnerability description, labels its role, and assigns it a reading priority and 3) a gated cross-attention adapter that propagates the memory, weighted by those priorities, into later decoder layers to guide answer generation. On 1,000 held-out descriptions, SecFlow raises grounding F1 over the unadapted Qwen2.5-Coder-7B from 33.08% to 49.59%.

Critical strengths
  1. The preliminary study is a valuable evaluation since it evaluates 7 frontier models on 15,365 OSV vulnerabilities in a common agent framework. It also separates 2 tasks of extracting security facts from a description and using repository evidence. The study also includes failure analysis which is helpful, though the failure categories are not defined anywhere. The study is useful to the community regardless of the SecFlow adapter.
  2. The idea of training a supervised adapter for a small open weight model is appealing for solving a problem specific to patch-localization. This is cheaper than using frontier models in an agent loop.
Critical weaknesses
  1. The paper fails to define or explain its components, and lists words or specific metaphors. This makes understanding what is actually happening unclear. Section 3 relies on terms such as "behavior flows" or "flow builder" without definitions or formulas, so it's unclear how memory is actually computed. It's also unclear where the annotations that are used for supervision come from.
  2. The task input and output for SecFlow are not clearly specified. Section 3.1 says that SecFlow receives “the available textual or program evidence,” without explaining what that evidence is for patch localization or how candidate commits are obtained. Section 4.1 states that SecFlow "follows the same evaluation environment as the Preliminary Study", which tests models in a common agentic harness environment. But in the released code at data.py in the prompt() function suggests that SecFlow ranks a pre-supplied list of candidate commits. The frontier models baselines are evaluated using an agentic harness with access to full repositories and commit histories, so the comparison between SecFlow and frontier models is not in similar experiment scenarios. SecFlow skips the retrieval step where several frontier models failed, and Hit@5 depends heavily on how many candidates are provided. Thus, this unfair evaluation comparison makes it unclear how SecFlow is able to beat frontier models.
Questions for authors
  1. Can you provide a clear formalization of what the task, input, and output for SecFlow are?
  2. Can you evaluate SecFlow and the frontier models under the same setting with the same candidate commits, and report how many candidates are provided per instance?
  3. Where do the annotations come from?
  4. How was the tag vocabulary chosen?
  5. In the preliminary study, what exact task, prompt, and tools were given to the agents?
  6. Can you provide the formulas for each of the metrics used in your study and evaluation?
  7. Can you define the failure categories F1-F8?
  8. How would SecFlow handle vulnerabilities whose dataflow is across multiple files or functions? What languages does the benchmark cover? Can you run more experiments on different languages and varying complexity of vulnerabilities?
Reviewer_pJSvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates large language models’ ability to understand security facts and make evidence-driven decisions in software vulnerability analysis, evaluating seven commercial models on vulnerability information extraction and patch localization. The authors propose SecFlow, which organizes textual and code evidence into source-linked behavior flows and combines event-conditioned routing with gated cross-attention to guide answer generation while keeping the backbone model frozen. Experiments show that SecFlow improves fact extraction F1, outperforming a supervised fine-tuning baseline using the same backbone.

Critical strengths
  1. Focused and practically relevant problem. The paper targets two specific challenges in software vulnerability analysis: understanding security facts and identifying genuine fixes from code evidence. Both are important for reliable LLM-assisted code security analysis.
  2. Thorough investigation and clear motivation. Beyond reporting performance across seven commercial models, the paper examines execution traces to characterize their failure patterns. Cases where models inspect the correct patch but still reject it clearly motivate better evidence interpretation and use
Critical weaknesses
  1. The need for the proposed complexity is not sufficiently justified.The paper introduces several trainable components, including behavior-flow modeling, event-conditioned routing, and gated attention. Given the focused problem and detailed failure analysis, I question whether such a complex solution is necessary. The observed failures might also be mitigated through a training-free agent workflow with explicit evidence verification and staged decision-making, but the paper does not investigate this alternative. This is my main concern.
  2. The patch-localization baselines are insufficiently comprehensive. Although the paper compares against several commercial models, PatchSeeker, and a same-backbone SFT baseline, it omits representative specialized methods discussed in the related work, such as PatchFinder and VFCFinder. This limits the evidence for its advantages over existing patch-localization approaches.
Questions for authors
  1. Can the authors provide evidence that SecFlow offers benefits beyond a training-free agent workflow tailored to the identified failure modes? 2.How does SecFlow compare with specialized patch-localization methods such as PatchFinder and VFCFinder? The first question is my primary concern. If the authors provide a convincing justification, I would consider raising my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.