acceptodds
Withdrawn from ICLR 2027

Security-Impact-Validated Traces for Module-Level RTL Vulnerability Detection with Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Detecting security weaknesses at the register-transfer level (RTL) can reduce costly post-fabrication repairs. Large language models (LLMs) offer automated detection, but require module-level data with reliable security labels. We present a Trace-aware detection approach and a data-construction pipeline combining Common Weakness Enumeration (CWE) scenario identification, structure-adaptive semantic mutation, whole-module syntax checks, and CWE-specific security-impact validation. The pipeline produces 2,444 deduplicated secure–vulnerable pairs across seven open-source platforms and eight hardware CWEs. Aligned views provide code-only inputs (L0) or code with Security Trace annotations (L1): validation-derived code blocks, control conditions, signal relationships, and statically resolvable dependencies, excluding endpoint values that could reveal labels. We compare code-only detection (No Trace), reference evidence (Gold Trace), and model-generated evidence (Self-Generated Trace), using source-lineage-isolated splits and platform holdout to assess detection and generalization. On 492 test modules, four models with source-ordered chunk aggregation achieve 96.54%–97.76% accuracy with Gold Trace; replacing it with generated Trace while retaining the same reference-trained classifiers yields 85.77%–92.68%. These results support the utility of structured security evidence while highlighting the gap between reference and generated Traces.

Primary area:
datasets and benchmarks
Keywords:
Hardware Security, RTL Vulnerability Detection, RTL Vulnerability Dataset, Large Language Models, Security-Impact Validation, Structured Security Evidence, Hardware CWE, RISC-V

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9oK8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper constructs a validated RTL vulnerability dataset and uses structured Security Traces to improve LLM-based vulnerability detection. The authors construct 2444 secure-vulnerable pairs from seven open-source hardware platforms. Each pair undergoes syntax checking and CWE-specific security-impact validation. The resulting Security Traces contain relevant code blocks, control conditions, and signal dependencies. Experiments with four LLMs compare code-only detection, reference Traces (Gold Trace), and model-generated Traces (Self Trace). The results show that Security Traces substantially improve detection accuracy over code-only inputs.

Critical strengths
  1. The authors construct a dataset consisting of 2444 secure-vulnerable pairs from seven real open-source RTL projects.
  2. The authors go beyond syntax checking by using CWE-specific simulation to verify security-relevant behavioral differences between paired RTL modules. This improves the reliability of the constructed dataset.
  3. The authors propose Security Traces to assist LLM-based detection. The experiments show that both reference and generated Traces improve detection accuracy over code-only inputs.
  4. The authors evaluate four LLMs and conduct several ablation studies to investigate the effects of Trace inputs, training strategies, and input organization.
Critical weaknesses
  1. All vulnerabilities are introduced using predefined mutation rules. Although the authors validate their security impact through simulation, it is still unclear whether the models learn general vulnerability characteristics or merely recognize the patterns associated with these mutations. Evaluation on independently identified vulnerabilities or unseen mutation types would provide stronger evidence of practical effectiveness.
  2. The paper describes the mutation rules and CWE-specific security-impact tests mainly at a high level. The paper lacks sufficient implementation details and concrete examples illustrating how vulnerabilities are introduced and how their security impact is validated for different CWEs. This makes it difficult to assess the reliability and reproducibility of the data construction process.
  3. The paper uses many lengthy and sometimes unnecessary compound terms, such as "structure-adaptive semantic transformations", "security-impact-validated", and "source-lineage-isolated splits", which make the presentation overly dense. Using simpler and more conventional terminology would improve readability.
  4. Each experiment is conducted only once with a single random seed, so it is difficult to assess the stability of the reported improvements.
Questions for authors
  1. Do the training and test sets share the same mutation rules? Have the authors evaluated the models on independently identified vulnerabilities or unseen mutation types? This would help clarify whether the models learn general vulnerability characteristics or patterns specific to the data construction process.
  2. Could the authors provide more concrete examples of the mutation rules and security-impact validation procedures for different CWEs?
Reviewer_h2obOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents an LLM-based approach to detect security vulnerabilities at the register transfer level (RTL) for system-on-chip applications. The authors construct a dataset by sourcing open-source Verilog/SystemVerilog design files, parsing them, identifying candidate sites to introduce vulnerabilities, and then introducing the target Common Weakness Enumeration (CWE) weaknesses at those sites. The data is then annotated (traces) to help LLMs detect vulnerabilities. Moreover, the authors propose an LLM-based trace generation method that annotates the data before the fine-tuned open-source models identify the vulnerabilities. The dataset includes 2,444 pairs of secure and vulnerable design files (the vulnerable ones generated from the secure ones). The experiments show that models given the reference traces from the dataset reach about 96% accuracy, compared with roughly 50% for commercial models. When the traces are generated by an LLM instead, accuracy drops to about 86-93%. The authors also report that the approach holds up when an entire platform (OpenTitan) is held out of training.

Critical strengths

The main strength of the paper is the construction of the dataset. The authors parse real open-source design files and carefully introduce well-known CWE vulnerabilities at identified sites. Pairing each vulnerable file with its secure original also gives a balanced dataset where the only difference is the introduced weakness, and the authors say they will release it, which could be useful to the community.

Critical weaknesses

The main weakness is that the best results depend on the reference trace from the dataset, which would not exist for a new design. In the case where the LLM generates the trace, accuracy drops to 86-93%. The technical contribution is also limited, since the detection side is mostly standard fine-tuning of existing open-source models, and the main novelty lies in the dataset rather than in the method.

Questions for authors
  • The trace is derived from the same data construction step that created the label and introduced vulnerabilities. Could you explain why the trace does not simply point the model to the place where the bug was introduced? An example of a secure sample’s trace next to its vulnerable counterpart and traces would help.
  • How close are the automated LLM generated traces to the reference ones? The paper reports only the final accuracy, so it is hard to tell whether the difference in error comes from poor traces or from the model.
  • Is it possible (is there data available) to be able to test the model on known vulnerabilities in real designs (instead of introduced vulnerabilities into secure design)?
Reviewer_RD28OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a dataset and a Trace-assisted method for module-level RTL vulnerability detection. It constructs 2,444 secure–vulnerable pairs from seven open-source hardware platforms and eight hardware CWE categories. The construction pipeline combines scenario identification, semantic mutation, whole-module syntax checks, and CWE-specific security-impact tests. The dataset provides aligned code-only inputs, called L0, and inputs with source-linked Security Traces, called L1.

The paper evaluates four fine-tuned language models and compares reference Traces, generated Traces, and code-only detection. On 492 test modules with source-lineage separation, reference Traces yield 96.54% to 97.76% accuracy. Generated Traces combined with classifiers trained on reference Traces yield 85.77% to 92.68%. The paper also reports Trace controls, aggregation comparisons, and an OpenTitan platform holdout. Its main contribution is a data-construction framework with structured security evidence, together with evidence of its detection utility on the resulting benchmark.

Critical strengths
  1. The data-construction framework addresses important weaknesses in RTL vulnerability datasets. The pipeline requires both syntax validity and an expected security-related behavioral difference between paired versions. This provides a stronger basis for constructing examples than syntax checks alone. The aligned L0 and L1 views also support comparisons that preserve code, labels, and partitions.

  2. The paper includes meaningful controls for the utility of generated Traces. Table 2A compares L0 and L1 under the same model and OH aggregation setting. Generated Traces with a Gold Answerer improve accuracy over these L0 controls across all four models. Table 2D also examines alternative input organizations, so the evidence extends beyond a comparison against unadapted or zero-shot models.

Critical weaknesses
  1. The central security-validation claim lacks enough detail to establish the meaning and reliability of the module-level labels. Section 3.2.3 permits validation of selected expressions or paths under specific stimuli and, in some cases, scalarized signal abstractions. Section 3.2.4 then assigns labels to complete-module records. The relation between these two scopes remains insufficiently clear. In particular, the paper does not precisely define whether “secure” means that the selected site satisfies the target property, that the module contains no instance of the target CWE, or that the module contains none of the eight target weaknesses. These are different learning targets. For vulnerable samples, the paper also gives limited detail on how the local test conditions and abstractions correspond to feasible behavior in the original module.

  2. The strongest practical configuration lacks an independent and matched generalization test. Section 4.2 states that the cross-input comparisons follow inspection of initial test results. Section 4.4C then uses these comparisons to motivate the main-table combination of Self Trace and Gold Answerer. Thus, the same test evidence helps select the configuration and assess its performance. Keeping model weights and thresholds fixed does not remove selection through the choice of pipeline. This choice is consequential: Table 2C reports an advantage of 4.47 to 7.11 percentage points for the selected Answerer on generated Traces.

Questions for authors
  1. What exactly does a “secure” label mean? Does it apply to a selected site and property, to the target CWE throughout the module, or to all eight target CWE categories? How does the validation procedure connect local stimuli and scalarized signals to feasible behavior in the original module? A complete paired example with its property, mutation, assumptions, stimuli, and observed outputs helps resolve this concern.

  2. What independent checks support the labels and the effective diversity of the dataset? Does an expert audit or a separate checker assess a representative sample across CWEs and mutation rules? What disagreement rate does that check produce?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.