- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper constructs a validated RTL vulnerability dataset and uses structured Security Traces to improve LLM-based vulnerability detection. The authors construct 2444 secure-vulnerable pairs from seven open-source hardware platforms. Each pair undergoes syntax checking and CWE-specific security-impact validation. The resulting Security Traces contain relevant code blocks, control conditions, and signal dependencies. Experiments with four LLMs compare code-only detection, reference Traces (Gold Trace), and model-generated Traces (Self Trace). The results show that Security Traces substantially improve detection accuracy over code-only inputs.
Critical strengths
- The authors construct a dataset consisting of 2444 secure-vulnerable pairs from seven real open-source RTL projects.
- The authors go beyond syntax checking by using CWE-specific simulation to verify security-relevant behavioral differences between paired RTL modules. This improves the reliability of the constructed dataset.
- The authors propose Security Traces to assist LLM-based detection. The experiments show that both reference and generated Traces improve detection accuracy over code-only inputs.
- The authors evaluate four LLMs and conduct several ablation studies to investigate the effects of Trace inputs, training strategies, and input organization.
Critical weaknesses
- All vulnerabilities are introduced using predefined mutation rules. Although the authors validate their security impact through simulation, it is still unclear whether the models learn general vulnerability characteristics or merely recognize the patterns associated with these mutations. Evaluation on independently identified vulnerabilities or unseen mutation types would provide stronger evidence of practical effectiveness.
- The paper describes the mutation rules and CWE-specific security-impact tests mainly at a high level. The paper lacks sufficient implementation details and concrete examples illustrating how vulnerabilities are introduced and how their security impact is validated for different CWEs. This makes it difficult to assess the reliability and reproducibility of the data construction process.
- The paper uses many lengthy and sometimes unnecessary compound terms, such as "structure-adaptive semantic transformations", "security-impact-validated", and "source-lineage-isolated splits", which make the presentation overly dense. Using simpler and more conventional terminology would improve readability.
- Each experiment is conducted only once with a single random seed, so it is difficult to assess the stability of the reported improvements.
Questions for authors
- Do the training and test sets share the same mutation rules? Have the authors evaluated the models on independently identified vulnerabilities or unseen mutation types? This would help clarify whether the models learn general vulnerability characteristics or patterns specific to the data construction process.
- Could the authors provide more concrete examples of the mutation rules and security-impact validation procedures for different CWEs?