- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper finds that LLMs generate related code security vulnerabilities across applications for the same given implementation feature classes. For instance, for different applications that are all equipped with a login feature, this specific login feature is likely to exhibit the same security vulnerability across the independently generated applications. The evaluation relies on static analyzers, and shows that on two independent splits of applications, the LLMs generate a largely overlapping set of vulnerabilities as found by the SAST tools.
Critical strengths
The observation that LLMs have a "typical set" of vulnerabilities that they generate across similar settings is interesting and could be transformer into valuable contributions, or be used to demonstrate speed-ups/efficiency gains in cyberattacks against LLM-generated applications.
Critical weaknesses
- The paper is extremely hard to understand, and its writing strongly resembles typical weak patterns of inlegible AI writing. Frequent non-linear argumentation, contrastive sentences such as X and not Y, and other typical patterns hint LLM writing. But even beyond this, the paper is very difficult to follow, to the extent to which I am unable to verify any of it claims, as it is unclear what is being even presented.
- Other presentation flaws not aiding understanding are the unclear placement of figures in the middle of the text, the abrupt ending of the paper, the convolutedly and unexplainedly defined metrics and measures in Section 3, and the convoluted and unclear explanation of the experimental setting (it sounds like only 5 samples are used from each benchmark for testing?).
- The paper's initial idea of trying to categorize LLM-generated vulnerabilities in a predictive framework is neat, but the actual execution and especially the evaluation completely underdelivers. It would have been more interesting if it could be demonstrated how an actual adversary with the goal of exploiting the applications can use this lookup table to speed up their attacks. Additionally, for any such lookup table, to validate their construction, it would need to be demonstrated that it is not just better than no table, but also, it is better than a generic vulnerability to exploit map (as even independently of LLMs, for frontend/backend features we can already expect a quite small set of typical vulnerabilities that are well-documented in vulnerability databases such as the MITRE CWE classification).
Overall, this paper is clearly below the bar.
Questions for authors
No questions.