- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies evasion attacks against LLM-based vulnerability detectors using source-code transformations such as identifier substitution, comments, preprocessor constructs, macros, and dead branches. The attacks are optimized on a surrogate model and tested for transfer across multiple open and closed models. The paper further evaluates compilation preservation and several simple defenses.
Critical strengths
he threat model is meaningful: attacks are evaluated only on vulnerabilities that the detector originally identifies correctly, and the transformations operate directly in source code rather than through arbitrary prompt injection.
The cross-model results are also convincing. Different detectors fail under different carriers, and universal strings optimized on one surrogate transfer to several other detectors, making the result difficult to dismiss as a model-specific artifact.
Critical weaknesses
he preservation claim is somewhat narrower than the framing suggests. Compilation is verified on a 1,434-function subset rather than the full benchmark, and runtime behavioral equivalence is not tested.
I am also less convinced by the practical strength of the identifier attack. The strongest strings are unusually long, while the shorter naming-policy-compliant variant is substantially more variable and is not evaluated for transfer to the main target models.
Questions for authors
Could the authors report transfer results for the ≤32-character identifier attack on the main target models, ideally over several independently optimized support sets? This would materially affect my assessment of how much of the vulnerability survives a simple source-code hygiene rule.