- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the natural transformation's impact on AI agents' capability on vulnerability analysis. The authors observe that agents can produce different results across code versions containing the same vulnerability. Motivated by this observation, they mined common transformation patterns from real-wolrd commits. They select 21 common transformations for evaluation and apply these transformations on agentic vulnerability analysis. The results show that transformation generally reduce agents' success rates.
Critical strengths
- The studies question is practical as it conerns about whether vulnerability analysis agents remain reliable
- The evaluated dataset is based on real-world repositories and executable PoC validation. It also include multiple models and harnesses.
Critical weaknesses
- The novelty is limited. The findings are consistent with prior studies on robustness and semantics-perserving code transformations. This paper does not establish the novel or unique benefits of mining natural transformations from real commits.
- The checking of semantic preservation is based on existing tests and ground-truth PoC replay, while this paper does not manual validation.
- The paper does not report the confidence intervalas and paired significance tests on the original vs. transformed code.
Questions for authors
- What is the unique benefit of mining transformations from real commits over manually specified refactorings or directly prompint an LLM to restructure code?
- Have the authors manually inspect a sample of the transformed repositories to assess semantic preservation beyong existing tests and PoC replay?
6 comments
Q1, W1: What is the unique benefit of mining transformations from real commits over manually specified refactorings or directly prompint an LLM to restructure code?
Prior work has studied manually designed syntactic perturbations and code obfuscation techniques [1, 2]. These transformations rely on predefined rules that leave recognizable patterns that are easier to identify and reverse, as existing reasoning LLMs have demonstrated [3, 4]. Our evaluation shows that these simple transformations are not robust when the adaptive attacker agents are allowed to proactively transform the code, and quickly become ineffective when the attacker agents are explicitly told the code is transformed so they can identify and reverse the transformations (Section 4.2).
In contrast, we take a data-driven approach by mining natural transformations from real-world software commits. By extracting transformation patterns from real code commits, we can capture a much more diverse set of subtle coding patterns that expand significantly beyond manually designed rules. Natural transformations are far more stealthy because they resemble routine software development patterns, and leave fewer predictable cues for LLMs to identify and reverse. Our experiments in Section 4.2 show that natural transformations remain effective even when an adaptive agent attempts to reverse the changes before vulnerability analysis. Therefore, we consider natural transformation as complementary to existing approaches using fixed transformations.
Our proposed natural transformation differs from existing works that directly prompt an LLM to restructure a codebase. Such transformations give the user zero control and cannot ensure that changes reflect common software development practices. In contrast, we use cluster-based mining to identify recurring transformations in real code, and distill them into reusable skills that guide LLM code transformations. This provides a systematic basis to evaluate how different transformations affect agentic vulnerability discovery.
That said, we are also intrigued by the finding that natural transformation, which is supposed to be much less out-of-distribution for existing models extensively pre-trained on real-world commits [5] and should thus be more amenable (and less disruptive) for agents to analyze, still effectively confuses the state-of-the-art agents, including Claude Mythos 5.2 + Claude Code (see our results below). Therefore, with this surprising effectiveness, we hope to share the potential of leveraging them to build more robust defense (exactly because of the stealthy nature of natural transformations) against attacker agents, and complement existing adversarial evaluations of agentic vulnerability analysis.
References
[1] Xiao Li, Yue Li, Hao Wu, Yue Zhang, Yechao Zhang, Fengyuan Xu, and Sheng Zhong. A systematic study of code obfuscation against llm-based vulnerability detection. arXiv preprint arXiv:2512.16538, 2025.
[2] Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al. Recode: Robustness evaluation of code generation models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13818–13843, 2023.
[3] David Beste, Grégoire Menguy, Hossein Hajipour, Mario Fritz, Antonio Emanuele Cinà, Sébastien Bardin, Thorsten Holz, Thorsten Eisenhofer, and Lea Schönherr. Exploring the potential of llms for code deobfuscation. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, pages 267–286. Springer, 2025.
[4] Shan Jiang, Pranoy Kovuri, David Tao, and Zhixun Tan. Cascade: Llm-powered javascript deobfuscator at google. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice, pages 292–302, 2026.
[5] Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In International Conference on Learning Representations, volume 2024, pages 7604–7623, 2024.
Q2, W2: Have the authors manually inspected a sample of the transformed repositories to assess semantic preservation beyond existing tests and PoC replay?
Thanks for this question, which prompted us to look into the transformations more closely. We agree manual inspection would strengthen our validation of semantic preservation.
To answer this question, we selected three repositories (skcms, sndfile, and c-blosc2) that we are familiar with for manual inspection. We organized the transformations by file and manually reviewed each modified region against its original implementation, including the surrounding code and relevant call sites. For each change, we inspected the affected control flow and data dependencies, checking whether branch conditions, return values, side effects, or error-handling behavior could alter the program’s functionality. We found no semantic differences in these inspections.
To further validate our inspection, we applied fuzzing-based checks for all evaluated programs. Specifically, we fuzzed each original program for 48 hours to collect input-output pairs, then replayed the same inputs on the transformed version and compared the outputs to detect potential semantic differences. The fuzzing scripts and all collected input-output pairs have been uploaded to our anonymous GitHub repository to support reproducibility.
We have added these validation procedures to the revised paper (Section 4.1).
Thank you for the constructive comments!
Thank you so much for taking the time and effort to read our paper and leave constructive comments. We have uploaded the revised PDF, with additions marked in blue and deletions in red. Some deletions in the main text are not shown due to the limited space.