- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper propose a new benchmark called CROSSPLOIT to test if LLM agents can migrate exploit PoC codes to unrecorded historical software versions. They collect 127 real cases and test many state-of-the-art models. The results show models have large action space but struggle a lot when the attack path need to be changed.
Critical strengths
- The execution-driven evaluation framework is very solid because it actually run the code in sandbox instead of just doing static matching.
- It target a very practical security problem. Finding missing vulnerable versions for CVE databases is very useful for real world security.
Critical weaknesses
- The benchmark is too overfitted to Python Web ecosystem. If it is about lower level C++ memory vulnerabilities, performance might be totally different.
- The experiment restrict agents from modifying the target source code. In real life security audits we often change the source code a bit to do print debugging, so this makes it a bit unrealistic.
- The baseline they compare with is a bit of a strawman. VULSCOPE-WEB is originally for C/C++ memory bugs, so of course it perform very bad on Python Web. A modern DAST tool would be a much fairer baseline.
- The evaluation metric is too flat. It only show success or fail rate. For complex agent benchmarks, it should also measure things like average token cost or steps taken to see where exactly the models fail.
Questions for authors
- How is the oracle implemented exactly? Different software versions change log formats, how do you make sure the oracle don't give false negatives?
- What exactly is in the input prompt for the agents? Could you detail it?
- Big models might already seen discussions about these old version bugs in their training data. How did you check for data contamination?
- If the two human raters disagree on a migration result, how do you resolve it and what is the inter-rater reliability score?
- How much compute and money cost to run all these experiments?