acceptodds
Withdrawn from ICLR 2027

CrossPloit: Can LLM Agents Migrate Exploits Across Software Versions?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cross-version proof-of-concept (PoC) migration requires an agent to adapt a working exploit to another software version while preserving the underlying vulnerability. The task combines source-code analysis, environment repair, and execution-based verification. We introduce CrossPloit, a benchmark of 127 migration instances across 45 CVEs in the Python web ecosystem. Each instance includes a version-pinned container, a reference PoC, and a vulnerability-specific oracle; two validators check that the observed effect arises from the disclosed vulnerability. Across 18 agent configurations (nine models and two harnesses), the best configuration resolves 58.3% of instances, compared with 8.7% for an adapted trace-guided baseline. Attack-vector shifts are particularly difficult: no configuration resolves more than 37.9% of these instances. Higher resolve rates are associated with a smaller share of missed-API failures and a larger share of incomplete executions among the remaining failures. In one configuration, withholding the reference PoC and its environment specification reduces resolve rate from 50.4% to 15.0%, highlighting the value of the supplied reference package. CrossPloit provides an execution-based evaluation of agents' ability to preserve exploit semantics across software versions.

Primary area:
datasets and benchmarks
Keywords:
Proof of Concept, Cyber Security, Software Security, Vulnerability Affected Version

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6sfiOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose a new benchmark called CROSSPLOIT to test if LLM agents can migrate exploit PoC codes to unrecorded historical software versions. They collect 127 real cases and test many state-of-the-art models. The results show models have large action space but struggle a lot when the attack path need to be changed.

Critical strengths
  • The execution-driven evaluation framework is very solid because it actually run the code in sandbox instead of just doing static matching.
  • It target a very practical security problem. Finding missing vulnerable versions for CVE databases is very useful for real world security.
Critical weaknesses
  • The benchmark is too overfitted to Python Web ecosystem. If it is about lower level C++ memory vulnerabilities, performance might be totally different.
  • The experiment restrict agents from modifying the target source code. In real life security audits we often change the source code a bit to do print debugging, so this makes it a bit unrealistic.
  • The baseline they compare with is a bit of a strawman. VULSCOPE-WEB is originally for C/C++ memory bugs, so of course it perform very bad on Python Web. A modern DAST tool would be a much fairer baseline.
  • The evaluation metric is too flat. It only show success or fail rate. For complex agent benchmarks, it should also measure things like average token cost or steps taken to see where exactly the models fail.
Questions for authors
  • How is the oracle implemented exactly? Different software versions change log formats, how do you make sure the oracle don't give false negatives?
  • What exactly is in the input prompt for the agents? Could you detail it?
  • Big models might already seen discussions about these old version bugs in their training data. How did you check for data contamination?
  • If the two human raters disagree on a migration result, how do you resolve it and what is the inter-rater reliability score?
  • How much compute and money cost to run all these experiments?
Reviewer_bhLPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CrossPloit, a benchmark for evaluating whether LLM agents can migrate known exploits across software versions. The benchmark contains 127 migration tasks covering 45 CVEs in Python web software, with executable environments and vulnerability-specific validation. The authors evaluate 18 agent configurations, finding that the best resolves 58.3% of tasks, while a trace-guided baseline resolves only 8.7%. They also identify 149 vulnerable software versions missing from existing vulnerability databases.

Critical strengths
  • Careful benchmark construction. The authors have invested considerable effort in constructing executable environments and validating successful migrations. The combination of vulnerability-specific oracles and independent manual validation is a strong aspect of the work, especially since detecting successful exploitation is not always straightforward.

  • Broad evaluation with some interesting findings. The evaluation covers nine models and two agent harnesses. The breakdown by migration category is useful, particularly the observation that attack-vector shifts are substantially harder than API changes. The discovery of 149 previously unlisted vulnerable versions is also a compelling practical outcome.

Critical weaknesses
  • Selection bias in benchmark construction. The benchmark only includes instances for which the authors successfully found a working migration. Of the 297 candidate versions examined, 87 had inconclusive outcomes. This raises questions about how representative the benchmark is of real-world exploit migration tasks. In particular, evaluating only confirmed vulnerable versions does not tell us how agents perform when they must also determine whether a vulnerability exists in the first place. Including negative examples would make the benchmark substantially more useful.

  • Lack of repeated experiments. Each model and harness configuration is evaluated only once per task. Agent behavior can vary considerably between runs, and several reported performance differences are quite small. Without repeated runs or uncertainty estimates, it is difficult to assess how meaningful these differences are, particularly for the smaller migration categories.

  • Novelty relative to existing benchmarks could be clearer. Prior work, particularly Diffploit, has already studied LLM-based cross-version exploit migration. CrossPloit extends this setting to Python web applications with different validation requirements, but I would have liked a stronger explanation of what fundamentally new agent capabilities this benchmark measures compared with existing work.

Questions for authors

How stable are the reported results across multiple runs? In particular, could the authors repeat the evaluation for a subset of models to establish whether the relatively small differences between configurations are meaningful?

Reviewer_YY4KOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

CROSSPLOIT evaluates adapting a working vulnerability PoC to another software version while preserving the same vulnerability and exploitation effect. The benchmark contains 127 migrations across 45 Python-web CVEs, with pinned environments, reference PoCs, per-vulnerability execution checks, and manual review of causal validity. Eighteen model–harness configurations are evaluated once per task; the best resolves 74/127. A reference-package ablation reduces one configuration from 64/127 to 19/127, but removes both exploit code and environment information. The main contribution is the executable benchmark and its validation, rather than a new migration algorithm.

Critical strengths
  1. The task is concrete and useful: adapt an existing exploit without changing the affected product or substituting a different effect. Some instances require finding another route to the same vulnerable behavior, which goes beyond updating a dependency or import. The resulting tasks can be reused to study agent adaptation.

  2. Success is checked more carefully than a passing command or an agent’s declaration. The validators inspect whether the disclosed vulnerability caused the observed effect and whether the output obeyed the modification constraints. They are blinded to the producing configuration, and the paper reports agreement. This effort directly supports the reliability of the benchmark’s main deliverable.

Critical weaknesses
  1. The novelty and interpretation of the capability results are limited. Prior work, including the cited Diffploit benchmark, already studies exploit migration. The case for CROSSPLOIT therefore depends on the value of its Python-web tasks and validation, rather than introducing migration itself. The category labels describe human-produced adaptations, and it is unclear whether every valid solution needs those changes. If agents can solve a labeled attack-vector-shift instance through a simpler adaptation, Table 2 does not isolate difficulty of that required skill. Similarly, removing both reference code and environment information does not identify the contribution of exploit-level reasoning.

  2. The comparative evaluation supports broad observations more than precise rankings. Each configuration runs once, multiple targets share a CVE, and categories overlap. The input-only baseline cannot perform several changes the agents are allowed to make, and it receives oracle-only rather than the same causal review. Its 8.7% result is therefore not a matched test of trace-guided versus agentic migration. The benchmark also contains only manually validated positive tasks. That is appropriate for measuring adaptation success, but does not establish affected-version auditing accuracy or the ability to recognize an unaffected version.

Questions for authors
  1. Category meaning. Do the 29 Attack Vector Shift labels indicate changes required by every valid solution, or changes used in the canonical human migration? Did any accepted agent solve a labeled task without switching the exploitation path? Please report these cases. This determines whether the category result supports a claim about a particular missing capability or only identifies a difficult group of tasks.

  2. Reference ablation. When the environment specification is retained and only exploit code is withheld, how many tasks does OpenHands + Sonnet solve? Among the original 64 successes that fail without the full reference package, how many fail during environment setup versus after the environment is runnable? This would separate exploit adaptation support from historical-dependency support.

  3. What drives the ranking? Please provide success counts by CVE, treating each CVE equally as well as reporting the current instance-weighted rate, and repeat the leading model–harness configurations on the same tasks. Do the main ordering and the attack-vector-shift conclusion persist? This is particularly important because several versions of one CVE can reward the same adaptation and should not be treated as independent evidence of general capability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.