acceptodds
Under review as a conference paper at ICLR 2027

When the Environment Lies: Evaluating Autonomous Penetration Testing Agents under Adversarial Environments

Abstract

Large language models (LLMs) are increasingly used to build autonomous penetration testing (AutoPT) systems, making rigorous evaluation of their capabilities and risks essential for understanding their real-world reliability. However, existing evaluations primarily assess whether agents can complete predefined attack tasks under trusted environments, leaving their robustness to unreliable environmental feedback largely unexplored. In real-world settings, AutoPT agents must infer target states from heterogeneous observations that may be inaccurate, inconsistent, or misleading. In this paper, we introduce CyberAEBench, the first adversarial environment benchmark for evaluating AutoPT agents under uncertain environmental conditions. CyberAEBench preserves target systems, attack objectives, and evaluation criteria while systematically modifying only the environmental evidence available to agents. It constructs 394 adversarial environments across four representative scenarios, including web penetration, CVE exploitation, chained exploitation, and enterprise network penetration. Evaluating four frontier LLMs within the PentestGPT framework, we find that current AutoPT agents are highly vulnerable to adversarial environmental information: misleading evidence can distort beliefs, persist through long-horizon planning, increase execution costs, and cause task failures. Further analysis reveals that adversarial impact depends on both information characteristics and observation channels, where consistent multi-source evidence can amplify incorrect beliefs through false confirmation. Beyond performance degradation, we identify emerging behavioral risks where adversarial environments induce unintended objective shifts and harmful operations. These findings demonstrate that trusted-environment evaluations alone cannot fully characterize AutoPT agents and highlight the need to assess their capabilities and risks under realistic uncertain environments.

Primary area:
datasets and benchmarks
Keywords:
Autonomous Penetration Testing, Agent Evaluation, Environmental Robustness, Adversarial Environments

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_p25wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CyberAEBench to evaluate autonomous penetration-testing agents under misleading environmental observations. The benchmark contains 394 adversarial variants derived from 14 base environments covering web penetration, CVE exploitation, chained exploitation, and enterprise-network tasks. Perturbations are organized by information type and observation channel while retaining the underlying vulnerabilities and attack objectives. Four models are evaluated within PentestGPT. The paper reports increased task failure and execution costs, and examines how misleading information affects agents' beliefs, memory, and subsequent actions.

Critical strengths
  1. Comparing trusted environments with adversarial variants addresses a relevant limitation of capability-only evaluations. Keeping the vulnerability and attack objective fixed provides a useful design for examining whether an agent can complete an otherwise solvable task when its observations are misleading.

  2. The benchmark distinguishes the content of a manipulation from the channel through which it is observed. Knowledge, instruction, and strategy manipulations are evaluated through direct content, communication metadata, and API responses. This organization supports more detailed analysis than a single aggregate attack-success score. Using the same agent framework across four models also reduces differences arising from agent architecture.

Critical weaknesses
  1. The reported failure rates are difficult to reconcile with the stated aggregation rule. Section 5.1 defines task success as retrieving all required flags in any of five trials, and defines I4 as the proportion of failed environments. With 30 EnterpriseNet variants, this environment-level rate should change in increments of approximately 3.33 percentage points. The 68.7% reported for DeepSeek-V4-Pro in Table 1 cannot be obtained under that definition. A run-level average or another aggregation could explain the value, but would represent a different metric. Raw counts and explicit aggregation formulas are needed to interpret the main results.

  2. The measurements supporting the claims about agent beliefs are insufficiently specified. I1 requires determining whether an agent has formed an incorrect belief, but the paper does not describe the labeling procedure or its validation. I2 measures how often adversarial information appears in memory; however, mentioning a misleading claim does not establish that the agent accepts it or continues to rely on it. The claim could instead be quoted, questioned, or rejected. Without distinguishing these cases and validating the measurements against execution traces, the conclusions about persistent belief contamination remain uncertain.

Questions for authors
  1. Are the I4 values in Table 1 calculated per environment, per trial, or through another aggregation? Please provide the underlying success/failure counts and reconcile the EnterpriseNet values with the stated any-of-five success criterion. Corresponding trusted-environment success counts would also clarify the comparison.

  2. How are I1 and I2 computed from agent traces? Please specify the annotation or automated decision rules and provide manually checked examples that distinguish acceptance and subsequent use of misleading information from quotation, uncertainty, or rejection.

Reviewer_w1beOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CyberAEBench to evaluate autonomous penetration testing agents under adversarial environmental observations. The benchmark constructs 394 adversarial variants from 14 base scenarios spanning web penetration, CVE exploitation, chained exploitation, and enterprise networks. It organizes manipulations by information type and observation channel, with the stated goal of preserving the underlying task while changing the evidence available to the agent. Four LLMs are evaluated within a shared PentestGPT framework using five trials per scenario. The paper reports environmental misperception, persistent belief contamination, increased execution costs, and task failures, and analyzes differences across manipulation types and placements.

Critical strengths

S1. An important and practically motivated evaluation question.

The paper addresses whether autonomous penetration testing agents can remain reliable when environmental observations are misleading, rather than evaluating only whether they can complete an exploitation task. This is a meaningful dimension of agent robustness because successful operation depends on interpreting observations and deciding which evidence to trust. The proposed pairing of trusted environments with adversarial variants is a useful starting point for investigating this question.

S2. A structured experimental framework for examining environmental manipulation.

The distinction between factual misinformation, embedded instructions, and strategic guidance, together with different observation channels, provides a useful organization of the problem. Using a common agent framework, repeated trials, and flag-based task verification also offers a basis for systematic comparison. These elements make the benchmark potentially valuable if the measurement procedures and experimental interpretations are clarified and validated.

Critical weaknesses

W1. The principal metrics and baseline comparisons are insufficiently specified, preventing a reliable interpretation of the reported degradation.

Section 5.1 defines I4 as failure under an adversarial environment despite success in the corresponding trusted environment, but Table 1 does not provide the model-specific baseline success counts, the adversarial configurations included in each aggregate, or the precise numerators and denominators. It is also unclear how the five trials are aggregated across trusted and adversarial conditions. Under the stated any-of-five success criterion, an environment-level failure should correspond to five unsuccessful trials, yet values such as 68.7% for 30 EnterpriseNet environments do not directly correspond to an integer failed-environment count. An alternative aggregation procedure may explain these values, but it needs to be specified. Appendix A.6 reports baseline time and token consumption rather than the success statistics needed to reconstruct the comparison.

The interpretation of I1 and I2 is similarly under-supported. The manuscript does not adequately describe their annotation procedures, validation, or treatment of disagreement. In particular, adversarial information appearing in an agent’s memory does not establish that the agent endorses it or continues acting on it: the agent may instead record the information to question or reject it. Without these distinctions, the evidence does not sufficiently support claims about persistent incorrect beliefs and limited recovery. These issues affect the paper’s central empirical conclusions, rather than merely the presentation of secondary details.

W2. The explanations of manipulation effects and the paper’s distinct contribution are not sufficiently established.

The comparisons across information types and placements do not report whether the manipulated content actually reached the evaluated agent’s model-visible context. Information placed in an environment may never be requested, may be omitted from tool output, or may be removed during processing. Consequently, differences in failure rates may reflect differences in exposure rather than differences in susceptibility after exposure. Likewise, the stronger effect of consistent multi-source manipulation may arise from increased exposure or repetition rather than the claimed illusion of independent confirmation. Trace-level exposure measurements and appropriately controlled comparisons are needed to substantiate these explanations.

More broadly, the introduction motivates the problem but does not clearly delineate the specific methodological, benchmark, and empirical contributions beyond existing prompt-injection and agent-robustness evaluation. The absence of a contribution list is not itself a concern; the substantive issue is that the reader is left uncertain about what is newly enabled or established by this work. A concise contribution statement, explicit positioning against related evaluations, and a clear mapping from each contribution to supporting evidence would substantially strengthen the paper. At present, the topic is compelling, but the contribution and validation are not sufficiently clear to support acceptance.

Questions for authors
  1. For Table 1, which model-specific trusted-environment successes form the baseline, which adversarial configurations are aggregated, and how do the five trials determine each I4 numerator and denominator?
  2. How are I1 and I2 annotated and validated, particularly when an agent records adversarial information without accepting it or allowing it to influence subsequent actions?
  3. What trace-level evidence confirms that manipulated information reached the model-visible context, and do the reported placement and consistency effects persist after accounting for exposure and repetition?
  4. What are the paper’s distinct methodological, benchmark, and empirical contributions beyond existing prompt-injection and agent-robustness evaluations, and which evidence supports each?
Reviewer_pJRCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces CyberAEBench to evaluate penetration-testing agents when observations contain adversarial information. It varies the type of manipulation, i.e., knowledge, instructions, or strategy, and its placement in content, metadata, or API responses. The benchmark contains 394 variants derived from web, CVE, chained-exploitation, and enterprise-network environments. Four models are evaluated through PentestGPT. The paper reports environmental misperception, persistent contamination, additional execution cost, and task failure, and studies how manipulation type and placement affect these outcomes.

Critical strengths

The paper asks a relevant question about autonomous cyber evaluation: how much apparent capability depends on trusting the environment's descriptions of itself? The motivating example, in which false migration information redirects the agent toward a decoy, makes the failure mode concrete. Holding the underlying target task fixed while changing observations is a useful experimental principle.

Using live software, multi-stage tasks, and flag-based completion gives the evaluation a concrete outcome. The common PentestGPT framework and documented task prompts also support comparison across the four models. A well-specified version of this benchmark could add a useful robustness dimension to capability evaluations.

Critical weaknesses

The central outcome measurements cannot currently be reconstructed. Section 5.1 does not explain how an incorrect belief is identified, which memory state is inspected for I2, who labels the trajectories, or how those labels are validated. Adversarial text remaining in memory could reflect quotation or explicit rejection. Its presence alone does not establish that the agent continues to believe or act on it.

The task-failure aggregation also needs reconciliation. The paper specifies five trials, with an environment successful if any trial succeeds. Appendix A lists 30 EnterpriseNet variants, yet Table 1 gives failure rates such as 68.7% and 58.4%, which are not rounded integer counts divided by 30. Undocumented weighting or exclusions could explain this, but the stated protocol does not. These gaps affect both the numerical results and their interpretation.

The claimed controlled intervention is insufficiently specified and validated. The 394 variants come from 14 base environments, including one enterprise environment. The paper does not provide the allocation of payloads, information types, and channels needed to reconstruct the comparisons in Figures 4–5. Differences in target composition, payload content, or exposure could therefore contribute to the reported type and consistency effects.

Validation is described for the trusted environments, with no reported reference-exploit check after each transformation. Since edits may affect status codes, cookies, and API responses, preserving the underlying vulnerability does not by itself show that the original workflow remains executable. This check is necessary before attributing the additional failures entirely to misleading evidence.

Questions for authors
  1. What exact annotation procedure produces I1 and I2? Please specify the inspected state, label source, agreement or validation checks, and how adopted misinformation is distinguished from quoted or rejected content.
  2. What numerators, denominators, and trial-to-environment aggregation produce Table 1, particularly the EnterpriseNet failure rates? How are unsuccessful trusted runs, timeouts, and failed adversarial trials handled in the cost and failure metrics?
  3. Can the authors provide the variant-to-base/type/channel/payload mapping, the matched consistent-versus-inconsistent comparisons, and reference-exploit validation for every transformed variant? These details would determine whether the main comparisons isolate the intended intervention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.