- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CyberAEBench to evaluate autonomous penetration-testing agents under misleading environmental observations. The benchmark contains 394 adversarial variants derived from 14 base environments covering web penetration, CVE exploitation, chained exploitation, and enterprise-network tasks. Perturbations are organized by information type and observation channel while retaining the underlying vulnerabilities and attack objectives. Four models are evaluated within PentestGPT. The paper reports increased task failure and execution costs, and examines how misleading information affects agents' beliefs, memory, and subsequent actions.
Critical strengths
-
Comparing trusted environments with adversarial variants addresses a relevant limitation of capability-only evaluations. Keeping the vulnerability and attack objective fixed provides a useful design for examining whether an agent can complete an otherwise solvable task when its observations are misleading.
-
The benchmark distinguishes the content of a manipulation from the channel through which it is observed. Knowledge, instruction, and strategy manipulations are evaluated through direct content, communication metadata, and API responses. This organization supports more detailed analysis than a single aggregate attack-success score. Using the same agent framework across four models also reduces differences arising from agent architecture.
Critical weaknesses
-
The reported failure rates are difficult to reconcile with the stated aggregation rule. Section 5.1 defines task success as retrieving all required flags in any of five trials, and defines I4 as the proportion of failed environments. With 30 EnterpriseNet variants, this environment-level rate should change in increments of approximately 3.33 percentage points. The 68.7% reported for DeepSeek-V4-Pro in Table 1 cannot be obtained under that definition. A run-level average or another aggregation could explain the value, but would represent a different metric. Raw counts and explicit aggregation formulas are needed to interpret the main results.
-
The measurements supporting the claims about agent beliefs are insufficiently specified. I1 requires determining whether an agent has formed an incorrect belief, but the paper does not describe the labeling procedure or its validation. I2 measures how often adversarial information appears in memory; however, mentioning a misleading claim does not establish that the agent accepts it or continues to rely on it. The claim could instead be quoted, questioned, or rejected. Without distinguishing these cases and validating the measurements against execution traces, the conclusions about persistent belief contamination remain uncertain.
Questions for authors
-
Are the I4 values in Table 1 calculated per environment, per trial, or through another aggregation? Please provide the underlying success/failure counts and reconcile the EnterpriseNet values with the stated any-of-five success criterion. Corresponding trusted-environment success counts would also clarify the comparison.
-
How are I1 and I2 computed from agent traces? Please specify the annotation or automated decision rules and provide manually checked examples that distinguish acceptance and subsequent use of misleading information from quotation, uncertainty, or rejection.