- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ProtoArena, a framework for evaluating whether LLM agents can generate observable security effects by interacting with multiple protocol participants, without access to the target source code. The authors constructed 80 environments covering 11 protocol or application domains and seven categories of attacks; the tasks are primarily derived from known attack mechanisms. The evaluation employs three information levels: L1 provides the mechanism, topology, and step-by-step guidance; L2 retains the mechanism and topology but omits the steps; and L3 provides the scope of access, constraints, and target effects. Success determination combines deterministic checks with LLM-based review of execution evidence. The paper evaluates five models using OpenHands, retaining a single result for each model-environment-information level combination. Table 3 reports that Claude Opus achieved the highest success rates: 48.8% for L1, 35.0% for L2, and 12.5% for L3. Additionally, optimization experiments for DNSBomb under known mechanisms report a peak downstream response rate up to approximately 171,268 times the average query input rate, alongside a case study of out-of-domain cache poisoning. The primary contributions of this paper are the construction of the benchmark and the design of the evaluation. The evidence demonstrates that some models can produce the specified effects within the constructed environments; however, it is insufficient to establish a stable ranking of models or to prove that models can generally infer vulnerability mechanisms from black-box observations. The paper explicitly acknowledges this latter limitation.
Critical strengths
-
The definition of the evaluation target holds clear scientific value. Section 3.1 emphasizes that merely deploying multiple services does not constitute a cross-component vulnerability task; the intended effect must rely on security-relevant interactions between components. DNSBomb—leveraging timing dependencies and discrepancies in HTTP request boundary interpretation—serves as a concrete example. This provides the benchmark with a clearly defined research object and establishes a sound direction for evaluating interaction capabilities beyond mere source code analysis.
-
The criteria for determining success prioritize observable effects and evidentiary sources. Sections 3.3 and 4.2, along with Appendices A and E, mandate deterministic verification, independent evidence, and mechanism alignment; they reject agent self-reporting as sufficient proof and prohibit LLMs from reclassifying runs that failed initial thresholds as successful. These principles are crucial for security agent benchmarks. Furthermore, the paper explicitly limits the scope of interpretation regarding DNSBomb rates and acknowledges the limitations inherent in single-instance results and existing mechanisms, thereby avoiding the pitfall of framing observations as broad conclusions about general capabilities.
Critical weaknesses
-
Appendix E claims that the final verdicts of the two LLM judges align with expert verification; however, it fails to report the number of verification samples, the selection method, whether experts made independent judgments, or whether initial "harness" failure cases were reviewed. Consequently, this claim does not adequately substantiate the judges' overall accuracy, particularly regarding the assessment of false negatives. Without addressing false negatives, it is difficult to determine the correctness and validity of the experimental results. Section 3.2 requires the submission of an executable PoC that achieves the desired effect within the specified budget; yet, Step 8 in Figure 5 indicates that the conversation reached the 60-turn limit and that the "Harness selects the live-session result." The figure does not demonstrate a successful replay of the submitted PoC from a clean state. Achieving an effect during interaction is a different evaluation criterion from delivering a reproducible exploit program; the current materials do not explain how this case satisfies both requirements, nor do they clarify its relationship to the main results matrix. This aspect is unclear to me.
-
The measurements for DNSBomb also require verification. Appendix D retains only the last 20,000 lines of the log increment, using this subset to calculate query input volume, time span, and peak response rates. If early query records were truncated, the denominator and time span values could be altered. The paper does not report whether truncation occurred for specific models or if recalculations based on full logs would yield consistent results. Figure 2 illustrates the process from query logs to pulse statistics—noting that query logs record only queries—whereas Appendix D relies on downstream response ([out]) records; the implementation details regarding data collection for both are not clearly explained. These questions directly concern the "success" reported in the paper and the prominently displayed attack intensity figures. While they do not prove the results incorrect, they render the current evidence insufficient to fully substantiate the reliability of the measurements.
-
Section 3.4 states that the instances, targets, judges, and budget remain unchanged, and that L3 continues to provide the permitted operations. However, while levels L1 and L2 in Appendix B explicitly state that the agent can run an authoritative service at 127.0.0.1:53535 and describe the forwarding relationship for
agent.lab, the L3 example specifies only the target service port without detailing the corresponding control permissions. While obscuring protocol mechanisms and network topology may be a reasonable way to set difficulty levels, the scope of agent control and the methods for discovering these interfaces within authorized limits must be clearly defined. Otherwise, a drop in success rates might reflect a lack of information regarding legitimate operational parameters, rather than merely the difficulty of crafting an exploit based on limited observations. Page 10 notes that the agent can read/taskand/opt/scenario.jsonbut does not show the content of these inputs across the different levels, making it impossible to verify whether information is missing or inadvertently exposed. Furthermore, L1 and L2 define a "qualifying pulse" as a one-second interval meeting a specific threshold, whereas L3 requires "distinct pulses." If the same detection logic is used for all three, the handling of this terminological discrepancy needs to be clarified. For a benchmark where information levels serve as the primary experimental variable, these are not merely peripheral implementation details; they are fundamental to ensuring fair comparisons and interpretable results.
Questions for authors
-
Is success contingent upon PoC replay? Please clarify whether the "live-session success" shown in Figure 5 is reflected in Tables 3–5, and whether the finally submitted PoC was subsequently executed from a clean state to reproduce the effect. Is the same success criterion applied across all tasks?
-
What is the evidence for the consistency between the verifiers and the experts? Please provide the number of cases fed into each verification stage, the scope and independence of expert verification, and the inspection results regarding cases where the harness failed. Does the consistency between the two judges and the experts refer to their initial independent assessments or their consensus following final adjudication?
-
Are the complete inputs and privileges comparable across the three information levels? Please specify what information is disclosed by the Prompt,
/task,/opt/scenario.json, and other readable files, respectively. How does DNSBomb L3 inform the agent of—or allow it to discover—control privileges over the authoritative service? Do the "one-second interval" and "distinct pulses" in the main task correspond to the same judgment criterion? -
Are the DNSBomb results robust against log truncation and data collection methods? For each of the five models, please provide the total query bytes, time span, peak downstream bytes, and the number of truncated logs; specify where the
[out]records are generated; and present the results recalculated using complete records or independent traffic evidence. -
Can you provide task-by-task records supporting the overall validity of the benchmark? The current PDF primarily presents aggregate counts by category. Please provide details for the 80 environments—including source mechanisms, software versions, key configurations, attacker privileges, success conditions, and summaries of expert PoC verification—and explain the rules regarding re-run limits, actual re-run counts, and result retention for instances involving tool errors, environment failures, or incomplete evidence.