acceptodds
Under review as a conference paper at ICLR 2027

CyberClear: A Benchmark for LLM Agent Systems on APT Attack Chain Provenance

Abstract

Large language model (LLM) agents have demonstrated promising capabilities in cybersecurity tasks, yet their ability to reconstruct complete Advanced Persistent Threat (APT) attack campaigns from complex security logs remains largely unexplored. Existing cybersecurity benchmarks for agents mainly focus on vulnerability discovery, exploitation, and security analysis tasks, leaving the evaluation of attack chain provenance under realistic security logs insufficiently studied. To address this gap, we introduce CyberClear, a benchmark for evaluating LLM agents and advanced agent systems on APT attack chain provenance from long-context security logs. CyberClear covers both single-step attacks and multi-stage attack chains, requiring agents to identify attack evidence, infer attack progression, and generate provenance graphs containing entities, causal relationships, MITRE ATT&CK techniques, and forensic evidence. To enable comprehensive evaluation, we develop an evaluation method tailored to APT attack chain provenance. Unlike conventional text similarity metrics that focus on surface-level matching, our evaluation examines whether reconstructed graphs preserve the semantics of attack chains across single-step behavior correctness, multi-step behavior identification, temporal and causal consistency, entity and relationship fidelity, and overall attack narrative consistency. Advanced multi-agent systems powered by state-of-the-art LLMs still struggle on CyberClear, motivating us to propose CyberProvenance, an agent cyber harness designed for multi-agents that augments LLM agents with evidence accumulation, execution-based validation, and feedback-guided refinement mechanisms for reliable attack-chain provenance. Extensive evaluations on CyberClear demonstrate the effectiveness of CyberProvenance in improving evidence reasoning, execution-grounded validation, and complete APT attack chain reconstruction. https://anonymous-hf.com/a/80jyozm7fh8d/Anonymous Benchmark The benchmark is partially available and will be fully released.

Primary area:
datasets and benchmarks
Keywords:
Advanced Persistent Threat, Agents System, Cybersecurity Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_37FMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CyberClear evaluates LLM agents on reconstructing attack chains from security logs. It contains 450 instances with reference provenance graphs and an LLM-based semantic evaluator. The paper also proposes CyberProvenance, which uses evidence memory, execution validation, and iterative refinement.

Critical strengths
  1. The benchmark targets attack-chain reconstruction rather than isolated attack detection, providing a useful evaluation task for LLM agents.

  2. The experiments cover multiple agent frameworks and model backbones, with component ablations and comparison against human judgments.

Critical weaknesses
  1. Reproducing an attack shows that it can run, but not that it explains the original logs. The paper does not specify how closely the reproduced traces must match the observed evidence before a step is accepted.

  2. A correct attack step may fail to reproduce because of missing dependencies or tool limitations. The paper does not explain how such failures are separated from incorrect predictions before the graph is revised.

  3. Reference graphs are built using attacker logs, while agents only receive defender logs. Expert review checks evidence coverage, but it is unclear whether every scored step is supported by the logs available to the agents.

  4. The high correlations with human scores are difficult to assess without knowing how many outputs were reviewed, how they were selected, and how closely the two experts agreed.

  5. CyberProvenance uses specialized execution tools, but the comparison reports neither their overhead nor inference costs. The results therefore show accuracy gains without establishing their computational cost.

  6. The log-length statistics in Table 2 do not agree. The overall mean is below every subset mean, and the hard-subset maximum cannot coexist with its reported mean and sample count if they measure the same quantity.

Questions for authors

Please refer to the concerns I raised above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 37FM (1/2)

Dear Reviewer 37FM,

Thank you for your thoughtful and constructive review. We appreciate your recognition of CyberClear’s value as a benchmark for attack-chain reconstruction and the breadth of our experiments across agent frameworks and model backbones. We address your concerns below.


W1: Trace alignment and step acceptance criteria.

We clarify that Sec. 4 requires predicted attack steps to be successfully reproduced and consistent with the collected evidence before the corresponding provenance graph is accepted. The Check Agent assesses graph structure, attack-step dependencies, and ATT&CK mappings. Appendix D.2 clarifies that the ATT&CK technique ID identifies the behavior category, while the observed logs guide the selection of a replay implementation consistent with the original execution context.

For benchmark evaluation, Appendix C provides explicit step-level criteria. Loose averages the semantic correctness of matches to reference attack steps regardless of order. Strict additionally requires attack events to appear in the correct temporal positions and penalizes unsupported steps. A correct step must preserve the actor, core action, target, and overall attack meaning.


W2: Execution failures versus prediction errors.

To distinguish replay failures from incorrect predictions before revising the graph, our workflow uses the existing validation_status field. We set it to failed when a dependency check or tool response identifies a missing dependency, an incompatible platform, or an unsupported operation. We also use failed when a replay times out, an expected event is not observed, or the cause remains unknown. The fields validation_failure_type and validation_failure_reason record the failure category and the available diagnostic evidence, respectively. The failed status describes the replay outcome. Before revising the predicted step, the Check Agent checks the recorded reason against the original log evidence.


W3: Defender-log support for reference attack steps.

Thank you for your question. As described in Section 3.2, during Automatic Verification, we use attacker logs as ground truth to check whether the entities, relationships, and attack timeline in each reference graph are consistent with the recorded attack. We also check graph readability, coverage of the required ATT&CK techniques, and format compliance. Graphs that fail any check are returned for regeneration. Experts then review the verified graphs and exclude samples with incomplete traces or insufficient evidence coverage. At evaluation time, agents receive only defender logs, and the judge compares the predicted graph with the reference graph without consulting attacker logs. During expert review, we checked each scored reference step against the defender logs provided to the agent and retained only steps supported by those logs.


W4: Human evaluation coverage and annotator agreement.

Our human evaluation covers all 4,950 outputs from 11 methods across 450 samples using the GPT-5.6-Terra backbone. Two experts independently scored each output three times using Appendix C, assessing step correctness, coverage, temporal order, graph details, and overall attack meaning. Scores range from 0 to 1, allowing semantically equivalent expressions and ignoring visual formatting differences. We averaged each expert’s repeated ratings and then averaged the two expert means. Correlations with automatic scores were calculated at the individual-output level. Absolute-agreement ICC(A,1) between the experts’ per-output means ranges from 0.9257 to 0.9516. All point estimates exceed the 0.90 threshold for excellent reliability illustrated in MIRAGE [1]. We have added the human evaluation protocol and inter-annotator agreement results to Appendix A.8 and marked the changes in blue.

DimensionICC(A,1)
Single0.9516
Loose0.9384
Strict0.9257
Detail0.9441
Scenario0.9340

[1] Dongre V, Gui C, Garg S, et al. MIRAGE: A benchmark for multimodal information-seeking and reasoning in agricultural expert-guided conversations. NeurIPS 2025.

AuthorsOpenReview ↗

Response to Reviewer 37FM (2/2)

W5: Execution overhead and inference costs.

We compared CyberProvenance with CAMEL and AgentScope-V2 across three backbones, using API calls for all inference. CyberProvenance used 106M tokens on Qwen3.8-Max, 103M on DeepSeek-V4-Flash, and 79M on GPT-5.6-Terra. Its largest saving was on GPT-5.6-Terra, where CAMEL used 90M tokens and AgentScope-V2 used 80M. Across the three backbones, CyberProvenance used 288M tokens in total, 2.4% fewer than CAMEL (295M) and 4.0% fewer than AgentScope-V2 (300M). Its reported inference cost was **\98.24forthethreebackbones∗∗.Thereportedcostsof98.24 for the three backbones**. The reported costs of \\92.98 for CAMEL and \$112.48 for AgentScope-V2 cover five backbones.

MethodQwen3.8-Max TokensDeepSeek-V4-Flash TokensGPT-5.6-Terra TokensTotal TokensTotal Cost
CAMEL105M100M90M295M$92.98
AgentScope-V2120M100M80M300M$112.48
CyberProvenance (Ours)106M103M79M288M$98.24

W6: Inconsistent log-length statistics in Table 2.

We sincerely apologize for the errors in the log-length statistics in Table 2 and the confusion they caused. We have corrected the affected entries in the revised manuscript and highlighted the changes in blue. We apologize again for this oversight and thank the reviewer for identifying it.


We hope our responses address your concerns. We have revised the manuscript accordingly and would be happy to answer any further questions. We respectfully ask you to consider adjusting your score.

AuthorsOpenReview ↗

Summary of Our Response to Reviewer 37FM

Dear Reviewer 37FM,

We wanted to share a brief summary of our response to your review. We clarified that CyberProvenance accepts a predicted step only when it is successfully reproduced and consistent with the original defender logs. When replay fails, the Check Agent checks the recorded failure reason against those logs before deciding whether to revise the graph.

Appendix A.8 reports agreement between the two experts who independently scored all 4,950 GPT-5.6-Terra outputs. We also reported CyberProvenance’s inference cost of $98.24 across three backbones. When building the reference graphs, our experts checked which attack steps they could identify from the defender logs given to the agents. We scored agents only on those steps.

Finally, we clarified that the overall mean log length was calculated across all 450 samples. We corrected the log-length statistics in Table 2 of Section 3.3 and marked the revised values in blue.

We hope the revisions address your concerns and would welcome any further comments or questions.

Dear Reviewer 37FM,

I hope this message finds you well. With one day remaining in the discussion period, we would appreciate hearing whether our response and revisions address your concerns. If any points need further clarification or you have additional suggestions, we would be happy to discuss them.

Thank you for your time and thoughtful review.

Reviewer_FRu5OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CyberClear, a benchmark for evaluating LLM agents on APT attack-chain provenance reconstruction from defender-side logs. The benchmark contains 450 instances and evaluates both DOT generation quality and the semantic correctness of reconstructed attack graphs.

Critical strengths
  1. The task addresses a meaningful security problem. Reconstructing attack behaviors and dependencies from defender-side logs requires integrating fragmented evidence into an interpretable account of an attack. CyberClear turns this practical investigation problem into a concrete benchmark, providing a useful foundation for evaluating cybersecurity agents.

  2. The evaluation distinguishes graph validity from attack correctness. The semantic dimensions examine attack behaviors, ordering, graph details, and scenario consistency alongside code-level metrics. This is consequential because a compilable DOT graph can still misrepresent the underlying attack. The evaluation therefore captures aspects of provenance reconstruction that surface-level similarity alone would miss.

  3. The proposed harness has a coherent domain motivation. Evidence Memory supports reasoning across distributed logs, Execution Validation supplies feedback about predicted attack steps, and Feedback Refinement enables local correction. These components form an understandable workflow for evidence-constrained reconstruction, making the method relevant to the specific demands of security investigation.

  4. The experiments provide several complementary perspectives. Comparisons across agent frameworks and LLM backbones establish useful reference results, while component ablations examine the contribution of the harness modules. The human-agreement analysis provides an additional check on the semantic evaluation protocol. Together, these analyses make the benchmark informative for subsequent research.

Critical weaknesses
  1. The benchmark contains a relatively modest number of instances. With 450 instances, readers would benefit from a more prominent explanation of the amount of log evidence associated with each case. Highlighting per-instance log volume alongside instance count would make the benchmark’s scale and task granularity easier to assess.

  2. The scope of human validation could be stated more explicitly. Appendix A.3 reports encouraging agreement with human judgments using GPT-5.6-Terra results. However, this analysis does not establish equivalent agreement across every backbone or attack category. The wording should more clearly reflect the coverage of the reported validation.

  3. The central methodological insight could be foregrounded. The presentation includes several agents, memory components, and security tools, which can draw attention away from the contribution of the overall workflow. A more concise explanation of how execution feedback participates in evidence-constrained refinement would help readers identify the key insight and interpret the component ablations appropriately.

  4. The operational requirements deserve greater visibility. Execution validation depends on compatible procedures, environment configurations, and telemetry. These details are discussed in the appendix, but a brief summary in the main text would clarify the conditions under which the method applies and the adaptation required for new environments.

Questions for authors
  1. CyberClear contains 450 instances. How much log data does each instance include, and how should this be considered when interpreting the benchmark’s scale?

  2. What is the intended scope of the human-agreement results in Appendix A.3? How should readers interpret high Pearson correlation in relation to individual-score agreement and reliability across other backbones?

  3. What is the central methodological insight of CyberProvenance beyond general-purpose multi-agent collaboration, and which existing results most directly support this insight?

  4. When adapting execution validation to a new operating system or ATT&CK technique, which procedures, environment settings, and telemetry components require modification?

  5. Could the authors explicitly distinguish information used during reference-graph construction from information available during inference, particularly the source of the Technique IDs received by the validation agent?

Two minor wording issues should also be corrected:

  • In Appendix A.1, page 16, “across all five Semantic Metrics metrics” contains a duplicated word. Please revise it to “across all five semantic metrics.”

  • The Table 2 caption on page 5, “Statistics of the CyberClear,” contains an unnecessary definite article. “Statistics of CyberClear” or “Statistics of the CyberClear benchmark” would be more natural.

4 comments
AuthorsOpenReview ↗

Response to Reviewer FRu5 (1/2)

Dear Reviewer FRu5,

Thank you for your thorough and constructive review. We appreciate your suggestions on clarifying the scale of CyberClear, the scope of our human evaluation, and the role of execution feedback in CyberProvenance. We address each concern below.


W1: Benchmark scale and the amount of log evidence per case.

Thank you for your question. We would like to clarify that CyberClear contains 450 attack-chain reconstruction cases comprise 2.1 GB of data. As shown in Table 2, cases contain an average of 5.78 log files, with up to 92 files in a single case.

CyberClear has more instances than most of the benchmarks listed in Table 1. CyberGym is larger, with 1,507 instances. However, its primary evaluation setting gives agents a description of the vulnerability to reproduce, making it unsuitable for evaluating APT attack-chain reconstruction. Every CyberClear case places additional evaluation demands on agents. They must identify attack events in defender logs without prior attack clues and, in multi-step cases, reconstruct their temporal order and causal dependencies.


W2: Scope of human validation across backbones and attack categories.

The human evaluation in Appendix A.3 covers all 4,950 outputs generated by 11 methods using the GPT-5.6-Terra backbone on 450 cases. Two experts independently scored each output three times using the rubric in Appendix C. For each output, we averaged each expert’s three ratings and then averaged the two expert means to obtain the human score. Figure 8 reports Pearson correlations between these human scores and the automatic scores, with attack categories pooled. We have added the mean number of log files per case and the observed range to the Dataset Scale paragraph of Section 3.3. The new text is marked in blue.


W3: The role of execution feedback in attack-chain refinement.

Thank you for the comment.CyberProvenance uses execution feedback together with the original defender logs to decide which parts of a reconstructed attack chain need revision. The Log Analysis Agent builds an initial graph from evidence accumulated across logs. The Check Agent then examines replay results alongside that evidence, revises the affected steps, and preserves validated nodes. As Section 4 states, the graph is retained when its steps are successfully reproduced and consistent with the collected evidence.


W4: Operational requirements for execution-based validation.

Thank you for suggesting that we make these requirements more visible in the main text. Appendices B.2 and D.2 describe three conditions for execution-based validation. The selected procedure must fit the execution context recorded in the defender logs, the replay environment must satisfy the procedure’s prerequisites, and the configured telemetry must capture the events needed for comparison.


Q1: Per-instance log volume across difficulty levels.

Table 2 shows how the overall average varies by difficulty. Easy cases average 3.26 log files, Medium cases average 5.99, and Hard cases average 83.2. Each Hard case contains between 79 and 92 files. These files jointly form the input for a single reconstruction: the agent must identify related events across the available logs and connect them in one provenance graph.


Q2: Interpretation of correlations between human and automatic scores.

Two experts reviewed all 4,950 outputs generated by 11 methods using GPT-5.6-Terra on 450 cases. Each expert independently scored each output three times using the rubric in Appendix C. We averaged each expert’s repeated ratings and then averaged the two expert means. Figure 8 reports output-level Pearson correlations of 0.9425 for Single, 0.9351 for Loose, 0.9445 for Strict, 0.9426 for Detail, and 0.9376 for Scenario.

These correlations show that outputs receiving higher human scores generally also received higher automatic scores in the GPT-5.6-Terra evaluation. Individual scores may still differ systematically while remaining highly correlated. To quantify agreement between the two human reviewers, we computed absolute-agreement ICC(A,1) using their per-output mean scores. Across the five dimensions, values range from 0.9257 to 0.9516, with all point estimates exceeding the 0.90 threshold for excellent reliability illustrated in MIRAGE [1]. Both analyses cover the same backbone and pool outputs across attack categories. We have added the human evaluation protocol and inter-annotator agreement results to Appendix A.8 and marked the additions in blue.

DimensionICC(A,1)
Single0.9516
Loose0.9384
Strict0.9257
Detail0.9441
Scenario0.9340

[1] Dongre V, Gui C, Garg S, et al. MIRAGE: A benchmark for multimodal information-seeking and reasoning in agricultural expert-guided conversations. NeurIPS 2025.

AuthorsOpenReview ↗

Response to Reviewer FRu5 (2/2)

Q3: Ablation evidence for evidence memory, execution validation, and refinement.

Table 5 provides the most direct evidence for the contribution of execution feedback. Removing Execution Validation lowers the mean semantic score from 0.7212 to 0.6472 and the Strict score from 0.6337 to 0.5485. The Strict result is especially relevant because it measures how much of the reference attack chain the agent reconstructs correctly in sequence.

Removing Evidence Memory lowers the mean semantic score to 0.6516, while removing Feedback Refinement lowers it to 0.6897. These ablations connect the workflow to the results: retaining evidence across logs, checking candidate steps through replay, and using the resulting feedback to revise the graph each contribute to reconstruction quality in the evaluated setting.


Q4: Adapting validation to new operating systems and attack techniques.

Appendix B.2 describes the use of Atomic Red Team for individual steps, CALDERA for steps with execution dependencies, and Ludus and Proxmox for environment provisioning. For a new operating system, the environment image and procedure implementation must support the target platform, the required dependencies must be available, and host telemetry must capture the expected events.

For a new ATT&CK technique on an existing platform, the technique ID identifies the behavior category, while the observed logs guide the choice of a compatible procedure, as described in Appendix D.2. Existing environment and telemetry configurations can be reused when they satisfy the procedure’s prerequisites and capture its expected behavior.


Q5: Source of ATT&CK technique IDs during inference.

The validation agent receives technique IDs predicted by the Log Analysis Agent from defender logs. During reference-graph construction, attacker logs and ATT&CK annotations are used to establish and verify the reference attack steps, as described in Section 3.2. During inference, the Log Analysis Agent identifies evidence in defender logs and predicts the attack steps and their technique IDs. The validation agent uses those predictions to select candidate procedures, while the environment configuration supplies platform and execution parameters.


Q6: Wording corrections in Appendix A.1 and the Table 2 caption.

Thank you for identifying both issues. We already change the phrase in Appendix A.1 to “across all five semantic metrics” and revise the Table 2 caption to “Statistics of CyberClear.”


Thank you again for recognizing the value of our work! We would be happy to answer any further questions.

Thank you for the responses. I have reviewed the response and revised manuscript, and the changes address my concerns about dataset scale, human validation, and execution feedback. I maintain my recommendation for acceptance.

Dear Reviewer FRu5,

Thank you for taking the time to read our response and the revised manuscript. We are glad the revisions addressed your concerns, and we appreciate your thoughtful feedback and continued support for our work.

Reviewer_VSKYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CyberClear, a benchmark that tests how well LLM agents can find and reconstruct cyberattack chains from security logs. It contains 450 examples covering 89 attack behaviors. The authors also propose CyberProvenance, a multi-agent system that uses stored evidence, attack testing, and repeated improvements to produce better attack graphs. The results show that CyberProvenance generally performs better than existing agent systems.

Critical strengths
  • The paper focuses on an important cybersecurity problem that is not well covered by existing LLM benchmarks.
  • The benchmark checks different parts of attack reconstruction, such as attack steps, their order, and relationships between events.
  • The authors test several LLMs and agent systems. The ablation studies also help show which parts of the proposed method are useful.
Critical weaknesses
  • Limited benchmark difficulty: Most examples (318 out of 450) contain only a single attack step, and only 10 are marked as hard. Since the paper focuses on complex attack chains, more difficult examples are needed to support this claim (Table 2).

  • Limited novelty: The benchmark is built using existing datasets, including PROVCON and CAM-LDS (Section 3.2). The proposed method also combines known ideas, such as storing evidence, testing attack steps, and improving results through feedback. It is not clear how much of the approach is actually new.

  • Concerns about evaluation: The authors use LLMs to generate and check the reference attack graphs and another LLM to score the results. Although they compare the scores with human judgments, more detailed human evaluation is needed to confirm that the scores are reliable.

  • Fairness of comparisons: CyberProvenance uses special tools for testing and checking attacks that the other methods do not have. Because of this, it is hard to tell whether the improvements come from the proposed method or from having access to better tools and more resources. A fair comparison using the same tools would make the results stronger.

  • Missing statistical analysis: The paper does not clearly show how much the results change across different runs. This makes it difficult to know how reliable the reported improvements are.

Questions for authors
  1. Benchmark diversity: How many unique attack campaigns are included in the 450 examples? Are some examples taken from the same attack campaign?

  2. Evaluation reliability: In Figure 8, are the human evaluation correlations calculated using individual examples or average scores across methods? How much did the two human reviewers agree with each other?

  3. Fairness of comparisons: Did the other agent systems have access to the same tools and computational resources as CyberProvenance? If not, how can we tell whether the improvements come from the proposed method or from the additional tools?

7 comments
AuthorsOpenReview ↗

Response to Reviewer VSKY (1/3)

Dear Reviewer VSKY,

We sincerely thank you for your thorough and constructive review. We appreciate your recognition of CyberClear’s importance, its multidimensional evaluation, and the breadth of our experiments. We address each concern below.


W1: Benchmark difficulty and insufficient hard examples.

We thank you for this thoughtful question. Advanced persistent threat (APT) campaigns are characterized by a high degree of stealth and the ability to persist for long periods [1], which makes traces of malicious activity difficult to identify. As Figures 7 and 10 show, existing agent systems still perform poorly on single-step attack provenance tasks. These results indicate that identifying stealthy APT activity remains challenging even in samples containing only a single attack step.

CyberClear contains 2.1 GB of data and 450 samples. Its sample count exceeds that of most existing cybersecurity benchmarks for agents listed in Table 1. CyberGym is larger, with 1,507 instances. However, CyberGym’s evaluation setting gives agents a description of the vulnerability to reproduce, making it unsuitable for evaluating APT attack-chain reconstruction. CyberClear’s all cases place additional demands on evaluation because agents must identify attack events without prior clues and reconstruct their temporal order and causal dependencies from security logs. For the dataset targeted for APT, the UNICORN SC-1 and SC-2 datasets for APT detection each contain 150 samples [2]. Within CyberClear, the 10 Hard samples capture realistic, large-scale APT scenarios. Expanding the Hard subset in the short term is difficult because it takes time to collect logs that capture how each stage of a prolonged, stealthy APT campaign leads to the next.

[1] Milajerdi S M, Gjomemo R, Eshete B, et al. Holmes: real-time apt detection through correlation of suspicious information flows. IEEE S&P 2019.

[2] Han X, Pasquier T, Bates A, et al. Unicorn: Runtime provenance-based detector for advanced persistent threats. NDSS 2020.


W2: Novelty of the benchmark and proposed method.

To our knowledge, CyberClear is the first benchmark for evaluating whether LLM agent systems can reconstruct provenance graphs of APT attack chains from security logs without prior attack clues. CyberClear reuses only the logs from PROVCON and CAM-LDS. We add expert-reviewed provenance graphs that capture attack steps, temporal order, and causal dependencies. We also propose an evaluation method that scores step identification separately from the reconstruction of order and dependencies.

CyberProvenance is a cybersecurity-specific agent harness for APT attack-chain reconstruction. Cybersecurity agents such as EnIGMA [1] and Co-RedTeam [2] retain observations, test candidate actions, and use execution feedback to refine subsequent actions. EnIGMA uses these mechanisms to solve CTF challenges, and Co-RedTeam uses them to discover vulnerabilities and validate exploits. CyberProvenance focuses on verifying causal dependencies in reconstructed APT attack chains. It uses replay to check whether an earlier step could have supplied a prerequisite for a later one, then checks whether the original defender logs support that link.

[1] Abramovich T, Udeshi M, Shao M, et al. EnIGMA: Interactive tools substantially assist LM agents in finding security vulnerabilities. ICML 2025.

[2] He P, Fox A, Miculicich L, et al. Co-redteam: Orchestrated security discovery and exploitation with llm agents. ICML 2026.


W3: Details of human evaluation.

Thank you for your comments. We reviewed all 4,950 outputs generated by the 11 methods using the GPT-5.6-Terra backbone on 450 samples. Two experts compared each predicted graph with its reference using the rubric in Appendix C. Single assesses the correctness of the core action, target, and meaning. Loose evaluates attack-step coverage regardless of order, while Strict assesses the longest consecutively correct prefix and penalizes unsupported actions. Detail evaluates entities, directed relations, and forensic attributes, while Scenario assesses the overall attack process, target, method, progression, and outcome. Scores range from 0 to 1. Experts accepted semantically equivalent expressions and ignored visual formatting differences. Missing, contradictory, and unsupported content was assessed under the corresponding criteria.

AuthorsOpenReview ↗

Response to Reviewer VSKY (2/3)

W4: Fairness of tools and computational resources.

We equipped CAMEL and AgentScope-V2 with the same tools as CyberProvenance and evaluated both using DeepSeek-V4-Flash, Qwen3.8-Max, and GPT-5.6-Terra. For CAMEL, its role-playing agents called the tools and incorporated execution feedback into their dialogue to refine their reconstructions. For AgentScope-V2, we registered the tools in its toolkit and shared execution feedback through structured messages to guide refinement. Following Appendix B.2, both used Atomic Red Team for individual attack steps and CALDERA for multi-step chains in the Ludus/Proxmox environment, with Sysmon, Linux Audit, and Zeek collecting execution traces.

MethodBackboneBLEUROUGE-LCodeBLEUPass@1SingleLooseStrictDetailScenario
CAMEL+ToolsDeepSeek-V4-Flash0.36790.56340.56180.92440.37670.31230.22810.24930.2155
Qwen3.8-Max0.42570.55760.57430.81780.68040.64110.48050.37260.5745
GPT-5.6-Terra0.39530.58900.42870.94220.54580.47020.30080.30160.4367
AgentScope-V2+ToolsDeepSeek-V4-Flash0.49730.62310.68430.86440.84590.72800.64250.56040.7274
Qwen3.8-Max0.50710.64120.68620.79110.83180.68630.60040.51380.6962
GPT-5.6-Terra0.50410.62770.72530.98670.86530.65940.61620.59870.7366
CyberProvenance (Ours)DeepSeek-V4-Flash0.49710.68730.68670.99330.86270.73250.63400.59810.6710
Qwen3.8-Max0.48100.68270.70520.89110.87640.70270.64770.57760.7087
GPT-5.6-Terra0.49520.69680.70160.93330.88970.74380.63370.61240.7263

Compared with Table 3, adding these tools yields modest average improvements in semantic scores. CyberProvenance maintains higher Single, Loose, and Detail scores across all three backbones.

These results indicate that CyberProvenance benefits from its harness, which integrates feedback on causal dependencies with specialized tools. We have revised Appendix A.6, Comparison with Tool-Equipped Baselines, to clarify this finding and marked the changes in blue.


W5: Run-to-run variability and statistical reliability.

Thank you for your question. Due to time and computational budget constraints, we evaluated CAMEL, AgentScope-V2, and CyberProvenance using DeepSeek-V4-Flash, Qwen3.8-Max, and GPT-5.6-Terra, with four runs for each method on each backbone. The table reports means and standard deviations. Standard deviations range from 0.0014 to 0.0425, indicating modest absolute run-to-run variation across the evaluated settings. We have added these results to Appendix A.7, Run-to-Run Variability, and marked the changes in blue.

MethodBackboneBLEUROUGE-LCodeBLEUPass@1SingleLooseStrictDetailScenario
CAMELDeepSeek-V4-Flash0.3559 (±0.0269)0.5650 (±0.0236)0.5623 (±0.0332)0.9206 (±0.0389)0.3764 (±0.0141)0.3033 (±0.0269)0.2353 (±0.0155)0.2209 (±0.0294)0.2074 (±0.0186)
Qwen3.8-Max0.4020 (±0.0275)0.5842 (±0.0054)0.5803 (±0.0126)0.7945 (±0.0121)0.6706 (±0.0275)0.6300 (±0.0396)0.4605 (±0.0425)0.3651 (±0.0195)0.5676 (±0.0338)
GPT-5.6-Terra0.3850 (±0.0391)0.5744 (±0.0110)0.4214 (±0.0143)0.9344 (±0.0312)0.5253 (±0.0344)0.4519 (±0.0256)0.2981 (±0.0154)0.3045 (±0.0249)0.4139 (±0.0014)
AgentScope-V2DeepSeek-V4-Flash0.4905 (±0.0372)0.6449 (±0.0205)0.6811 (±0.0143)0.8656 (±0.0289)0.8448 (±0.0137)0.7235 (±0.0391)0.6303 (±0.0197)0.5420 (±0.0344)0.6990 (±0.0122)
Qwen3.8-Max0.4963 (±0.0207)0.6496 (±0.0162)0.6816 (±0.0144)0.7872 (±0.0156)0.8386 (±0.0162)0.6750 (±0.0257)0.5869 (±0.0117)0.5194 (±0.0257)0.6956 (±0.0231)
GPT-5.6-Terra0.4992 (±0.0307)0.6273 (±0.0222)0.6919 (±0.0308)0.9867 (±0.0067)0.8596 (±0.0255)0.6577 (±0.0163)0.6114 (±0.0279)0.5773 (±0.0135)0.7169 (±0.0311)
CyberProvenance (Ours)DeepSeek-V4-Flash0.4905 (±0.0372)0.6749 (±0.0205)0.6811 (±0.0143)0.9956 (±0.0089)0.8548 (±0.0137)0.7235 (±0.0391)0.6303 (±0.0197)0.5920 (±0.0344)0.6790 (±0.0122)
Qwen3.8-Max0.4863 (±0.0207)0.6896 (±0.0162)0.6996 (±0.0144)0.8872 (±0.0156)0.8786 (±0.0162)0.6950 (±0.0257)0.6369 (±0.0117)0.5794 (±0.0257)0.6956 (±0.0231)
GPT-5.6-Terra0.4992 (±0.0307)0.6973 (±0.0222)0.6919 (±0.0308)0.9267 (±0.0067)0.8696 (±0.0255)0.7577 (±0.0163)0.6314 (±0.0279)0.6073 (±0.0135)0.7169 (±0.0311)
AuthorsOpenReview ↗

Response to Reviewer VSKY (3/3)

Q1: Independent campaign counts and instance overlap.

The 450 samples cover 89 attack behavior types across 12 attack scenarios. The examples below show that samples from the same campaign can differ in the number of attack steps, the participating devices, and the corresponding host IP addresses.

SourceScenarioAttack stepsLog-source devicesCorresponding host IPs
PROVCONAPT17CorporateServer192.168.56.11
PROVCONAPT18EmployeeWorkstation192.168.56.20
CAM-LDSScenario 3: Lateral Movement1reposerver172.17.100.122
CAM-LDSScenario 3: Lateral Movement3reposerver;linuxshare;inetfw;wazuh172.17.100.122; 192.168.100.23; 172.17.100.254; 192.168.100.130

Q2: Correlation calculation and inter-annotator agreement.

Correlations between automatic scores and human judgments are calculated at the individual-sample level. Two experts independently scored each output three times according to the rubric in Appendix C. We first averaged each expert’s repeated ratings and then averaged the two expert means. We further report inter-annotator agreement between the two experts’ per-output mean scores using absolute-agreement ICC(A,1). Values range from 0.9257 to 0.9516, with all point estimates exceeding the 0.90 threshold for excellent reliability illustrated in MIRAGE [1]. The revised manuscript reports the human evaluation protocol and inter-annotator agreement results in Appendix A.8, with the additions highlighted in blue.

DimensionICC(A,1)
Single0.9516
Loose0.9384
Strict0.9257
Detail0.9441
Scenario0.9340

[1] Dongre V, Gui C, Garg S, et al. MIRAGE: A benchmark for multimodal information-seeking and reasoning in agricultural expert-guided conversations. NeurIPS 2025.


Q3: Shared tools, matched budgets, and attribution of improvements.

We report total token usage and costs across the three backbones. CyberProvenance uses 106M tokens on Qwen3.8-Max, 103M on DeepSeek-V4-Flash, and 79M on GPT-5.6-Terra, totaling 288M tokens, compared with 329M for CAMEL+Tools and 332M for AgentScope-V2+Tools across the same three backbones. Together with W4, these results show that CyberProvenance achieves its performance gains under shared tools with lower aggregate token consumption. We have added a comparison of token usage and inference costs to Appendix B.3, Inference Costs under Shared Tool Access.

MethodQwen3.8-Max TokensDeepSeek-V4-Flash TokensGPT-5.6-Terra TokensTotal TokensTotal Cost
CAMEL105M100M90M295M$92.98
CAMEL+Tools114M112M103M329M$120.63
AgentScope-V2120M100M80M300M$112.48
AgentScope-V2+Tools119M107M106M332M$119.07
CyberProvenance (Ours)106M103M79M288M$98.24

We hope our responses have addressed your concerns and would be happy to clarify any remaining points. We respectfully ask you to consider adjusting your score in light of these clarifications. Thank you again for your thoughtful review.

Comments on the paper

1 comment

Dear Area Chair,

We sincerely appreciate your time and effort in coordinating the review of our submission. We introduce CyberClear, the first benchmark for evaluating LLM agents on APT attack-chain reconstruction from security logs without prior attack clues. CyberClear contains 2.1 GB of data across 450 cases covering 89 attack behaviors. Each case includes an expert-reviewed provenance graph that records the attack steps, their temporal order, and their causal dependencies. Our evaluation method scores step identification separately from the reconstruction of temporal order and causal dependencies.

We also introduce CyberProvenance, which integrates security tools into attack-chain reconstruction. It replays earlier steps to identify the conditions they can produce, then uses the original security logs to check proposed links to later steps and revise the graph.

We addressed three recurring concerns in the reviews.

First, we evaluated all three systems using the same tools and backbones. On the code-level metrics, CyberProvenance achieved the highest ROUGE-L on all three backbones and the highest CodeBLEU and Pass@1 on DeepSeek-V4-Flash and Qwen3.8-Max. AgentScope-V2+Tools led on BLEU across all three backbones and on CodeBLEU and Pass@1 with GPT-5.6-Terra. On the semantic metrics, CyberProvenance scored higher than CAMEL+Tools on all five metrics across all three backbones. It also scored higher than AgentScope-V2+Tools on Single, Loose, and Detail across all three backbones and on Strict with Qwen3.8-Max and GPT-5.6-Terra.

**Second, we reported token usage and API inference costs. Across the three backbones, CyberProvenance used 288M tokens at a cost of \98.24.∗∗Bothtool−equippedbaselinesusedmoretokensandincurredhighercosts.CAMEL+Toolsused329Mtokensatacostof98.24.** Both tool-equipped baselines used more tokens and incurred higher costs. CAMEL+Tools used 329M tokens at a cost of \\120.63, and AgentScope-V2+Tools used 332M tokens at a cost of \119.07.Withouttheadditionaltools,CAMELused295MtokensandAgentScope−V2used300Mtokensacrossthesamethreebackbones.TheirreportedAPIcostsacrossfivebackboneswere119.07. Without the additional tools, CAMEL used 295M tokens and AgentScope-V2 used 300M tokens across the same three backbones. Their reported API costs across five backbones were \\92.98 for CAMEL and \$112.48 for AgentScope-V2. Although CAMEL’s reported API cost was lower than CyberProvenance’s, CAMEL scored lower on all five semantic metrics across all three backbones.

Third, we quantified agreement between human experts. Two experts independently scored all 4,950 outputs generated by 11 methods on 450 cases using GPT-5.6-Terra. Across the five semantic metrics, the absolute-agreement ICC(A,1) between the experts’ per-output mean ratings ranged from 0.9257 to 0.9516.

We hope these results help inform your assessment of CyberClear and its contribution. Thank you once again for your valuable time!

Sincerely,

The authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.