- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CyberClear evaluates LLM agents on reconstructing attack chains from security logs. It contains 450 instances with reference provenance graphs and an LLM-based semantic evaluator. The paper also proposes CyberProvenance, which uses evidence memory, execution validation, and iterative refinement.
Critical strengths
-
The benchmark targets attack-chain reconstruction rather than isolated attack detection, providing a useful evaluation task for LLM agents.
-
The experiments cover multiple agent frameworks and model backbones, with component ablations and comparison against human judgments.
Critical weaknesses
-
Reproducing an attack shows that it can run, but not that it explains the original logs. The paper does not specify how closely the reproduced traces must match the observed evidence before a step is accepted.
-
A correct attack step may fail to reproduce because of missing dependencies or tool limitations. The paper does not explain how such failures are separated from incorrect predictions before the graph is revised.
-
Reference graphs are built using attacker logs, while agents only receive defender logs. Expert review checks evidence coverage, but it is unclear whether every scored step is supported by the logs available to the agents.
-
The high correlations with human scores are difficult to assess without knowing how many outputs were reviewed, how they were selected, and how closely the two experts agreed.
-
CyberProvenance uses specialized execution tools, but the comparison reports neither their overhead nor inference costs. The results therefore show accuracy gains without establishing their computational cost.
-
The log-length statistics in Table 2 do not agree. The overall mean is below every subset mean, and the hard-subset maximum cannot coexist with its reported mean and sample count if they measure the same quantity.
Questions for authors
Please refer to the concerns I raised above.
4 comments
Response to Reviewer 37FM (2/2)
W5: Execution overhead and inference costs.
We compared CyberProvenance with CAMEL and AgentScope-V2 across three backbones, using API calls for all inference. CyberProvenance used 106M tokens on Qwen3.8-Max, 103M on DeepSeek-V4-Flash, and 79M on GPT-5.6-Terra. Its largest saving was on GPT-5.6-Terra, where CAMEL used 90M tokens and AgentScope-V2 used 80M. Across the three backbones, CyberProvenance used 288M tokens in total, 2.4% fewer than CAMEL (295M) and 4.0% fewer than AgentScope-V2 (300M). Its reported inference cost was **\92.98 for CAMEL and \$112.48 for AgentScope-V2 cover five backbones.
| Method | Qwen3.8-Max Tokens | DeepSeek-V4-Flash Tokens | GPT-5.6-Terra Tokens | Total Tokens | Total Cost |
|---|---|---|---|---|---|
| CAMEL | 105M | 100M | 90M | 295M | $92.98 |
| AgentScope-V2 | 120M | 100M | 80M | 300M | $112.48 |
| CyberProvenance (Ours) | 106M | 103M | 79M | 288M | $98.24 |
W6: Inconsistent log-length statistics in Table 2.
We sincerely apologize for the errors in the log-length statistics in Table 2 and the confusion they caused. We have corrected the affected entries in the revised manuscript and highlighted the changes in blue. We apologize again for this oversight and thank the reviewer for identifying it.
We hope our responses address your concerns. We have revised the manuscript accordingly and would be happy to answer any further questions. We respectfully ask you to consider adjusting your score.
Summary of Our Response to Reviewer 37FM
Dear Reviewer 37FM,
We wanted to share a brief summary of our response to your review. We clarified that CyberProvenance accepts a predicted step only when it is successfully reproduced and consistent with the original defender logs. When replay fails, the Check Agent checks the recorded failure reason against those logs before deciding whether to revise the graph.
Appendix A.8 reports agreement between the two experts who independently scored all 4,950 GPT-5.6-Terra outputs. We also reported CyberProvenance’s inference cost of $98.24 across three backbones. When building the reference graphs, our experts checked which attack steps they could identify from the defender logs given to the agents. We scored agents only on those steps.
Finally, we clarified that the overall mean log length was calculated across all 450 samples. We corrected the log-length statistics in Table 2 of Section 3.3 and marked the revised values in blue.
We hope the revisions address your concerns and would welcome any further comments or questions.
Dear Reviewer 37FM,
I hope this message finds you well. With one day remaining in the discussion period, we would appreciate hearing whether our response and revisions address your concerns. If any points need further clarification or you have additional suggestions, we would be happy to discuss them.
Thank you for your time and thoughtful review.
Response to Reviewer 37FM (1/2)
Dear Reviewer 37FM,
Thank you for your thoughtful and constructive review. We appreciate your recognition of CyberClear’s value as a benchmark for attack-chain reconstruction and the breadth of our experiments across agent frameworks and model backbones. We address your concerns below.
We clarify that Sec. 4 requires predicted attack steps to be successfully reproduced and consistent with the collected evidence before the corresponding provenance graph is accepted. The Check Agent assesses graph structure, attack-step dependencies, and ATT&CK mappings. Appendix D.2 clarifies that the ATT&CK technique ID identifies the behavior category, while the observed logs guide the selection of a replay implementation consistent with the original execution context.
For benchmark evaluation, Appendix C provides explicit step-level criteria. Loose averages the semantic correctness of matches to reference attack steps regardless of order. Strict additionally requires attack events to appear in the correct temporal positions and penalizes unsupported steps. A correct step must preserve the actor, core action, target, and overall attack meaning.
To distinguish replay failures from incorrect predictions before revising the graph, our workflow uses the existing
validation_statusfield. We set it tofailedwhen a dependency check or tool response identifies a missing dependency, an incompatible platform, or an unsupported operation. We also usefailedwhen a replay times out, an expected event is not observed, or the cause remains unknown. The fieldsvalidation_failure_typeandvalidation_failure_reasonrecord the failure category and the available diagnostic evidence, respectively. Thefailedstatus describes the replay outcome. Before revising the predicted step, the Check Agent checks the recorded reason against the original log evidence.Thank you for your question. As described in Section 3.2, during Automatic Verification, we use attacker logs as ground truth to check whether the entities, relationships, and attack timeline in each reference graph are consistent with the recorded attack. We also check graph readability, coverage of the required ATT&CK techniques, and format compliance. Graphs that fail any check are returned for regeneration. Experts then review the verified graphs and exclude samples with incomplete traces or insufficient evidence coverage. At evaluation time, agents receive only defender logs, and the judge compares the predicted graph with the reference graph without consulting attacker logs. During expert review, we checked each scored reference step against the defender logs provided to the agent and retained only steps supported by those logs.
Our human evaluation covers all 4,950 outputs from 11 methods across 450 samples using the GPT-5.6-Terra backbone. Two experts independently scored each output three times using Appendix C, assessing step correctness, coverage, temporal order, graph details, and overall attack meaning. Scores range from 0 to 1, allowing semantically equivalent expressions and ignoring visual formatting differences. We averaged each expert’s repeated ratings and then averaged the two expert means. Correlations with automatic scores were calculated at the individual-output level. Absolute-agreement ICC(A,1) between the experts’ per-output means ranges from 0.9257 to 0.9516. All point estimates exceed the 0.90 threshold for excellent reliability illustrated in MIRAGE [1]. We have added the human evaluation protocol and inter-annotator agreement results to Appendix A.8 and marked the changes in blue.
[1] Dongre V, Gui C, Garg S, et al. MIRAGE: A benchmark for multimodal information-seeking and reasoning in agricultural expert-guided conversations. NeurIPS 2025.