- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies why large language models fail in cyber threat intelligence (CTI) analysis across four stages: contextualization, attribution, prediction, and mitigation. The authors combine CTIBench, SevenLLM-Bench, SWE-Bench, CyberTeam, and real-world threat reports to construct 28 tasks and evaluate 12 general-purpose and cybersecurity-specific models.
The authors rank responses by similarity to reference answers and divide them into strata of 5% each. They manually inspect a few samples to establish score ranges for identifying errors automatically, with additional review of boundary cases. Humans initialize the taxonomy, and multiple models classify failures over two rounds. Cases involving disagreement or changing labels receive human review. The taxonomy contains three categories—spurious correlations, contradictory knowledge, and constrained generalization—and 15 subtypes. Classification is evaluated by comparing category proportions with human annotations. A separate study uses three independent annotators per sample on 320 samples to measure agreement among humans.
The authors also propose C³, a counterfactual consistency check that changes temporal, sector, or environmental information, checks whether responses change, and combines multiple responses through weighted voting. They use it to analyze failure causes and compare task scores before and after subtype-specific defenses.
Critical strengths
Studying LLM errors in CTI analysis has practical value. The evaluation covers multiple models and tasks. The authors provide an annotation protocol, agreement analysis, and code. The result has some human calibration
Critical weaknesses
The paper aims to explain model failures in CTI, but there are problems with failure classification, counterfactual interventions, and evaluation. I therefore do not recommend acceptance.
First, failure identification is not validated. The authors use a few manually inspected samples to establish score ranges for labeling the remaining responses. However, similar scores do not imply the same correctness. The paper should report precision and recall for this procedure; otherwise, the subsequent classification and analysis of failure causes rest on an unreliable foundation.
Second, category proportions do not establish classification accuracy. Swapping all labels between two equally sized categories leaves their proportions unchanged. The agreement study on 320 samples compares human annotators, not AI predictions against human labels. The authors should compare AI and human labels instance by instance and report precision, recall, and F1 for each subtype. Reviewing only disputed or unstable predictions may also miss errors on which all models agree.
Third, some C³ transformations change the correct answer. Exploit likelihood and patch recommendations may depend on time and environment, yet the code replaces Windows Server with Linux Server while retaining the original reference answer. Changing the response may then be correct and should not be treated as an error.
Fourth, some ground truth and metrics are inappropriate. EPSS is itself a prediction model, so agreement with its scores does not establish accurate prediction of actual exploitation. YARA rules and event timelines are evaluated using BLEU, but textual similarity does not establish effective detection or factual correctness.
Fifth, the code does not match the annotation procedure. Filtering thresholds come from mean scores and fixed offsets rather than human labels. Boundary cases are decided heuristically by default but recorded as manual_boundary. The later classification stage allows human input but retains only one label, contrary to the multi-label formulation. The package lacks human labels and instance-level results. The authors should provide the code and annotation records actually used.
Questions for authors
On the 320 manually annotated samples, how well do the AI classifications match the human labels at the instance level, rather than just in category proportions?
How do the authors ensure that C³ transformations preserve the correct answer?
Thank you! And please see the point-wise response below as well as updated manuscript
Dear Reviewers, AC, SAC, and all editors,
We would like to take this opportunity to thanks for your time and efforts for organizing the review process. Please see below regarding our point-wise responses with intensive revision efforts. Besides:
We would greatly appreciate the reviewer's further assessment of these improvements and warmly welcome any additional feedback or discussion. And we sincerely hope that our responses and revisions have adequately addressed the concerns raised and further improved the clarity and overall quality of the manuscript.
All the best,
The author team