acceptodds
Under review as a conference paper at ICLR 2027

Uncovering Failure Modes of LLM-Assisted Cyber Threat Intelligence

Abstract

Large language models (LLMs) are increasingly used to help security analysts manage the surge of cyber threats, automating tasks from vulnerability assessment to incident response. Yet in CTI workflows, reliability gaps remain substantial. Existing explanations often point to generic model issues (e.g., hallucination), but we argue the dominant bottleneck is the threat landscape itself: CTI is heterogeneous, volatile, and fragmented. Under these conditions, evidence is intertwined, crowdsourced, and temporally unstable, which are properties that standard LLM-based studies rarely capture. This paper presents a comprehensive empirical study of LLM failure modes in CTI reasoning. We introduce a heterogeneous categorization framework that robustly labels failure modes across the CTI lifecycle, avoiding the brittleness of automated LLM-as-a-judge pipelines. We identify three domain-specific cognitive failures: spurious correlations from superficial metadata, contradictory knowledge from conflicting sources, and constrained generalization to emerging threats. We validate these mechanisms via causal interventions and show that targeted defenses reduce failure rates significantly. Together, these results offer a concrete roadmap for building resilient, domain-aware CTI agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Cyber Threat Intelligence, Large Language Models, LLM Reliability, Failure Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6icKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why large language models fail in cyber threat intelligence (CTI) analysis across four stages: contextualization, attribution, prediction, and mitigation. The authors combine CTIBench, SevenLLM-Bench, SWE-Bench, CyberTeam, and real-world threat reports to construct 28 tasks and evaluate 12 general-purpose and cybersecurity-specific models.

The authors rank responses by similarity to reference answers and divide them into strata of 5% each. They manually inspect a few samples to establish score ranges for identifying errors automatically, with additional review of boundary cases. Humans initialize the taxonomy, and multiple models classify failures over two rounds. Cases involving disagreement or changing labels receive human review. The taxonomy contains three categories—spurious correlations, contradictory knowledge, and constrained generalization—and 15 subtypes. Classification is evaluated by comparing category proportions with human annotations. A separate study uses three independent annotators per sample on 320 samples to measure agreement among humans.

The authors also propose C³, a counterfactual consistency check that changes temporal, sector, or environmental information, checks whether responses change, and combines multiple responses through weighted voting. They use it to analyze failure causes and compare task scores before and after subtype-specific defenses.

Critical strengths

Studying LLM errors in CTI analysis has practical value. The evaluation covers multiple models and tasks. The authors provide an annotation protocol, agreement analysis, and code. The result has some human calibration

Critical weaknesses

The paper aims to explain model failures in CTI, but there are problems with failure classification, counterfactual interventions, and evaluation. I therefore do not recommend acceptance.

First, failure identification is not validated. The authors use a few manually inspected samples to establish score ranges for labeling the remaining responses. However, similar scores do not imply the same correctness. The paper should report precision and recall for this procedure; otherwise, the subsequent classification and analysis of failure causes rest on an unreliable foundation.

Second, category proportions do not establish classification accuracy. Swapping all labels between two equally sized categories leaves their proportions unchanged. The agreement study on 320 samples compares human annotators, not AI predictions against human labels. The authors should compare AI and human labels instance by instance and report precision, recall, and F1 for each subtype. Reviewing only disputed or unstable predictions may also miss errors on which all models agree.

Third, some C³ transformations change the correct answer. Exploit likelihood and patch recommendations may depend on time and environment, yet the code replaces Windows Server with Linux Server while retaining the original reference answer. Changing the response may then be correct and should not be treated as an error.

Fourth, some ground truth and metrics are inappropriate. EPSS is itself a prediction model, so agreement with its scores does not establish accurate prediction of actual exploitation. YARA rules and event timelines are evaluated using BLEU, but textual similarity does not establish effective detection or factual correctness.

Fifth, the code does not match the annotation procedure. Filtering thresholds come from mean scores and fixed offsets rather than human labels. Boundary cases are decided heuristically by default but recorded as manual_boundary. The later classification stage allows human input but retains only one label, contrary to the multi-label formulation. The package lacks human labels and instance-level results. The authors should provide the code and annotation records actually used.

Questions for authors

On the 320 manually annotated samples, how well do the AI classifications match the human labels at the instance level, rather than just in category proportions?

How do the authors ensure that C³ transformations preserve the correct answer?

Reviewer_jXTFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates failure modes of LLM-assisted cyber threat intelligence (CTI) across four stages: contextualization, attribution, prediction, and mitigation. The authors evaluate general-purpose and cybersecurity-specialized LLMs using existing benchmarks and real-world threat data, and propose a human-in-the-loop framework for categorizing failures through iterative taxonomy refinement and multi-agent judgments. The analysis identifies three major failure categories: spurious correlations, contradictory knowledge, and constrained generalization, along with their underlying causes and effects on CTI workflows. The authors validate their categorization method against human annotations and automated baselines, conduct case studies and causal interventions, and evaluate an inference-time counterfactual consistency strategy that reportedly reduces failures across several CTI tasks.

Critical strengths

The paper addresses an important practical problem by systematically investigating LLM failures in cyber threat intelligence. A key strength is its broad empirical evaluation across four CTI stages, multiple benchmarks, and 12 general-purpose and cybersecurity-specialized models. Furthermore, the proposed taxonomy organizes observed failures into three major categories and 15 finer-grained subtypes, providing a structured view of how different limitations manifest across CTI tasks. This combination of broad evaluation and detailed categorization could provide a useful foundation for understanding and addressing failures in LLM-based CTI systems.

Critical weaknesses
  1. Limited novelty of the identified failure modes. The three main categories, spurious correlations, contradictory knowledge, and constrained generalization, are already well-known limitations of LLMs. While the paper provides CTI-specific examples and subcategories, the analysis remains largely descriptive. It is unclear what fundamentally new insights or failure mechanisms are uncovered beyond applying these known limitations to cybersecurity tasks.

  2. Lack of competitive baselines to establish the significance of the identified failures. The evaluation does not systematically compare against stronger CTI systems incorporating RAG, external knowledge verification, or other established grounding approaches. Although Table 11 includes a retrieval-only mitigation ablation and shows that retrieval can substantially reduce certain failures, the paper lacks comprehensive task-level comparisons against such approaches. Consequently, it is difficult to determine whether the reported failures reflect fundamental limitations of LLMs in CTI or shortcomings that existing methods could already address.

Questions for authors
  1. What new CTI-specific insights does the proposed taxonomy provide beyond already known LLM failure modes?

  2. How would the identified failure rates change when using stronger RAG-based or evidence-verification approaches?

  3. Can the authors report instance-level agreement between the automated failure categorization and human annotations, beyond the aggregate distributions in Table 3?

Reviewer_DzawOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability challenges of large language models (LLMs) in cyber threat intelligence (CTI) reasoning, emphasizing the heterogeneous, volatile, and fragmented nature of threat intelligence rather than generic model limitations. The authors conduct a comprehensive empirical study and introduce a heterogeneous failure categorization framework, identifying three domain-specific failure modes: spurious correlations from superficial metadata, contradictory knowledge from conflicting sources, and limited generalization to emerging threats. Through causal interventions and targeted defenses, the study validates these failure mechanisms and demonstrates significant reductions in failure rates, providing insights for developing more reliable and domain-aware CTI agents.

Critical strengths
  1. The research topic is valuable and meaningful, regarding both LLM and its application in CTI analysis.
  2. The studied problem of LLM's reliability does make sense.
  3. The overall structure and design of this study is good, following the evaluation, failure categorization, analysis, and mitigation. This is a good way to systematically trouble shooting and problem solving.
  4. The conduct study and experiments are extensive and the findings could be useful and helpful to guide practice.
Critical weaknesses
  1. This study lacks research depth. It is more like a technical report or an engineering practice, instead of a principled research.
  2. The major contribution lies in the overall bundle of the efforts, from evaluation and analysis to improvement, which is quite scattered. It would be better to more focus on certain part of the overall pipeline.
  3. The technical novelty is a bit weak. For example, the challenges of the problem should be well justified and strengthened, why simple solutions or existing methods cannot or hardly address these challenges? Otherwise, the technical contribution of the proposed method would be weak.
Questions for authors

please see the weaknesses above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you! And please see the point-wise response below as well as updated manuscript

Dear Reviewers, AC, SAC, and all editors,

We would like to take this opportunity to thanks for your time and efforts for organizing the review process. Please see below regarding our point-wise responses with intensive revision efforts. Besides:

  • we have uploaded revised manuscript, wherein Blue text indicates new updates or revised texts and Orange text indicates highlight to original contents to make it easier to be caught.
  • We have also integrated some new experimental results, with corresponding tables and figures updated. Please see below for detailed explanations of those efforts.

We would greatly appreciate the reviewer's further assessment of these improvements and warmly welcome any additional feedback or discussion. And we sincerely hope that our responses and revisions have adequately addressed the concerns raised and further improved the clarity and overall quality of the manuscript.

All the best,

The author team

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.