acceptodds
Withdrawn from ICLR 2027

Polar: LLM-Powered Synthesis of Real-World Cyber Evidence for Prioritization and Mitigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cyber threat analysis increasingly depends on evidence distributed across vendor advisories, vulnerability databases, and threat intelligence sources. Turning these fragmented observations into timely decisions requires models to connect technical severity with evolving exploitation evidence and available defensive actions. We present Polar, an LLM-powered framework for synthesizing real-world cyber evidence into threat-centric assessments for prioritization and mitigation. Polar first disentangles overlapping incidents and grounds each threat in source-linked evidence. For prioritization, it infers severity metrics from cyber evidence and combines the resulting assessment with temporally ordered exploitation signals to estimate near-term exploitation likelihood. For mitigation, it links the synthesized threat data to authoritative remediation knowledge and organizes applicable actions according to threat urgency and operational constraints. We evaluate Polar on real-world vulnerability evidence collected from public resources and compare it with multiple baselines. Across heterogeneous incidents and zero-day settings, Polar improves threat ranking and mitigation retrieval while producing evidence-linked intermediate assessments that support analyst inspection. The results establish evidence synthesis as a practical foundation for LLM-based cyber decision support across related security tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Cyber Threat Intelligence, Vulnerability Management, Large Language Models, Exploit Prediction, Risk-Aware Decision Making, Security Analytics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EiCkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents POLAR, an LLM pipeline that combines unstructured reports (advisories, blogs, CTI) with sources like NVD, MITRE ATT&CK, CISA KEV, and Exploit-DB. It splits incidents into threats, links them to evidence, predicts CVSS metrics, estimates a 30-day exploitation probability, and recommends mitigations. The authors evaluate on about 12.8K CVE-linked incidents, in full-information and zero-day settings, and report that POLAR beats all baselines.

Critical strengths

The problem is practical and worth studying.

POLAR can use unstructured text that fixed-feature models cannot, and it gives three useful outputs: CVSS, exploitation probability, and mitigations.

The authors compare with several baselines and run ablations on multiple backbones.

Critical weaknesses

Zero-day setup not described: The authors do not say how zero-day samples were built, how many there are, or what time point defines “zero-day.” Appendix E.1 requires every case to have a CVE and NVD entry, which conflicts with this setting. Table 4 cannot be checked.

Memorization: The backbones have likely seen these CVEs, CVSS vectors, and exploitation history during training. The authors do not report the CVE time range, do not split by model cutoff, and include no memorization check.

Circular evaluation: The reference answers come from the same sources POLAR retrieves from (NVD, KEV, vendor advisories). High scores may reflect lookup rather than reasoning.

Weak check of p_k: The probability is just a number returned by the LLM, compared to one fixed EPSS snapshot. EPSS changes daily and is itself a prediction, not real exploitation.

Missing details: The “reference urgency order,” ranking group size, and how mitigations are matched to references are not explained. Gemini’s results differ between Table 1 and Table 7.

No real-world or human evaluation: No analyst checks the outputs, and nothing tests whether top-ranked CVEs were later exploited.

Suspicious pattern in Tables 5 and 6: For every model, the 90-day results are almost a fixed multiple of the 30-day results. Within each trend type, 90-day RMSE is about 1.25 times the 30-day RMSE (about 1.24 for abrupt changes), 90-day direction accuracy is about 0.92 times the 30-day value (about 0.87 for stable and abrupt), and the improvement over the backbone shrinks by about 0.82. This holds across all five general LLMs and all three security models. Independent runs of different models would not give such constant ratios, and a simple rescaling cannot explain the fixed drop in direction accuracy. Do the authors care to explain how the 90-day results were obtained and to share the raw predictions ?

Questions for authors

Were zero-day samples made by hiding data from known CVEs? Which fields were hidden, and what is the ground truth for Table 4?

Can you report results only on CVEs disclosed after each model’s knowledge cutoff?

If POLAR is rerun at day 0, 7, and 30, does p_k move like EPSS or real exploitation? How much does it vary across repeated runs?

How were the ATT&CK mappings for all 12.8K CVEs created, and will the dataset be released?

Reviewer_4RfwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Why does BATA combine ensemble predictions differently during calibration and candidate selection? Section 4.1 averages the ensemble members’ predictions before converting them to ranks and learning the mixture weight. Section 4.3 instead ranks each member’s predictions separately, then uses these ranks to select candidates. Averaging predictions before ranking can produce a different ordering from ranking each member separately. Please explain why a weight learned using the first procedure is appropriate for the second, and whether this difference materially affects the selected candidates or optimization results. How were the initial measured variants and the two evaluation cohorts chosen? Sections 5.1 and 5.3 describe Main-35 and a separate sensitivity evaluation with 70 initializations for GB1 and TrpB. Were the initial 96 variants sampled uniformly from each candidate pool, or selected using restrictions such as fitness or distance from a reference sequence? Were both cohorts generated using the same sampling procedure, and how was Main-35 designated as the primary evaluation?

Critical strengths
  • While prior work has focused separately on either threat prioritization or mitigation planning, this paper presents the first end-to-end system that addresses both within a unified framework, which is a meaningful contribution to the field. Also, the incorporation of temporal exploitation context, such as PoC release dates and malware observation timestamps, into exploitation probability estimation reflects the realities of operational security, where the urgency of a threat evolves over time rather than remaining static.
  • The evaluation is conducted across five representative LLM backbones spanning both general-purpose and cybersecurity-specialized models, which strengthens the generalizability of the reported results and demonstrates that POLAR’s gains are not specific to a single model. Furthermore, the system integrates evidence from a diverse set of authoritative public sources (e.g. NVD, MITRE ATT&CK, CISA KEV, Exploit-DB, and VirusTotal) reflecting the heterogeneous nature of real-world threat intelligence workflows.
Critical weaknesses
  • The system uses EPSS as the ground truth for evaluating exploitation prediction accuracy; however, EPSS is itself a predictive model rather than an empirical observation of real-world exploitation. Evaluating one model against another raises concerns about circular validation. The inclusion of complementary ground truth signals, such as confirmed in-the-wild exploitation records or CISA KEV membership as a binary label, would substantially strengthen the evaluation. Also, the experiments rely exclusively on publicly available data sources, without any simulation or case study grounded in real enterprise environments. It remains unclear whether POLAR would deliver meaningful performance gains when applied to settings that include internal asset inventories, proprietary threat feeds, or organization-specific telemetry. A controlled pilot study or realistic enterprise simulation would help bridge this gap.
  • The paper presents a well-engineered pipeline, but the core components, namely prompt design, retrieval-augmented generation, and structured extraction, build entirely on existing techniques. It is not sufficiently clear what new, generalizable knowledge this work contributes beyond a carefully assembled application system. The significance of the work to the broader research community would be strengthened by articulating more explicitly what transferable insights or methodological advances other researchers can build upon.
Questions for authors

Does POLAR incorporate any safeguards against cases where the LLM produces confidently incorrect exploitation probability estimates due to flawed reasoning or hallucination? It would be helpful to know whether the authors tracked false positive rates or other error statistics during their experiments, and if so, to what extent such errors occurred.

Reviewer_Kj8aOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents POLAR, an LLM-based framework for synthesizing heterogeneous cybersecurity evidence to support threat prioritization and mitigation recommendation. The framework integrates information from vulnerability databases, vendor advisories, exploit repositories, and threat intelligence sources. It employs LLMs to disentangle threat information, assess vulnerability severity, estimate exploitation likelihood, and recommend mitigation actions. The authors evaluate POLAR across multiple LLM backbones and cybersecurity baselines, reporting improvements in threat ranking and mitigation retrieval.

Critical strengths
  1. The paper addresses a practically important problem in cybersecurity, namely synthesizing heterogeneous threat intelligence for vulnerability prioritization and mitigation.

  2. The evaluation covers multiple LLM backbones and includes comparisons with existing cybersecurity-oriented methods.

Critical weaknesses
  1. Limited technical novelty and insufficient methodological contribution for ICLR. My primary concern is that the technical contribution appears rather limited relative to the expectations of ICLR. The proposed framework mainly combines existing LLM capabilities with external information retrieval, structured prompting, CVSS-based severity assessment, temporal evidence organization, and mitigation recommendation. While integrating these components into a unified cybersecurity workflow may have practical value, the paper does not appear to introduce a substantially new learning algorithm, model architecture, reasoning mechanism, or principled approach to improving LLM reliability. The central contribution seems to be demonstrating that LLMs can synthesize heterogeneous cybersecurity evidence and achieve better performance on threat prioritization and mitigation retrieval when provided with structured information and task-specific instructions. However, this observation is not particularly surprising given the established capabilities of modern LLMs and retrieval-augmented reasoning systems. It is also unclear what generalizable methodological insights can be drawn from the proposed framework beyond this particular application domain. In its current form, the work appears closer to an application-oriented system design and empirical study of LLM capabilities in cybersecurity than a substantial machine learning contribution. Consequently, I question whether the paper is sufficiently aligned with ICLR.

  2. Insufficient validation of evidence reliability and LLM hallucination risks. A fundamental concern is whether POLAR can produce trustworthy decisions from large-scale, heterogeneous, and potentially unreliable online cybersecurity information. The authors recognize the importance of evidence provenance and discuss mechanisms for handling conflicting information. However, preserving source provenance and applying heuristic conflict-resolution rules do not necessarily guarantee the correctness of the underlying evidence or the faithfulness of the LLM's conclusions. In real-world cybersecurity settings, information collected from online sources can be incomplete, outdated, contradictory, or even deliberately misleading. An LLM may incorrectly interpret retrieved information, combine unrelated evidence, overstate the credibility of uncertain reports, or generate conclusions that are not actually supported by the cited sources. Although the paper describes mechanisms intended to mitigate these issues, I do not find sufficiently rigorous validation showing that they effectively prevent such failures.

  3. Evaluation does not convincingly establish real-world decision correctness. The experimental results demonstrate improvements in threat ranking and mitigation retrieval. However, I am not convinced that the chosen evaluation metrics adequately establish whether POLAR makes reliable and correct cybersecurity decisions. The evaluation of threat prioritization raises concerns regarding whether the reported performance reflects genuine predictive capability. For example, the authors use existing cybersecurity assessment signals, including EPSS, as references and evaluate prioritization against reference urgency rankings. However, agreement with existing risk assessments does not necessarily demonstrate the ability to accurately forecast future exploitation events. Since POLAR explicitly estimates 30-day exploitation probabilities, it would be important to clarify whether these predictions are evaluated against independently observed exploitation outcomes. For mitigation recommendation, the evaluation mainly uses F1 against database-derived reference mitigation items. However, correctly retrieving or reproducing known mitigation information does not necessarily demonstrate that a recommendation is appropriate, safe, or effective for a specific deployment environment.

  4. The paper includes ablation studies showing performance degradation when individual pipeline components are removed. While useful, these experiments do not fully establish why POLAR outperforms existing methods. In particular, the observed gains could partly arise from providing the LLM with richer external evidence, more structured prompts, or additional inference-time computation rather than from a fundamentally new reasoning methodology.

Questions for authors
  1. What is the fundamental methodological contribution of POLAR beyond combining existing LLM reasoning, retrieval, and structured cybersecurity knowledge? What generalizable machine learning insights does the framework provide?

  2. Although POLAR retains source provenance and describes conflict-resolution mechanisms, how do the authors independently verify that generated assessments are faithful to their cited evidence? Can they provide quantitative measurements of hallucinations, unsupported claims, and incorrect evidence attribution? How does POLAR handle contradictory, outdated, or deliberately misleading threat intelligence? Have the authors evaluated performance under controlled evidence corruption or conflicting-source settings?

  3. Are the predicted 30-day exploitation probabilities validated against actual future exploitation events?

  4. How much of POLAR's performance improvement can be attributed to additional information retrieval, prompt engineering, or inference-time computation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.