- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents POLAR, an LLM pipeline that combines unstructured reports (advisories, blogs, CTI) with sources like NVD, MITRE ATT&CK, CISA KEV, and Exploit-DB. It splits incidents into threats, links them to evidence, predicts CVSS metrics, estimates a 30-day exploitation probability, and recommends mitigations. The authors evaluate on about 12.8K CVE-linked incidents, in full-information and zero-day settings, and report that POLAR beats all baselines.
Critical strengths
The problem is practical and worth studying.
POLAR can use unstructured text that fixed-feature models cannot, and it gives three useful outputs: CVSS, exploitation probability, and mitigations.
The authors compare with several baselines and run ablations on multiple backbones.
Critical weaknesses
Zero-day setup not described: The authors do not say how zero-day samples were built, how many there are, or what time point defines “zero-day.” Appendix E.1 requires every case to have a CVE and NVD entry, which conflicts with this setting. Table 4 cannot be checked.
Memorization: The backbones have likely seen these CVEs, CVSS vectors, and exploitation history during training. The authors do not report the CVE time range, do not split by model cutoff, and include no memorization check.
Circular evaluation: The reference answers come from the same sources POLAR retrieves from (NVD, KEV, vendor advisories). High scores may reflect lookup rather than reasoning.
Weak check of p_k: The probability is just a number returned by the LLM, compared to one fixed EPSS snapshot. EPSS changes daily and is itself a prediction, not real exploitation.
Missing details: The “reference urgency order,” ranking group size, and how mitigations are matched to references are not explained. Gemini’s results differ between Table 1 and Table 7.
No real-world or human evaluation: No analyst checks the outputs, and nothing tests whether top-ranked CVEs were later exploited.
Suspicious pattern in Tables 5 and 6: For every model, the 90-day results are almost a fixed multiple of the 30-day results. Within each trend type, 90-day RMSE is about 1.25 times the 30-day RMSE (about 1.24 for abrupt changes), 90-day direction accuracy is about 0.92 times the 30-day value (about 0.87 for stable and abrupt), and the improvement over the backbone shrinks by about 0.82. This holds across all five general LLMs and all three security models. Independent runs of different models would not give such constant ratios, and a simple rescaling cannot explain the fixed drop in direction accuracy. Do the authors care to explain how the 90-day results were obtained and to share the raw predictions ?
Questions for authors
Were zero-day samples made by hiding data from known CVEs? Which fields were hidden, and what is the ground truth for Table 4?
Can you report results only on CVEs disclosed after each model’s knowledge cutoff?
If POLAR is rerun at day 0, 7, and 30, does p_k move like EPSS or real exploitation? How much does it vary across repeated runs?
How were the ATT&CK mappings for all 12.8K CVEs created, and will the dataset be released?