- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents CTIFoundry, a system that reorganizes cyber threat intelligence for use by an LLM agent. It combines official relations between security knowledge bases, entities and aliases extracted from vendor reports, links to supporting text, and hybrid retrieval. Seven tools and task-specific playbooks make this information accessible to an agent running a shared harness.
The evaluation covers nine CTIConnect tasks and four models. Compared with an agent accessing the corpus through a shell, CTIFoundry improves the average score by 0.190–0.275. The paper also evaluates extraction and entity resolution, examines the contributions of tools and playbooks, and reports construction and query costs. The largest improvements occur on entity-linking tasks whose answers can be obtained through official knowledge-base relations.
Critical strengths
1.Clear improvements across four models, particularly on entity linking.
Table 2 reports higher overall scores for all four models, including an increase from 0.610 to 0.829 for GPT-5.4 and from 0.470 to 0.745 for Haiku. Several entity-linking tasks approach perfect accuracy. The consistency across models supports the practical value of making official knowledge-base relations directly accessible to an agent.
2.The evaluation examines several parts of the system.
Beyond final answer scores, the paper evaluates extraction quality across six models, entity resolution against 50 report clusters, agreement between the synthesis judge and human assessments, and construction and query costs. The 2×2 tool/playbook ablation also provides useful evidence that different task families benefit from different parts of the system.
Critical weaknesses
1.The shell-based comparison does not establish the advantage over relevant retrieval and graph-query baselines.
The main baseline accesses raw corpus files through a shell. CTIFoundry adds hybrid retrieval, direct lookup of official relations, extracted entities, and task-specific playbooks. This is a useful comparison of the complete systems, but it leaves open whether a simpler agent equipped with standard retrieval and relation-lookup tools could obtain similar gains.
This is consequential because CTIConnect already provides domain-specific retrieval strategies, including entity-based report retrieval. A matched comparison with a relevant baseline would help establish what the proposed scaffold adds. The 2×2 ablation is informative, but it removes all tools together and therefore does not separate the value of basic retrieval access from that of the more specialized corpus design. Appendix H also indicates that the playbooks retained in the shell-only condition refer to operations unavailable there; how these instructions were adapted needs clarification. The evidence currently supports “this complete system improves on raw-file access” more strongly than the broader claim that corpus structure, rather than agent capability, is the main bottleneck.
2.The report layer’s contribution to answer quality remains unclear, and its reliability is interpreted too broadly.
Alias resolution and evidence-linked report indexing are important parts of the proposed method, but their downstream contribution is not isolated. In Table 3, GPT-5.4 obtains very similar scores on all three synthesis tasks with the full system and without the tools. Appendix H.5 also allows the agent to read report IDs supplied in the question, which can reduce the need to discover relevant reports through the entity index. The improved cluster-reconstruction results are useful, but do not by themselves show that this layer improves the final answers.
The extraction results also do not support the statement that smaller models lose quality “almost entirely in recall.” For GPT-5.4-mini, the model used to build the scaffold, Table 1 reports entity-extraction precision of 0.708 and TTP-grounding precision of 0.611, compared with 0.846 and 0.749 for GPT-5.4. Precision therefore declines substantially as well. The zero-fabricated-identifier check is valuable, but establishes identifier validity rather than the correctness of an entity type or a behavior-to-technique assignment. A targeted report-layer ablation and an analysis of these remaining errors would strengthen this part of the paper.
Questions for authors
1.Could you compare CTIFoundry with a relevant domain-specific retrieval baseline, or a simpler agent given hybrid search and official-relation lookup, under the same model and evaluation settings? In the “w/o tools” condition, how were the playbooks adapted to the available shell operations?
2.How do the synthesis results change when alias resolution and entity-based report collection are disabled while report reading remains available? What proportion of synthesis questions already provides the relevant report IDs?
3.Section 3.2 states that the report layer emits no relational triples, while Appendices G.2 and G.5 describe triple extraction and validation. Are these intermediate outputs that are later discarded, or do they contribute to the evaluated system? Please also clarify how the static task-to-playbook mapping in Appendix H relates to the statement in Section 4.4 that tool-use behavior is unprompted by a router.
4.Please specify the size of the development slice mentioned in Section 4.1 and its separation from the 691-question main set and the 1,168-question scale set. Does the submitted supplement contain the split IDs, frozen playbooks, corpus snapshots, and configurations needed to reproduce Tables 2 and 3?
5.Is Figure 3(a) intended as a schematic illustration or a quantitative plot of the reported runs? Some GPT-5.4 points appear lower than the corresponding values in Tables 2 and 3. If the figure is schematic, please state this in the caption; otherwise, please check that it uses the same experimental results and averaging rule as the tables. I regard this as a presentation clarification rather than evidence against the reported results.
4 comments
Author response (part 2)
W2 (first paragraph) and Q2. The report layer's contribution to synthesis answers.
We appreciate the reviewer raising this concern, and we ran the ablation that the reviewer suggests.
Setup. No aliases or entity index: resolve_entity matches canonical names and identifiers only, chunks_mentioning is removed, and read_report, search_chunks and the KB tools stay. We run it with gpt-5.4 on the MDS questions of a stratified random sample of 150 main-set questions. On the share of questions that supply report ids: No main-set MDS question supplies report ids. The first step of the synthesis playbook in App. H.5 is conditional on a question naming reports and did not apply.
Table R3. MDS mean (CSC, TAP, MLA) on the sampled questions. Difference = full minus reduced, 95% paired bootstrap.
| MDS mean | gpt-5.4 |
|---|---|
| Full report layer | 0.855 |
| No aliases or entity index | 0.845 |
| Difference | +0.010 [-0.025, +0.046] |
Table R3 shows that the two conditions are level, as are the full system and the w/o tools arm in Table 3. The reason is that MDS is close to standard retrieval and summarization over text, where a general agent is already strong: the base agent scores 0.826 on MDS with gpt-5.4, against 0.498 on EL, so little room is left for any layer to raise the score. What the report layer adds on these tasks is what §4.2 measures: it reconstructs the cross-vendor report cluster and links every statement to its evidence. We have revised §4.5, marked in blue: "MDS needs neither" now reads "MDS is on par in all four configurations", and App. K.4 reports the ablation.
W2 (second paragraph). Extraction precision, and identifier validity.
We thank the reviewer for the careful reading of Table 1, and we analysed the remaining errors as suggested.
Setup. We labelled by hand every false positive of gpt-5.4-mini, the build model, on the Table 1 gold set: 155 for entities and 63 for TTP grounding.
Table R4. Causes of gpt-5.4-mini false positives, share within each layer.
| Layer | Cause | Share |
|---|---|---|
| Entity | wrong type | 38% |
| Entity | span boundary | 29% |
| Entity | spurious mention | 25% |
| Entity | other | 8% |
| TTP | sibling technique | 35% |
| TTP | parent/child confusion | 19% |
| TTP | defender behavior | 11% |
| TTP | should have abstained | 24% |
| TTP | other | 11% |
Table R4 shows that no single cause dominates. Wrong type leads for entities and sibling technique for TTPs, the categories also reported for LLM extraction [2] and for LLM ATT&CK mapping [3, 4]. These errors do not reach the EL and EA results: the ontology layer is built from the official knowledge bases with no extraction model (§3.2), and the EL and EA playbooks use only the KB tools. They can reach synthesis answers, where the report-layer tools are used (Fig. 2(b)). We have revised the paper, with the changes marked in blue: §4.2 and App. E.1 now report the precision and recall of gpt-5.4-mini together (entity 0.708 and 0.781, TTP 0.611 and 0.692, against 0.846, 0.872, 0.749 and 0.841 for gpt-5.4), and state that the build guarantees identifier and span validity while types and technique assignments rest on the extraction model. App. K.5 reports Table R4.
Q3. Triples in App. G, and the static playbook map.
We thank the reviewer for cross-checking the appendices. The evaluated system uses no relational triples from reports. Only the alias_of statements of the App. G.2 prompt are kept, at build time, as alias evidence for entity resolution (App. A, Step 3 (d)). Every other relation statement is discarded and never materialized in the scaffold. The only relations a tool returns are the official KB cross-references that ontology_neighbors traverses. App. G.5 was used while developing an earlier pipeline that did materialize report triples, and it is not part of the build behind Tables 1 to 3. The task-to-playbook map is static: it is fixed before any question is read and involves no model call. The playbook recommends the opening calls, and every call inside the loop is the model's own choice (first-call compliance is 0.45 on ATA and 0.90 on VCA, Fig. 2(b)). We have revised the wording: §3.2 says that the layer exposes no relational triples to the agent and uses alias statements only as resolution evidence, App. G.5 is labelled as an earlier stage outside the evaluated build, and §4.4 says that a static task-to-playbook map selects each playbook.
Author response (part 3)
Q4. Development slice and the supplement.
We appreciate the reviewer raising this concern. The development slice has 92 questions, sampled at random from the scale set before any tuning, so it is disjoint from the 691-question main set behind Tables 2 and 3. The scale-set table in App. E.2 included these questions. We removed them and recomputed the table, and no task's F1 moves by more than 0.010. On the four items the reviewer lists: the split IDs of the main set (691), the scale set (1,168) and the development slice (92) are at https://anonymous.4open.science/r/repro_package-D241/. The frozen playbooks are in the submitted supplement and in App. H. The corpus is public: CTIConnect's released corpus and the four upstream knowledge bases. The configurations are the flags of the harness entry point in the supplement, and its README gives the commands for the rows of Table 3. The revised paper marks these changes in blue: §4.1 states the slice, and Table 6 in App. E.2 reports the scale set without it.
Q5. Figure 3(a).
We thank the reviewer for spotting this. Figure 3(a) is a quantitative plot of the reported runs. Its gpt-5.4 points used the MDS scores of an earlier judging pass, which places them about 0.02 below the tables. Tables 2 and 3 use the final pass and are unaffected. We have redrawn the figure from the table values in the revised paper, and its caption now states the averaging rule.
References:
- [1] CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligence. Cheng, Liu, Li, Song and Gao. KDD 2026.
- [2] Han et al.: Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors. Han, Peng, Yang, Wang, Liu and Wan. arXiv:2305.14450, 2023.
- [3] TechniqueRAG: Retrieval Augmented Generation for Adversarial Technique Annotation in Cyber Threat Intelligence Text. Lekssays, Shukla, Sencar and Parvez. Findings of ACL 2025.
- [4] Rule-ATT&CK Mapper (RAM): Mapping SIEM Rules to TTPs Using LLMs. Wudali, Kravchik, Malul, Gandhi, Elovici and Shabtai. arXiv:2502.02337, 2025.
Thank you for the additional experiments. The comparison with retrieval alone addresses my concern about the shell baseline. It shows that giving the agent direct access to official cross-references improves entity linking. My remaining reservation is how much the other parts of CTIFoundry contribute to answer quality. Even though none of the main-set synthesis questions supplies report IDs, disabling alias resolution and entity-based report lookup made little difference to the answers’ scores. Using a shared agent across three task families has practical value, but its scores are close to those of CTIConnect’s dedicated pipelines. I now have more confidence in the entity-linking result, though I still find the overall contribution too narrow for acceptance and maintain my rating of 2 (Weak Reject).
Author response (part 1)
We thank the reviewer for the detailed and constructive review.
W1 and Q1. Retrieval and relation-lookup baselines, and the playbooks in the w/o tools arm.
We appreciate the reviewer raising this concern, and we are happy to provide additional comparisons.
Setup. All runs use gpt-5.4 on the same 150 main-set questions, a stratified random sample in which each task keeps its share of the main set. Retrieval: the base agent equipped with
search_kb, hybrid dense and BM25 search over KB entries, andsearch_chunks, dense search over report chunks, with no skills. This is the simpler agent with standard retrieval that the reviewer describes. Retrieval + relation lookup: addsget_entityandontology_neighbors, which read the official cross-references. This is the agent with hybrid search and official-relation lookup that the reviewer describes. CTIConnect pipelines [1]: vanilla RAG and the domain-specific pipelines, Extract-then-Retrieve (EtR) for entity linking (EL), Decompose-then-Retrieve (DtR) for attribution (EA), and CSKG-guided RAG, its entity-based report retrieval, for synthesis (MDS), with the release's settings over the same records. Base + procedural guidance: the base agent with the playbooks, in which the steps that call typed tools are translated into natural language.Table R1. gpt-5.4, 150 sampled questions. Family means, Overall is the nine-task mean.
Table R2. gpt-5.4, 150 sampled questions, family means.
Table R1 shows that standard retrieval alone does not close the gap. Retrieval adds +0.027 [-0.030, +0.084] on EL over base (95% paired bootstrap), 11% of the gain from base to CTIFoundry w/o skill. The gain arrives with the official-relation lookup: Retrieval + relation lookup adds a further +0.206 [+0.104, +0.308] on EL. This lookup is what CTIFoundry builds: the official cross-references of the four knowledge bases, materialized as typed edges that a tool can traverse (§3.2). This agent is therefore CTIFoundry's ontology layer without the report layer and the skills, and the skills add a further +0.159 on EL. Table R2 shows that CTIFoundry is on par with CTIConnect's dedicated pipelines. The differences are small in every family (-0.003 on EL, +0.025 on EA and +0.024 on MDS), and none is distinguishable from zero (95% paired bootstrap). This is expected, because our playbooks build the design of these pipelines into the agent: the attribution playbooks decompose a passage into behaviors and restate each in the target taxonomy's idiom, as DtR does, and the entity linking playbook keeps EtR's restated-query search as its fallback. These designs carry the know-how of the CTI domain, and CTIFoundry keeps it. What CTIFoundry changes is the form: it turns three fixed retrieval pipelines, each built around one family's question format, into one flexible agentic scaffold. A single agent answers all nine tasks with the same tools, and a new task is added by plugging in a new playbook. On the playbooks in the w/o tools arm: that arm received the playbooks verbatim, and App. H notes that the steps that call typed tools "become inert there". With these steps translated into natural language, the base agent reaches 0.699 on EL, against 0.568 with the verbatim playbooks and 0.923 for CTIFoundry, so procedure alone leaves most of the EL gap open. We have revised the paper accordingly, with the changes marked in blue: §1 and §4.3 state the claim as one about access to the records, App. K.1 and K.2 report Tables R1 and R2, and App. I reports the playbook variants.