acceptodds
Under review as a conference paper at ICLR 2027

CTIFoundry: An Agent-Native Corpus Scaffold for Cyber Threat Intelligence

Abstract

Cyber threat intelligence (CTI) is increasingly consumed not by human analysts but by LLM agents that compose multi-step investigations at query time. The harness side of this shift has matured rapidly (planning loops, tool protocols, context management), but the corpus side has not: threat reports and vulnerability databases are still packaged for retrieval-augmented generation, as opaque chunks behind an embedding index. We argue that this substrate, not model capability, is the bottleneck on agentic CTI investigation, and present CTIFoundry, an agent-native corpus scaffold. At build time, CTIFoundry materializes the latent structure of a CTI corpus: a deterministic ontology graph over four authoritative knowledge bases (CVE, CWE, CAPEC, ATT&CK) whose official cross-references become typed, traversable edges; a span-grounded report layer whose canonical, alias-resolved cross-vendor entities index provenance-carrying chunks; and hybrid dense+lexical retrieval surfaces. At query time this structure is exposed through seven typed tools and three procedural skills mounted on a stock, widely-used open-source agent harness. On the public CTIConnect benchmark (nine tasks over entity linking, attribution, and multi-document synthesis), swapping only the action surface lifts the identically-harnessed agent from 0.610 to 0.829 overall F1 with gpt-5.4 and from 0.470 to 0.745 with claude-haiku-4-5: a small model on CTIFoundry surpasses a flagship model on the flat substrate. The scaffolded agent is simultaneously more accurate and more efficient: on both Claude models it answers with roughly half the tool calls per question. Together these results give design principles for corpus scaffolding over data of different modalities, in CTI and beyond.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
cyber threat intelligence, domain-specific scaffolding, knowledge graph, agentic retrieval, retrieval-augmented generation, tool use, agent-computer interface, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pbBPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CTIFoundry, a system that reorganizes cyber threat intelligence for use by an LLM agent. It combines official relations between security knowledge bases, entities and aliases extracted from vendor reports, links to supporting text, and hybrid retrieval. Seven tools and task-specific playbooks make this information accessible to an agent running a shared harness.

The evaluation covers nine CTIConnect tasks and four models. Compared with an agent accessing the corpus through a shell, CTIFoundry improves the average score by 0.190–0.275. The paper also evaluates extraction and entity resolution, examines the contributions of tools and playbooks, and reports construction and query costs. The largest improvements occur on entity-linking tasks whose answers can be obtained through official knowledge-base relations.

Critical strengths

1.Clear improvements across four models, particularly on entity linking.

Table 2 reports higher overall scores for all four models, including an increase from 0.610 to 0.829 for GPT-5.4 and from 0.470 to 0.745 for Haiku. Several entity-linking tasks approach perfect accuracy. The consistency across models supports the practical value of making official knowledge-base relations directly accessible to an agent.

2.The evaluation examines several parts of the system.

Beyond final answer scores, the paper evaluates extraction quality across six models, entity resolution against 50 report clusters, agreement between the synthesis judge and human assessments, and construction and query costs. The 2×2 tool/playbook ablation also provides useful evidence that different task families benefit from different parts of the system.

Critical weaknesses

1.The shell-based comparison does not establish the advantage over relevant retrieval and graph-query baselines.

The main baseline accesses raw corpus files through a shell. CTIFoundry adds hybrid retrieval, direct lookup of official relations, extracted entities, and task-specific playbooks. This is a useful comparison of the complete systems, but it leaves open whether a simpler agent equipped with standard retrieval and relation-lookup tools could obtain similar gains.

This is consequential because CTIConnect already provides domain-specific retrieval strategies, including entity-based report retrieval. A matched comparison with a relevant baseline would help establish what the proposed scaffold adds. The 2×2 ablation is informative, but it removes all tools together and therefore does not separate the value of basic retrieval access from that of the more specialized corpus design. Appendix H also indicates that the playbooks retained in the shell-only condition refer to operations unavailable there; how these instructions were adapted needs clarification. The evidence currently supports “this complete system improves on raw-file access” more strongly than the broader claim that corpus structure, rather than agent capability, is the main bottleneck.

2.The report layer’s contribution to answer quality remains unclear, and its reliability is interpreted too broadly.

Alias resolution and evidence-linked report indexing are important parts of the proposed method, but their downstream contribution is not isolated. In Table 3, GPT-5.4 obtains very similar scores on all three synthesis tasks with the full system and without the tools. Appendix H.5 also allows the agent to read report IDs supplied in the question, which can reduce the need to discover relevant reports through the entity index. The improved cluster-reconstruction results are useful, but do not by themselves show that this layer improves the final answers.

The extraction results also do not support the statement that smaller models lose quality “almost entirely in recall.” For GPT-5.4-mini, the model used to build the scaffold, Table 1 reports entity-extraction precision of 0.708 and TTP-grounding precision of 0.611, compared with 0.846 and 0.749 for GPT-5.4. Precision therefore declines substantially as well. The zero-fabricated-identifier check is valuable, but establishes identifier validity rather than the correctness of an entity type or a behavior-to-technique assignment. A targeted report-layer ablation and an analysis of these remaining errors would strengthen this part of the paper.

Questions for authors

1.Could you compare CTIFoundry with a relevant domain-specific retrieval baseline, or a simpler agent given hybrid search and official-relation lookup, under the same model and evaluation settings? In the “w/o tools” condition, how were the playbooks adapted to the available shell operations?

2.How do the synthesis results change when alias resolution and entity-based report collection are disabled while report reading remains available? What proportion of synthesis questions already provides the relevant report IDs?

3.Section 3.2 states that the report layer emits no relational triples, while Appendices G.2 and G.5 describe triple extraction and validation. Are these intermediate outputs that are later discarded, or do they contribute to the evaluated system? Please also clarify how the static task-to-playbook mapping in Appendix H relates to the statement in Section 4.4 that tool-use behavior is unprompted by a router.

4.Please specify the size of the development slice mentioned in Section 4.1 and its separation from the 691-question main set and the 1,168-question scale set. Does the submitted supplement contain the split IDs, frozen playbooks, corpus snapshots, and configurations needed to reproduce Tables 2 and 3?

5.Is Figure 3(a) intended as a schematic illustration or a quantitative plot of the reported runs? Some GPT-5.4 points appear lower than the corresponding values in Tables 2 and 3. If the figure is schematic, please state this in the caption; otherwise, please check that it uses the same experimental results and averaging rule as the tables. I regard this as a presentation clarification rather than evidence against the reported results.

4 comments
AuthorsOpenReview ↗

Author response (part 1)

We thank the reviewer for the detailed and constructive review.

W1 and Q1. Retrieval and relation-lookup baselines, and the playbooks in the w/o tools arm.

We appreciate the reviewer raising this concern, and we are happy to provide additional comparisons.

Setup. All runs use gpt-5.4 on the same 150 main-set questions, a stratified random sample in which each task keeps its share of the main set. Retrieval: the base agent equipped with search_kb, hybrid dense and BM25 search over KB entries, and search_chunks, dense search over report chunks, with no skills. This is the simpler agent with standard retrieval that the reviewer describes. Retrieval + relation lookup: adds get_entity and ontology_neighbors, which read the official cross-references. This is the agent with hybrid search and official-relation lookup that the reviewer describes. CTIConnect pipelines [1]: vanilla RAG and the domain-specific pipelines, Extract-then-Retrieve (EtR) for entity linking (EL), Decompose-then-Retrieve (DtR) for attribution (EA), and CSKG-guided RAG, its entity-based report retrieval, for synthesis (MDS), with the release's settings over the same records. Base + procedural guidance: the base agent with the playbooks, in which the steps that call typed tools are translated into natural language.

Table R1. gpt-5.4, 150 sampled questions. Family means, Overall is the nine-task mean.

ConfigurationELEAMDSOverall
Base (bash)0.5110.5120.8290.617
Retrieval0.5380.5190.8430.635
Retrieval + relation lookup0.7440.5270.8080.717
CTIFoundry w/o skill0.7640.5440.8020.728
CTIFoundry0.9230.6350.8550.836

Table R2. gpt-5.4, 150 sampled questions, family means.

SystemELEAMDSOverall
Vanilla RAG0.6990.5270.8310.705
EtR / DtR / CSKG-guided0.9260.6100.8280.823
CTIFoundry0.9230.6350.8550.836

Table R1 shows that standard retrieval alone does not close the gap. Retrieval adds +0.027 [-0.030, +0.084] on EL over base (95% paired bootstrap), 11% of the gain from base to CTIFoundry w/o skill. The gain arrives with the official-relation lookup: Retrieval + relation lookup adds a further +0.206 [+0.104, +0.308] on EL. This lookup is what CTIFoundry builds: the official cross-references of the four knowledge bases, materialized as typed edges that a tool can traverse (§3.2). This agent is therefore CTIFoundry's ontology layer without the report layer and the skills, and the skills add a further +0.159 on EL. Table R2 shows that CTIFoundry is on par with CTIConnect's dedicated pipelines. The differences are small in every family (-0.003 on EL, +0.025 on EA and +0.024 on MDS), and none is distinguishable from zero (95% paired bootstrap). This is expected, because our playbooks build the design of these pipelines into the agent: the attribution playbooks decompose a passage into behaviors and restate each in the target taxonomy's idiom, as DtR does, and the entity linking playbook keeps EtR's restated-query search as its fallback. These designs carry the know-how of the CTI domain, and CTIFoundry keeps it. What CTIFoundry changes is the form: it turns three fixed retrieval pipelines, each built around one family's question format, into one flexible agentic scaffold. A single agent answers all nine tasks with the same tools, and a new task is added by plugging in a new playbook. On the playbooks in the w/o tools arm: that arm received the playbooks verbatim, and App. H notes that the steps that call typed tools "become inert there". With these steps translated into natural language, the base agent reaches 0.699 on EL, against 0.568 with the verbatim playbooks and 0.923 for CTIFoundry, so procedure alone leaves most of the EL gap open. We have revised the paper accordingly, with the changes marked in blue: §1 and §4.3 state the claim as one about access to the records, App. K.1 and K.2 report Tables R1 and R2, and App. I reports the playbook variants.

AuthorsOpenReview ↗

Author response (part 2)

W2 (first paragraph) and Q2. The report layer's contribution to synthesis answers.

We appreciate the reviewer raising this concern, and we ran the ablation that the reviewer suggests.

Setup. No aliases or entity index: resolve_entity matches canonical names and identifiers only, chunks_mentioning is removed, and read_report, search_chunks and the KB tools stay. We run it with gpt-5.4 on the MDS questions of a stratified random sample of 150 main-set questions. On the share of questions that supply report ids: No main-set MDS question supplies report ids. The first step of the synthesis playbook in App. H.5 is conditional on a question naming reports and did not apply.

Table R3. MDS mean (CSC, TAP, MLA) on the sampled questions. Difference = full minus reduced, 95% paired bootstrap.

MDS meangpt-5.4
Full report layer0.855
No aliases or entity index0.845
Difference+0.010 [-0.025, +0.046]

Table R3 shows that the two conditions are level, as are the full system and the w/o tools arm in Table 3. The reason is that MDS is close to standard retrieval and summarization over text, where a general agent is already strong: the base agent scores 0.826 on MDS with gpt-5.4, against 0.498 on EL, so little room is left for any layer to raise the score. What the report layer adds on these tasks is what §4.2 measures: it reconstructs the cross-vendor report cluster and links every statement to its evidence. We have revised §4.5, marked in blue: "MDS needs neither" now reads "MDS is on par in all four configurations", and App. K.4 reports the ablation.

W2 (second paragraph). Extraction precision, and identifier validity.

We thank the reviewer for the careful reading of Table 1, and we analysed the remaining errors as suggested.

Setup. We labelled by hand every false positive of gpt-5.4-mini, the build model, on the Table 1 gold set: 155 for entities and 63 for TTP grounding.

Table R4. Causes of gpt-5.4-mini false positives, share within each layer.

LayerCauseShare
Entitywrong type38%
Entityspan boundary29%
Entityspurious mention25%
Entityother8%
TTPsibling technique35%
TTPparent/child confusion19%
TTPdefender behavior11%
TTPshould have abstained24%
TTPother11%

Table R4 shows that no single cause dominates. Wrong type leads for entities and sibling technique for TTPs, the categories also reported for LLM extraction [2] and for LLM ATT&CK mapping [3, 4]. These errors do not reach the EL and EA results: the ontology layer is built from the official knowledge bases with no extraction model (§3.2), and the EL and EA playbooks use only the KB tools. They can reach synthesis answers, where the report-layer tools are used (Fig. 2(b)). We have revised the paper, with the changes marked in blue: §4.2 and App. E.1 now report the precision and recall of gpt-5.4-mini together (entity 0.708 and 0.781, TTP 0.611 and 0.692, against 0.846, 0.872, 0.749 and 0.841 for gpt-5.4), and state that the build guarantees identifier and span validity while types and technique assignments rest on the extraction model. App. K.5 reports Table R4.

Q3. Triples in App. G, and the static playbook map.

We thank the reviewer for cross-checking the appendices. The evaluated system uses no relational triples from reports. Only the alias_of statements of the App. G.2 prompt are kept, at build time, as alias evidence for entity resolution (App. A, Step 3 (d)). Every other relation statement is discarded and never materialized in the scaffold. The only relations a tool returns are the official KB cross-references that ontology_neighbors traverses. App. G.5 was used while developing an earlier pipeline that did materialize report triples, and it is not part of the build behind Tables 1 to 3. The task-to-playbook map is static: it is fixed before any question is read and involves no model call. The playbook recommends the opening calls, and every call inside the loop is the model's own choice (first-call compliance is 0.45 on ATA and 0.90 on VCA, Fig. 2(b)). We have revised the wording: §3.2 says that the layer exposes no relational triples to the agent and uses alias statements only as resolution evidence, App. G.5 is labelled as an earlier stage outside the evaluated build, and §4.4 says that a static task-to-playbook map selects each playbook.

AuthorsOpenReview ↗

Author response (part 3)

Q4. Development slice and the supplement.

We appreciate the reviewer raising this concern. The development slice has 92 questions, sampled at random from the scale set before any tuning, so it is disjoint from the 691-question main set behind Tables 2 and 3. The scale-set table in App. E.2 included these questions. We removed them and recomputed the table, and no task's F1 moves by more than 0.010. On the four items the reviewer lists: the split IDs of the main set (691), the scale set (1,168) and the development slice (92) are at https://anonymous.4open.science/r/repro_package-D241/. The frozen playbooks are in the submitted supplement and in App. H. The corpus is public: CTIConnect's released corpus and the four upstream knowledge bases. The configurations are the flags of the harness entry point in the supplement, and its README gives the commands for the rows of Table 3. The revised paper marks these changes in blue: §4.1 states the slice, and Table 6 in App. E.2 reports the scale set without it.

Q5. Figure 3(a).

We thank the reviewer for spotting this. Figure 3(a) is a quantitative plot of the reported runs. Its gpt-5.4 points used the MDS scores of an earlier judging pass, which places them about 0.02 below the tables. Tables 2 and 3 use the final pass and are unaffected. We have redrawn the figure from the table values in the revised paper, and its caption now states the averaging rule.

References:

  • [1] CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligence. Cheng, Liu, Li, Song and Gao. KDD 2026.
  • [2] Han et al.: Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors. Han, Peng, Yang, Wang, Liu and Wan. arXiv:2305.14450, 2023.
  • [3] TechniqueRAG: Retrieval Augmented Generation for Adversarial Technique Annotation in Cyber Threat Intelligence Text. Lekssays, Shukla, Sencar and Parvez. Findings of ACL 2025.
  • [4] Rule-ATT&CK Mapper (RAM): Mapping SIEM Rules to TTPs Using LLMs. Wudali, Kravchik, Malul, Gandhi, Elovici and Shabtai. arXiv:2502.02337, 2025.

Thank you for the additional experiments. The comparison with retrieval alone addresses my concern about the shell baseline. It shows that giving the agent direct access to official cross-references improves entity linking. My remaining reservation is how much the other parts of CTIFoundry contribute to answer quality. Even though none of the main-set synthesis questions supplies report IDs, disabling alias resolution and entity-based report lookup made little difference to the answers’ scores. Using a shared agent across three task families has practical value, but its scores are close to those of CTIConnect’s dedicated pipelines. I now have more confidence in the entity-linking result, though I still find the overall contribution too narrow for acceptance and maintain my rating of 2 (Weak Reject).

Reviewer_Ue4FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines the corpus side of CTI investigation and proposes CTIFoundry, a scaffold for agentic CTI investigation. It constructs an ontology graph from official knowledge-base cross-references, organizes report mentions into canonical entities with source-span provenance, and exposes these artifacts through hybrid retrieval, seven typed tools, and procedural skills. Experiments on CTIConnect using four LLMs with the mini-swe-agent harness show that CTIFoundry improves investigation performance over the baseline.

Critical strengths
  1. The motivation for examining the role of corpus structure in CTI investigation is clear and reasonable.

  2. The build-time pipeline, domain-specific tools, and procedural skills are clearly presented and easy to understand.

  3. Across the four query-time models, CTIFoundry consistently improves overall performance over the baseline. It also reduces tool calls for both Claude models.

Critical weaknesses
  1. The claim that corpus structure is the primary bottleneck in agentic CTI investigation is insufficiently justified. Since the baseline can access records containing the same official relations, its failure to use them may also reflect limitations in retrieval, parsing, or reasoning.

  2. The evaluation is confined to CTIConnect, so it remains unclear how well the findings generalize to CTI investigation more broadly. The gains are also concentrated in entity-linking tasks: the mean improvement on multi-document synthesis is only 0.02, within the paper’s stated 0.06 resolution for the judge, and there is no improvement on one synthesis task. The conclusions should therefore be scoped to the types of CTI tasks for which the scaffold provides useful structure unless further evidence demonstrates broader gains.

  3. Limited methodological generality. The contribution is primarily a CTI-specific integration of various datasets and tools. The paper suggests that its approach may extend to other domains with authoritative cross-references, but evaluates it only on CTIConnect. It remains unclear which design principles generalize beyond CTI and whether the scaffold would provide similar benefits in another domain.

Questions for authors

Regarding the above point 1. Could the authors provide targeted evidence that the baseline’s failures stem primarily from its corpus representation?

4 comments
AuthorsOpenReview ↗

Author response (part 1)

We thank the reviewer for the thoughtful comments.

W1 and Q1. Targeted evidence on why the base agent fails.

We appreciate the reviewer raising this concern, and we are happy to provide targeted evidence on why the base agent fails.

Setup. Failure classes: from the stored trajectories of the Table 2 base run with gpt-5.4, we assign every failed answer to one class. Not located: the record that holds the answer never appears in a tool output. Located but not extracted: the record appears, and the agent never states the gold identifier. Extracted, wrong: the agent states the gold identifier and the final answer drops it. Other: no parseable answer. Retrieval: the base agent equipped with search_kb, hybrid dense and BM25 search over KB entries, and search_chunks, dense search over report chunks, with no skills. We run it with gpt-5.4 on 150 main-set questions, a stratified random sample in which each task keeps its share of the main set, next to the existing runs on the same questions.

Table R1. Base agent, gpt-5.4, main set. Share of failed answers in each class.

FamilyFailedNot locatedLocated but not extractedExtracted, wrongOther
Entity linking (EL)2120.240.470.220.07
Attribution (EA)910.120.250.550.08

Table R2. gpt-5.4, 150 sampled questions. EL, EA and MDS are means of their task scores, Overall is the nine-task mean.

ConfigurationELEAMDSOverall
Base (bash)0.5110.5120.8290.617
Retrieval0.5380.5190.8430.635
CTIFoundry w/o skill0.7640.5440.8020.728
CTIFoundry0.9230.6350.8550.836

Table R3. gpt-5.4, EL, sampled questions. Share of the base failures of each class that each configuration answers correctly.

Base failure classRetrievalCTIFoundry w/o skillCTIFoundry
Not located0.180.550.82
Located but not extracted0.090.820.95
Extracted, wrong0.000.300.70

Table R1 shows that on EL the base agent fails at access, before reasoning. In 0.47 of its failures the record was on screen and the agent did not use the cross-reference in it, and in 0.24 it never reached the record. Only 0.22 are errors made after the answer was found. On EA the largest class is extracted, wrong (0.55), the choice among close candidates that the skills address (Table 3). Table R2 shows that better retrieval alone does not fix this. Retrieval adds +0.027 [-0.030, +0.084] on EL over base, 11% of the gain from base to CTIFoundry w/o skill, while the structured representation adds +0.226 [+0.112, +0.340] with the model, the records and the retrieval tools held fixed (95% paired bootstrap). Table R3 shows where the gain comes from: Retrieval answers 0.09 of the located-but-not-extracted failures, and CTIFoundry w/o skill answers 0.82. This is direct evidence that the failures stem from the corpus representation: with the model and the information fixed, changing only the representation of the records removes most of the failures. The retrieval and parsing limits that the reviewer names are real, and they are the limits of an agent over a flat corpus, where models miss information that is already in their input (Liu et al. [1]). The scaffold removes them without a stronger model. We have revised §1 and §4.3, marked in blue, to state the claim in these terms: on tasks whose answers are recorded in the knowledge bases, the agent is limited by access to the records. App. K.3 and K.1 report these tables.

W2. Gains concentrated in entity linking, and synthesis within the judge's resolution.

This is a great question, and it pushed us to look at where a general agent is already strong. On MDS the base agent already performs well, so little room is left for any scaffold to add. MDS questions ask the agent to find and summarize what several reports say about one entity, which is close to standard retrieval and summarization over text, where current LLMs are already strong. Table 2 shows this: with gpt-5.4 the base agent scores 0.826 on MDS, against 0.498 on EL and 0.510 on EA. EL and EA are where a general agent falls short, and where CTIFoundry adds the most: their answers are identifiers recorded as cross-references between knowledge bases, and CTIFoundry raises EL from 0.498 to 0.918 and EA from 0.510 to 0.628. On MDS, CTIFoundry is on par with the base agent for all four models, with differences of +0.016 (gpt-5.4), +0.003 (gpt-5.4-mini), +0.032 (claude-sonnet-4-6) and +0.035 (claude-haiku-4-5). We have revised the paper to say so, with the changes marked in blue: §4.3 and §4.5 state that the gain is in EL and EA and that MDS is on par because the base agent is already strong there.

AuthorsOpenReview ↗

Author response (part 2)

W3. Methodological generality.

We appreciate the reviewer raising this concern. We have tightened this claim in the revised paper. Three statements reached beyond CTI: "in CTI and beyond" in the abstract, "The lesson transfers to any vertical" in the third contribution, and "The claim extends past CTI" in §6. They are now limited to CTI, marked in blue. We take CTI as our entry point and study the design of LLM scaffolds in depth within this domain. To our knowledge, CTIConnect is the only released CTI benchmark that evaluates retrieval over a released corpus (App. C), which makes it well suited to our setup. What we want to convey is a design rationale: a general agent does well on tasks that read like text retrieval and summarization (MDS) and much worse on tasks whose answers are recorded as cross-references (EL and EA), and typed tools and skills over a structured scaffold close much of that gap. We offer this as an insight for domains with a similar setup.

Reference:

  • [1] Liu et al.: Lost in the Middle: How Language Models Use Long Contexts. Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang. TACL 2024.

Thank you for the additional analyses and clarifications. The trajectory analysis and retrieval-plus-relation-lookup ablation strengthen the evidence that explicit cross-reference access improves performance on CTIConnect’s entity-linking tasks. I also appreciate the authors narrowing their claim to tasks whose answers are recorded in the knowledge bases. However, the new evidence remains within CTIConnect and does not establish generalization beyond this benchmark or validate the proposed design principles in another setting. Results on multi-document synthesis remain largely unchanged. I therefore remain unconvinced that the evidence establishes the broader significance claimed, and maintain my original score.

AuthorsOpenReview ↗

Follow-up response

We thank the reviewer for reading our response and for acknowledging that it addressed some of their concerns. We would like to clarify the remaining points.

To our knowledge, CTIConnect is the only released benchmark for corpus-grounded CTI investigation. Earlier CTI benchmarks such as CTIBench probe what a model already knows and release no corpus for an agent to investigate (App. F.5), so they cannot test a corpus scaffold. Within CTIConnect the evaluation is broad: nine expert-verified tasks in three families (entity linking, attribution and multi-document synthesis), over four knowledge bases and 321 vendor reports. If they have another benchmark of this kind in mind, we would be glad to run it. The small improvement on multi-document synthesis does not indicate a shortcoming. It shows a strength of the existing agent and harness, which are already optimized for generic tasks such as synthesis, in contrast to how they perform on the remaining CTI tasks. We report all nine tasks to give an honest picture of agentic CTI investigation and of how a domain-specific scaffold improves it.

We are grateful for their constructive review, and we hope this addresses their remaining concerns.

Reviewer_gYJdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CTIFoundry, which supports LLM agents in conducting cyber threat intelligence analysis by integrating official knowledge base cross-references, cross-vendor entity aliases, and traceable report evidence, combined with hybrid retrieval, specialized tools, and investigative workflows. Using the same model and agent framework, this method outperforms the baseline approaches based on raw files and shell access on the CTIConnect dataset, with improvements primarily concentrated in entity linking tasks that rely on official cross-references, demonstrating the practical value of agent-oriented corpus organization.

Critical strengths

The method described in this paper achieves significant improvements on the entity linking task, as supported by ablation experiments. GPT-5.4’s scores on ATD and RCM rose from 0.317 and 0.588, respectively, to 1.000. Even after removing the skill, substantial improvements were maintained on some tasks, such as ESD reaching 0.975, indicating that the gains cannot be attributed solely to the additional process prompts. The structure and tool layers themselves make substantial contributions.

This paper conducts a direct quantitative evaluation of data construction quality. The authors report downstream performance and evaluate entity resolution on 50 gold-standard report clusters, achieving an F1 score that improved from 0.878 to 0.913 compared to the original benchmark graph, while the number of exactly reconstructed clusters increased from 21 to 27. Furthermore, a comparison of six extraction models provides useful guidance for model selection in the pipeline.

The comparison of the main loop in this paper is effectively controlled, and the improvements show consistent trends across multiple models. Both groups used the same mini-swe-agent main loop, models, step budgets, and temperatures, thereby minimizing interference caused by custom harnesses. The experiments covered four models from two vendors across different capability tiers, with consistent overall improvements. Notably, Claude Haiku 4.5 achieved a score of 0.745 when using CTIFoundry, surpassing GPT-5.4’s score of 0.610 on the original document baseline. These results suggest practical value, at least on CTIConnect.

Critical weaknesses

I acknowledge the paper’s substantial improvements in the structured entity linking task, but the current experiments are insufficient to establish that "the binding constraint on agentic CTI investigation is the corpus substrate, not agent capability". The baselines primarily rely on shell-based literal retrieval, whereas the complete system simultaneously incorporates hybrid retrieval, structured access, and dedicated workflows. Although ablation studies have been conducted on existing tools and skills, the contributions of the toolkit’s internal retrieval capabilities and structured representations have not yet been disentangled. I recommend adding a baseline with comparable retrieval capabilities.

Skills encompass both general investigation procedures and knowledge specific to the benchmark, such as the number of answers, scoring rules, and question formats. Ablation results show that the improvement in EA is primarily driven by skills, while the gains from MDS are relatively limited. Therefore, the success observed in EL does not yet support the claim that all types of CTI tasks are primarily constrained by corpus structure. I suggest that the authors more clearly distinguish the respective roles and conditions of application for structured access, procedural guidance, and benchmark adaptation.

The combined number of questions in the main set and the scale set covers the entire benchmark; however, the paper also mentions a “held-out development slice” used to freeze prompts, tools, and skills, with the skills derived from failure trajectory analysis. The current description does not sufficiently clarify whether the development questions and related trajectories overlap with the final evaluation. I recommend explicitly stating the source, size, and partitioning method of the development set to confirm the independence of the evaluation.

Questions for authors
  1. Could you provide an additional baseline to distinguish how much of the improvement comes from retrieval capabilities and how much from structured representations?

  2. The original CTIConnect report presented results for several RAG configurations. How does CTIFoundry compare to the best of these RAG configurations on the same questions?

  3. What is the source of the "held-out development slice" used to freeze prompts, tools, and skills? What is its size, and how was it partitioned? Since the main set and scale set together already comprise all 1,859 questions, was the development set drawn from these?

  4. The values of the interaction terms in the 2×2 ablation experiment seem relatively low. Are there variance estimates or confidence intervals from multiple runs to support the “super-additive” conclusion?

4 comments
AuthorsOpenReview ↗

Author response (part 1)

We thank the reviewer for the careful reading and the concrete suggestions.

W1 and Q1. Disentangling retrieval capability from structured representation.

We appreciate the reviewer raising this concern, and we are happy to provide additional ablations.

Setup. We introduce an additional configuration, Retrieval: the base agent equipped with two of the seven tools of Table 5, search_kb, hybrid dense and BM25 search over KB entries, and search_chunks, dense search over report chunks, and with no skills. All configurations are compared on the same 150 main-set questions, a stratified random sample in which each task keeps its share of the main set.

Table R1. gpt-5.4, 150 main-set questions (stratified random sample).

ConfigurationELEAMDSOverall
Base (bash)0.5110.5120.8290.617
Retrieval0.5380.5190.8430.635
CTIFoundry w/o skill0.7640.5440.8020.728
CTIFoundry0.9230.6350.8550.836

Table R1 shows that retrieval capability alone recovers little of the entity linking (EL) gain. Retrieval minus base is +0.027 [-0.030, +0.084], 11% of the gain from base to CTIFoundry w/o skill, while CTIFoundry w/o skill minus Retrieval, the gain at matched retrieval, is +0.226 [+0.112, +0.340] (95% paired bootstrap, 10,000 resamples within task, Koehn [1]). The skills add a further +0.159 on EL on top of CTIFoundry w/o skill. Without skills the configurations move EA little, and the skill recovers most of the EA gain (see W2). MDS changes little on every configuration. Changing only the interface with the model fixed follows SWE-agent [2]. In short, stronger retrieval over the same records recovers only 7% of CTIFoundry's EL gain over the shell baseline. Most of the gain comes from the structured representation and the skills. The revised paper reports this comparison in App. K.1 and refers to it in §4.5, marked in blue.

W3 and Q3. Source, size and separation of the development slice.

We appreciate the reviewer raising this concern.

Setup. The development slice has 92 questions, sampled at random from the scale set before any tuning, and prompts, tools and skills were iterated only on these questions. The scale-set table in App. E.2 included them. We removed these 92 questions from the scale set and recomputed n and F1 for every task.

Table R2. CTIFoundry on the scale set (gpt-5.4), with the 92 development questions removed.

TasknCTIFoundry F1
RCM1750.990
WIM1920.695
ATD1480.957
ESD1660.833
ATA920.588
VCA1190.492
CSC550.927
TAP740.824
MLA550.848

Table R2 shows that removing the development questions leaves the scale-set results essentially unchanged. No task moves by more than 0.010 from the submitted Table 6. The slice comes from the scale set only, so it is disjoint from the 691-question main set, and Tables 2 and 3 never included a development question. The evaluation is therefore independent of the development slice. We have revised the paper accordingly, with the changes marked in blue: §4.1 now states the source, size and selection of the slice, and Table 6 in App. E.2 reports the scale set without the development questions. We thank the reviewer for checking our data so carefully, which gave us very useful feedback.

AuthorsOpenReview ↗

Author response (part 2)

W2. Investigation procedure versus benchmark adaptation in the skills.

We appreciate the reviewer pointing out that the skills entangle general investigation procedure with knowledge specific to the benchmark. To clearly distinguish the respective roles and conditions of application of structured access, procedural guidance and benchmark adaptation, we ran additional experiments.

Setup. We give the base agent two variants of the playbooks. Base + benchmark adaptation: only the benchmark adaptation, that is, the general statements of how each task is posed and answered, together with its rules. Base + procedural guidance: adds the procedure of the playbooks, with the steps that call typed tools translated into natural language, so the base agent carries them out with its own shell commands. Both run on gpt-5.4, on the 150 sampled questions of Table R1.

Table R3. gpt-5.4, 150 sampled questions. EL and MDS are family means.

ConfigurationELATAVCAMDSOverall
Base0.5110.5100.5140.8290.617
Base + benchmark adaptation0.4980.5570.5490.8570.630
Base + procedural guidance0.6990.6190.6690.8180.726
CTIFoundry0.9230.6310.6380.8550.836

Table R3 shows that benchmark adaptation alone does little for the base agent. EL stays at 0.498 (base 0.511), and ATA and VCA rise to 0.557 and 0.549. With procedural guidance, the base agent closes the EA gap and part of the EL gap. EA reaches 0.619 (ATA) and 0.669 (VCA), close to CTIFoundry's 0.631 and 0.638, and EL rises to 0.699. CTIFoundry, with the same procedure and the typed tools, is still +0.224 [+0.125, +0.324] higher on EL (95% paired bootstrap). MDS is on par in every configuration. The three roles therefore separate as follows: benchmark adaptation accounts for about a third (31%) of the EA gain and none of the EL gain, procedural guidance for the rest of the EA gain and 47% of the EL gain, and structured access for the remaining EL gain. We have revised the paper accordingly, with the changes marked in blue: §1 and §4.3 restate the central claim as one about access to the records (finding them and reading the relations in them), §3.3 separates procedural guidance from benchmark adaptation, App. H lists the adaptation lines of each playbook, and the new App. I reports both configurations.

Q4. Variance and the 2x2 interaction.

We appreciate the reviewer raising this concern, and we computed paired intervals for the interaction and repeated every cell.

Setup. On the paper's four gpt-5.4 runs we resample questions within each task, identically for all four cells, with 95% percentile intervals over 10,000 resamples [1], a paired, question-level analysis as recommended in [4]. Each cell was also run twice more on the 150 sampled questions of Table R1.

Table R4. gpt-5.4, Overall. Estimates are Table 3 differences.

ContrastEstimate95% CI
Playbook on bash (w/o tools - base)+0.062[+0.031, +0.093]
Tools alone (w/o skill - base)+0.136[+0.106, +0.166]
Both (CTIFoundry - base)+0.219[+0.192, +0.245]
Interaction+0.021[-0.018, +0.060]

Table R5. The other three models, Overall. Single-factor cells are the runs plotted in Fig. 3(a).

ModelBasew/o skillw/o toolsCTIFoundryInteraction [95% CI]
gpt-5.4-mini0.5680.5800.6080.758+0.138 [+0.095, +0.180]
claude-sonnet-4-60.5590.6530.5750.781+0.112 [+0.073, +0.151]
claude-haiku-4-50.4700.6420.5320.745+0.041 [-0.001, +0.083]

Table R4 shows that each component helps on its own, and that the interaction is not distinguishable from zero. Over three runs per cell on the sampled questions, the sd of Overall is at most 0.018, below every gap between cells, while the interaction's sd is 0.036, at least as large as the interaction itself. Table R5 shows a positive interaction for the other three models, with intervals that exclude zero for gpt-5.4-mini and claude-sonnet-4-6 and include it for claude-haiku-4-5. For gpt-5.4 the +0.021 interaction is therefore not distinguishable from zero, so we have removed "super-additive" and now describe the two components as roughly additive for that model. The revised paper marks these changes in blue: §1 and §4.5 carry the new wording, and the new App. J reports both tables.

AuthorsOpenReview ↗

Author response (part 3)

Q2. CTIConnect's RAG configurations on the same questions.

We appreciate the reviewer raising this question, and we ran CTIConnect's pipelines on our questions.

Setup. CTIConnect [3] compares closed-book answering, vanilla RAG and one domain-specific pipeline per family: Extract-then-Retrieve (EtR) for EL, Decompose-then-Retrieve (DtR) for EA, and CSKG-guided RAG for MDS, which ranks reports by BM25 over their extracted entity sets. In CTIConnect's own GPT-4o comparison (its Table V), retrieve-then-rerank and IRCoT score below the domain-specific pipeline in every family, so we compare against vanilla RAG and the three domain-specific pipelines. We ran the released pipelines with gpt-5.4 on the 150 sampled questions of Table R1, with the release's retrieval settings (text-embedding-3-large, top 5 passages for EL and EA, top 10 for MDS) over the same released records, scored with the identifier F1 and the claim-level judge of Table 2. The release ships no MDS answering step and no query script for CSKG-guided RAG, so we implemented both from CTIConnect's description and release that code.

Table R6. gpt-5.4, 150 sampled questions, family means of task scores, Overall is the nine-task mean.

SystemELEAMDSOverall
Vanilla RAG0.6990.5270.8310.705
EtR / DtR / CSKG-guided0.9260.6100.8280.823
Base agent0.5110.5120.8290.617
CTIFoundry0.9230.6350.8550.836

Table R6 shows that CTIFoundry is on par with CTIConnect's dedicated pipelines on EL, attribution (EA) and synthesis (MDS). The differences are small in every family (-0.003 on EL, +0.025 on EA and +0.024 on MDS), and none is distinguishable from zero (95% paired bootstrap). Vanilla RAG lands well below both on EL. This is expected, because our playbooks build the design of these pipelines into the agent: the attribution playbooks decompose a passage into behaviors and restate each in the target taxonomy's idiom, as DtR does, and the entity linking playbook keeps EtR's restated-query search as its fallback. These designs carry the know-how of the CTI domain, and CTIFoundry keeps it. What CTIFoundry changes is the form: it turns three fixed retrieval pipelines, each built around one family's question format, into one flexible agentic scaffold. A single agent answers all nine tasks with the same tools, and a new task is added by plugging in a new playbook. Vanilla RAG scores above the base agent, which has no retriever and reads the records through the shell. The revised paper reports this comparison in App. K.2 and refers to it in §4.3, marked in blue.

References:

  • [1] Koehn: Statistical Significance Tests for Machine Translation Evaluation. Koehn. EMNLP 2004.
  • [2] SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. Yang et al. NeurIPS 2024.
  • [3] CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligence. Cheng, Liu, Li, Song and Gao. KDD 2026.
  • [4] Miller: Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Miller. arXiv:2411.00640, 2024.

Thank you for the thoughtful rebuttal and the additional experiments. The clarification about the development slice and its removal from the revised scale set results address my concern about evaluation independence. I also appreciate the uncertainty analysis and the revision of the super additivity claim for GPT-5.4.

The retrieval only ablation strengthens the evidence that structured access contributes to the EL gains beyond hybrid retrieval alone. The comparison with CTIConnect’s task specific RAG pipelines is also informative: no clear performance advantage is established over those pipelines, but achieving comparable observed performance across the three task families with a shared agent and toolset is a useful systems contribution. I view the evidence as strongest for structured EL; the benefits of this design for new tasks or workloads remain less directly demonstrated.

Overall, the rebuttal addresses several of my concerns and increases my confidence in the paper’s narrower claims. I maintain my Weak Accept (3) rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.