- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper reframes long-tailed APT attribution from fixed-label classification as cross-graph entity alignment between a query CTI graph with masked actor identities and a reference graph containing known actors. It introduces HEAA (501 reports, 84 actors) and analyzes the limited overlap and differing discriminative value of malware, attack patterns, and IoCs. ThreatAlign comprises (i) evidence saliency estimation using local graph statistics and external knowledge priors, (ii) recursive, LLM-based profiling across an IoC/attack-pattern/malware/actor hierarchy, and (iii) candidate retrieval from profile similarity and shared IoCs followed by LLM evidence arbitration. The reported HEAA macro-F1 is 43.5%, versus 26.9% for the best attribution baseline. Additional experiments cover AADM+, ICEWS-WIKI, a chronological split, single-report attribution, and component ablations.
Critical strengths
-
A relevant and difficult evaluation setting. HEAA directly targets scarce-reference-report attribution, where conventional supervised classification is often ill-posed. The paper reports cross-graph correspondence and topology statistics, including only 3.02% structural similarity, and separates discriminative, ambiguous, and nonmatching evidence. This analysis is useful beyond the proposed method because it clarifies why exact identifiers and common attack techniques fail for different reasons.
-
An interpretable modular pipeline with informative diagnostics. The progression from evidence selection to recursive profiling to evidence arbitration is coherent. Ablations attribute substantial performance differences to lower-level profile reuse (43.5% to 32.6% macro-F1 when removed) and external retrieval (43.5% to 31.8%). The chronological, single-report, and pretraining-memory diagnostics strengthen the empirical presentation, even though they do not settle the central confounding concerns below.
Critical weaknesses
-
The attribution benchmark may admit direct or indirect identity recovery, and the masking protocol is not sufficiently demonstrated. Appendix A.1 claims that query-side actor names, aliases, and identifying textual mentions are masked. Yet Figure 4's query-side report excerpt reads "unknown apt group, also known as GOLDEN CHICKENS," while the ground-truth aligned actor is Venom Spider. Golden Chickens is a publicly documented alternative name for Venom Spider. The figure does not establish whether this unmasked excerpt was actually supplied to the model, so I am not alleging confirmed leakage. However, it creates a concrete and serious ambiguity about the masking procedure and the validity of the showcased result. Furthermore, the actor-profiling prompt (Appendix B.5.3) explicitly asks the LLM to infer the standardized actor name using its internal knowledge, and the external knowledge graph contains actor–malware–technique relations. The resulting profiles can therefore contain the answer itself. Separating report files between the external text index and HEAA (Appendix B.1) does not prevent overlapping actor identities and threat facts. A rigorous identity-leakage audit is necessary before the main attribution numbers can be interpreted confidently.
-
The comparisons do not isolate the proposed alignment mechanism from privileged external information. ThreatAlign can exploit MITRE ATT&CK/Malpedia entity relations, retrieved report text, and LLM knowledge of named threat actors. The strongest classification and general-purpose alignment baselines do not appear to have equivalent access to this information or a comparable retrieval-and-reasoning pipeline. The direct-LLM baseline in Appendix D.3 uses the same query graph and candidate list but bypasses both hierarchical profiling and cross-graph matching; it is not a controlled comparison that provides the same retrieved knowledge. Thus, the 16.6-point HEAA advantage may substantially reflect access to external identity-bearing knowledge rather than the intrinsic benefit of entity-level semantic alignment or the specific saliency/recursive architecture. This concern is reinforced by the zero macro-F1 reported for all methods in the NN-only evidence group (Table 3), which limits the evidence for generalization when no validated direct correspondence is available.
Questions for authors
-
In Figure 4, is the phrase "also known as GOLDEN CHICKENS" part of any actual inference input (report text, extracted graph attributes, retrieved context, or LLM prompt), or is it included only in an illustrative raw-report excerpt? Please provide the exact model-facing query input for this example, explain why the phrase survives the displayed anonymization, and report a dataset-wide audit for residual canonical actor names and aliases at every stage of inference.
-
What is ThreatAlign's HEAA macro-F1 when external actor names, aliases, and explicit actor–malware/actor–technique identity links are withheld from the knowledge base and model-facing retrieved content, while retaining generic malware and technique descriptions? Please compare against at least one simple retrieval/LLM baseline with access to precisely the same external corpus, actor candidate set, and identity-masking rules.
-
For how many of the 84 HEAA actors does the external knowledge base contain the ground-truth actor under its canonical name or alias, together with an explicit connection to any query-side malware, tool, or technique? Please stratify attribution results by that knowledge-base coverage. The pretraining probe in Appendix D.4 is appreciated, but failing to elicit memorized text is not evidence that pretrained actor knowledge is absent.
6 comments
(2/5) Author Rebuttal
With external knowledge, RAG-Direct improves M-F1 by 5.2 percentage points over the Direct LLM Prediction baseline in the appendix, confirming that the knowledge base contributes to attribution performance. Nevertheless, despite access to the same LLM internal knowledge and external knowledge, RAG-Direct remains 17.0 percentage points below ThreatAlign. This indicates that directly recovering standardized actor names from identity-masked query graphs using internal memory and external knowledge remains challenging for the LLM. Flat-Profile provides further evidence: despite receiving additional reference-side information and having the same scope of knowledge access as ThreatAlign, it performs 6.6 percentage points below RAG-Direct. This indicates that flat profiling without hierarchical organization can exacerbate interference from noise in long-tailed data.
We further conducted a simple controlled experiment to examine the effect of removing the actor-name inference instruction from the prompt. Under both configurations, we also measured the proportion of profiles containing the correct identity and its relationship with attribution performance.
| Configuration | M-P (%) | M-R (%) | M-F1 (%) |
|---|---|---|---|
| Original configuration | 40.7 ± 2.3 | 50.8 ± 1.5 | 43.5 ± 2.0 |
| Without the actor-name inference instruction | 42.4 ± 3.6 | 52.0 ± 3.7 | 45.0 ± 3.6 |
| Configuration | Profile category | Proportion | ThreatAlign M-R | APT-ATT M-R |
|---|---|---|---|---|
| Original configuration | Correct identity recovered | 26.2% | 89.4 | 53.0 |
| Original configuration | Correct identity not recovered | 73.8% | 37.1 | 26.9 |
| Without the actor-name inference instruction | Correct identity recovered | 15.5% | 92.3 | 47.0 |
| Without the actor-name inference instruction | Correct identity not recovered | 84.5% | 44.6 | 31.3 |
Direct identity recovery by the LLM occurs in only a minority of cases. Removing the instruction reduces the proportion of profiles recovering the correct identity by 10.7 percentage points, without a corresponding decrease in M-R. This indicates that attribution performance does not directly depend on recovering actor identities in the profiles. ThreatAlign achieves substantially higher attribution M-R when the correct identity is recovered, but it still outperforms APT-ATT in the group where the correct identity is not recovered. APT-ATT also performs better on the group where the LLM recovers the correct identity, suggesting that the two groups may differ in task difficulty. Together, this controlled experiment, the pretraining-knowledge probe in Appendix D.4, and the simplified-baseline experiments represent attempts to remove the influence of pretraining knowledge and demonstrate the benefits of the method’s design beyond pretraining knowledge.
W1c (Q3). Knowledge Coverage Effects on Attribution.
We have added statistics on structured knowledge-base coverage of HEAA actors and evaluated attribution performance by coverage group. Actors with explicit associations to query-side entities achieve higher attribution performance. For actors without such associations, ThreatAlign still outperforms the classification baselines evaluated.
We checked whether the knowledge base contains each HEAA actor under its canonical name or an alias, and whether that actor has an explicit structured association with any query-side malware, tool, or attack technique. Among the 84 actors, 48.8% have both an actor identity entry and such an association; 47.6% have an identity entry but no such association; and 3.6% are absent from the knowledge base. We combine the latter two groups for evaluation according to whether an explicit association with query-side entities is found.
| Structured knowledge coverage | Number of actors | ThreatAlign M-F1 (%) | APT-ATT M-F1 (%) | CyBERT M-F1 (%) |
|---|---|---|---|---|
| Explicit association with query-side entities | 41 | 67.2 ± 1.8 | 40.4 ± 1.6 | 27.6 ± 5.0 |
| No such association found | 43 | 20.9 ± 2.7 | 14.1 ± 1.2 | 14.2 ± 2.0 |
ThreatAlign performs better on actors with explicit structured associations, indicating a relationship between knowledge coverage and attribution performance. The classification baselines also obtain lower scores in the group without such associations, suggesting that the groups may additionally differ in evidence availability or task difficulty. Nevertheless, ThreatAlign retains an advantage over both classification baselines on the 43 actors without explicit associations, demonstrating its effectiveness in cases where no verifiable query-entity association is found in the knowledge base.
The influence of LLM internal knowledge is discussed in W1b.
(3/5) Author Rebuttal
W2a (Q2). Knowledge Access Confounds Alignment Gains.
Existing classification-based attribution methods have input constraints that make it difficult to incorporate external knowledge for comparisons with equivalent knowledge access. W1b reports simplified baselines under equivalent information-access conditions. Here, we additionally report an experiment removing external actor-identity information from ThreatAlign. Together, these experiments examine the role of external actor associations and the complete framework’s additional benefits over simpler knowledge-enhanced methods.
For all 84 query actors and 84 reference actors in HEAA, we disable retrieval from the external knowledge base’s actor index and retrieval of structured relations involving actors. Before retrieved text passages enter the profiling module, we remove known actor names, aliases, explicit attribution statements, and identity metadata associated with the entities to which the passages belong. We also set the external knowledge prior in saliency scoring to zero and recompute the graph-structure-based scores. These interventions aim to exclude, as far as possible, the influence of external actor identities and their associated information.
| Configuration | M-P (%) | M-R (%) | M-F1 (%) |
|---|---|---|---|
| ThreatAlign | 40.7 ± 2.3 | 50.8 ± 1.5 | 43.5 ± 2.0 |
| Removing external actor identities and associations | 32.0 ± 3.5 | 43.7 ± 5.5 | 35.2 ± 4.1 |
Explicit external actor priors contribute an 8.3-percentage-point improvement, consistent with the RAG-Direct results in W1b. Nevertheless, after their removal, ThreatAlign retains an M-F1 of 35.2%, exceeding APT-ATT by 8.3 percentage points and also outperforming RAG-Direct. This indicates that ThreatAlign remains competitive after external actor relations and identity cues are removed, relying only on generic technical descriptions and recursive cross-graph alignment.
W2b. Unclear Generalization without Direct Correspondences.
ThreatAlign targets long-tail actors for which usable semantic or relational connections remain available. As discussed in W1b, W2a, and W1c (Q3), its advantages arise from both internal and external knowledge and the framework’s design choices. Its central goal is to make effective use of existing information that can support cross-graph associations. The all-zero results in the NN-only group indicate a boundary of the method’s applicability.
References
[1] Perry, Lior, Bracha Shapira, and Rami Puzis. “No-Doubt: Attack Attribution Based on Threat Intelligence Reports.” 2019 IEEE International Conference on Intelligence and Security Informatics (ISI). IEEE, 2019.
[2] Cai, Saihua, et al. “APT-ATT: An Efficient APT Attribution Model Based on Heterogeneous Threat Intelligence Representation and CTGAN.” Computer Networks 270 (2025): 111511.
[3] Plohmann, Daniel, et al. “Malpedia: A Collaborative Effort to Inventorize the Malware Landscape.” The Journal on Cybercrime and Digital Investigations 3.1 (2018): 1–19.
[4] Strom, Blake E., et al. MITRE ATT&CK: Design and Philosophy. Technical report, 2018.
[5] King, Isaiah J., et al. “TRAIL: A Knowledge Graph-Based Approach for Attributing Advanced Persistent Threats.” 2025 IEEE 41st International Conference on Data Engineering (ICDE). IEEE, 2025.
[6] Jiang, Xuhui, et al. “Unlocking the Power of Large Language Models for Entity Alignment.” Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
[7] Cheng, Jingwei, et al. “EasyEA: Large Language Model Is All You Need in Entity Alignment Between Knowledge Graphs.” Findings of the Association for Computational Linguistics: ACL 2025. 2025.
[8] Xiao, Nan, et al. “TAA-EPLMR: Threat Actor Attribution via Evidence Path-Enhanced Large Language Model Reasoning.” 2025 IEEE International Conference on Big Data (BigData). IEEE, 2025.
(1/5) Author Rebuttal
We sincerely thank the reviewer for their careful review of our manuscript.
W1a (Q1). Insufficient Validation of Identity Masking.
The raw report excerpt in Figure 4 is included only for illustration and is not part of the input data provided by HEAA. HEAA consists of reference and query graphs, with actor names, aliases, and identifying textual mentions masked on the query side. Compared with other widely used public benchmarks, such as AADM [1] and AADM+ [2], identity masking is an additional contribution of HEAA.
For classification baselines that require report text, we separately anonymize the original reports using a combination of an alias dictionary and an LLM. GOLDEN CHICKENS was not included in the Venom Spider alias set constructed from Malpedia [3], MITRE ATT&CK [4], and prior work [5], and was therefore not fully masked. This omission may have inflated the performance of the affected text-based classification baselines.
We further inspected the raw input fields in the neighborhoods of all 84 query actors in HEAA. Except for the previously identified malware entities whose names match actor names (12/412), we found no explicit mentions of the target actors’ canonical names or verified aliases. We retained these malware entities because malware–actor usage relations can be many-to-many, and name equality alone is insufficient to treat a malware entity as an actor-identity label. Nevertheless, we acknowledge that such malware names may provide identity cues. We therefore additionally report results after masking the corresponding malware entities. Compared with the original configuration, M-F1 decreases by 4.4 percentage points but remains 12.2 percentage points above the strongest baseline, APT-ATT.
We will revise Figure 4 to clearly distinguish the illustrative report excerpt from the model input. The complete input used to generate the query-side actor profile in Figure 4 is provided in the Supplementary Material.
W1b. Potential Answer Leakage in Profiles.
For long-tail actor attribution under sparse evidence, using the LLM’s existing knowledge to assist attribution is a necessary design choice. ChatEA [6], EASY-EA [7], and TAA-EPLMR [8] also emphasize using LLM capabilities. ThreatAlign follows this research direction by using existing knowledge to supplement limited observations. Consequently, its generated profiles may indeed contain the target actor’s identity through external knowledge or the LLM’s internal knowledge.
As the reviewer correctly points out, however, the contributions of internal and external knowledge must be distinguished from those of the framework’s other design choices. We therefore added simplified baselines under equivalent information-access conditions. Both baselines use the same DeepSeek-V3.2 model, external corpus, complete candidate actor pool, and query-side actor-identity masking rules as ThreatAlign. The input limit is 122,880 tokens and the output limit is 8,192 tokens, with a total context length of at most 131,072 tokens.