acceptodds
Under review as a conference paper at ICLR 2027

Align to Attribute: APT Attribution via Entity-Level Semantic Alignment

Abstract

Threat actor attribution from cyber threat intelligence (CTI) reports commonly predicts actor labels from historical reports. However, many advanced persistent threat (APT) actors have few reports, with evidence fragmented across heterogeneous entities and unstructured text, making reliable attribution difficult for tail actors. We explore entity-level semantic alignment for attribution, comparing unknown and known actors through associated entities and graph contexts. We construct HEAA as a benchmark for long-tail attribution evaluation. Our analysis quantifies the trade-off between evidence discriminativeness and cross-report generalizability and identifies limitations of individual evidence types for attribution. Based on these findings, we proposeThreatAlign, a large language model (LLM)-based alignment framework that recursively integrates selected graph evidence and external knowledge into semantic profiles. The framework combines profile semantics with supporting evidence for cross-graph entity alignment to identify unknown actors. ThreatAlign achieves 43.5% macro-F1 on HEAA, 16.6 percentage points above the strongest evaluated baseline, and remains close to the best methods on conventional attribution (AADM+) and general-purpose entity alignment (ICEWS-WIKI). Our code is available at https://anonymous.4open.science/r/ThreatAlign-31F6/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Threat Actor Attribution, Entity Alignment, Cyber Threat Intelligence,Long-tailed data, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_fjDCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper reframes long-tailed APT attribution from fixed-label classification as cross-graph entity alignment between a query CTI graph with masked actor identities and a reference graph containing known actors. It introduces HEAA (501 reports, 84 actors) and analyzes the limited overlap and differing discriminative value of malware, attack patterns, and IoCs. ThreatAlign comprises (i) evidence saliency estimation using local graph statistics and external knowledge priors, (ii) recursive, LLM-based profiling across an IoC/attack-pattern/malware/actor hierarchy, and (iii) candidate retrieval from profile similarity and shared IoCs followed by LLM evidence arbitration. The reported HEAA macro-F1 is 43.5%, versus 26.9% for the best attribution baseline. Additional experiments cover AADM+, ICEWS-WIKI, a chronological split, single-report attribution, and component ablations.

Critical strengths
  1. A relevant and difficult evaluation setting. HEAA directly targets scarce-reference-report attribution, where conventional supervised classification is often ill-posed. The paper reports cross-graph correspondence and topology statistics, including only 3.02% structural similarity, and separates discriminative, ambiguous, and nonmatching evidence. This analysis is useful beyond the proposed method because it clarifies why exact identifiers and common attack techniques fail for different reasons.

  2. An interpretable modular pipeline with informative diagnostics. The progression from evidence selection to recursive profiling to evidence arbitration is coherent. Ablations attribute substantial performance differences to lower-level profile reuse (43.5% to 32.6% macro-F1 when removed) and external retrieval (43.5% to 31.8%). The chronological, single-report, and pretraining-memory diagnostics strengthen the empirical presentation, even though they do not settle the central confounding concerns below.

Critical weaknesses
  1. The attribution benchmark may admit direct or indirect identity recovery, and the masking protocol is not sufficiently demonstrated. Appendix A.1 claims that query-side actor names, aliases, and identifying textual mentions are masked. Yet Figure 4's query-side report excerpt reads "unknown apt group, also known as GOLDEN CHICKENS," while the ground-truth aligned actor is Venom Spider. Golden Chickens is a publicly documented alternative name for Venom Spider. The figure does not establish whether this unmasked excerpt was actually supplied to the model, so I am not alleging confirmed leakage. However, it creates a concrete and serious ambiguity about the masking procedure and the validity of the showcased result. Furthermore, the actor-profiling prompt (Appendix B.5.3) explicitly asks the LLM to infer the standardized actor name using its internal knowledge, and the external knowledge graph contains actor–malware–technique relations. The resulting profiles can therefore contain the answer itself. Separating report files between the external text index and HEAA (Appendix B.1) does not prevent overlapping actor identities and threat facts. A rigorous identity-leakage audit is necessary before the main attribution numbers can be interpreted confidently.

  2. The comparisons do not isolate the proposed alignment mechanism from privileged external information. ThreatAlign can exploit MITRE ATT&CK/Malpedia entity relations, retrieved report text, and LLM knowledge of named threat actors. The strongest classification and general-purpose alignment baselines do not appear to have equivalent access to this information or a comparable retrieval-and-reasoning pipeline. The direct-LLM baseline in Appendix D.3 uses the same query graph and candidate list but bypasses both hierarchical profiling and cross-graph matching; it is not a controlled comparison that provides the same retrieved knowledge. Thus, the 16.6-point HEAA advantage may substantially reflect access to external identity-bearing knowledge rather than the intrinsic benefit of entity-level semantic alignment or the specific saliency/recursive architecture. This concern is reinforced by the zero macro-F1 reported for all methods in the NN-only evidence group (Table 3), which limits the evidence for generalization when no validated direct correspondence is available.

Questions for authors
  1. In Figure 4, is the phrase "also known as GOLDEN CHICKENS" part of any actual inference input (report text, extracted graph attributes, retrieved context, or LLM prompt), or is it included only in an illustrative raw-report excerpt? Please provide the exact model-facing query input for this example, explain why the phrase survives the displayed anonymization, and report a dataset-wide audit for residual canonical actor names and aliases at every stage of inference.

  2. What is ThreatAlign's HEAA macro-F1 when external actor names, aliases, and explicit actor–malware/actor–technique identity links are withheld from the knowledge base and model-facing retrieved content, while retaining generic malware and technique descriptions? Please compare against at least one simple retrieval/LLM baseline with access to precisely the same external corpus, actor candidate set, and identity-masking rules.

  3. For how many of the 84 HEAA actors does the external knowledge base contain the ground-truth actor under its canonical name or alias, together with an explicit connection to any query-side malware, tool, or technique? Please stratify attribution results by that knowledge-base coverage. The pretraining probe in Appendix D.4 is appreciated, but failing to elicit memorized text is not evidence that pretrained actor knowledge is absent.

6 comments
AuthorsOpenReview ↗

(1/5) Author Rebuttal

We sincerely thank the reviewer for their careful review of our manuscript.


W1a (Q1). Insufficient Validation of Identity Masking.

The raw report excerpt in Figure 4 is included only for illustration and is not part of the input data provided by HEAA. HEAA consists of reference and query graphs, with actor names, aliases, and identifying textual mentions masked on the query side. Compared with other widely used public benchmarks, such as AADM [1] and AADM+ [2], identity masking is an additional contribution of HEAA.

For classification baselines that require report text, we separately anonymize the original reports using a combination of an alias dictionary and an LLM. GOLDEN CHICKENS was not included in the Venom Spider alias set constructed from Malpedia [3], MITRE ATT&CK [4], and prior work [5], and was therefore not fully masked. This omission may have inflated the performance of the affected text-based classification baselines.

We further inspected the raw input fields in the neighborhoods of all 84 query actors in HEAA. Except for the previously identified malware entities whose names match actor names (12/412), we found no explicit mentions of the target actors’ canonical names or verified aliases. We retained these malware entities because malware–actor usage relations can be many-to-many, and name equality alone is insufficient to treat a malware entity as an actor-identity label. Nevertheless, we acknowledge that such malware names may provide identity cues. We therefore additionally report results after masking the corresponding malware entities. Compared with the original configuration, M-F1 decreases by 4.4 percentage points but remains 12.2 percentage points above the strongest baseline, APT-ATT.

ConfigurationM-F1 (%)
Original configuration43.5 ± 2.0
Masking malware entities sharing actor names39.1 ± 1.7

We will revise Figure 4 to clearly distinguish the illustrative report excerpt from the model input. The complete input used to generate the query-side actor profile in Figure 4 is provided in the Supplementary Material.


W1b. Potential Answer Leakage in Profiles.

For long-tail actor attribution under sparse evidence, using the LLM’s existing knowledge to assist attribution is a necessary design choice. ChatEA [6], EASY-EA [7], and TAA-EPLMR [8] also emphasize using LLM capabilities. ThreatAlign follows this research direction by using existing knowledge to supplement limited observations. Consequently, its generated profiles may indeed contain the target actor’s identity through external knowledge or the LLM’s internal knowledge.

As the reviewer correctly points out, however, the contributions of internal and external knowledge must be distinguished from those of the framework’s other design choices. We therefore added simplified baselines under equivalent information-access conditions. Both baselines use the same DeepSeek-V3.2 model, external corpus, complete candidate actor pool, and query-side actor-identity masking rules as ThreatAlign. The input limit is 122,880 tokens and the output limit is 8,192 tokens, with a total context length of at most 131,072 tokens.

  • RAG-Direct: The anonymized query graph, retrieved external context, and candidate actor names and identifiers are provided directly to the LLM to predict the actor. This baseline uses the same query-side information, external knowledge, and LLM internal knowledge as ThreatAlign, together with an explicit candidate actor list, serving as a direct RAG baseline.
  • Flat-Profile: This baseline additionally uses reference-graph information. It organizes the graph evidence associated with each actor into a single input for profile generation, retrieves candidate actors through vector search, and then uses the LLM to complete attribution. It serves as a simpler entity-alignment baseline.
MethodQuery-graph informationReference-graph informationExternal corpusLLM internal knowledgeM-F1 (%)
Direct-LLM✓✗✗✓21.3 ± 1.5
RAG-Direct✓✗✓✓26.5 ± 1.5
Flat-Profile✓✓✓✓19.9 ± 1.1
ThreatAlign✓✓✓✓43.5 ± 2.0
AuthorsOpenReview ↗

(2/5) Author Rebuttal

With external knowledge, RAG-Direct improves M-F1 by 5.2 percentage points over the Direct LLM Prediction baseline in the appendix, confirming that the knowledge base contributes to attribution performance. Nevertheless, despite access to the same LLM internal knowledge and external knowledge, RAG-Direct remains 17.0 percentage points below ThreatAlign. This indicates that directly recovering standardized actor names from identity-masked query graphs using internal memory and external knowledge remains challenging for the LLM. Flat-Profile provides further evidence: despite receiving additional reference-side information and having the same scope of knowledge access as ThreatAlign, it performs 6.6 percentage points below RAG-Direct. This indicates that flat profiling without hierarchical organization can exacerbate interference from noise in long-tailed data.

We further conducted a simple controlled experiment to examine the effect of removing the actor-name inference instruction from the prompt. Under both configurations, we also measured the proportion of profiles containing the correct identity and its relationship with attribution performance.

ConfigurationM-P (%)M-R (%)M-F1 (%)
Original configuration40.7 ± 2.350.8 ± 1.543.5 ± 2.0
Without the actor-name inference instruction42.4 ± 3.652.0 ± 3.745.0 ± 3.6
ConfigurationProfile categoryProportionThreatAlign M-RAPT-ATT M-R
Original configurationCorrect identity recovered26.2%89.453.0
Original configurationCorrect identity not recovered73.8%37.126.9
Without the actor-name inference instructionCorrect identity recovered15.5%92.347.0
Without the actor-name inference instructionCorrect identity not recovered84.5%44.631.3

Direct identity recovery by the LLM occurs in only a minority of cases. Removing the instruction reduces the proportion of profiles recovering the correct identity by 10.7 percentage points, without a corresponding decrease in M-R. This indicates that attribution performance does not directly depend on recovering actor identities in the profiles. ThreatAlign achieves substantially higher attribution M-R when the correct identity is recovered, but it still outperforms APT-ATT in the group where the correct identity is not recovered. APT-ATT also performs better on the group where the LLM recovers the correct identity, suggesting that the two groups may differ in task difficulty. Together, this controlled experiment, the pretraining-knowledge probe in Appendix D.4, and the simplified-baseline experiments represent attempts to remove the influence of pretraining knowledge and demonstrate the benefits of the method’s design beyond pretraining knowledge.


W1c (Q3). Knowledge Coverage Effects on Attribution.

We have added statistics on structured knowledge-base coverage of HEAA actors and evaluated attribution performance by coverage group. Actors with explicit associations to query-side entities achieve higher attribution performance. For actors without such associations, ThreatAlign still outperforms the classification baselines evaluated.

We checked whether the knowledge base contains each HEAA actor under its canonical name or an alias, and whether that actor has an explicit structured association with any query-side malware, tool, or attack technique. Among the 84 actors, 48.8% have both an actor identity entry and such an association; 47.6% have an identity entry but no such association; and 3.6% are absent from the knowledge base. We combine the latter two groups for evaluation according to whether an explicit association with query-side entities is found.

Structured knowledge coverageNumber of actorsThreatAlign M-F1 (%)APT-ATT M-F1 (%)CyBERT M-F1 (%)
Explicit association with query-side entities4167.2 ± 1.840.4 ± 1.627.6 ± 5.0
No such association found4320.9 ± 2.714.1 ± 1.214.2 ± 2.0

ThreatAlign performs better on actors with explicit structured associations, indicating a relationship between knowledge coverage and attribution performance. The classification baselines also obtain lower scores in the group without such associations, suggesting that the groups may additionally differ in evidence availability or task difficulty. Nevertheless, ThreatAlign retains an advantage over both classification baselines on the 43 actors without explicit associations, demonstrating its effectiveness in cases where no verifiable query-entity association is found in the knowledge base.

The influence of LLM internal knowledge is discussed in W1b.

AuthorsOpenReview ↗

(3/5) Author Rebuttal


W2a (Q2). Knowledge Access Confounds Alignment Gains.

Existing classification-based attribution methods have input constraints that make it difficult to incorporate external knowledge for comparisons with equivalent knowledge access. W1b reports simplified baselines under equivalent information-access conditions. Here, we additionally report an experiment removing external actor-identity information from ThreatAlign. Together, these experiments examine the role of external actor associations and the complete framework’s additional benefits over simpler knowledge-enhanced methods.

For all 84 query actors and 84 reference actors in HEAA, we disable retrieval from the external knowledge base’s actor index and retrieval of structured relations involving actors. Before retrieved text passages enter the profiling module, we remove known actor names, aliases, explicit attribution statements, and identity metadata associated with the entities to which the passages belong. We also set the external knowledge prior in saliency scoring to zero and recompute the graph-structure-based scores. These interventions aim to exclude, as far as possible, the influence of external actor identities and their associated information.

ConfigurationM-P (%)M-R (%)M-F1 (%)
ThreatAlign40.7 ± 2.350.8 ± 1.543.5 ± 2.0
Removing external actor identities and associations32.0 ± 3.543.7 ± 5.535.2 ± 4.1

Explicit external actor priors contribute an 8.3-percentage-point improvement, consistent with the RAG-Direct results in W1b. Nevertheless, after their removal, ThreatAlign retains an M-F1 of 35.2%, exceeding APT-ATT by 8.3 percentage points and also outperforming RAG-Direct. This indicates that ThreatAlign remains competitive after external actor relations and identity cues are removed, relying only on generic technical descriptions and recursive cross-graph alignment.


W2b. Unclear Generalization without Direct Correspondences.

ThreatAlign targets long-tail actors for which usable semantic or relational connections remain available. As discussed in W1b, W2a, and W1c (Q3), its advantages arise from both internal and external knowledge and the framework’s design choices. Its central goal is to make effective use of existing information that can support cross-graph associations. The all-zero results in the NN-only group indicate a boundary of the method’s applicability.

References

[1] Perry, Lior, Bracha Shapira, and Rami Puzis. “No-Doubt: Attack Attribution Based on Threat Intelligence Reports.” 2019 IEEE International Conference on Intelligence and Security Informatics (ISI). IEEE, 2019.

[2] Cai, Saihua, et al. “APT-ATT: An Efficient APT Attribution Model Based on Heterogeneous Threat Intelligence Representation and CTGAN.” Computer Networks 270 (2025): 111511.

[3] Plohmann, Daniel, et al. “Malpedia: A Collaborative Effort to Inventorize the Malware Landscape.” The Journal on Cybercrime and Digital Investigations 3.1 (2018): 1–19.

[4] Strom, Blake E., et al. MITRE ATT&CK: Design and Philosophy. Technical report, 2018.

[5] King, Isaiah J., et al. “TRAIL: A Knowledge Graph-Based Approach for Attributing Advanced Persistent Threats.” 2025 IEEE 41st International Conference on Data Engineering (ICDE). IEEE, 2025.

[6] Jiang, Xuhui, et al. “Unlocking the Power of Large Language Models for Entity Alignment.” Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

[7] Cheng, Jingwei, et al. “EasyEA: Large Language Model Is All You Need in Entity Alignment Between Knowledge Graphs.” Findings of the Association for Computational Linguistics: ACL 2025. 2025.

[8] Xiao, Nan, et al. “TAA-EPLMR: Threat Actor Attribution via Evidence Path-Enhanced Large Language Model Reasoning.” 2025 IEEE International Conference on Big Data (BigData). IEEE, 2025.

Reviewer_HNerOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies threat-actor attribution when only a few CTI reports are available per actor. It introduces HEAA, comprising 501 reports for 84 actors with actor labels and cross-graph entity correspondences, and formulates attribution as semantic alignment between query and reference graphs. ThreatAlign combines saliency-based evidence selection, recursive LLM-generated entity profiles enriched with MITRE ATT&CK/Malpedia knowledge, and candidate retrieval followed by LLM arbitration. It reports 43.5% macro-F1 on HEAA versus 26.9% for the strongest evaluated attribution baseline. Additional results include single-report attribution, a chronological split, component ablations, AADM+, general-domain entity alignment, and a probe of report-specific LLM knowledge.

Critical strengths
  1. The paper studies a consequential and concretely characterized sparse-evidence setting. HEAA includes 84 actors, with 30 having one report on each side, and supports both attribution and entity-correspondence analysis (§3; Appendix A). The distinction among discriminative, ambiguous, and non-matching evidence helps explain why standard actor classifiers and direct entity matching can struggle. This is a useful evaluation contribution beyond simply reporting aggregate attribution accuracy.

  2. The improvement is substantial and is supported by several relevant checks. Besides 43.5% versus 26.9% actor-level macro-F1 (Table 2), ThreatAlign achieves 34.2% versus 15.3% for the strongest single-report baseline (Table 10) and 41.3% versus 24.5% on the chronological split (Table 11). Table 4 controls the node budget when removing saliency ranking and retains selected entities, relations, and external context when removing lower-level profile reuse. These results provide credible evidence that the complete system and its profiling choices are useful in the evaluated setting.

Critical weaknesses

The main comparison does not adequately isolate the benefit over an equally informed, simpler LLM/retrieval system. The external knowledge base explicitly contains malware-actor relations, and the worked example obtains the target actor association from this knowledge before cross-graph alignment (Appendix B.1, pp. 15-16; §D.5, p. 24). The actor-profiling prompt also asks the LLM to infer a standardized actor name using its internal knowledge (p. 19). This information access is a legitimate design choice, but it creates a substantial alternative explanation for the advantage over reference-report-trained classifiers. The direct-LLM baseline (§D.3) specifies the same filtered query graph and candidate set, without establishing access to the same retrieved knowledge. The H1/H2 ablations are valuable, but do not compare against a flat RAG or direct relation-lookup baseline with matched evidence. The related-work discussion identifies an especially relevant LLM evidence-path method, TAA-EPLMR, but no experimental comparison or exclusion rationale is provided. This leaves the incremental contribution of recursive cross-graph alignment.

Questions for authors

See the weakness part.

5 comments
AuthorsOpenReview ↗

(1/2) Author Rebuttal

We thank the reviewer for the valuable comments.


W1a. Missing Equally Informed Simpler Baselines.

We have added two simplified baselines, one that directly uses retrieved evidence and another that performs non-recursive profile matching, to examine whether ThreatAlign’s complete design is needed under the same information-access conditions.

Baseline definitions and control conditions. Both baselines use the same DeepSeek-V3.2 model, external corpus, complete candidate actor pool, and query-side actor-identity masking rules as ThreatAlign. The input limit is 122,880 tokens and the output limit is 8,192 tokens, with a total context length of at most 131,072 tokens.

  • RAG-Direct: The anonymized query graph, retrieved external context, and candidate actor names and identifiers are provided directly to the LLM to predict the actor. This baseline uses the same query-side information, external knowledge, and LLM internal knowledge as ThreatAlign, together with an explicit candidate actor list, serving as a direct RAG baseline.
  • Flat-Profile: This baseline additionally uses reference-graph information. It organizes the graph evidence associated with each actor into a single input for profile generation, retrieves candidate actors through vector search, and then uses the LLM to complete attribution. It serves as a simpler entity-alignment baseline.
MethodQuery-graph informationReference-graph informationExternal corpusLLM internal knowledgeM-F1 (%)
Direct-LLM✓✗✗✓21.3 ± 1.5
RAG-Direct✓✗✓✓26.5 ± 1.5
Flat-Profile✓✓✓✓19.9 ± 1.1
ThreatAlign✓✓✓✓43.5 ± 2.0

Adding external retrieval improves RAG-Direct’s M-F1 by 5.2 percentage points over the Direct LLM Prediction baseline in Appendix D.3, showing that external knowledge is helpful. Nevertheless, RAG-Direct remains 17.0 percentage points below ThreatAlign, indicating that directly using this knowledge does not achieve the performance of the complete pipeline. Flat-Profile has the same scope of information access as ThreatAlign but achieves an M-F1 score that is 23.6 percentage points lower. Providing the same information sources and performing single-pass profile matching is therefore insufficient to match the complete method. Its lower performance than RAG-Direct also indicates that additional reference-side information does not automatically translate into better performance.

These comparisons support ThreatAlign’s additional benefits over the simplified pipelines evaluated. Furthermore, the ablation study retains the selected entities, relations, and external context while replacing lower-level profiles with raw attributes, reducing M-F1 from 43.5% to 32.6%. Together, the simplified baselines and this controlled experiment support the indispensable roles of the complete pipeline and recursive representations.


W1b. Pretraining Knowledge as Alternative Explanation.

For long-tail actor attribution under sparse evidence, using the LLM’s existing knowledge to assist attribution is a necessary design choice. ChatEA [1], EASY-EA [2], and TAA-EPLMR [3] also emphasize using LLM capabilities. ThreatAlign follows this research direction by using existing knowledge to supplement limited observations.

As the reviewer correctly points out, however, the contribution of internal knowledge must be distinguished from that of the framework’s other design choices. The simplified baselines in W1a examine this alternative explanation. RAG-Direct and Flat-Profile use the same DeepSeek-V3.2 backbone as ThreatAlign and have access to the same external corpus, achieving 26.5% and 19.9% macro-F1, respectively, compared with 43.5% for the complete method. The available evidence supports the view that how the framework organizes evidence, constructs representations, and uses accessible knowledge is an important source of ThreatAlign’s performance gains.

We further conducted a simple controlled experiment to examine the effect of removing the actor-name inference instruction from the prompt. The results are shown below.

ConfigurationM-P (%)M-R (%)M-F1 (%)
Original configuration40.7 ± 2.350.8 ± 1.543.5 ± 2.0
Without the actor-name inference instruction42.4 ± 3.652.0 ± 3.745.0 ± 3.6

Removing the instruction does not reduce the observed mean performance. Together, this controlled experiment, the pretraining-knowledge probe in Appendix D.4, and the simplified-baseline experiments represent attempts to remove the influence of pretraining knowledge and demonstrate the benefits of the method’s design beyond pretraining knowledge.

AuthorsOpenReview ↗

(2/2) Author Rebuttal


W1c. Missing TAA-EPLMR Comparison or Justification.

TAA-EPLMR was not included in the experimental comparison primarily because its code is not publicly available, and its paper omits some details of the prompts, path retrieval, and pruning procedures. Moreover, it reports results only on the self-constructed, non-public Dataset-Full benchmark [4] and its Dataset-Incomplete and Dataset-Noise variants. These limitations make it difficult to include TAA-EPLMR directly as an experimental baseline.


References

[1] Jiang, Xuhui, et al. “Unlocking the Power of Large Language Models for Entity Alignment.” Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

[2] Cheng, Jingwei, et al. “EasyEA: Large Language Model Is All You Need in Entity Alignment Between Knowledge Graphs.” Findings of the Association for Computational Linguistics: ACL 2025. 2025.

[3] Xiao, Nan, et al. “TAA-EPLMR: Threat Actor Attribution via Evidence Path-Enhanced Large Language Model Reasoning.” 2025 IEEE International Conference on Big Data (BigData). IEEE, 2025.

[4] Xiao, Nan, et al. “APT-MMF: An Advanced Persistent Threat Actor Attribution Method Based on Multimodal and Multilevel Feature Fusion.” Computers & Security 144 (2024): 103960.

Thank you for the author’s reply. My question has been basically resolved, and I have decided to maintain the score.

Reviewer_SZjbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the long-tail problem in APT attribution based on Cyber Threat Intelligence (CTI) reports. It reframes attribution as an entity-level semantic alignment task between an unattributed query graph and an attributed reference graph.

The authors propose THREATALIGN, which consists of three main components:

  1. Evidence Saliency Estimation: Assigns saliency scores to entities based on their discriminability and corroboration.
  2. Hierarchical Semantic Profiling: Uses an LLM to aggregate selected lower-level evidence into textual profiles, progressively building representations up to the APT actor level.
  3. Evidence-Constrained Alignment: Retrieves candidate actors based on profile similarity and IoC overlap, followed by LLM-based arbitration under a compound acceptance rule.

The authors also construct a new dataset, HEAA, and evaluate their method on HEAA, AADM+, and ICEWS-WIKI.

Critical strengths
  1. Well-motivated problem formulation. The paper provides a meaningful analysis of the long-tail problem in APT attribution and proposes a feasible approach to address this challenge.

  2. New dataset construction. The authors construct a dedicated dataset, HEAA, to evaluate the proposed method, which provides a useful benchmark for studying long-tail APT attribution.

  3. Comprehensive ablation studies. The paper includes extensive module-level ablation experiments and analyses, which help illustrate the contributions of different components of the proposed framework.

Critical weaknesses
  1. Limited generalization to public benchmarks. The main conclusions of the paper are largely supported by results on the newly constructed HEAA dataset. However, the proposed method does not demonstrate a clear performance advantage over existing approaches on the public AADM and ICEWS-WIKI datasets. This raises concerns about the generalizability of the proposed approach beyond the authors' own dataset.

  2. Insufficient differentiation from existing work. The overall research idea and methodology share considerable similarities with TAA-EPLMR. The paper should more clearly explain the fundamental differences between the two approaches and clarify its novel contributions.

  3. Missing efficiency evaluation. Since the proposed method relies on LLM-based reasoning, the paper should report computational efficiency metrics, particularly token consumption and inference latency. These metrics are important for understanding the practical applicability of the proposed approach.

  4. Dependence on manual intervention. Some steps of the proposed algorithm rely on manual selection, which may limit its scalability and applicability in practical APT attribution scenarios.

Questions for authors
  1. More reliable baseline comparisons. Could the authors include additional and more reliable baselines, or provide further explanations for the current baseline results? In particular, the performance of APT-MMF and TRAIL is below that of random prediction, which is unexpected. Such results make it difficult to assess the effectiveness of the proposed method through these comparisons.

  2. Ablation studies on public datasets. Could the authors provide additional ablation studies on the AADM+ dataset? Currently, most of the ablation analyses are conducted on HEAA, which is entirely constructed by the authors. Additional experiments on public benchmarks would provide more convincing evidence for the effectiveness of individual components.

  3. LLM generalization. The proposed method relies heavily on LLM-based reasoning. Does it remain effective when using LLMs other than DeepSeek? Could the authors provide experimental results with alternative LLM backbones to demonstrate the generality of the approach?

  4. Potential data leakage. The report-level separation strategy does not necessarily guarantee the absence of data leakage. Could the authors provide additional evidence or analyses to demonstrate that there is no overlap or information leakage between the training and evaluation data?

  5. Contribution of Malpedia and the need for RAG baselines. The proposed method relies on Malpedia data, making it difficult to determine whether the observed performance improvements primarily come from the proposed algorithm, the external knowledge provided by Malpedia, or the reasoning capabilities of the LLM itself. Could the authors include a RAG-based baseline to better isolate the contributions of these components?

  6. Additional evaluation metrics. Although the paper primarily focuses on the long-tail problem, could the authors also report Micro-F1 and other overall performance metrics? This would provide a more comprehensive understanding of the method's performance across different frequency distributions of APT groups.

5 comments
AuthorsOpenReview ↗

(1/4) Author Rebuttal

We thank you for your careful and constructive feedback.


W1. Limited Generalization Beyond HEAA.

To evaluate whether the method generalizes beyond our newly constructed dataset, we selected AADM+ and ICEWS-WIKI, two widely used benchmarks for report-based attribution and heterogeneous entity alignment, respectively. Identifying additional suitable benchmarks is difficult given the limited research on report-based attribution: the datasets used by APT-MMF [1] and TAA-EPLMR [2] are not publicly available, while the TRAIL [3] dataset does not target report-based attribution.

ThreatAlign is primarily designed for long-tail APT attribution under extremely limited report coverage, while also generalizing to other public benchmarks. Although ThreatAlign does not achieve the best performance on either public benchmark, it remains close to the best-performing methods. On AADM+, ThreatAlign achieves 97.8% M-F1, compared with 98.0% for APT-ATT. On ICEWS-WIKI, the adapted ThreatAlign achieves 98.18% Hits@1, compared with 99.50% for EASY-EA. HEAA evaluates the target long-tail setting, in which existing baselines experience substantial performance degradation, further supporting ThreatAlign’s intended scope.


W2. Insufficient Differentiation from TAA-EPLMR.

TAA-EPLMR [2] and ThreatAlign differ substantially in three respects: (i) task assumptions, (ii) attribution formulation, and (iii) evidence representation.

Specifically, (i) TAA-EPLMR relies on retrievable IoC association paths in a graph, whereas ThreatAlign targets long-tail conditions with limited actor information, fragmented behavioral evidence, and heterogeneous expressions across reports, where available association paths are limited. (ii) TAA-EPLMR aggregates retrieved and pruned paths by candidate actor and uses an LLM to predict the actor’s name. ThreatAlign instead formulates attribution as entity alignment between query and reference graphs, identifying actor correspondences through profile retrieval and comparison. (iii) TAA-EPLMR uses association paths as evidence units. ThreatAlign uses a hierarchical structure and recursive profiling to progressively aggregate fragmented behavioral context into actor representations that can be compared across graphs.

TAA-EPLMR has no publicly available code, and its paper omits some details of the prompts, path retrieval, and pruning procedures. Moreover, it reports results only on the non-public Dataset-Full [1] benchmark and its Dataset-Incomplete and Dataset-Noise variants. These limitations make it difficult to include TAA-EPLMR directly as an experimental baseline.

ThreatAlign’s main contributions are: (i) using entity-level semantic alignment to exploit limited actor evidence; (ii) introducing a four-level evidence hierarchy and recursive actor profiling; and (iii) jointly designing evidence-selection and matching mechanisms tailored to APT evidence. The ablation studies validate the effectiveness of these designs.


W3. Missing Efficiency Evaluation.

We have added a cost comparison between ThreatAlign and two simplified LLM-based baselines described in Q5. All methods use DeepSeek-V3.2, and we measure token consumption and total batch runtime on HEAA. The results demonstrate the practical feasibility of ThreatAlign as an offline attribution workflow: the complete framework achieves better attribution performance at a higher computational cost than the simplified baselines.

MethodTotal tokens (K)Amortized tokens per actor (K)Total runtime (min)Amortized runtime per actor (s)
ThreatAlign5,381.3064.1103.073.6
RAG-Direct2,216.226.414.910.6
Flat-Profile3,992.847.519.914.2

Compared with Flat-Profile, which has access to the same information sources, ThreatAlign increases token consumption by 34.8% while achieving a relative M-F1 improvement of 118.6%. Compared with RAG-Direct, which does not use reference-graph information, the corresponding increases are 142.8% and 64.2%. Based on official pricing, the estimated additional cost per actor is approximately USD 0.007 and USD 0.016, respectively, and the upper bound on the total batch token cost of ThreatAlign is approximately USD 2.26. The additional calls therefore incur a low absolute monetary cost while improving attribution performance.

Regarding practical applicability, Yuldoshkhujaev et al. [4] compiled 1,509 deduplicated reports from three public report collections covering 2014–2023, averaging approximately 151 reports per year or 12.6 per month. On HEAA, which contains 501 reports, ThreatAlign completes attribution in approximately 1.7 hours, averaging 12.3 seconds per report or an amortized 73.6 seconds per actor across 84 actors. These results demonstrate its feasibility for offline batch processing.

AuthorsOpenReview ↗

(2/4) Author Rebuttal


W4. Dependence on Manual Intervention.

ThreatAlign’s attribution inference pipeline is fully automated. Human involvement is limited to benchmark construction.


Q1. Reliability of Baseline Comparisons.

We previously used only the GNN-based variant, which achieved the best performance reported in the original TRAIL paper, as a baseline. To broaden the comparison, we have added the label-propagation variant, which achieved the second-best performance in that paper, and denote it as TRAIL-LP. We have renamed the original GNN-based baseline as TRAIL-GNN.

As shown below, the baselines exhibit clear differences in performance. Feature-based methods generally retain some predictive capability, whereas the graph-learning-based methods APT-MMF and TRAIL-GNN perform poorly. Analysis of their outputs indicates that the low scores arise from extremely long-tailed conditions, where scarce supervision and limited discriminative neighborhoods jointly constrain learning for tail actors.

MethodMethod categoryM-F1 (%)
SMOBI+SVMMachine-learning-based attribution15.6 ± 0.0
SMOBI+RFMachine-learning-based attribution20.9 ± 1.2
SMOBI+XGBoostMachine-learning-based attribution20.5 ± 0.0
SMOBI+LightGBMMachine-learning-based attribution21.2 ± 0.0
APT-ATTMachine-learning-based attribution26.9 ± 1.4
APT-MMFGraph-learning-based attribution0.6 ± 0.3
TRAIL-GNNGraph-learning-based attribution2.1 ± 1.0
TRAIL-LP (added)Graph-based attribution20.4 ± 1.4
ThreatAlignEntity alignment for APT attribution43.5 ± 2.0

Specifically, 59 of the 84 actors in HEAA (70.24%) have only one reference instance used for training. Neither APT-MMF nor TRAIL-GNN correctly attributes any of these 59 actors in any of the three runs. For TRAIL-GNN, same-class training references are reachable within two hops for only 15–17 of the 84 actors, increasing to 29–34 within four hops. Although APT-MMF connects 76–77 of the 84 actors to same-class references, only 3.23%–3.25% of the query–training connections in its combined meta-path graph are same-class connections, close to the random-pairing prior of 2.94%. Under seed 42, the higher-purity paths based on shared domains, URLs, and IP addresses cover only 16, 2, and 3 actors, respectively. Ultimately, APT-MMF predicts only 3, 10, and 7 distinct classes across the three runs, while TRAIL-GNN predicts only 4 classes in every run.

In contrast, TRAIL-LP achieves 20.4% ± 1.4% M-F1. Under the same graph and evaluation protocol, label propagation substantially outperforms TRAIL-GNN and approaches the performance of the feature-based methods. Even when restricted to the same two-hop range as the two-layer GNN, it achieves 19.05% M-F1. This indicates that the graph still contains useful attribution signals, but the GNN fails to learn effective prediction rules from long-tailed supervision.

Overall, despite extensive data and method adaptation, the generalization difficulties of graph models under long-tailed supervision directly lead to the unusually low scores of the existing graph baselines. We will include TRAIL-LP as an additional effective baseline in the comparison.


Q2. Component Ablations on AADM+.

We have added single-component ablations on the public attribution dataset AADM+ and compared the results with those on HEAA.

ConfigurationM-P (%)M-R (%)M-F1 (%)
ThreatAlign98.2 ± 1.398.2 ± 1.397.8 ± 1.6
w/o E1E_195.7 ± 3.196.4 ± 2.395.3 ± 3.3
w/o E2E_289.8 ± 5.891.7 ± 4.589.1 ± 6.2
w/o H2H_281.9 ± 12.684.7 ± 10.981.8 ± 12.4
w/o H1H_191.2 ± 4.389.8 ± 4.087.9 ± 5.2
w/o S1S_188.0 ± 6.889.8 ± 5.286.5 ± 7.2

Retrieved context (H2H_2) substantially affects attribution performance and stability across runs. Removing it reduces M-F1 on AADM+ by 16.0 percentage points and increases the standard deviation from 1.6 to 12.4. Retrieved context provides more stable factual grounding for profile generation and candidate evaluation; without it, model outputs vary more across runs.

Saliency ranking (S1S_1) has a substantially larger effect on AADM+ than on HEAA, which can be explained by the characteristics of its input graphs. The query and reference graphs of AADM+ contain 17 and 22 relation types, respectively, with a structural similarity of only 0.55%. The standard deviation of the absolute cross-graph degree differences for aligned actors reaches 409.09, indicating substantial differences in the scale and structure of their associations across the two graphs. Under these conditions, saliency ranking allocates the limited profiling budget according to discriminability and corroboration from lower-level evidence, reducing interference from low-value associations in actor representations.

AuthorsOpenReview ↗

(3/4) Author Rebuttal

After removing IoC matching (E1E_1), M-F1 remains at 95.3%, indicating that the remaining semantic processing can support most attributions. On HEAA, where reports are scarce and cross-graph heterogeneity is substantial, the direct-correspondence channel provides complementary evidence that is harder to replace with semantic matching. The remaining component ablations show trends consistent with those on HEAA.


Q3. Generalization Across LLM Backbones.

The additional experiments support the effectiveness of ThreatAlign with other LLM backbones. We expanded the evaluation to cover additional model families and parameter scales. All added configurations achieve mean M-F1 scores no lower than the original DeepSeek-V3.2 configuration. Notably, the 27B model achieves attribution performance comparable to the original configuration.

LLM backboneM-P (%)M-R (%)M-F1 (%)
DeepSeek-V3.240.7 ± 2.350.8 ± 1.543.5 ± 2.0
DeepSeek V4.1 Flash45.7 ± 3.956.7 ± 3.048.8 ± 3.5
GLM-5.3-Flash47.8 ± 2.958.3 ± 1.951.0 ± 2.7
Qwen3.8-Flash43.7 ± 1.455.6 ± 2.047.2 ± 1.6
MiMo-V2.6-Flash45.6 ± 2.456.3 ± 2.048.9 ± 2.3
Qwen3.8-27B41.5 ± 3.750.8 ± 2.844.1 ± 3.4

The results with GLM, Qwen, and MiMo demonstrate that the framework remains effective beyond the DeepSeek family. All added Flash configurations improve mean M-F1, indicating that changing the backbone can further improve the performance of the complete pipeline. For consistency with the main paper, thinking mode is disabled for all backbones except GLM-5.3-Flash, which does not support disabling it. Regarding parameter scale, Qwen3.8-27B achieves a slightly higher mean M-F1 than DeepSeek-V3.2, demonstrating that the proposed pipeline remains effective with a smaller backbone and suggesting greater potential for offline deployment of ThreatAlign.


Q4. Potential Data Leakage.

We agree that report-level separation alone cannot rule out all information leakage. Accordingly, before submission, we manually examined the original reports on the query and reference sides to avoid duplicate reports or direct data leakage.


Q5. RAG Baselines and Knowledge Contributions.

We have added two simplified knowledge-enhanced baselines to examine whether the complete framework provides additional benefits under the same LLM backbone, external-corpus access, and candidate actor pool. The results show that external knowledge helps attribution, while ThreatAlign retains an advantage over the simplified methods evaluated.

Baseline definitions and control conditions. Both baselines use the same DeepSeek-V3.2 model, external corpus, complete candidate actor pool, and query-side actor-identity masking rules as ThreatAlign. The input limit is 122,880 tokens and the output limit is 8,192 tokens, with a total context length of at most 131,072 tokens.

  • RAG-Direct: The anonymized query graph, retrieved external context, and candidate actor names and identifiers are provided directly to the LLM to predict the actor. This baseline uses the same query-side information, external knowledge, and LLM internal knowledge as ThreatAlign, together with an explicit candidate actor list, serving as a direct RAG baseline.
  • Flat-Profile: This baseline additionally uses reference-graph information. It organizes the graph evidence associated with each actor into a single input for profile generation, retrieves candidate actors through vector search, and then uses the LLM to complete attribution. It serves as a simpler entity-alignment baseline.

Results and analysis.

MethodQuery-graph informationReference-graph informationExternal corpusLLM internal knowledgeM-F1 (%)
Direct-LLM✓✗✗✓21.3 ± 1.5
RAG-Direct✓✗✓✓26.5 ± 1.5
Flat-Profile✓✓✓✓19.9 ± 1.1
ThreatAlign✓✓✓✓43.5 ± 2.0

With external knowledge, RAG-Direct improves M-F1 by 5.2 percentage points over the Direct LLM Prediction baseline in Appendix D.3, confirming that the knowledge base contributes to attribution performance. Nevertheless, despite access to the same LLM internal knowledge and external knowledge, RAG-Direct remains 17.0 percentage points below ThreatAlign. This indicates that directly recovering standardized actor names from identity-masked query graphs using internal memory and external knowledge remains challenging for the LLM. Under the same scope of knowledge access, the complete framework outperforms the evaluated Flat-Profile baseline, supporting the additional benefits of evidence organization and matching. These experiments further demonstrate the necessity of ThreatAlign’s design choices and support their additional benefits beyond the use of LLM internal knowledge and an external knowledge base.

Reviewer_ypYQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

How I used AI

I first wrote my assessment of this paper myself in Chinese. English is not my first language, so I then used an AI language model to polish these comments and translate them into English. The Chinese text reproduced below contains my original assessment, including its strengths, concerns, questions, and recommendation. My original recommendation was weak acceptance (3), with confidence 3.

The purpose of the AI request was translation and language polishing. I provide my original text so that the ACs and SACs can assess my own comments directly.

Prompt used

润色以上审稿意见并且将其翻译成英文

English translation of the prompt: “Polish the above review comments and translate them into English.”

The Chinese assessment reproduced below was the review text supplied for this request.

Original human-written Chinese assessment supplied for translation and polishing

44123 — THREATALIGN

长尾 APT 归因,改成跨图实体对齐。先挑证据,再做 profile,最后检索、比较候选。这个切入点有实际意义,很多 actor 本来就没几篇报告,直接做分类比较困难。

好的地方是把 malware、行为等细粒度证据汇总起来做归因。HEAA 这个数据也有用,43.5 提升比较大。消融和不同设置都做了,作为偏应用的工作算是有价值的。

不过方法新意一般。证据筛选、LLM 写 profile、检索重排、LLM 再判断,这些步骤都比较熟悉。组合得合理,但最核心的新对齐机制在哪里?需要把针对 APT 的设计和一般的工程组合讲清楚。已有消融能说明组件有用,我还想知道,同样信息和候选范围下,普通 LLM 对齐流程离它有多远。

数据只有 501 篇报告、84 个 actor,很多 actor 的报告很少。这也是长尾任务的特点,但每个组的结果会不会容易受少数样本影响?标注质量还需要具体数字。malware 对应这部分是 LLM 先给候选,人工核验,有分歧再处理。流程可以,但两人一致性多少,多少候选被否掉或改过?只检查候选的话,LLM 没提出来的正确对应可能还是漏的。

核心框架只验证了 DeepSeek-V3.2。提升有多少来自流程,有多少依赖这个模型已有的安全知识和语义能力,不太好分。希望至少换一个模型系列或更小的模型看一下。当前模型上的消融还不能说明换模型后也成立。

评估这里读的时候需要停一下。actor-level 和拿到一篇报告就归因,不是同一个难度。分类基线用了 any-hit,相关数字不能直接理解成独立报告的识别能力。43.5 和 34.2 分别说明什么、两个层级的设置怎么对应实际使用,要解释得更直白。

给作者的问题主要是标注一致性和人工修改比例、普通 LLM 对齐及各模块的收益、换模型后的表现,以及两个评估层级的差别。

整体偏弱接收,3 分。创新性不强,但问题有意义,数据和结果能支持一定贡献。写法有点密,几个模块的关系需要来回看。confidence 3。


English translation accompanying the original notes

This is the English translation included with my Chinese notes. It is provided for readability. The Chinese text is my original assessment; this English text is its translation.

44123 — THREATALIGN

Long-tail APT attribution, turned into cross-graph entity alignment. First select evidence, then make profiles, and finally retrieve and compare candidates. This starting point has practical meaning. Many actors only have a few reports to begin with, so directly doing classification is fairly difficult.

The good part is bringing together fine-grained evidence such as malware and behaviors for attribution. HEAA is also useful, and the improvement to 43.5 is fairly large. Ablations and different settings have both been tested. As a more application-oriented piece of work, it has value.

But the novelty of the method is average. Evidence selection, LLM-written profiles, retrieval and reranking, and then another LLM judgment are all fairly familiar steps. The combination is reasonable, but where is the core new alignment mechanism? The designs specific to APT and the general engineering combination need to be explained clearly. The existing ablations can show that the components are useful. I still want to know how far an ordinary LLM alignment pipeline is from it, given the same information and candidate range.

The data has only 501 reports and 84 actors, and many actors have very few reports. This is also a feature of the long-tail task, but could the results for each group be easily affected by a small number of samples? Annotation quality still needs specific numbers. For the malware correspondences, the LLM first gives candidates, humans check them, and disagreements are then handled. The process is fine, but how much do the two people agree, and how many candidates were rejected or changed? If only the candidates are checked, correct correspondences that the LLM did not propose may still be missed.

The core framework has only been tested with DeepSeek-V3.2. It is not easy to separate how much of the improvement comes from the pipeline and how much depends on this model's existing security knowledge and semantic ability. I hope they at least try another model family or a smaller model. Ablations on the current model do not yet show that the results will also hold after changing models.

I need to pause a bit when reading the evaluation here. Actor-level attribution and attributing an actor from a single report are not equally difficult. The classification baselines use any-hit, so the related numbers cannot be directly understood as the ability to identify actors from independent reports. What 43.5 and 34.2 each mean, and how the settings at the two levels correspond to actual use, need to be explained more directly.

The main questions for the authors are annotation agreement and the proportion of manual changes, ordinary LLM alignment and the gains from each module, performance after changing models, and the difference between the two evaluation levels.

Overall, leaning toward weak acceptance, score 3. The novelty is not strong, but the problem is meaningful, and the data and results support some contribution. The writing is a bit dense, and understanding the relationships among the modules takes some back-and-forth reading. confidence 3.

Critical strengths

1. This paper targets the practical challenge of identifying long-tail APT actors in threat attribution. The proposed entity-level semantic alignment approach fits real-world threat intelligence workflows and shows good practical potential.

2. The authors construct a dedicated evaluation dataset and validate the method across a set of comparative experiments. The experimental setup is reasonably thorough to back up their main claims.

Critical weaknesses
  • The proposed framework is primarily an engineering combination of existing techniques, including heuristic evidence selection, LLM-based semantic profiling, retrieval and reranking, and LLM-based arbitration. The paper does not introduce a new learning objective, alignment mechanism, or theoretical formulation. As a result, the methodological contribution beyond applying existing techniques to the APT attribution domain is relatively limited.

  • The HEAA benchmark is relatively small, containing only 501 reports and 84 APT actors. Many actors have very limited reports, which may reduce the statistical reliability of the evaluation. In addition, the paper does not report inter-annotator agreement for the cross-graph entity alignment annotations. The LLM-propose-then-human-verify annotation procedure may also introduce candidate generation bias and annotation noise.

  • All experiments rely on a single LLM, DeepSeek-V3.2, without sufficient evaluation across different model families or model sizes. It is therefore unclear whether the reported improvements mainly result from the proposed framework or the underlying domain knowledge and semantic capabilities of the selected LLM.

  • The actor-level evaluation uses an any-hit aggregation rule, where an actor is considered correctly attributed if any of its reports receives the correct prediction. This evaluation protocol may overestimate the actual report-level attribution capability of the method. More emphasis should be placed on independent single-report attribution performance.

Questions for authors
  1. Could the authors report inter-annotator agreement for HEAA and the proportion of LLM-generated annotations that require human correction?

  2. How much does each proposed component contribute compared with a standard LLM-based entity alignment baseline?

  3. Do the reported improvements remain consistent across different LLM families and model sizes?

  4. Could the authors provide a more detailed comparison between actor-level and single-report attribution performance?

6 comments
AuthorsOpenReview ↗

(1/3) Author Rebuttal

We thank the reviewer for the careful review and for recognizing the practical importance of addressing long-tail APT attribution.


W1. Limited Methodological Novelty.

ThreatAlign’s design indeed draws inspiration from established areas such as RAG and LLM-as-Judge. The methodological contributions of this work are: i) leveraging limited actor evidence through entity-level semantic alignment; ii) jointly designing a four-level evidence structure and recursive actor profiling; and iii) developing evidence selection and matching mechanisms tailored to APT evidence characteristics and integrating them into a coordinated framework. These specific designs are introduced in this work, and the ablation experiments demonstrate their effectiveness.

Following your suggestion, we additionally introduce Flat-Profile, a baseline that, under the same conditions, generates actor profiles using external corpus augmentation and the candidate actor set, retrieves candidates, and uses an LLM to make attribution decisions.

ThreatAlign outperforms Flat-Profile by 23.6 percentage points, indicating a clear additional benefit over simply combining basic designs.

SettingM-F1 (%)Decrease
ThreatAlign43.5 ± 2.0—
w/o S1S_138.4 ± 3.35.1
w/o H1H_132.6 ± 1.610.9
w/o E1E_134.6 ± 2.98.9
w/o E2E_234.9 ± 4.58.6
Flat-Profile19.9 ± 1.123.6

W2a. Benchmark Size and Statistical Reliability.

HEAA substantially expands the scale of existing benchmarks. Because threat reports are difficult to obtain, benchmarks for report-based attribution typically contain hundreds to just over a thousand reports. Taking the only two publicly available benchmarks known to us as examples, AADM [1] contains only 12 actors and 238 reports, while AADM+ [2] contains 12 actors and 271 reports. HEAA contains 84 actors and 501 reports.

In real-world settings, as shown by our statistical analysis of Malpedia in the main paper, 64.48% of 960 actors are associated with no more than two reports, while 12.08% are associated with more than five reports. HEAA preserves this characteristic to better reflect real-world conditions. To avoid evaluation bias, we also employ multiple evaluation settings, including report-count buckets, a temporal split, and independent single-report attribution. We additionally conduct three independent runs and report the mean and standard deviation.

[1] Perry, Lior, Bracha Shapira, and Rami Puzis. "No-doubt: Attack attribution based on threat intelligence reports." 2019 IEEE International Conference on Intelligence and Security Informatics (ISI). IEEE, 2019.

[2] Cai, Saihua, et al. "APT-ATT: An efficient APT attribution model based on heterogeneous threat intelligence representation and CTGAN." Computer Networks 270 (2025): 111511.


W2b (Q1). Annotation Reliability and Potential Bias.

Independent verification by two annotators in HEAA primarily concerns candidate cross-graph correspondences between malware entities. Two cybersecurity researchers independently determine whether each candidate pair refers to the same entity, with disagreements adjudicated by a third researcher. Statistical analysis shows an agreement rate of 96.7% between the two annotators on report annotations, with Cohen’s κ of 0.894. LLM-assisted annotation tends to produce excessive matches when one query entity corresponds to multiple reference entities. According to our statistics, approximately 17.0% of malware candidate entities were manually excluded. We use human verification to prevent incorrect cross-graph entity correspondences.

AuthorsOpenReview ↗

(2/3) Author Rebuttal

W3 (Q3). Limited Cross-Model Evaluation.

To address this concern, we added experiments with different LLM families and model sizes. The results show that ThreatAlign retains its attribution capability when the backbone is changed. All newly evaluated models achieve a mean M-F1 at least as high as the original DeepSeek V3.2 configuration, and the smaller 27B model also achieves comparable performance.

Backbone modelM-PM-RM-F1
DeepSeek V3.240.7 ± 2.350.8 ± 1.543.5 ± 2.0
DeepSeek V4.1 Flash45.7 ± 3.956.7 ± 3.048.8 ± 3.5
GLM-5.3-Flash47.8 ± 2.958.3 ± 1.951.0 ± 2.7
Qwen3.8-Flash43.7 ± 1.455.6 ± 2.047.2 ± 1.6
MiMo-V2.6-Flash45.6 ± 2.456.3 ± 2.048.9 ± 2.3
Qwen3.8-27B41.5 ± 3.750.8 ± 2.844.1 ± 3.4

All newly evaluated Flash models achieve higher mean M-F1, indicating that the framework can benefit from backbone differences to further improve performance. As in the main paper, thinking mode is disabled for all models except GLM-5.3-Flash, for which it cannot be disabled. Qwen3.8-27B achieves a slightly higher mean M-F1 than DeepSeek V3.2, demonstrating ThreatAlign’s compatibility with smaller models and strengthening its practicality for offline deployment.


W4 (Q4). Any-Hit Bias and Single-Report Evaluation.

Any-hit evaluation can indeed overestimate attribution capability. However, this evaluation protocol is intended to allow report-level methods to benefit from multiple reports in joint attribution; it is not applied to ThreatAlign. The actor-level advantage reported in the paper is therefore obtained while granting classification baselines this favorable condition.

ThreatAlign supports actor-level attribution using multiple reports about the same unknown actor jointly. However, classification baselines take individual reports as input, do not support joint attribution across multiple reports, and are also subject to input-length constraints. To give these baselines the opportunity to use multiple reports in actor-level comparisons, we apply any-hit aggregation to their report-level attribution results.

Evaluation settingThreatAlignClassification baselineGap
Actor-level attribution43.5 ± 2.0APT-ATT: 26.9 ± 1.4+16.6
Single-report attribution34.2 ± 1.3CyBERT: 15.3 ± 2.3+18.9

Performance in the single-report attribution setting is established by the experiments in Appendix D.1. ThreatAlign’s M-F1 exceeds that of the strongest baseline, CyBERT, by 18.9%. This margin is larger than in the actor-level evaluation. These results indicate that ThreatAlign has advantages in both multi-report evidence aggregation and single-report attribution. We will further emphasize the single-report results and clarify which methods use any-hit and the motivation for this evaluation setting.

AuthorsOpenReview ↗

(3/3) Author Rebuttal

Q2. Contributions Beyond Standard Entity Alignment.

Compared with the entity alignment baseline EASY-EA, ThreatAlign differs fundamentally in actor representation generation and candidate actor recall. The knowledge-base augmentation experiment, profile-substitution experiment, and component ablations demonstrate the effectiveness of ThreatAlign’s design. We additionally report candidate recall to clarify these differences.

Specifically, adding the same knowledge base K\mathcal{K} used by ThreatAlign to the original EASY-EA increases Hits@1 by only 3%, indicating that the gap is far from being attributable solely to the external knowledge base. Further replacing EASY-EA’s actor representations with actor profiles generated by ThreatAlign increases candidate recall from 8.47% to 32.20% and Hits@1 from 5.08% to 28.81%. This indicates that actor representation generation in conventional alignment methods does not readily adapt to the complex setting of long-tail APT actors. Furthermore, even when using ThreatAlign’s actor representations, there remains a gap of 29.7 in candidate recall compared with ThreatAlign, demonstrating a difference in candidate actor retrieval.

MethodCandidate recallHits@1
EASY-EA8.55.1
EASY-EA + K\mathcal{K}22.08.5
EASY-EA + ThreatAlign actor profiles32.228.8
ThreatAlign61.950.8

The roles of ThreatAlign’s individual components within the complete framework are evaluated through ablation experiments in Section 5.4 of the main paper. The results support dependencies among the components and indicate that the framework’s overall gains cannot be obtained by simply adding independent component gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.