- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces APTInvestBench, a benchmark designed to evaluate the robustness of LLM-based agents performing autonomous advanced persistent threat (APT) investigations under varying telemetry conditions. Unlike existing cybersecurity benchmarks that primarily focus on vulnerability exploitation or attack-chain reconstruction, APTInvestBench examines whether agents can consistently acquire and cite sufficient evidence when log availability, retention, and sampling conditions change.
The benchmark contains 370 investigation cases across seven SOC-inspired telemetry conditions, constructed from 56 report-informed attack reconstructions and approximately 16.4 million log records. It evaluates eleven LLMs and four agent frameworks using an investigation environment in which agents query telemetry and produce evidence-grounded reports. A central methodological feature is the separation of evidence recoverability, evidence acquisition, and formal citation support, allowing the authors to distinguish environmental limitations from agent investigation and reporting failures.
The experiments reveal substantial weaknesses in current autonomous investigation agents. On average, agents acquire sufficient evidence for 44.3% of recoverable attack actions, while only 25.0% receive sufficient formal citation support. Furthermore, relatively small changes in aggregate coverage can conceal substantial instability at the individual-action level. The paper argues that cross-telemetry robustness and evidence retention should be evaluated explicitly when assessing the reliability of autonomous cybersecurity investigation agents.
Critical strengths
- Important and practically relevant research problem.
The paper addresses an important challenge in autonomous cybersecurity investigation: whether LLM-based agents can reliably investigate the same security incidents under varying telemetry conditions. Since real-world security operations centers differ in log availability, retention, and sampling practices, evaluating cross-telemetry robustness is practically meaningful and relevant to the deployment of reliable security agents.
- Well-structured evaluation framework with fine-grained failure analysis.
A key strength is the distinction between evidence availability, evidence acquisition, and formal citation support. By applying consistent action-level evidence requirements across different telemetry conditions, the proposed benchmark enables a more detailed analysis of investigation failures than aggregate coverage metrics alone. In particular, it can distinguish cases where sufficient evidence is unavailable from cases where an agent fails to acquire or preserve available evidence.
- Substantial benchmark construction and experimental coverage.
The benchmark includes 370 investigation cases derived from 56 report-informed attack reconstructions, seven telemetry conditions, and approximately 16.4 million log records. The evaluation of eleven LLMs and four agent frameworks provides a relatively broad empirical assessment of autonomous investigation behavior.
- Interesting observations regarding hidden instability in investigation performance.
The experiments reveal that relatively small changes in aggregate citation coverage can conceal substantial changes in which individual attack actions are adequately supported. This finding highlights the limitations of relying exclusively on aggregate performance metrics and motivates more fine-grained evaluation of investigation consistency.
Critical weaknesses
- Insufficient validation of whether the proposed metrics reflect actual investigation competence.
My primary concern is the validity of the proposed evaluation metrics as measures of autonomous APT investigation capability. The benchmark primarily evaluates whether agents retrieve and formally cite sufficient supporting records. However, retrieving relevant record identifiers or citing sufficient evidence does not necessarily demonstrate that an agent correctly understands the evidence, reconstructs the attack sequence, or accurately identifies the underlying malicious activities.
Although the authors distinguish evidence acquisition from report interpretation, it remains unclear how strongly the primary evaluation metrics correlate with actual investigation correctness. For example, an agent could retrieve and cite relevant records without correctly interpreting their causal or temporal relationships. Independent validation against expert security analysts' assessments would substantially strengthen the credibility of the proposed benchmark.
- Limited demonstration of practical improvements over existing evaluation methodologies.
While the paper introduces a structured evaluation framework, it does not sufficiently demonstrate that the proposed metrics provide more accurate, reliable, or practically actionable assessments than existing cybersecurity investigation benchmarks. The comparison with prior benchmarks primarily emphasizes differences in benchmark design and supported evaluation features rather than directly establishing the empirical advantages of the proposed methodology.
A more convincing evaluation would compare the proposed metrics against existing assessment approaches on the same investigation outputs and demonstrate whether the new metrics better identify meaningful investigation failures or predict expert-assessed investigation quality. Without such evidence, the practical benefit of the proposed benchmark remains insufficiently established.
- Limited methodological novelty beyond benchmark construction and metric decomposition.
The main contribution consists of constructing a benchmark with controlled telemetry variations and decomposing investigation outcomes into evidence availability, acquisition, and citation support. Although this is a useful evaluation perspective, the paper does not introduce a fundamentally new investigation method or learning algorithm. Furthermore, related benchmarks have already considered variations in evidence sources, telemetry availability, and investigation diagnostics.
The authors should more clearly establish what constitutes a substantial methodological advance beyond combining these existing evaluation concepts. In particular, the novelty of the action-level evidence matching framework should be justified through stronger comparisons with closely related benchmark methodologies.
- Limited insight into how the identified failures can be addressed.
The experiments reveal that LLM agents frequently fail to acquire or preserve sufficient evidence, even when relevant supporting records remain available. However, the paper provides limited analysis of which specific agent capabilities or design choices are responsible for these failures and how they might be improved.
Although proposing a new investigation algorithm is not a requirement for a benchmark paper, stronger diagnostic experiments would increase the significance of the contribution. For example, evaluating whether improved retrieval strategies, evidence management mechanisms, or report-generation procedures systematically reduce the identified failures would help establish the practical usefulness of the benchmark beyond documenting existing limitations.
- Uncertain generalizability to real-world security operations.
The benchmark relies on report-informed attack reconstructions and generated background activity. While this design enables controlled experiments, it remains unclear whether the resulting investigation environments adequately reflect the complexity and uncertainty of real-world SOC operations.
Real enterprise environments often contain incomplete ground truth, heterogeneous logging systems, ambiguous evidence, and evolving attack behaviors. Without additional validation using realistic operational datasets or expert assessments, the extent to which the reported findings generalize to practical security investigations remains uncertain.
- Limited independence of evaluation cases and potential sensitivity to benchmark construction.
Although the benchmark contains 370 investigation cases, these cases are derived from 56 underlying attack reconstructions under different telemetry conditions. Consequently, the nominal number of evaluation cases does not represent an equivalent number of independent security incidents.
The reported findings may also depend on the selected attack scenarios, predefined evidence requirements, and telemetry transformation procedures. More explicit incident-level statistical analyses and sensitivity studies would help establish whether the conclusions are robust to these benchmark design choices.
Questions for authors
Major Questions
- Validation of the proposed evaluation metrics
The benchmark primarily measures evidence availability, acquisition, and formal citation support. However, retrieving and citing sufficient supporting records does not necessarily imply that an agent correctly understands the evidence or reconstructs the underlying attack sequence.
How do the authors validate that these metrics reliably reflect actual APT investigation competence? In particular, could the authors provide a comparison with expert security analysts' assessments of investigation correctness, including the accuracy of reconstructed attack activities and their relationships? How strongly do the proposed metrics correlate with these expert judgments?
- Practical advantages over existing evaluation methodologies
The paper compares APTInvestBench with existing cybersecurity investigation benchmarks, but the comparison primarily focuses on differences in benchmark design and evaluation features.
Can the authors demonstrate empirically that the proposed evaluation framework provides more reliable, informative, or actionable assessments than existing benchmark metrics? For example, could the authors evaluate the same investigation outputs using conventional aggregate metrics and the proposed action-level evidence metrics, and show whether the latter better identify meaningful investigation failures or distinguish agents with different investigation capabilities?
- Methodological novelty relative to closely related benchmarks
Several existing benchmarks already consider variations in evidence sources, telemetry availability, and investigation diagnostics. The main novelty of APTInvestBench appears to lie in controlled cross-telemetry comparisons and fixed action-level evidence requirements.
Could the authors clarify which components of the proposed framework constitute fundamentally new methodological contributions relative to ExCyTIn-Bench, DiagChain, SynthChain, and RAG-SIA? In particular, what capabilities or insights are uniquely enabled by the proposed action-level evidence matching procedure that cannot be obtained through existing evaluation methodologies?
- Practical implications and improvement of investigation agents
The experiments reveal substantial gaps between available evidence, acquired evidence, and formally cited evidence. However, the paper provides limited evidence regarding how these findings can guide the development of more reliable investigation agents.
Could the authors provide additional diagnostic experiments identifying the primary causes of these failures? For example, are the observed gaps mainly attributable to ineffective query strategies, insufficient evidence interpretation, context-management limitations, or failures in report generation? Would relatively simple interventions, such as improved evidence tracking or citation management, substantially reduce the identified gaps?
- Validity and completeness of the evidence sufficiency requirements
The proposed evaluation relies on predefined action-level evidence requirements and alternative sufficient evidence sets. These requirements directly determine whether an investigation is considered successful at the evidence-support level.
How were these requirements validated, and how can the authors ensure that they adequately represent the evidence considered sufficient by professional security analysts? Could the authors report inter-annotator agreement or independent expert validation? Additionally, how sensitive are the benchmark results and model rankings to alternative reasonable definitions of sufficient evidence?
- Generalizability to realistic SOC environments
The benchmark is constructed using report-informed attack reconstructions and generated background activity, enabling controlled telemetry comparisons. However, real-world security operations frequently involve incomplete ground truth, heterogeneous logging configurations, ambiguous evidence, and substantially more complex background activity.
To what extent do the authors expect the observed cross-telemetry robustness gaps to generalize to real operational environments? Could the authors provide validation using real-world investigation datasets or analyze how the results change under more realistic noise and evidence incompleteness?
3 comments
Official Comment by Authors
W3/Q3. Methodological novelty relative to existing benchmarks
We do not claim that telemetry variation or investigation diagnostics are individually new. As summarized in Table 1 and Section 6, ExCyTIn-Bench evaluates interactive investigation QA under source and time variations; DiagChain diagnoses retrieval, incorporation, and attack-chain reconstruction across source profiles; SynthChain studies source combinations and sampling in supply-chain reconstruction; and RAG-SIA supports forensic QA and reconstruction with retrieval diagnostics. APTInvestBench addresses a different measurement problem: whether sufficient support for the same attack actions survives telemetry changes, is acquired by the agent, and reaches its formal report citations.
Its methodological contribution has two parts. First, each reference action has a support rule that requires complementary records while accepting alternative sufficient witnesses. The same rule is applied to available logs, query returns, and formal citations, distinguishing unavailable support from failed acquisition and citation loss. Second, paired comparisons preserve incident and action identities across telemetry views, with stricter comparisons additionally holding registered candidate records and sufficient witnesses unchanged.
These capabilities enable empirical distinctions beyond aggregate scores. As reported in the main paper, citation coverage decreases by only 1.6 pp from Full to endpoint-only telemetry, yet only 64.5% of previously covered actions remain covered. Table 5 further shows lower citation coverage for all four frameworks on actions with unchanged registered supporting records, while Table 6 locates losses between acquisition and formal citation.
Thus, our contribution is not another telemetry-variation benchmark or attack-chain diagnostic in isolation, but a consistent, action-level measurement protocol that connects telemetry availability, autonomous evidence acquisition, and report preservation under controlled cross-telemetry comparisons.
W4/Q4. Diagnostic experiments and practical improvement of agents
Our original experiments identify two distinct failure points: agents fail to acquire available support and fail to preserve acquired support in formal citations (Table 6). We conducted additional controlled interventions to examine whether these gaps can be reduced through changes to retrieval and reporting strategies.
Query planning. We implemented an enhanced BFS with scoped process IDs, sessions, canonical paths, and query prioritization. On 24 additional Full/endpoint-only cases from 12 APT-source groups, fixed leads and original BFS each acquire sufficient support for 25/206 actions, while enhanced BFS reaches 41/206 under the same 739-page budget. The gain is concentrated in two groups, with a net improvement of 7.77 pp (95% interval [0.00, 20.97]). Replaying saved agent trajectories under common record limits further shows that retrieval advantages depend on the budget:
| Record limit | Original BFS | Enhanced BFS | LLM agent |
|---|---|---|---|
| 100 | 19/128 | 31/128 | 20/128 |
| 250 | 19/128 | 35/128 | 35/128 |
| 500 | 19/128 | 35/128 | 50/128 |
These results show that structured pivots improve deterministic retrieval in some cases, while autonomous investigation provides complementary gains at larger budgets.
Citation and report generation. We tested citation repair on ten archived drafts exceeding the 64-unique-ID limit. Model-based revision prompts preserve sufficient citations for 0/17 action opportunities, whereas deterministic filtering of valid record identifiers preserves 10/17 and makes all ten drafts scorable. In a separate test of 20 over-limit responses, rotating citation selection across report stages preserves support for 27/30 actions, compared with 18/30 under report-order selection. However, neither strategy changes coverage in 18 held-out reports that do not reach the original limit. These gains therefore concern citation constraints.
Together, these interventions demonstrate that APTInvestBench can identify and test improvements at different investigation stages. Structured retrieval can improve evidence acquisition, while citation filtering and selection can reduce specific reporting failures. Their benefits remain conditional on search budgets and reporting constraints.
Official Comment by Authors
W5/Q6. Generalizability to realistic security operations
Validation on independently collected telemetry. Section 3 describes how we derive attack behaviors from 6,217 APT reports while explicitly separating reported behaviors from generated environmental details. To test whether our evidence relationships extend beyond EvidenceForge-generated logs, we audited three independent telemetry sources:
| Dataset | Validation results |
|---|---|
| OTRF | Two host collections (1,041 and 2,657 records), together with packet captures, provide alternative observations for remote task creation and modification. Removing related records breaks parts of the logon and remote RPC evidence chains. |
| Splunk Attack Range | Process records support one registry modification and four file creations. Removing state-change records eliminates sufficient support for all five actions, although command invocations remain visible. |
| OpTC | Across 6,730,332 eCAR records from three hosts and three attack days, seven of nine predefined facts are supportable under full collection. Removing file, endpoint-flow, process/thread, or registry records reduces support to 4/9, 5/9, 5/9, and 5/9, respectively. |
These audits show that complementary records, cross-source relationships, and evidence loss under incomplete telemetry also occur in independently collected security data. They validate the applicability of our support rules beyond the generated corpus.
We acknowledge that these audits do not constitute end-to-end agent evaluations in production SOCs. They do not establish generalization to all enterprise logging configurations, evolving attacks, or real analyst decisions. We therefore limit our claim to the transferability of the evidence-support evaluation and the observed failure mechanisms; their operational prevalence and magnitude require further validation on production SOC investigation traces.
W6. Incident independence and sensitivity to benchmark construction
We agree that the 370 applicable scenario–view cases do not represent 370 independent incidents. The complete resource contains 56 attack reconstructions from 46 APT-source groups, whereas the main evaluation uses ten reconstructions from nine groups, with the two APT27 reconstructions sharing one source group. The seven views are repeated observations of the same incidents. Accordingly, our bootstrap analyses resample source groups, keeping related reconstructions and views together.
Incident-selection sensitivity. We additionally evaluated 12 attacks from 12 source groups with no overlap with the main study. Across 24 planned Full–endpoint-only pairs and 196 common-recoverable action opportunities, sufficient citation delivery decreases from 38/196 to 22/196 (−8.16 pp, 95% CI [−19.59, −0.41]). Leave-one-group-out estimates remain negative, ranging from −10.23 to −5.32 pp. However, only 18 of 48 planned runs passed scoring and runtime audits, and only five pairs had usable audits on both sides; their conditional estimate has a wider interval [−43.75, +8.70]. We therefore distinguish actual task delivery from performance conditional on valid investigations.
A separate analysis of 42 reconstructions from 37 source groups outside the main study finds seven lost and six newly covered actions across 82 Ω6 paired opportunities (28/82 to 27/82). After excluding source groups used in the additional 12-attack evaluation, five pairs from five groups still show six losses and five gains (19/61 to 18/61).
Sensitivity to evidence requirements and incident composition. Adding two source-verified candidate witness sets changes scores in 20 of 97 archived runs, adding 12 sufficient acquisitions and 15 sufficient citations, while preserving the direction of the four paired telemetry comparisons. Excluding eight disputed actions likewise preserves these directions. However, close model rankings remain sensitive to incident composition: MiniMax-M3 exceeds DeepSeek-V4-Pro by only 0.4 pp (95% CI [−14.4, +13.7]), with leave-one-group-out differences ranging from −5.5 to +5.7 pp.
Overall, the action-level support changes are observed beyond the main evaluation cohort and persist under the tested evidence-rule alternatives. However, effect magnitudes and close system rankings remain sensitive to incident composition and evaluation choices. We will clarify the distinction between independent attack reconstructions, repeated observation views, and evaluation opportunities, and report protocol-specific denominators throughout.
Official Comment by Authors
We sincerely thank the reviewer for their time and valuable comments. We address the comments and questions below.
W1/Q1/Q5. Metric validity: Do evidence-support metrics reflect actual investigation correctness?
We agree that retrieving and citing sufficient records does not necessarily establish correct interpretation of attack activities or their relationships. Our metrics measure a necessary dimension of investigation competence: whether agents acquire and formally cite complete support for observable attack actions, rather than the correctness of every report assertion.
Human validation of report correctness. The original paper (§4 and Appendix A.4) defines and verifies action-level support using registered facts, entity links, and temporal constraints. We additionally verified 959 replayable runs against their saved reports and scoring inputs. To assess semantic correctness, two independent security analysts reviewed 18 reports from nine APT-source groups without access to benchmark scores or reference action labels.
At the report level, sufficient citation coverage correlates with activity-fact accuracy (Spearman's ρ = 0.54, 95% CI [0.01, 0.84], 17 reports) and more strongly with relationship accuracy (ρ = 0.76, 95% CI [0.24, 0.87], 14 reports). This supports the relevance of evidence coverage to human-assessed investigation correctness, while confirming that citation sufficiency alone does not guarantee correct interpretation.
Validation of evidence-sufficiency rules. As described in Section 3 and Appendix A.4, reference actions are bound to required facts, entity links, and temporal constraints, with automated checks and annotation-owner review. The two analysts additionally assessed 100 report–action opportunities, agreeing on 82/100 (Cohen's κ = 0.61), with both judging 35 as sufficiently supported.
We compared their adjudicated judgments against three scoring rules on the same outputs:
The complete-witness rule achieves the highest agreement, avoiding the false sufficiency of arbitrary record overlap while allowing complementary records across formal stages. The stricter single-stage rule reduces false sufficiency but increases false insufficiency.
Finally, our existing sensitivity analysis shows that adding two source-verified alternative witness sets changes scores in 20 of 97 runs but preserves the direction of all four paired telemetry comparisons. Excluding eight disputed actions likewise preserves these directions.
Together, the original rule-construction and reproducibility checks, human assessments, and sensitivity analyses support the validity of evidence-support measurement as a distinct dimension of investigation competence.
W2/Q2. Practical advantages over aggregate evaluation methods
We evaluate the same investigation outputs using aggregate citation coverage and action-level metrics. From Full to endpoint-only telemetry, aggregate citation coverage decreases by only 1.6 percentage points. However, only 64.5% of actions covered in Full-telemetry reports remain covered, as newly covered actions offset lost ones. Thus, similar aggregate scores can conceal substantial changes in the attack activities being substantiated.
A stricter comparison further shows that citation coverage decreases for all four frameworks even when the registered candidate records and sufficient witnesses remain unchanged (Table 5). Query audits also locate these losses: among endpoint-only model-panel omissions, 72.7% lack sufficient acquired support, while 27.3% have acquired support that is not preserved in formal citations (Table 6). Moreover, on 77 identical action opportunities acquired by all four frameworks, citation retention ranges from 75.3% to 90.9% (Appendix H.3).
These results demonstrate that action-level evaluation provides information beyond aggregate coverage: it identifies which attack activities are lost and whether the gaps arise during acquisition or reporting, enabling more targeted assessment of investigation capabilities.