acceptodds
Under review as a conference paper at ICLR 2027

APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry

Abstract

Large language model (LLM) agents could help security operations centers (SOCs) investigate advanced persistent threats (APTs) by turning weak leads into evidence for intrusion scoping and response. Yet success under one telemetry setting does not establish robustness to changes in log collection, retention, or sampling. We introduce APTInvestBench, a benchmark for evaluating cross-telemetry robustness in autonomous APT investigation. It comprises 370 cases across seven SOC-inspired conditions, derived from 56 report-informed attack reconstructions with 16.4 million log records. Agents investigate unverified leads and submit reports with record-level citations. Fixed action-level support requirements track sufficient evidence across available logs, query returns, and formal citations, separating telemetry limitations from acquisition and reporting gaps. Across eleven LLMs, agents acquire sufficient evidence for 44.3% of recoverable attack actions on average, while formal citations support only 25.0%. More importantly, aggregate coverage can conceal substantial instability: from Full to endpoint-only telemetry, coverage declines by only 1.6 percentage points, yet 35.5% of previously covered actions lose sufficient citation support despite remaining recoverable. Across four frameworks, such losses persist even when registered supporting records remain unchanged. APTInvestBench provides reusable investigation environments and diagnostic evaluation for identifying these gaps and developing more reliable defensive agents.

Primary area:
datasets and benchmarks
Keywords:
APT Investigation, LLM Agents, Evidence-Grounded Evaluation, Cross-Telemetry Robustness, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WoEwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces APTInvestBench, a benchmark designed to evaluate the robustness of LLM-based agents performing autonomous advanced persistent threat (APT) investigations under varying telemetry conditions. Unlike existing cybersecurity benchmarks that primarily focus on vulnerability exploitation or attack-chain reconstruction, APTInvestBench examines whether agents can consistently acquire and cite sufficient evidence when log availability, retention, and sampling conditions change.

The benchmark contains 370 investigation cases across seven SOC-inspired telemetry conditions, constructed from 56 report-informed attack reconstructions and approximately 16.4 million log records. It evaluates eleven LLMs and four agent frameworks using an investigation environment in which agents query telemetry and produce evidence-grounded reports. A central methodological feature is the separation of evidence recoverability, evidence acquisition, and formal citation support, allowing the authors to distinguish environmental limitations from agent investigation and reporting failures.

The experiments reveal substantial weaknesses in current autonomous investigation agents. On average, agents acquire sufficient evidence for 44.3% of recoverable attack actions, while only 25.0% receive sufficient formal citation support. Furthermore, relatively small changes in aggregate coverage can conceal substantial instability at the individual-action level. The paper argues that cross-telemetry robustness and evidence retention should be evaluated explicitly when assessing the reliability of autonomous cybersecurity investigation agents.

Critical strengths
  1. Important and practically relevant research problem.

The paper addresses an important challenge in autonomous cybersecurity investigation: whether LLM-based agents can reliably investigate the same security incidents under varying telemetry conditions. Since real-world security operations centers differ in log availability, retention, and sampling practices, evaluating cross-telemetry robustness is practically meaningful and relevant to the deployment of reliable security agents.

  1. Well-structured evaluation framework with fine-grained failure analysis.

A key strength is the distinction between evidence availability, evidence acquisition, and formal citation support. By applying consistent action-level evidence requirements across different telemetry conditions, the proposed benchmark enables a more detailed analysis of investigation failures than aggregate coverage metrics alone. In particular, it can distinguish cases where sufficient evidence is unavailable from cases where an agent fails to acquire or preserve available evidence.

  1. Substantial benchmark construction and experimental coverage.

The benchmark includes 370 investigation cases derived from 56 report-informed attack reconstructions, seven telemetry conditions, and approximately 16.4 million log records. The evaluation of eleven LLMs and four agent frameworks provides a relatively broad empirical assessment of autonomous investigation behavior.

  1. Interesting observations regarding hidden instability in investigation performance.

The experiments reveal that relatively small changes in aggregate citation coverage can conceal substantial changes in which individual attack actions are adequately supported. This finding highlights the limitations of relying exclusively on aggregate performance metrics and motivates more fine-grained evaluation of investigation consistency.

Critical weaknesses
  1. Insufficient validation of whether the proposed metrics reflect actual investigation competence.

My primary concern is the validity of the proposed evaluation metrics as measures of autonomous APT investigation capability. The benchmark primarily evaluates whether agents retrieve and formally cite sufficient supporting records. However, retrieving relevant record identifiers or citing sufficient evidence does not necessarily demonstrate that an agent correctly understands the evidence, reconstructs the attack sequence, or accurately identifies the underlying malicious activities.

Although the authors distinguish evidence acquisition from report interpretation, it remains unclear how strongly the primary evaluation metrics correlate with actual investigation correctness. For example, an agent could retrieve and cite relevant records without correctly interpreting their causal or temporal relationships. Independent validation against expert security analysts' assessments would substantially strengthen the credibility of the proposed benchmark.

  1. Limited demonstration of practical improvements over existing evaluation methodologies.

While the paper introduces a structured evaluation framework, it does not sufficiently demonstrate that the proposed metrics provide more accurate, reliable, or practically actionable assessments than existing cybersecurity investigation benchmarks. The comparison with prior benchmarks primarily emphasizes differences in benchmark design and supported evaluation features rather than directly establishing the empirical advantages of the proposed methodology.

A more convincing evaluation would compare the proposed metrics against existing assessment approaches on the same investigation outputs and demonstrate whether the new metrics better identify meaningful investigation failures or predict expert-assessed investigation quality. Without such evidence, the practical benefit of the proposed benchmark remains insufficiently established.

  1. Limited methodological novelty beyond benchmark construction and metric decomposition.

The main contribution consists of constructing a benchmark with controlled telemetry variations and decomposing investigation outcomes into evidence availability, acquisition, and citation support. Although this is a useful evaluation perspective, the paper does not introduce a fundamentally new investigation method or learning algorithm. Furthermore, related benchmarks have already considered variations in evidence sources, telemetry availability, and investigation diagnostics.

The authors should more clearly establish what constitutes a substantial methodological advance beyond combining these existing evaluation concepts. In particular, the novelty of the action-level evidence matching framework should be justified through stronger comparisons with closely related benchmark methodologies.

  1. Limited insight into how the identified failures can be addressed.

The experiments reveal that LLM agents frequently fail to acquire or preserve sufficient evidence, even when relevant supporting records remain available. However, the paper provides limited analysis of which specific agent capabilities or design choices are responsible for these failures and how they might be improved.

Although proposing a new investigation algorithm is not a requirement for a benchmark paper, stronger diagnostic experiments would increase the significance of the contribution. For example, evaluating whether improved retrieval strategies, evidence management mechanisms, or report-generation procedures systematically reduce the identified failures would help establish the practical usefulness of the benchmark beyond documenting existing limitations.

  1. Uncertain generalizability to real-world security operations.

The benchmark relies on report-informed attack reconstructions and generated background activity. While this design enables controlled experiments, it remains unclear whether the resulting investigation environments adequately reflect the complexity and uncertainty of real-world SOC operations.

Real enterprise environments often contain incomplete ground truth, heterogeneous logging systems, ambiguous evidence, and evolving attack behaviors. Without additional validation using realistic operational datasets or expert assessments, the extent to which the reported findings generalize to practical security investigations remains uncertain.

  1. Limited independence of evaluation cases and potential sensitivity to benchmark construction.

Although the benchmark contains 370 investigation cases, these cases are derived from 56 underlying attack reconstructions under different telemetry conditions. Consequently, the nominal number of evaluation cases does not represent an equivalent number of independent security incidents.

The reported findings may also depend on the selected attack scenarios, predefined evidence requirements, and telemetry transformation procedures. More explicit incident-level statistical analyses and sensitivity studies would help establish whether the conclusions are robust to these benchmark design choices.

Questions for authors

Major Questions

  1. Validation of the proposed evaluation metrics

The benchmark primarily measures evidence availability, acquisition, and formal citation support. However, retrieving and citing sufficient supporting records does not necessarily imply that an agent correctly understands the evidence or reconstructs the underlying attack sequence.

How do the authors validate that these metrics reliably reflect actual APT investigation competence? In particular, could the authors provide a comparison with expert security analysts' assessments of investigation correctness, including the accuracy of reconstructed attack activities and their relationships? How strongly do the proposed metrics correlate with these expert judgments?

  1. Practical advantages over existing evaluation methodologies

The paper compares APTInvestBench with existing cybersecurity investigation benchmarks, but the comparison primarily focuses on differences in benchmark design and evaluation features.

Can the authors demonstrate empirically that the proposed evaluation framework provides more reliable, informative, or actionable assessments than existing benchmark metrics? For example, could the authors evaluate the same investigation outputs using conventional aggregate metrics and the proposed action-level evidence metrics, and show whether the latter better identify meaningful investigation failures or distinguish agents with different investigation capabilities?

  1. Methodological novelty relative to closely related benchmarks

Several existing benchmarks already consider variations in evidence sources, telemetry availability, and investigation diagnostics. The main novelty of APTInvestBench appears to lie in controlled cross-telemetry comparisons and fixed action-level evidence requirements.

Could the authors clarify which components of the proposed framework constitute fundamentally new methodological contributions relative to ExCyTIn-Bench, DiagChain, SynthChain, and RAG-SIA? In particular, what capabilities or insights are uniquely enabled by the proposed action-level evidence matching procedure that cannot be obtained through existing evaluation methodologies?

  1. Practical implications and improvement of investigation agents

The experiments reveal substantial gaps between available evidence, acquired evidence, and formally cited evidence. However, the paper provides limited evidence regarding how these findings can guide the development of more reliable investigation agents.

Could the authors provide additional diagnostic experiments identifying the primary causes of these failures? For example, are the observed gaps mainly attributable to ineffective query strategies, insufficient evidence interpretation, context-management limitations, or failures in report generation? Would relatively simple interventions, such as improved evidence tracking or citation management, substantially reduce the identified gaps?

  1. Validity and completeness of the evidence sufficiency requirements

The proposed evaluation relies on predefined action-level evidence requirements and alternative sufficient evidence sets. These requirements directly determine whether an investigation is considered successful at the evidence-support level.

How were these requirements validated, and how can the authors ensure that they adequately represent the evidence considered sufficient by professional security analysts? Could the authors report inter-annotator agreement or independent expert validation? Additionally, how sensitive are the benchmark results and model rankings to alternative reasonable definitions of sufficient evidence?

  1. Generalizability to realistic SOC environments

The benchmark is constructed using report-informed attack reconstructions and generated background activity, enabling controlled telemetry comparisons. However, real-world security operations frequently involve incomplete ground truth, heterogeneous logging configurations, ambiguous evidence, and substantially more complex background activity.

To what extent do the authors expect the observed cross-telemetry robustness gaps to generalize to real operational environments? Could the authors provide validation using real-world investigation datasets or analyze how the results change under more realistic noise and evidence incompleteness?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for their time and valuable comments. We address the comments and questions below.

W1/Q1/Q5. Metric validity: Do evidence-support metrics reflect actual investigation correctness?

We agree that retrieving and citing sufficient records does not necessarily establish correct interpretation of attack activities or their relationships. Our metrics measure a necessary dimension of investigation competence: whether agents acquire and formally cite complete support for observable attack actions, rather than the correctness of every report assertion.

Human validation of report correctness. The original paper (§4 and Appendix A.4) defines and verifies action-level support using registered facts, entity links, and temporal constraints. We additionally verified 959 replayable runs against their saved reports and scoring inputs. To assess semantic correctness, two independent security analysts reviewed 18 reports from nine APT-source groups without access to benchmark scores or reference action labels.

Report-level judgmentAgreementCohen's κBoth judged correct
Activity facts69/760.5464/76
Explicit relationships65/690.7458/69
Event judgments17/180.7615/18

At the report level, sufficient citation coverage correlates with activity-fact accuracy (Spearman's ρ = 0.54, 95% CI [0.01, 0.84], 17 reports) and more strongly with relationship accuracy (ρ = 0.76, 95% CI [0.24, 0.87], 14 reports). This supports the relevance of evidence coverage to human-assessed investigation correctness, while confirming that citation sufficiency alone does not guarantee correct interpretation.

Validation of evidence-sufficiency rules. As described in Section 3 and Appendix A.4, reference actions are bound to required facts, entity links, and temporal constraints, with automated checks and annotation-owner review. The two analysts additionally assessed 100 report–action opportunities, agreeing on 82/100 (Cohen's κ = 0.61), with both judging 35 as sufficiently supported.

We compared their adjudicated judgments against three scoring rules on the same outputs:

Scoring ruleAgreementHuman insufficient, rule sufficientHuman sufficient, rule insufficient
Any candidate-record overlap74/100197
Complete registered witness86/10077
Complete witness within one stage82/100414

The complete-witness rule achieves the highest agreement, avoiding the false sufficiency of arbitrary record overlap while allowing complementary records across formal stages. The stricter single-stage rule reduces false sufficiency but increases false insufficiency.

Finally, our existing sensitivity analysis shows that adding two source-verified alternative witness sets changes scores in 20 of 97 runs but preserves the direction of all four paired telemetry comparisons. Excluding eight disputed actions likewise preserves these directions.

Together, the original rule-construction and reproducibility checks, human assessments, and sensitivity analyses support the validity of evidence-support measurement as a distinct dimension of investigation competence.

W2/Q2. Practical advantages over aggregate evaluation methods

We evaluate the same investigation outputs using aggregate citation coverage and action-level metrics. From Full to endpoint-only telemetry, aggregate citation coverage decreases by only 1.6 percentage points. However, only 64.5% of actions covered in Full-telemetry reports remain covered, as newly covered actions offset lost ones. Thus, similar aggregate scores can conceal substantial changes in the attack activities being substantiated.

A stricter comparison further shows that citation coverage decreases for all four frameworks even when the registered candidate records and sufficient witnesses remain unchanged (Table 5). Query audits also locate these losses: among endpoint-only model-panel omissions, 72.7% lack sufficient acquired support, while 27.3% have acquired support that is not preserved in formal citations (Table 6). Moreover, on 77 identical action opportunities acquired by all four frameworks, citation retention ranges from 75.3% to 90.9% (Appendix H.3).

These results demonstrate that action-level evaluation provides information beyond aggregate coverage: it identifies which attack activities are lost and whether the gaps arise during acquisition or reporting, enabling more targeted assessment of investigation capabilities.

AuthorsOpenReview ↗

Official Comment by Authors

W3/Q3. Methodological novelty relative to existing benchmarks

We do not claim that telemetry variation or investigation diagnostics are individually new. As summarized in Table 1 and Section 6, ExCyTIn-Bench evaluates interactive investigation QA under source and time variations; DiagChain diagnoses retrieval, incorporation, and attack-chain reconstruction across source profiles; SynthChain studies source combinations and sampling in supply-chain reconstruction; and RAG-SIA supports forensic QA and reconstruction with retrieval diagnostics. APTInvestBench addresses a different measurement problem: whether sufficient support for the same attack actions survives telemetry changes, is acquired by the agent, and reaches its formal report citations.

Its methodological contribution has two parts. First, each reference action has a support rule that requires complementary records while accepting alternative sufficient witnesses. The same rule is applied to available logs, query returns, and formal citations, distinguishing unavailable support from failed acquisition and citation loss. Second, paired comparisons preserve incident and action identities across telemetry views, with stricter comparisons additionally holding registered candidate records and sufficient witnesses unchanged.

These capabilities enable empirical distinctions beyond aggregate scores. As reported in the main paper, citation coverage decreases by only 1.6 pp from Full to endpoint-only telemetry, yet only 64.5% of previously covered actions remain covered. Table 5 further shows lower citation coverage for all four frameworks on actions with unchanged registered supporting records, while Table 6 locates losses between acquisition and formal citation.

Thus, our contribution is not another telemetry-variation benchmark or attack-chain diagnostic in isolation, but a consistent, action-level measurement protocol that connects telemetry availability, autonomous evidence acquisition, and report preservation under controlled cross-telemetry comparisons.

W4/Q4. Diagnostic experiments and practical improvement of agents

Our original experiments identify two distinct failure points: agents fail to acquire available support and fail to preserve acquired support in formal citations (Table 6). We conducted additional controlled interventions to examine whether these gaps can be reduced through changes to retrieval and reporting strategies.

Query planning. We implemented an enhanced BFS with scoped process IDs, sessions, canonical paths, and query prioritization. On 24 additional Full/endpoint-only cases from 12 APT-source groups, fixed leads and original BFS each acquire sufficient support for 25/206 actions, while enhanced BFS reaches 41/206 under the same 739-page budget. The gain is concentrated in two groups, with a net improvement of 7.77 pp (95% interval [0.00, 20.97]). Replaying saved agent trajectories under common record limits further shows that retrieval advantages depend on the budget:

Record limitOriginal BFSEnhanced BFSLLM agent
10019/12831/12820/128
25019/12835/12835/128
50019/12835/12850/128

These results show that structured pivots improve deterministic retrieval in some cases, while autonomous investigation provides complementary gains at larger budgets.

Citation and report generation. We tested citation repair on ten archived drafts exceeding the 64-unique-ID limit. Model-based revision prompts preserve sufficient citations for 0/17 action opportunities, whereas deterministic filtering of valid record identifiers preserves 10/17 and makes all ten drafts scorable. In a separate test of 20 over-limit responses, rotating citation selection across report stages preserves support for 27/30 actions, compared with 18/30 under report-order selection. However, neither strategy changes coverage in 18 held-out reports that do not reach the original limit. These gains therefore concern citation constraints.

Together, these interventions demonstrate that APTInvestBench can identify and test improvements at different investigation stages. Structured retrieval can improve evidence acquisition, while citation filtering and selection can reduce specific reporting failures. Their benefits remain conditional on search budgets and reporting constraints.

AuthorsOpenReview ↗

Official Comment by Authors

W5/Q6. Generalizability to realistic security operations

Validation on independently collected telemetry. Section 3 describes how we derive attack behaviors from 6,217 APT reports while explicitly separating reported behaviors from generated environmental details. To test whether our evidence relationships extend beyond EvidenceForge-generated logs, we audited three independent telemetry sources:

DatasetValidation results
OTRFTwo host collections (1,041 and 2,657 records), together with packet captures, provide alternative observations for remote task creation and modification. Removing related records breaks parts of the logon and remote RPC evidence chains.
Splunk Attack RangeProcess records support one registry modification and four file creations. Removing state-change records eliminates sufficient support for all five actions, although command invocations remain visible.
OpTCAcross 6,730,332 eCAR records from three hosts and three attack days, seven of nine predefined facts are supportable under full collection. Removing file, endpoint-flow, process/thread, or registry records reduces support to 4/9, 5/9, 5/9, and 5/9, respectively.

These audits show that complementary records, cross-source relationships, and evidence loss under incomplete telemetry also occur in independently collected security data. They validate the applicability of our support rules beyond the generated corpus.

We acknowledge that these audits do not constitute end-to-end agent evaluations in production SOCs. They do not establish generalization to all enterprise logging configurations, evolving attacks, or real analyst decisions. We therefore limit our claim to the transferability of the evidence-support evaluation and the observed failure mechanisms; their operational prevalence and magnitude require further validation on production SOC investigation traces.

W6. Incident independence and sensitivity to benchmark construction

We agree that the 370 applicable scenario–view cases do not represent 370 independent incidents. The complete resource contains 56 attack reconstructions from 46 APT-source groups, whereas the main evaluation uses ten reconstructions from nine groups, with the two APT27 reconstructions sharing one source group. The seven views are repeated observations of the same incidents. Accordingly, our bootstrap analyses resample source groups, keeping related reconstructions and views together.

Incident-selection sensitivity. We additionally evaluated 12 attacks from 12 source groups with no overlap with the main study. Across 24 planned Full–endpoint-only pairs and 196 common-recoverable action opportunities, sufficient citation delivery decreases from 38/196 to 22/196 (−8.16 pp, 95% CI [−19.59, −0.41]). Leave-one-group-out estimates remain negative, ranging from −10.23 to −5.32 pp. However, only 18 of 48 planned runs passed scoring and runtime audits, and only five pairs had usable audits on both sides; their conditional estimate has a wider interval [−43.75, +8.70]. We therefore distinguish actual task delivery from performance conditional on valid investigations.

A separate analysis of 42 reconstructions from 37 source groups outside the main study finds seven lost and six newly covered actions across 82 Ω6 paired opportunities (28/82 to 27/82). After excluding source groups used in the additional 12-attack evaluation, five pairs from five groups still show six losses and five gains (19/61 to 18/61).

Sensitivity to evidence requirements and incident composition. Adding two source-verified candidate witness sets changes scores in 20 of 97 archived runs, adding 12 sufficient acquisitions and 15 sufficient citations, while preserving the direction of the four paired telemetry comparisons. Excluding eight disputed actions likewise preserves these directions. However, close model rankings remain sensitive to incident composition: MiniMax-M3 exceeds DeepSeek-V4-Pro by only 0.4 pp (95% CI [−14.4, +13.7]), with leave-one-group-out differences ranging from −5.5 to +5.7 pp.

Overall, the action-level support changes are observed beyond the main evaluation cohort and persist under the tested evidence-rule alternatives. However, effect magnitudes and close system rankings remain sensitive to incident composition and evaluation choices. We will clarify the distinction between independent attack reconstructions, repeated observation views, and evaluation opportunities, and report protocol-specific denominators throughout.

Reviewer_SoaROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When an LLM-based agent investigates an APT (Advanced Persistent Threat), can it still locate sufficient evidence, and include that evidence in its final report, if the log collection conditions change? The paper introduces APTInvestBench, a benchmark for evaluating LLM agents on autonomous APT investigation. It focuses on cross-telemetry robustness: whether agents still acquire and report sufficient evidence for the same attack actions when log collection, retention, or sampling changes.

Critical strengths
  1. The problem formulation is new with originality.
  2. The decomposition is clear and intuitive.
Critical weaknesses
  1. On the experiments side, small effective sample sizes since only 10 of 56 scenarios are evaluated, from 9 source-incident groups. Synthetic logs and missing realism validation. It's possible to receive contamination from public reports. And the baselines are too limited. A simple deterministic pivoting baseline (e.g., BFS over shared identifiers from the leads), or a provenance-graph method, would show whether LLM agents add value beyond mechanical correlation.
  2. There're problems on the structure. The paper is dense. It uses many distinct analysis populations (47 vs. 56 shared cases, per-view sets, paired sets, unchanged-witness sets, historical audits with older rules), which makes it hard to know which number refers to which population. And too many typos in Figure 2: "supperssion", "Acquistion", "revoverable", "accuqired".
Questions for authors
  1. Do you have, or can you run, two independent runs of the same view (Full–Full, and ideally Ω6–Ω6) for a few systems? What are the loss and gain rates there?
  2. Why does matched benign removal, which preserves target support, produce a −8.2 pp framework decline with an interval excluding zero?
  3. How was the "one selected recorded run" chosen when multiple accepted runs existed? Could the runtime-audit selection flag, or the exclusion of failed runs (e.g., 8 failed framework runs under network-only), bias the paired comparisons?
  4. Can models identify the actor or predict subsequent actions from the leads alone, without querying? Does renaming artifacts change results?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for their time and valuable comments. We address the comments and questions below.

W1. Effective sample size, synthetic-log realism, potential contamination, and baseline comparisons

Evaluation scope. Section 3 reports 56 attack scenarios, 523 annotated action instances, 16 log formats, and six environment types. It also states that the main comparisons use ten selected scenarios. Section 5 explains the rationale:

"To manage the cost of multi-round investigations across systems and telemetry conditions, we concentrate the main evaluation on ten of the 56 scenarios. This allocation prioritizes paired comparisons and query-level analysis of where agents fail to acquire or preserve attack evidence."

These ten scenarios represent ten distinct APT attacks from nine APT groups. The two APT27 scenarios are separate attacks attributed to the same organization. Similarly, the full resource contains 56 distinct attacks from 46 APT groups. The 47 model-panel cases and 56 framework-panel cases are shared scenario–view combinations.

Synthetic-log realism and Potential contamination from public reports. Our scenarios separate report-grounded attack behaviors from generated environmental details and telemetry. As described in Section 3, we screened 6,217 APT reports and constructed 56 multi-step scenarios from selected reports. Each scenario distinguishes reported attack behavior, configured environment details, and generated execution context. Reference actions are bound to required facts, entity links, and temporal constraints, with automatic checks and review by annotation owners (Appendix A.4). For telemetry generation, we build on EvidenceForge's open-source event model and renderers (Cisco Talos, 2026), extending them with scenario-specific templates, command matching, parameter pools, and benign-activity patterns. The pipeline normalizes events, resolves entity references, adds prerequisite events, and renders correlated endpoint, network, and application records. These mechanisms establish a report-grounded and structurally controlled generation process, but do not by themselves establish statistical equivalence to production SOC telemetry.

Deterministic baseline. Appendix D.1 already includes a fixed-lead reference. Following the reviewer's suggestion, we additionally implemented original and enhanced BFS baselines using the same initial leads, read-only query interface, and frozen support evaluator. The enhanced BFS adds typed entity pivots through process IDs, sessions, and canonical paths, together with query prioritization and pagination.

On 12 additional attack scenarios, the results show the incremental value of these search strategies. Enhanced BFS gains 16 actions over original BFS under the same 739-page budget.

MethodSufficiently supported actions
Fixed initial leads25/206
Original BFS25/206
Enhanced BFS41/206

To assess whether LLM agents provide additional value beyond deterministic correlation, we also replayed Codex–MiniMax-M3 investigation trajectories under common query-prefix limits. Across 128 matched action opportunities:

MethodSufficiently supported actions (32-query limit)
Original BFS19/128
Enhanced BFS35/128
Codex–MiniMax-M350/128

At the 32-query limit, the agent recovers 15 more actions than enhanced BFS (+11.72 pp, 95% interval [0.00, 22.39]). LLM-guided investigation provides additional and complementary discoveries under constrained search budgets.

W2. Dense presentation and multiple analysis populations

We agree that the different evaluation populations should be easier to distinguish. Appendix A.1 already defines a scenario as one attack reconstruction and a case as a scenario paired with an applicable observation condition. We will consolidate the populations as follows:

PopulationScenariosAPT groupsCases
Complete benchmark5646370
Controlled study10970
Model panel10947
Framework panel10956

The model and framework panels use different common-case intersections because their qualifying runs differ. Per-view, paired, and unchanged-witness populations apply progressively stricter action-eligibility conditions. We will state each population's denominator alongside its results and shorten the main-text descriptions.

We will also correct the four Figure 2 typos: "suppression," "acquisition," "recoverable," and "acquired."

AuthorsOpenReview ↗

Official Comment by Authors

Q1. Independent runs under the same observation view

We conducted two independent runs per configuration using Codex and Ref-ReAct with MiniMax-M3 across Full, endpoint-only, and Ω6 conditions. Among the 60 planned same-view pairs, 21 had usable audits on both runs: six Full–Full, eight endpoint-only–endpoint-only, and seven Ω6–Ω6 pairs.

We compare the first and second runs on their common recoverable action opportunities. A loss occurs when an action has sufficient support in the first run but not the second, while a gain occurs in the reverse direction. The results are:

ViewFrameworkOpportunitiesAcquisition loss/gainCitation loss/gain
Full–FullCodex219.5%/0%9.5%/9.5%
Ref-ReAct224.5%/9.1%9.1%/0%
Endpoint-onlyCodex200%/0%0%/10.0%
Ref-ReAct248.3%/12.5%12.5%/4.2%
Ω6–Ω6Codex2213.6%/4.5%9.1%/0%
Ref-ReAct160%/18.8%6.2%/0%

Pooling the two frameworks within each view gives:

Same-view comparisonOpportunitiesAcquisition loss/gainCitation loss/gain
Full–Full437.0% / 4.7%9.3% / 4.7%
Endpoint-only444.5% / 6.8%6.8% / 6.8%
Ω6–Ω6387.9% / 10.5%7.9% / 0%

These results show that investigation outcomes can change even when telemetry remains unchanged. In particular, repeated runs can lose previously supported actions and, in some cases, gain support for previously uncovered actions. This variability occurs under all three views, including Full and Ω6, and should therefore be considered when interpreting cross-view differences.

Q2. Why does matched benign removal, which preserves target support, produce a −8.2 pp framework decline with an interval excluding zero?

The key distinction is that preserving target evidence does not preserve the investigation context. Under Ω6, we remove 76 non-candidate background records while retaining all 356 registered candidate records with identical IDs, contents, and metadata. The registered sufficient-witness sets therefore remain unchanged. Nevertheless, across 32 paired runs and 232 common-recoverable action opportunities, citation coverage decreases from 88/232 (37.9%) to 69/232 (29.7%), a net decline of 8.2 percentage points. This combines 36 lost actions and 17 newly covered actions.

We audited where the 36 losses occur:

Failure locationCountShare
Complete support not acquired through queries2672.2%
Acquired support not preserved in formal citations1027.8%
Total36100%

Among the 26 acquisition losses, 23 have no returned witness member and three have only partial witnesses. Among the ten post-acquisition losses, seven cite no valid witness member and three cite only part of a sufficient combination.

The query traces indicate that background removal is associated with changes in early investigation paths. All 32 pairs diverge in query commands or parameters within their first five record queries. Three Full runs also retrieve records subsequently removed in Ω6. In contrast, fixed-lead queries and the original BFS return identical record IDs across all ten Full/Ω6 scenario pairs, supporting the same 10/73 action opportunities. To examine whether the query backend explains the divergence, we replayed 64 commands at their first divergence, yielding 128 view-specific queries. Sixty-three commands reproduce the archived record-ID order and return length, with byte-identical native responses across views. The remaining show command is rejected in both views because its record ID is unavailable. These results suggest that the losses are associated with divergent adaptive investigation paths, rather than the removal of registered target witnesses or inconsistent responses to matched queries.

The negative estimate also persists under alternative aggregation and run-selection rules: −6.9 pp with equal source-group weighting, −8.6 pp with equal scenario weighting, and −5.4 pp using either system-specific qualifying pairs or additional accepted runs (Appendix E.4).

The reported 95% interval [−12.5, −2.5] is obtained from 20,000 source-group bootstrap draws. It measures sensitivity to source-group composition, not run-to-run variability. Our repeated-run analysis in Q1 confirms that same-view variability also exists. We therefore interpret −8.2 pp as a negative paired difference in the evaluated runs, rather than a fully isolated causal effect of benign-record removal.

The main finding is that preserving the availability of sufficient target evidence does not guarantee stable investigation outcomes: changes in background context can coincide with different query paths and different actions being substantiated.

AuthorsOpenReview ↗

Official Comment by Authors

Q3. How was the "one selected recorded run" chosen when multiple accepted runs existed? Could the runtime-audit selection flag or exclusion of failed runs bias the paired comparisons?

Appendix C.1 specifies that one attempt per configuration and case was selected from a frozen ledger, independently of capability scores, rather than choosing the highest-scoring retry. All 70 selected runs per server configuration are marked as first attempts. The primary analysis requires an accepted report, a usable evidence audit, and a passing runtime-audit flag. Accepted empty reports receive zero coverage, while failed or unavailable runs are excluded because their acquisition status cannot be established.

To assess potential selection bias, we compared three inclusion policies:

ComparisonShared qualifying runsSystem-specific runsAll planned tasks
Full → Endpoint-only−25.00 pp−22.67 pp−22.27 pp
Full → Network-only+17.65 pp+7.22 pp0.00 pp
Full → Ω6−8.19 pp−5.42 pp−7.19 pp

The eight Network-only framework failures are execution failures, not runtime-audit exclusions. Full and Network-only have one and seven missing valid reports, respectively. When all 40 planned pairs are included, both views achieve 39/112 sufficient citations, with 15 losses and 15 gains.

These results show that the apparent Network-only improvement depends on run eligibility and disappears under planned-task accounting. In contrast, the Endpoint-only and Ω6 declines remain negative across all three comparisons. We will report this distinction between performance on valid runs and actual task delivery.

Q4. Can models identify the actor or predict subsequent actions from leads alone? Does renaming artifacts change results?

First, we tested whether MiniMax-M3 could identify the source or predict subsequent attack categories using only initial lead metadata, without querying logs. Across ten scenarios and nine candidate source labels, source identification was 0/10 with forced selection and the model abstained in all ten cases when allowed. Using dates alone yielded 1/10. For subsequent-category prediction, the model achieved 1/4, below the majority-class baseline of 3/4. Second, we renamed 21 preselected artifact identifiers across eight Full-view scenarios while preserving record identities, reference actions, and support rules. Formal citation delivery was 34/126 with original identifiers and 35/126 after renaming, a difference of +0.79 pp (95% interval [−15.00, +18.63]). These results provide no indication that the tested lead metadata or exact artifact identifiers alone explain the investigation outcomes.

Reviewer_QGroOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the robustness of LLM-based agents in autonomous advanced persistent threat (APT) investigation, particularly under varying telemetry conditions in security operations centers (SOCs). The authors introduce APTInvestBench, a benchmark comprising 370 investigation cases across seven telemetry conditions, with an evidence-based evaluation framework that distinguishes telemetry limitations from evidence acquisition and reporting failures. Experiments across eleven LLMs and four agent frameworks reveal substantial weaknesses in evidence acquisition and citation reliability, highlighting that seemingly stable aggregate performance can conceal significant instability across telemetry settings.

Critical strengths
  1. The target problem is of great value regarding both AI and cyber security, both research and practical applications.
  2. The presentation is of high quality, with good structure and easy to follow.
  3. The point of extending the benchmark to the varying telemetry conditions is interesting and novel.
  4. The work is solid, with extensive experiments conducted and useful findings obtained.
Critical weaknesses

The technical depth of this work could be strengthened. Even though this is typical for benchmark works, it is still important to reveal profound technical insights and contributions. It is not sufficient to identify a novel angle, design a benchmark, conduct the evaluation, report the results and some analysis. What are the scientific problems behind the weankensses identified by the empricial quantitative studies, thereafter, what capabilities should the LLMs and agents be further strengthened, particularly from this application.

Questions for authors

see the weakness above.

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for their time and valuable comments. We address the comments and questions below.

W1/Q1. What scientific problems underlie the observed weaknesses, and which capabilities should be strengthened?

Thank you for this important comment. We would like to clarify that APTInvestBench's central methodological contribution is a unified, action-level sufficient-support evaluation. As formalized in Sections 4.1–4.3, the same support predicate handles complementary records, alternative witnesses, and temporal requirements, and is applied consistently to available records, query returns, and formal citations. We further introduce three levels of cross-telemetry comparison (per-view, common-recoverable, and unchanged-witness) and action-state transitions to distinguish changes in evidence availability from failures during investigation and reporting.

These existing designs and experiments reveal three important problems in autonomous APT investigation.

1. Search adaptation under changing telemetry.

Section 5.3 (RQ3), Figure 6, and Appendix E.5 examine whether agents can find sufficient support when the supporting records remain unchanged. The strict Full-to-endpoint-only comparison verifies identical target record IDs, contents, schemas, and registered minimal-witness sets. Nevertheless, citation coverage decreases for all four frameworks on the 37 shared qualifying actions. The APT27 Ref-ReAct trace provides a concrete explanation: under Full telemetry, a host query retrieves the complete witness for the creation of error2.aspx. Under endpoint-only telemetry, the agent queries an unavailable web-access source and misses the same witness in the endpoint records.

This reveals a problem beyond evidence availability: agents do not always adapt their search strategies to the sources that remain available. The relevant capability is source-aware query planning, including cross-source pivots and redirection when a query path is unavailable.

2. Preserving acquired support in final reports.

Section 5.4 (RQ4) and Appendix F distinguish insufficient acquisition from citation-retention failures. The existing transition analysis in Appendix E.3 (Table 22) further localizes the losses in the four-framework matched-removal comparison:

Full-to-changed transitionCountInterpretation
C → U26Complete support not acquired through queries
C → A10Acquired support not preserved in formal citations
Total36Lost citation support

Here, C denotes sufficient formal citation support, U denotes insufficient query acquisition, and A denotes sufficient acquisition without sufficient formal citations. The analysis covers 232 framework system–action pairs.

Appendix F.3 (Table 29) provides a complementary comparison on 77 action–view opportunities acquired by all four frameworks. Ref-ReAct preserves sufficient support for 70 (90.9%), whereas each of the other three frameworks preserves it for 58 (75.3%). Although the returned records and investigation contexts are not necessarily identical, this comparison controls the acquired-action opportunities. The Spyder audit in Appendix G.4 provides further confirmation: complete record content was returned for five actions that still lacked sufficient formal citations.

Together, these results show that acquiring sufficient evidence and preserving it through report generation are distinct capabilities. The relevant direction is to maintain action-to-record associations and sufficient supporting combinations throughout investigation, so that following new leads does not cause previously acquired support to disappear from the final report.

3. Selecting record combinations that satisfy evidence requirements.

Our support-rule analysis also reveals why record relevance or quantity alone is insufficient. Appendix F.5 (Table 33) distinguishes missing candidates, insufficient record counts, and compatibility failures for recurring-network actions. Among model-panel opportunities with acquired support but insufficient formal citations, 34.0% cite no candidate, 50.0% fail the count requirement, and 16.0% satisfy the count but fail compatibility.

This finding follows directly from the sufficient-witness formulation in Section 2.2: evidence support is a property of a compatible record combination. Agents therefore need stronger cross-source entity binding, temporal reasoning, and verification of complete supporting combinations before submitting claims.

We also performed supplementary offline analyses on saved investigation outputs. An equal-limit retrieval control returned 16 additional result pages without completing another sufficient witness. A citation-selection stress test under the same 64-ID cap preserved support for 27 of 30 actions when selection was rotated across report stages, compared with 18 under report-order selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.