- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SecRespond, a benchmark for evaluating LLM agents on post-compromise incident response, specifically using a cyber range system setup. The authors build a set of cyber ranges by executing attacks against Linux (CentOS/Ubuntu) and Windows hosts, and then provide agents with forensic snapshots and security alerts. The agents need to investigate the compromised systems, identify malicious activity, and propose remediation plans. The benchmark evaluates different aspects of the investigation using fine-grained rubrics. The paper also introduces RETrace, an RL method that uses evidence cited in generated reports to assign rewards to earlier investigation steps.
I overall like the idea of evaluating agents on actual compromised environments. This is certainly more useful than another benchmark based on isolated security questions or narrow CTFs. But I have quite a few reservations about the evaluation and the proposed training method. Specifically, the paper makes fairly broad claims about improving autonomous incident investigation, etc., while much of the evidence comes from a small collection of constructed scenarios/environments and rubric-based assessment of written reports. I'm also not convinced why these environments are representative of realistic environments (in particular, no quantitative comparison of the attack patterns, alert distributions, or environmental complexity against those observed in practice). The paper's distinction from some existing incident-response benchmarks and works is also not entirely convincing to me. Overall, I lean weak reject.
Critical strengths
-
I appreciate the cyber-range setup overall. The fact that the authors actually execute attacks and preserve the resulting forensic artifacts (including persistence mechanisms, etc.) is promising. It seems like a fairly useful setting for evaluating incident investigation.
-
The introduced benchmark covers multiple aspects of incident response as the authors say rather than just attack detection. I particularly liked the distinction between alert-driven and proactive investigation (important in practice in SOC-like envs).
-
There's a decent amount of experimental work across different models. The RETrace ablations are also useful, esp. in understanding which parts of the method contribute most to the improvements.
Critical weaknesses
-
I have some major concerns about relying on final reports for evaluation (agents largely scored based on whether their reports satisfy certain checkpoints). Does satisfying these checkpoints really reflect investigation quality? The separate evidence retrieval evaluation helps, but I'm not sure it's sufficient, esp. given RETrace's report-based rewards.
-
I am not particularly convinced by the credit assignment in RETrace. The method traces evidence in the final report back to earlier tool outputs, but how would that necessarily mean those actions were actually useful? Don't the authors think this is a fairly strong assumption? It is not clear how (if it holds) reliable this attribution really is.
-
The evaluation is limited to 10 single-host ranges, with no external validation (tested on held-out scenarios from the same collection). Remediation is also only proposed and not executed. So there's essentially no validation of whether the remediation plans actually work to recover the systems or not.
-
I lastly have some reservations about the novelty. Previous work such as SIR-Bench, CyberSleuth, CFA-Bench, and IRCopilot (out of which only IRCopilot has been cited by the authors; maybe they can explain??) has already explored agent-based incident investigation, including evaluations using evidence from actual attacks. While the filesystem snapshots and detailed rubrics are useful, I don't see a particularly clear advancement over these works. The paper could do a better job discussing these differences.
-
Additionally, as related to my previous point, the authors' "the first benchmark for evaluating LLM agents to autonomously conduct the post-compromise incident-response workflow in an end-to-end manner" looks unfounded to me. Maybe they can explain this claim as well clearly.
Questions for authors
-
See questions raised in my weaknesses section.
-
How do the authors distinguish improvements in actual investigation from improvements in report generation? This seems particularly important given how RETrace is trained.
-
Why is tracing evidence back to the earliest matching tool output a reliable way to assign credit? What happens if and when multiple investigation potentially steps contribute to a finding? Do you even study such settings?
-
Have the authors considered actually executing the generated remediation plans in the cyber ranges and seeing how and if they work? What reasons as to why they do or don't? It looks like the most natural those resolutions to the incidents even make sense.
-
How well would RETrace transfer to incident scenarios outside the benchmark? Especially ones constructed independently of the authors' setup.
-
Please clearly explain the novelty over the works I mentioned and provide a more detailed survey of other related work not covered currently.
5 comments
Response to Reviewer QDrV [Part II]
W4: "I lastly have some reservations about the novelty..."
Thank you for highlighting these related works. SecRespond's intended contribution lies in benchmarking autonomous command-line investigation of a full host disk snapshot with capability-oriented evaluation, and proposing a training recipe that credits steps with key evidence. The agent must decide where to investigate and reconstruct the attack chain from artifacts distributed across the host. Existing works differ from SecRespond in their evidence scope or execution protocol. CFA-Bench [1] focuses on network traces, CyberSleuth [2] analyzes network traces and application logs, and SIR-Bench [3] investigates cloud telemetry. None of them exposes the agent to the compromised host itself, where persistence mechanisms, deliberately erased traces, and concurrent host activity are kept. IRBench [4] uses human-executed commands in its baseline LLM evaluation, whereas SecRespond agents execute investigative commands autonomously. We have added the discussion to the related work in the revised paper.
W5: Additionally, as related to my previous point, the authors' "the first benchmark for evaluating LLM agents to autonomously conduct the post-compromise incident-response workflow in an end-to-end manner" looks unfounded to me. Maybe they can explain this claim as well clearly.
(1) Complete Post-compromise Host: The agent is exposed to the compromised host left after the attack, which is a full disk snapshot with attack artifacts, persistence mechanisms, erased traces mixed with normal system and business files, rather than an isolated environment that keeps only a certain attack and filters out other files, as in previous work [1,2,3]. (2) Autonomous and open-ended investigation: The agent autonomously walks through this host with CLI commands and delivers forensic reports that reconstruct the attack chain and a remediation plan that proposes the fixes. Existing works either predefine specific questions with hints about the artifact, such as the exploited CVE or the attacker IP, or rely on a human to execute the commands [4].
[1] CFA-bench: Cybersecurity Forensic LLM Agent Benchmark and Testing, WTMC 2025 [2] CyberSleuth: Autonomous Blue-Team LLM Agent for Web Attack Forensics, arXiv.2508.20643 [3] SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents, arXiv.2604.12040 [4] IRCopilot: Automated Incident Response with Large Language Models, arXiv.2505.20945
Q1: "How do the authors distinguish improvements in actual investigation from improvements in report generation?"
(1) RETrace improves the sub-scores that can only be earned by investigation. As discussed in W1, the detection score of each checkpoint is the sum of three sub-scores, Discovery, Evidence, and Attribution. On OOD ranges, RETrace improves Evidence by 7.3% and Attribution by 3.0%. We have also included these detailed sub-scores in Appendix F.5.
| Model | Domain | Discovery | Evidence | Attribution |
|---|---|---|---|---|
| SFT | ID | 74.8 | 67.3 | 65.4 |
| RETrace | ID | 78.8 | 70.8 | 68.5 |
| SFT | OOD | 67.8 | 56.3 | 54.1 |
| RETrace | OOD | 72.9 | 63.5 | 57.1 |
(2) The trajectories show deeper investigation, independent of the report and the judge. As discussed in W1, the number of required evidence items reported in Figure 4(a) that appear in the tool outputs of a trajectory is computed from the trajectory alone. On the OOD ranges, RETrace retrieves 19.0 items against 17.9 for SFT and 16.8 for GRPO, without more tool calls or longer responses. Figure 4(b) further reports the improvement in detecting silent artifacts, which induce no alert and require the agent to search and locate them on the disk. The gain of RETrace concentrates on the silent checkpoints of the OOD ranges (57.9% vs. 52.7% for SFT).
Q2: "Why is tracing evidence back to the earliest matching tool output a reliable way to assign credit?..."
Choosing the earliest step for the same artifact. When the same artifact appears in several tool outputs, the earliest step first brings this evidence into the agent's context. Crediting later steps that re-read the evidence would reinforce redundant actions. The traced steps are also reliable in practice since 91.4% of the credited steps retrieve the artifact that the attack actually left on the disk as discussed in W2.
Multiple steps contributing to one finding. The steps that retrieve the evidence cited for a checkpoint additionally receive credit. Each evidence item in an accepted paragraph is traced separately, so a finding built from evidence retrieved by different steps credits all of them with equal weight. We examined 1.4K training rollouts, and 32.4% of the traced checkpoints are credited to two or more steps. Extending additional credit to steps that contribute to findings through joint interpretation of several tool outputs, without directly containing the evidence, is a natural next step.
Response to Reviewer QDrV [Part III]
Q3: "Have the authors considered actually executing the generated remediation plans in the cyber ranges..."
Please refer to the response to W3.
Q4: "How well would RETrace transfer to incident scenarios outside the benchmark? Especially ones constructed independently of the authors' setup."
As discussed in W3, we use OOD results to reflect generalization to attacks with unseen stacks, vulnerabilities, and persistence mechanisms given the construction cost of each range. Particularly, we further identified the 114 held-out checkpoints (approximately 41% of the total checkpoints) whose software stacks, persistence mechanisms, and CVEs never appear in the training fold (Table 22). On these checkpoints, RETrace improves over SFT by 4.3% on Detection and 4.2% on Planning and over GRPO by 10.6% and 10.7%.
| Method | OOD Fold A Det | OOD Fold A Plan | OOD Fold B Det | OOD Fold B Plan | Average Det | Average Plan |
|---|---|---|---|---|---|---|
| Qwen3.5-9B (base) | 49.8 ± 3.9 | 25.0 ± 2.4 | 50.6 ± 1.9 | 31.4 ± 1.8 | 50.2 | 28.2 |
| Qwen3.5-9B + skill prompt | 46.6 ± 2.5 | 33.9 ± 2.0 | 48.2 ± 2.2 | 32.9 ± 1.2 | 47.4 | 33.4 |
| SFT (skill-guided trajectories) | 57.5 ± 3.2 | 43.0 ± 3.2 | 48.8 ± 1.4 | 36.7 ± 2.4 | 53.1 | 39.9 |
| SFT + GRPO | 48.0 ± 2.0 | 34.0 ± 3.1 | 45.9 ± 4.1 | 32.8 ± 3.2 | 47.0 | 33.4 |
| SFT + RETrace | 60.9 ± 1.8 | 48.0 ± 3.1 | 54.0 ± 3.0 | 40.2 ± 1.1 | 57.5 | 44.1 |
Q5: "Please clearly explain the novelty over the works I mentioned and provide a more detailed survey of other related work not covered currently."
Please refer to the response to W4 and W5.
Response to Reviewer QDrV [Part I]
Thank you to the reviewers for their time, effort, and detailed comments, which have helped us significantly improve the paper. In the following, we clarify several points that may not have been sufficiently clear in the original manuscript and provide additional evidence to further support our claims.
First, the checkpoints are grounded and calibrated. Each checklist is derived from the blueprint, the post-attack snapshot, and the security-product findings and is audited by experts for completeness. Ambiguous or non-discriminative checkpoints were rewritten or split during pilot runs. Detection separates Discovery, Evidence, and Attribution, requiring the agent not only to name the attack but also to locate the artifact and identify the correct root cause. Besides, 130 detection checkpoints are silent, requiring evidence absent from the alerted security findings. These design choices make checkpoint scores a meaningful measure of investigation quality.
Second, the analysis in Figure 4 examines investigation quality through trajectory evidence and silent-artifact detection, and verifies that training improved the investigation rather than fitting the reward. Figure 4(a) counts required evidence items appearing in tool outputs (19.0 for RETrace vs. 17.9 for SFT and 16.8 for GRPO) on OOD ranges, without increased tool calls or response length. Figure 4(b) shows that silent checkpoint detection improves from 52.7 to 57.9. The retrieval measure is computed before the final report, indicating that the additional evidence comes from more accurate and targeted investigation.
We would like to point out that the credited step is required to contain evidence grounded in the disk snapshot. A step is credited only when its tool output contains the concrete artifact (file path, hash, IP, or log line) cited in a report paragraph that the judge accepts for a checkpoint. We examined 1.4K rollouts and found that in 91.4% of the credited steps, the tool output contains the artifact that the attack actually left on the disk. Thus, the credit encourages the model to reliably find attack evidence, and removing the traceback lowers OOD detection by 5.2% as shown in Table 3.
First, we would like to clarify that the evaluation within each range is much richer than the range count suggests. Every range is produced by executing a multi-stage attack against a deployed vulnerable stack through a real, publicly known weakness. The ten ranges span 4 entry-point types and over 15 persistence mechanisms, with 280 checkpoints covering 52 capability items and 21 ATT&CK techniques.
Second, given the construction cost of each range, we adopted two-fold cross-validation so that every range can be evaluated as held-out. The two folds are trained independently, and OOD evaluation covers all ten ranges. Generalization can be assessed on the held-out ranges, since different folds share no common software stack and exploited vulnerability, while more than half of the persistence mechanisms and capability items appear in only one fold, as in the statistics in Table 22 in Appendix F.5. Thus, the OOD results can measure generalization to attacks with unseen stacks, vulnerabilities, and persistence mechanisms.
Third, executing the plans would provide a stronger test of operational effectiveness. The current task positions the agent as an assistant that reconstructs the intrusion and delivers a remediation plan for analyst review. Executing remediation on a compromised production host is destructive and should stay under human control, so we evaluate the remediation plans with Correctness and Completeness in the rubrics, which assess whether the plan eradicates the threat and maintains the service. Developing a safe environment to evaluate remediation execution is a natural extension that we leave to future work.