acceptodds
Under review as a conference paper at ICLR 2027

SecRespond: Benchmarking and Training AI Agents for Real-World Post-Compromise Incident Response

Abstract

Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to assess their security capabilities. However, existing cybersecurity benchmarks mostly focus on pre-compromise settings where agents operate in a clean environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SecRespond, the first benchmark for evaluating LLM agents to autonomously conduct the post-compromise incident-response workflow in an end-to-end manner. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans. We further propose a training recipe for post-compromise agents with skill internalization and RETrace, a report-anchored credit assignment, which assigns each rubric checkpoint's score to the corresponding output span and the investigation step. On out-of-domain ranges, RETrace improves over outcome-reward RL by 8.8% in detection and 7.2% in planning.

Primary area:
datasets and benchmarks
Keywords:
LLM Agent, Incident Response, Agentic RL

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_QDrVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SecRespond, a benchmark for evaluating LLM agents on post-compromise incident response, specifically using a cyber range system setup. The authors build a set of cyber ranges by executing attacks against Linux (CentOS/Ubuntu) and Windows hosts, and then provide agents with forensic snapshots and security alerts. The agents need to investigate the compromised systems, identify malicious activity, and propose remediation plans. The benchmark evaluates different aspects of the investigation using fine-grained rubrics. The paper also introduces RETrace, an RL method that uses evidence cited in generated reports to assign rewards to earlier investigation steps.

I overall like the idea of evaluating agents on actual compromised environments. This is certainly more useful than another benchmark based on isolated security questions or narrow CTFs. But I have quite a few reservations about the evaluation and the proposed training method. Specifically, the paper makes fairly broad claims about improving autonomous incident investigation, etc., while much of the evidence comes from a small collection of constructed scenarios/environments and rubric-based assessment of written reports. I'm also not convinced why these environments are representative of realistic environments (in particular, no quantitative comparison of the attack patterns, alert distributions, or environmental complexity against those observed in practice). The paper's distinction from some existing incident-response benchmarks and works is also not entirely convincing to me. Overall, I lean weak reject.

Critical strengths
  • I appreciate the cyber-range setup overall. The fact that the authors actually execute attacks and preserve the resulting forensic artifacts (including persistence mechanisms, etc.) is promising. It seems like a fairly useful setting for evaluating incident investigation.

  • The introduced benchmark covers multiple aspects of incident response as the authors say rather than just attack detection. I particularly liked the distinction between alert-driven and proactive investigation (important in practice in SOC-like envs).

  • There's a decent amount of experimental work across different models. The RETrace ablations are also useful, esp. in understanding which parts of the method contribute most to the improvements.

Critical weaknesses
  • I have some major concerns about relying on final reports for evaluation (agents largely scored based on whether their reports satisfy certain checkpoints). Does satisfying these checkpoints really reflect investigation quality? The separate evidence retrieval evaluation helps, but I'm not sure it's sufficient, esp. given RETrace's report-based rewards.

  • I am not particularly convinced by the credit assignment in RETrace. The method traces evidence in the final report back to earlier tool outputs, but how would that necessarily mean those actions were actually useful? Don't the authors think this is a fairly strong assumption? It is not clear how (if it holds) reliable this attribution really is.

  • The evaluation is limited to 10 single-host ranges, with no external validation (tested on held-out scenarios from the same collection). Remediation is also only proposed and not executed. So there's essentially no validation of whether the remediation plans actually work to recover the systems or not.

  • I lastly have some reservations about the novelty. Previous work such as SIR-Bench, CyberSleuth, CFA-Bench, and IRCopilot (out of which only IRCopilot has been cited by the authors; maybe they can explain??) has already explored agent-based incident investigation, including evaluations using evidence from actual attacks. While the filesystem snapshots and detailed rubrics are useful, I don't see a particularly clear advancement over these works. The paper could do a better job discussing these differences.

  • Additionally, as related to my previous point, the authors' "the first benchmark for evaluating LLM agents to autonomously conduct the post-compromise incident-response workflow in an end-to-end manner" looks unfounded to me. Maybe they can explain this claim as well clearly.

Questions for authors
  • See questions raised in my weaknesses section.

  • How do the authors distinguish improvements in actual investigation from improvements in report generation? This seems particularly important given how RETrace is trained.

  • Why is tracing evidence back to the earliest matching tool output a reliable way to assign credit? What happens if and when multiple investigation potentially steps contribute to a finding? Do you even study such settings?

  • Have the authors considered actually executing the generated remediation plans in the cyber ranges and seeing how and if they work? What reasons as to why they do or don't? It looks like the most natural those resolutions to the incidents even make sense.

  • How well would RETrace transfer to incident scenarios outside the benchmark? Especially ones constructed independently of the authors' setup.

  • Please clearly explain the novelty over the works I mentioned and provide a more detailed survey of other related work not covered currently.

5 comments
AuthorsOpenReview ↗

Response to Reviewer QDrV [Part I]

Thank you to the reviewers for their time, effort, and detailed comments, which have helped us significantly improve the paper. In the following, we clarify several points that may not have been sufficiently clear in the original manuscript and provide additional evidence to further support our claims.

W1: "I have some major concerns about relying on final reports for evaluation..."

First, the checkpoints are grounded and calibrated. Each checklist is derived from the blueprint, the post-attack snapshot, and the security-product findings and is audited by experts for completeness. Ambiguous or non-discriminative checkpoints were rewritten or split during pilot runs. Detection separates Discovery, Evidence, and Attribution, requiring the agent not only to name the attack but also to locate the artifact and identify the correct root cause. Besides, 130 detection checkpoints are silent, requiring evidence absent from the alerted security findings. These design choices make checkpoint scores a meaningful measure of investigation quality.

Second, the analysis in Figure 4 examines investigation quality through trajectory evidence and silent-artifact detection, and verifies that training improved the investigation rather than fitting the reward. Figure 4(a) counts required evidence items appearing in tool outputs (19.0 for RETrace vs. 17.9 for SFT and 16.8 for GRPO) on OOD ranges, without increased tool calls or response length. Figure 4(b) shows that silent checkpoint detection improves from 52.7 to 57.9. The retrieval measure is computed before the final report, indicating that the additional evidence comes from more accurate and targeted investigation.

W2: "I am not particularly convinced by the credit assignment in RETrace..."

We would like to point out that the credited step is required to contain evidence grounded in the disk snapshot. A step is credited only when its tool output contains the concrete artifact (file path, hash, IP, or log line) cited in a report paragraph that the judge accepts for a checkpoint. We examined 1.4K rollouts and found that in 91.4% of the credited steps, the tool output contains the artifact that the attack actually left on the disk. Thus, the credit encourages the model to reliably find attack evidence, and removing the traceback lowers OOD detection by 5.2% as shown in Table 3.

W3: "The evaluation is limited to 10 single-host ranges..."

First, we would like to clarify that the evaluation within each range is much richer than the range count suggests. Every range is produced by executing a multi-stage attack against a deployed vulnerable stack through a real, publicly known weakness. The ten ranges span 4 entry-point types and over 15 persistence mechanisms, with 280 checkpoints covering 52 capability items and 21 ATT&CK techniques.

Second, given the construction cost of each range, we adopted two-fold cross-validation so that every range can be evaluated as held-out. The two folds are trained independently, and OOD evaluation covers all ten ranges. Generalization can be assessed on the held-out ranges, since different folds share no common software stack and exploited vulnerability, while more than half of the persistence mechanisms and capability items appear in only one fold, as in the statistics in Table 22 in Appendix F.5. Thus, the OOD results can measure generalization to attacks with unseen stacks, vulnerabilities, and persistence mechanisms.

Third, executing the plans would provide a stronger test of operational effectiveness. The current task positions the agent as an assistant that reconstructs the intrusion and delivers a remediation plan for analyst review. Executing remediation on a compromised production host is destructive and should stay under human control, so we evaluate the remediation plans with Correctness and Completeness in the rubrics, which assess whether the plan eradicates the threat and maintains the service. Developing a safe environment to evaluate remediation execution is a natural extension that we leave to future work.

AuthorsOpenReview ↗

Response to Reviewer QDrV [Part II]

W4: "I lastly have some reservations about the novelty..."

Thank you for highlighting these related works. SecRespond's intended contribution lies in benchmarking autonomous command-line investigation of a full host disk snapshot with capability-oriented evaluation, and proposing a training recipe that credits steps with key evidence. The agent must decide where to investigate and reconstruct the attack chain from artifacts distributed across the host. Existing works differ from SecRespond in their evidence scope or execution protocol. CFA-Bench [1] focuses on network traces, CyberSleuth [2] analyzes network traces and application logs, and SIR-Bench [3] investigates cloud telemetry. None of them exposes the agent to the compromised host itself, where persistence mechanisms, deliberately erased traces, and concurrent host activity are kept. IRBench [4] uses human-executed commands in its baseline LLM evaluation, whereas SecRespond agents execute investigative commands autonomously. We have added the discussion to the related work in the revised paper.

W5: Additionally, as related to my previous point, the authors' "the first benchmark for evaluating LLM agents to autonomously conduct the post-compromise incident-response workflow in an end-to-end manner" looks unfounded to me. Maybe they can explain this claim as well clearly.

(1) Complete Post-compromise Host: The agent is exposed to the compromised host left after the attack, which is a full disk snapshot with attack artifacts, persistence mechanisms, erased traces mixed with normal system and business files, rather than an isolated environment that keeps only a certain attack and filters out other files, as in previous work [1,2,3]. (2) Autonomous and open-ended investigation: The agent autonomously walks through this host with CLI commands and delivers forensic reports that reconstruct the attack chain and a remediation plan that proposes the fixes. Existing works either predefine specific questions with hints about the artifact, such as the exploited CVE or the attacker IP, or rely on a human to execute the commands [4].

[1] CFA-bench: Cybersecurity Forensic LLM Agent Benchmark and Testing, WTMC 2025 [2] CyberSleuth: Autonomous Blue-Team LLM Agent for Web Attack Forensics, arXiv.2508.20643 [3] SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents, arXiv.2604.12040 [4] IRCopilot: Automated Incident Response with Large Language Models, arXiv.2505.20945

Q1: "How do the authors distinguish improvements in actual investigation from improvements in report generation?"

(1) RETrace improves the sub-scores that can only be earned by investigation. As discussed in W1, the detection score of each checkpoint is the sum of three sub-scores, Discovery, Evidence, and Attribution. On OOD ranges, RETrace improves Evidence by 7.3% and Attribution by 3.0%. We have also included these detailed sub-scores in Appendix F.5.

ModelDomainDiscoveryEvidenceAttribution
SFTID74.867.365.4
RETraceID78.870.868.5
SFTOOD67.856.354.1
RETraceOOD72.963.557.1

(2) The trajectories show deeper investigation, independent of the report and the judge. As discussed in W1, the number of required evidence items reported in Figure 4(a) that appear in the tool outputs of a trajectory is computed from the trajectory alone. On the OOD ranges, RETrace retrieves 19.0 items against 17.9 for SFT and 16.8 for GRPO, without more tool calls or longer responses. Figure 4(b) further reports the improvement in detecting silent artifacts, which induce no alert and require the agent to search and locate them on the disk. The gain of RETrace concentrates on the silent checkpoints of the OOD ranges (57.9% vs. 52.7% for SFT).

Q2: "Why is tracing evidence back to the earliest matching tool output a reliable way to assign credit?..."

Choosing the earliest step for the same artifact. When the same artifact appears in several tool outputs, the earliest step first brings this evidence into the agent's context. Crediting later steps that re-read the evidence would reinforce redundant actions. The traced steps are also reliable in practice since 91.4% of the credited steps retrieve the artifact that the attack actually left on the disk as discussed in W2.

Multiple steps contributing to one finding. The steps that retrieve the evidence cited for a checkpoint additionally receive credit. Each evidence item in an accepted paragraph is traced separately, so a finding built from evidence retrieved by different steps credits all of them with equal weight. We examined 1.4K training rollouts, and 32.4% of the traced checkpoints are credited to two or more steps. Extending additional credit to steps that contribute to findings through joint interpretation of several tool outputs, without directly containing the evidence, is a natural next step.

AuthorsOpenReview ↗

Response to Reviewer QDrV [Part III]

Q3: "Have the authors considered actually executing the generated remediation plans in the cyber ranges..."

Please refer to the response to W3.

Q4: "How well would RETrace transfer to incident scenarios outside the benchmark? Especially ones constructed independently of the authors' setup."

As discussed in W3, we use OOD results to reflect generalization to attacks with unseen stacks, vulnerabilities, and persistence mechanisms given the construction cost of each range. Particularly, we further identified the 114 held-out checkpoints (approximately 41% of the total checkpoints) whose software stacks, persistence mechanisms, and CVEs never appear in the training fold (Table 22). On these checkpoints, RETrace improves over SFT by 4.3% on Detection and 4.2% on Planning and over GRPO by 10.6% and 10.7%.

MethodOOD Fold A DetOOD Fold A PlanOOD Fold B DetOOD Fold B PlanAverage DetAverage Plan
Qwen3.5-9B (base)49.8 ± 3.925.0 ± 2.450.6 ± 1.931.4 ± 1.850.228.2
Qwen3.5-9B + skill prompt46.6 ± 2.533.9 ± 2.048.2 ± 2.232.9 ± 1.247.433.4
SFT (skill-guided trajectories)57.5 ± 3.243.0 ± 3.248.8 ± 1.436.7 ± 2.453.139.9
SFT + GRPO48.0 ± 2.034.0 ± 3.145.9 ± 4.132.8 ± 3.247.033.4
SFT + RETrace60.9 ± 1.848.0 ± 3.154.0 ± 3.040.2 ± 1.157.544.1

Q5: "Please clearly explain the novelty over the works I mentioned and provide a more detailed survey of other related work not covered currently."

Please refer to the response to W4 and W5.

Reviewer_cRrhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces SECRESPOND, a benchmark for evaluating LLM agents on post-compromise incident response. It contains 10 cyber ranges covering five operating systems, four entry-point types, and 21 ATT&CK techniques. Agents receive forensic disk snapshots, security alerts, vulnerability scans, and baseline checks, and must produce intrusion, vulnerability, baseline, and remediation reports.

Critical strengths

Unlike many cybersecurity benchmarks focused on CTFs, exploitation, or clean environments, this work evaluates forensic investigation and remediation after a host has already been compromised.

Critical weaknesses

The evaluation relies on three LLM judges, while human validation covers only 60 checkpoints. This is not enough to fully establish the reliability of 280 checkpoints, complex remediation plans, or model rankings. The paper also lacks statistical significance tests across judges, seeds, and inference runs.

Questions for authors

Please report refusal rates separately and consider a safety-preserving evaluation protocol for models that refuse certain cybersecurity ranges, rather than simply excluding those ranges from aggregate scores.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cRrh [Part I]

Thank you for highlighting evaluation reliability and the handling of incident data. We address these points below.

W1: The evaluation relies on three LLM judges, while human validation covers only 60 checkpoints. This is not enough to fully establish the reliability of 280 checkpoints, complex remediation plans, or model rankings. The paper also lacks statistical significance tests across judges, seeds, and inference runs.

Human validation is larger in scale. The randomly selected 60 checkpoints cover all 10 ranges and both detection and planning axes. The expert graded 10 trajectories per checkpoint without seeing the judge scores, producing 600 checkpoint–trajectory assessments. On this sample, Pearson correlation is 0.96 and quadratic-weighted kappa is 0.94, as reported in Appendix D.5.

Statistical Robustness. Our evaluation is scored by the same three judges, and averaged to reduce the variance of any single judge. Appendix D.5 shows that the judges differ mainly in overall strictness rather than in preference for particular models. We further add significance tests on the existing results. For the benchmark results in Section 4, paired analyses reported p < 0.02 for the comparisons across 23 models. For training, t-tests over inference runs yield p < 0.03 for all tested improvements of RETrace over SFT and GRPO.

Additional evaluation across training seeds. We further conducted an additional experiment with Gemma-4-E4B using three independent training seeds comparing SFT against SFT + RETrace on 8 OOD Linux cyber ranges. RETrace improves mean detection from 53.4% to 56.6% and planning from 41.8% to 44.5%, with p-values of 0.135 and 0.134, respectively.

MethodMetric (%)Next.js-RCENPM-WormShiro-FastjsonSSH-MinerDocker-EscapeJenkins-RCELog4j-RCERedis-RCEAverage
SFTDetection48.0 ± 18.340.5 ± 2.724.8 ± 6.074.9 ± 3.863.4 ± 8.843.3 ± 2.783.3 ± 0.349.1 ± 2.553.4 ± 2.8
SFTPlanning38.1 ± 12.934.0 ± 3.721.1 ± 2.865.7 ± 3.543.0 ± 8.325.4 ± 3.162.0 ± 3.345.1 ± 4.541.8 ± 2.2
SFT + RETraceDetection61.8 ± 2.844.2 ± 5.328.5 ± 5.579.2 ± 1.466.7 ± 1.337.4 ± 4.283.8 ± 0.451.4 ± 6.856.6 ± 1.4
SFT + RETracePlanning49.4 ± 2.639.1 ± 1.921.9 ± 5.567.6 ± 2.438.6 ± 3.126.3 ± 3.363.9 ± 4.749.0 ± 2.144.5 ± 1.2

Q1: Please report refusal rates separately and consider a safety-preserving evaluation protocol for models that refuse certain cybersecurity ranges, rather than simply excluding those ranges from aggregate scores.

Across our experiments, the only refusal we observed is Claude Opus 4.7 on NPM-Worm, where it declined every attempt and produced no report. On the 9 valid ranges, Claude Opus 4.7 achieves detection/planning scores of 79.0%/65.7%. Giving the refused range zero score changes the detection/planning means to 71.1%/59.1% over ten ranges.

AuthorsOpenReview ↗

Response to Reviewer cRrh [Part II]

Ethical Concerns

We have declared these concerns in the Ethics Statement section. We would like to explain them point-by-point as well.

whether data derived from 372 real compromised cloud hosts involved appropriate authorization and privacy safeguards

The 372 hosts served as a reference for common incident types and scenario selection. We built each benchmark range from scratch on a fresh host deployed and attacked in an isolated sandbox, so its artifacts were not copied from the reference collection.

whether skills derived from customer incidents could expose proprietary operational knowledge

The skill is a general procedural guide written by our team, containing investigation routines and report templates. It does not contain malicious paths, known vulnerability identifiers, or operational knowledge from any range.

whether releasing forensic artifacts and attack chains could facilitate offensive use or security-product evasion

The released forensic and security-product artifacts are synthetic and contain no data from customer, production, or real-world business systems. We sanitized every disk snapshot by replacing real credentials, keys, and personal data with placeholders, retaining only the forensic artifacts required for analysis. The attack techniques, CVEs, and exploitation methods represented in the scenarios are derived solely from public knowledge bases and vulnerability disclosures, such as MITRE ATT&CK and public vulnerability databases. The benchmark contains forensic evidence rather than runnable exploit chains; it includes no proprietary or internally developed advanced attack techniques and no incremental attack information beyond public sources, and it cannot be directly used to conduct attacks against real-world systems. The benchmark is intended exclusively for research.

whether third-party model evaluation and output retention comply with provider terms and data policies

We accessed all commercial models through provider-authorized APIs and gave them the same task prompt. Everything sent to the APIs was sandbox content with no personal or customer information. Model outputs are used only for evaluation and never for training.

Reviewer_cRrhOpenReview ↗

Response

Thanks for the authors' detailed reply, I will keep my positive score.

Thank you for taking the time to review our rebuttal. We greatly appreciate your feedback and your recognition of the additional evidence provided by our new analyses.

Reviewer_6VDJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SecRespond is a benchmark for evaluating LLM agents onend-to-end post-compromise incident response across ten cyber ranges,covering tasks including intrusion investigation, risk assessment, andremediation planning. The authors evaluate 23 LLMs and find that currentagents can identify problems exposed by security alerts but struggle toproactively discover hidden intrusion evidence and produce comprehensiveremediation plans. To address these limitations, the paper also proposes a RLbased training approach called RETrace, which assigns fine-grained credit toreport segments and their supporting investigation steps, improving detectionand planning scores over outcome-based reinforcement learning by 8.8% and7.2% respectively.

Critical strengths
  1. From the reviewer’s point of view the paper does not just identify a practical problem in LLM-based cybersecurity, but also builds a benchmark to evaluate it and proposes a method to address it. The overall motivation is clear, and the different parts of the paper fit together well.

  2. The benchmark evaluation is quite comprehensive, covering 23 models from eight model series, including several recent frontier models. For RETrace, the authors also compare it with different training methods, including SFT, GRPO, and GSPO. I think these comparisons provide convincing evidence of the benefits of the proposed approach.

  3. I particularly like how the authors break down the incident-response task into finer-grained checkpoints and different capability dimensions. Rather than reporting only an overall score, the benchmark helps us understand where agents perform well and where they still struggle, making the evaluation more informative and useful.

Critical weaknesses
  1. RETrace uses LLM-based scores to guide training, while the final evaluation also relies on LLM judges using the same rubric. This raises a concern that the model may learn to write reports that score better without actually becoming better at investigating incidents. For example, a more detailed report does not necessarily mean the agent has found more evidence or can provide more reliable fixes. Although the authors compare the LLM judges with human assessments, it would be helpful to provide further evidence that RETrace improves the agent's actual investigation capabilities, rather than simply helping it produce reports that better match the scoring criteria.

  2. The comparison with traditional security scanners provides useful evidence of the potential benefits of LLM agents, but the baseline is relatively limited.Traditional scanners primarily rely on static pattern matching, whereas agents can actively inspect files and reason across multiple sources of evidence. Therefore, outperforming this baseline does not necessarily demonstrate that the proposed agents are competitive with experienced security analysts or more comprehensive incident-response workflows that combine multiple security tools.

  3. The training experiments focus on the Qwen 3.5 model family, including a9B model and a larger 27B model. While the results across these scales are encouraging, from the reviewer’s point of view, it is still unclear on whetherRETrace provides similar benefits for agents built on other model families with different tool-use and reasoning behaviors. A preliminary evaluation on a model from another family such as Muse Glimmer or Gemma 4 (these two models are within 30B size which is similar to Qwen 3.5 27B and were released before the paper cutoff date) would help establish the generalizability of the proposed method.

  4. A tiny issue: Some figures, particularly Fig. 1 and Fig. 2(b), become pixelated when I zoomed in. I would suggest the authors export them as pdf format or another vector format to improve readability as it may affect the presentation quality of the paper.

Questions for authors
  1. Since LLM-based agents may introduce additional reasoning and tool-use overhead compared with traditional security tools, could the authors provide a comparison of their efficiency, such as execution time and cost? This would help clarify the practical trade-offs of using LLM-based methods for incident response.
  2. The training experiments focus on the Qwen 3.5 family. Could the authors provide a preliminary evaluation on a model from another family to see whether RETrace generalizes beyond Qwen? I understand that full RL experiments can be expensive, so even a small-scale experiment would be helpful.
  3. I noticed that Claude Opus 4.7 refuses the NPM-Worm scenario and is evaluated on only nine ranges. Given the growing interest in modifying model safety guardrails for cybersecurity applications, I am curious how frequently such refusals occur across the evaluated models. Could the authors report the refusal rates and clarify how refusals are handled in the overall comparison? Some approaches such as OpenAI Daybreak can alsoenable the cyber capabilities of these commercial models.
  4. Could the authors clarify how much the training and held-out ranges overlap in terms of ATT&CK techniques, CVEs, software stacks, and persistence mechanisms? This would help readers better understand how challenging the OOD evaluation is and how well the method generalizes to unseen attack scenarios.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 6VDJ [Part I]

We sincerely thank the reviewer for the constructive feedback and suggestions for additional comparisons, which have significantly improved our work.

W1: "...it would be helpful to provide further evidence that RETrace improves the agent's actual investigation capabilities..."

We agree that a higher judge score on the trained model does not establish better investigation behavior, and we examine what the agent retrieves before writing its final report. Figure 4(a) counts required evidence items in tool outputs. On OOD ranges, RETrace retrieves 19.0 items versus 17.9 for SFT and 16.8 for GRPO, demonstrating that the gain comes from reaching more evidence. Besides, Figure 4(b) separates checkpoints whose evidence is exposed by security findings from those requiring further investigation. RETrace improves detection on the latter checkpoints from 52.7% to 57.9%. These analyses examine the investigation process and show that RETrace improves how the agent investigates, instead of simply adapting to the scoring rubric.

W2: "The comparison with traditional security scanners provides useful evidence of the potential benefits of LLM agents, but the baseline is relatively limited..."

We agree that outperforming the agentless scanner does not show that agents are competitive with experienced analysts. The scanner is included as a component baseline representing a deployed tool for disk analysis. This comparison asks where an agent can complement static inspection, rather than treating the scanner as a substitute for an experienced analyst or comprehensive workflows. On the same snapshots, the scanner scores 2.1% on persistence mechanisms and 20.7% on baseline risks. These tasks involve inspecting configuration files and correlating evidence, which highlights the potential of proactive investigation to uncover evidence beyond the scanner's findings list.

W3: "The training experiments focus on the Qwen 3.5 model family, ..."

Thank you for the suggestion. The original experiments cover Qwen3.5-9B and Qwen3.5-27B; we therefore added a Gemma-4-E4B comparison between SFT and SFT + RETrace, using three training seeds and evaluating on eight held-out Linux ranges. As shown in the table below, RETrace increases the average detection score from 53.4% to 56.6% and the average planning score from 41.8% to 44.5%. Paired t-tests yield p = 0.135 for detection and p = 0.134 for planning. This shows that RETrace’s improvement remains on an additional model family across 3 different training seeds.

MethodMetric (%)Next.js-RCENPM-WormShiro-FastjsonSSH-MinerDocker-EscapeJenkins-RCELog4j-RCERedis-RCEAverage
SFTDetection48.0 ± 18.340.5 ± 2.724.8 ± 6.074.9 ± 3.863.4 ± 8.843.3 ± 2.783.3 ± 0.349.1 ± 2.553.4 ± 2.8
SFTPlanning38.1 ± 12.934.0 ± 3.721.1 ± 2.865.7 ± 3.543.0 ± 8.325.4 ± 3.162.0 ± 3.345.1 ± 4.541.8 ± 2.2
SFT + RETraceDetection61.8 ± 2.844.2 ± 5.328.5 ± 5.579.2 ± 1.466.7 ± 1.337.4 ± 4.283.8 ± 0.451.4 ± 6.856.6 ± 1.4
SFT + RETracePlanning49.4 ± 2.639.1 ± 1.921.9 ± 5.567.6 ± 2.438.6 ± 3.126.3 ± 3.363.9 ± 4.749.0 ± 2.144.5 ± 1.2

W4: "...Some figures become pixelated..."

Thanks for your suggestion. We have replaced the figures with clearer versions in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer 6VDJ [Part II]

Q1: "Since LLM-based agents may introduce additional reasoning and tool-use overhead compared with traditional security tools, could the authors provide a comparison of their efficiency, such as execution time and cost? This would help clarify the practical trade-offs of using LLM-based methods for incident response."

Thank you for pointing this out. In our evaluated setup, the scanner takes 3–5 minutes per run, compared with 20–40 minutes for an agent. Agent runtime varies with API latency and rate limits. Figure 8(a) reports average steps, input and output tokens, and cost per run. For example, Qwen3.7 Plus averages 16.8 steps and 0.92 million tokens per run, scoring comparably to the Claude series for under one dollar. Agents take longer and cost more but produce investigation reports and remediation plans, whereas the scanner returns a static findings list.

Q2: "The training experiments focus on the Qwen 3.5 family. Could the authors provide a preliminary evaluation on a model from another family to see whether RETrace generalizes beyond Qwen? I understand that full RL experiments can be expensive, so even a small-scale experiment would be helpful."

Please refer to the response to W3.

Q3: "I noticed that Claude Opus 4.7 refuses the NPM-Worm scenario and is evaluated on only nine ranges. Given the growing interest in modifying model safety guardrails for cybersecurity applications, I am curious how frequently such refusals occur across the evaluated models..."

Thanks for the detailed review. Across our experiments, the only refusal we observed is Claude Opus 4.7 on NPM-Worm, where it declined every attempt and produced no report. We therefore measure average performance over the other nine ranges for this model. On the judge side, Claude Opus 4.7 also declined to grade the NPM-Worm reports, so for this range we replace the Claude judge with Opus 4.6, and keep the other two judges unchanged.

Q4: "Could the authors clarify how much the training and held-out ranges overlap in terms of ATT&CK techniques, CVEs, software stacks, and persistence mechanisms..."

To clarify the overlap, we add Table 22 in Appendix F.5 to list the ATT&CK techniques and persistence mechanisms that appear in only one fold. We therefore also examine the 114-checkpoint subset (approximately 41% of all checkpoints) selected for components absent from the corresponding training fold. On this subset, RETrace improves over SFT by 4.3% on Detection and 4.2% on Planning and over GRPO by 10.6% and 10.7%, consistent in direction on both folds, which is shown in the table below.

MethodOOD Fold A DetOOD Fold A PlanOOD Fold B DetOOD Fold B PlanAverage DetAverage Plan
Qwen3.5-9B (base)49.8 ± 3.925.0 ± 2.450.6 ± 1.931.4 ± 1.850.228.2
Qwen3.5-9B + skill prompt46.6 ± 2.533.9 ± 2.048.2 ± 2.232.9 ± 1.247.433.4
SFT (skill-guided trajectories)57.5 ± 3.243.0 ± 3.248.8 ± 1.436.7 ± 2.453.139.9
SFT + GRPO48.0 ± 2.034.0 ± 3.145.9 ± 4.132.8 ± 3.247.033.4
SFT + RETrace60.9 ± 1.848.0 ± 3.154.0 ± 3.040.2 ± 1.157.544.1
Reviewer_Mj3GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SecRespond, a benchmark for evaluating whether LLM agents can autonomously carry out the end-to-end post-compromise incident-response workflow. The workflow runs from offline host forensics to remediation planning, inside a CLI-based interactive environment. The benchmark has 10 cyber ranges. Each provides a forensic disk snapshot together with alerts, vulnerability scans, and baseline checks from a host security product. Three frontier models act as LLM judges and score agent outputs against range-specific checkpoints along two axes, Detection and Planning. The authors evaluate 23 LLMs under the OpenCode harness. They find that models handle issues directly exposed by alerts reasonably well, but struggle to proactively uncover silent intrusion traces and to produce complete remediation plans. Building on these findings, the authors propose a two-stage training recipe. First, the model is fine-tuned on skill-guided teacher trajectories (SFT). Second, it is trained with RETrace, which anchors each rubric checkpoint’s score to the corresponding report span and traces detection credit back to the investigative step that surfaced the evidence. Most training and ablation experiments use Qwen3.5-9B, and the method is further validated on Qwen3.5-27B.

Critical strengths
  1. Well-motivated problem and task design. Existing cybersecurity benchmarks mostly target pre-compromise settings such as CTF challenges, vulnerability exploitation, or patching. End-to-end post-compromise incident response has not been evaluated systematically. SecRespond requires agents to actively inspect the disk and correlate evidence across files. It also deliberately lets only 30–40% of attack actions trigger alerts. This design separates agents that merely follow alerts from those that conduct proactive root-cause investigation. Covering both Linux and Windows hosts with several entry-point types adds further diversity.

  2. Well-structured hierarchical evaluation and large-scale experiments. Detection (discovery, evidence, attribution) and Planning (correctness, completeness) are scored separately and then mapped to five capability dimensions. This makes it possible to tell whether a failure happens during investigation or during remediation planning. The authors use three judges from different model families and analyze their agreement and self-preference. A human expert also blindly graded 60 checkpoints spanning all ranges. Together, these give useful initial support for the scoring protocol. The evaluation of 23 models, with per-checkpoint results and cost analysis, is extensive.

Critical weaknesses
  1. Small benchmark scale and limited statistical robustness. All conclusions rest on 10 single-host ranges. In 8 of them, the attack chain ends in cryptomining, and only 2 involve Windows. This limits how well the results represent other incident types (e.g., ransomware, data exfiltration, multi-host lateral movement) and other system environments. Training uses a two-fold split: each run trains on five fixed ranges and tests on the other five. The results therefore mainly show transfer across ranges within this benchmark. The in-domain (ID) results measure performance on ranges whose rubrics were used as training rewards, so they are not evidence of generalization. More importantly, Appendix F.1 states that each configuration comes from a single training run. The reported standard errors reflect only three inference runs and do not capture training variance. RETrace's OOD gains over SFT (+5.4 / +5.0 points, Table 3) are only a few times larger than these inference-only standard errors. Without multiple training seeds or a significance test, it is unclear whether these gains are robust.

  2. Concerns about evaluation validity. According to Appendices C.3 and C.4, the main evaluation has LLM judges read the generated reports and score them against a checklist. The judge is given only the checklist and the reports. The paper does not say whether the file paths, log lines, and configurations cited in the reports are checked against the raw disk snapshot or the agent's tool outputs. The rubric penalizes fabricated evidence, wrong attribution, and dangerous advice. However, it does not explain how false positives, or valid findings outside the checklist, affect the score. Because the same rubric is used as the training reward, a model might raise its score by covering more rubric items rather than by investigating more accurately.

    In addition, the Planning axis mainly assesses whether the written plan is technically correct, complete, and includes verification steps. No execution-based validation of the generated plans is reported. This supports comparing plan quality as text. It does not show that the plans eradicate the threat while keeping services available. The phrase "verified remediation plans" in the abstract and conclusion should therefore be qualified. Finally, the Windows task prompt (Appendix C.2) includes detailed forensic guidance and execution-risk constraints that the Linux prompt (Appendix C.1) lacks. If cross-OS differences are to be interpreted as capability differences, the authors should test how this extra guidance affects results.

Questions for authors

Please address the questions raised in W1–W2.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Mj3G [Part I]

We sincerely thank the reviewer for the thoughtful and constructive feedback.

Benchmark scale: "All conclusions rest on 10 single-host ranges...This limits how well the results represent other incident types."

Thank you for raising this concern. We would like to clarify that each range presents a host-wide investigation space, with artifacts scattered across system, service, and user paths, while the evaluation within each range is much richer than the range count suggests. The number of scenarios in our benchmark is comparable to prior agent benchmarks built on realistic attack environments. For example, ExCyTIn-Bench [1] evaluates cyber threat investigation across eight multi-stage attack scenarios. In our benchmark, every range is produced by executing a multi-stage attack against a deployed vulnerable stack through a real, publicly known weakness. The ten ranges span four entry-point types and over 15 persistence mechanisms, with 280 checkpoints covering 52 capability items and 21 ATT&CK techniques. Each range has 20–45 checkpoints. Examples beyond cryptomining include exfiltration (Log4j-RCE CHK-05), credential dumping (Jenkins-RCE CHK-09), worm propagation (NPM-Worm CHK-08), and log wiping (Shiro-Fastjson CHK-32). This depth supports detailed evaluation of the covered forensic capabilities, while the incident distribution remains concentrated on ten single-host ranges, including two Windows environments and predominantly cryptomining outcomes. Extending the benchmark to ransomware and other incident types is an important direction for future work.

[1] ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation, ICML 2026

Generalization beyond the training ranges: "Training uses a two-fold split...they are not evidence of generalization."

The evidence of generalization comes from out-of-domain ranges separated from training and analysis of unseen attack characteristics. Given the construction cost of each range, we adopted two-fold cross-validation. The two folds are trained independently, and OOD evaluation covers all ten ranges.

Generalization is assessed on the held-out ranges, since OOD ranges include different entry vectors, software stacks, CVEs, and persistence mechanisms that never appear in the training fold. We additionally analyzed 114 held-out checkpoints, approximately 41% of all 280 checkpoints, selected for components absent from the corresponding training fold (Table 22). On these checkpoints RETrace achieves mean detection and planning scores of 57.5 and 44.1, outperforming SFT and GRPO in both folds, as shown below. Our method transfers to components absent from the training fold, with consistent improvements across both folds. We add this analysis to Appendix F.5 as well.

MethodOOD Fold A DetOOD Fold A PlanOOD Fold B DetOOD Fold B PlanAverage DetAverage Plan
Qwen3.5-9B (base)49.8 ± 3.925.0 ± 2.450.6 ± 1.931.4 ± 1.850.228.2
Qwen3.5-9B + skill prompt46.6 ± 2.533.9 ± 2.048.2 ± 2.232.9 ± 1.247.433.4
SFT (skill-guided trajectories)57.5 ± 3.243.0 ± 3.248.8 ± 1.436.7 ± 2.453.139.9
SFT + GRPO48.0 ± 2.034.0 ± 3.145.9 ± 4.132.8 ± 3.247.033.4
SFT + RETrace60.9 ± 1.848.0 ± 3.154.0 ± 3.040.2 ± 1.157.544.1

Statistical robustness: "More importantly...it is unclear whether these gains are robust."

The existing Qwen results show gains across both folds and model scales in Table 3. RETrace improves OOD detection/planning over SFT on both folds at 9B and 27B scales, with average gains of 5.4%/5.0% and 6.4%/6.5%, respectively. We additionally conducted t-tests over scores of inference runs, finding that all improvements of RETrace over SFT and GRPO yielded p < 0.03.

To provide additional evidence across training seeds, we also conducted a Gemma-4-E4B experiment with three training seeds and evaluated it on eight OOD Linux cyber ranges. RETrace improves average detection from 53.4% to 56.6% and planning from 41.8% to 44.5%. This demonstrates improvements under repeated training.

MethodMetric (%)Next.js-RCENPM-WormShiro-FastjsonSSH-MinerDocker-EscapeJenkins-RCELog4j-RCERedis-RCEAverage
SFTDetection48.0 ± 18.340.5 ± 2.724.8 ± 6.074.9 ± 3.863.4 ± 8.843.3 ± 2.783.3 ± 0.349.1 ± 2.553.4 ± 2.8
SFTPlanning38.1 ± 12.934.0 ± 3.721.1 ± 2.865.7 ± 3.543.0 ± 8.325.4 ± 3.162.0 ± 3.345.1 ± 4.541.8 ± 2.2
SFT + RETraceDetection61.8 ± 2.844.2 ± 5.328.5 ± 5.579.2 ± 1.466.7 ± 1.337.4 ± 4.283.8 ± 0.451.4 ± 6.856.6 ± 1.4
SFT + RETracePlanning49.4 ± 2.639.1 ± 1.921.9 ± 5.567.6 ± 2.438.6 ± 3.126.3 ± 3.363.9 ± 4.749.0 ± 2.144.5 ± 1.2
AuthorsOpenReview ↗

Response to Reviewer Mj3G [Part II]

Evaluation validity: "According to Appendices C.3 and C.4, the main evaluation has LLM judges read the generated reports and score them against a checklist. The judge is given only the checklist and the reports. The paper does not say whether the file paths, log lines, and configurations cited in the reports are checked against the raw disk snapshot or the agent's tool outputs."

We would like to clarify that the judge can check the evidence in the report against the disk through the checklist. Each checklist records reference artifacts derived from the attack blueprint, post-attack snapshot, and security-product findings. The judge compares report citations against these references. We additionally audited the grading of the Evidence sub-score. For checkpoints whose reference evidence is a concrete file path, 97.1% of audited positive Evidence cases correctly cited the corresponding disk artifact.

False positives and findings outside the checklist: "The rubric penalizes fabricated evidence, wrong attribution, and dangerous advice. However, it does not explain how false positives, or valid findings outside the checklist, affect the score. Because the same rubric is used as the training reward, a model might raise its score by covering more rubric items rather than by investigating more accurately."

The rubric checklist specifies the evaluation targets for each attack scenario, and findings outside these targets do not affect the score. Within each checkpoint, naming a finding earns only the Discovery point. Evidence and Attribution require a concrete citation and correct attribution. False findings identified by the judge are subject to explicit deduction rules, and findings outside the checklist receive no additional checkpoint credit.

Thus, just covering more items without investigating earns less reward, while a more effective way to raise the score is to report grounded findings and correlate intrusions. The training results demonstrate improvement in investigation behavior. On OOD ranges, RETrace improves over SFT by 7.3% on Evidence and 3.0% on Attribution, as shown in Table 21. We also examine investigation behavior independently of rubric grading. The number of required evidence items retrieved in the agent's tool outputs also increases (Appendix F.4) while response length does not increase. This indicates that the policy learned to ground its findings rather than to cover more items.

Planning axis: "In addition, the Planning axis mainly assesses whether the written plan is technically correct, complete, and includes verification steps. No execution-based validation of the generated plans is reported. This supports comparing plan quality as text. It does not show that the plans eradicate the threat while keeping services available. The phrase "verified remediation plans" in the abstract and conclusion should therefore be qualified."

SecRespond positions the agent as an assistant to the security team. The task requires the agent to reconstruct the intrusion, assess risks, and deliver a remediation plan for analyst review, since remediation execution on a compromised production host is destructive and should stay under human control. The Planning rubric evaluates Correctness and Completeness, including whether proposed steps address the identified artifacts, specify verification, and account for side effects, assessing plan quality rather than observed recovery.

Prompts for ranges on Windows OS: "Finally, the Windows task prompt (Appendix C.2) includes detailed forensic guidance and execution-risk constraints that the Linux prompt (Appendix C.1) lacks. If cross-OS differences are to be interpreted as capability differences, the authors should test how this extra guidance affects results."

Thanks for your detailed review. Our benchmark does not interpret differences between Windows and Linux ranges as evidence of capability differences specific to the operating system. All models receive the same prompt within a range, and Windows and Linux ranges use different prompts and environments. The additional section in the Windows prompt exists because the Windows disk is provided as a directory tree on a Linux analysis machine. The prompt explains how to read these artifact classes with Linux tools with no leakage of exact path, file, payload, or attack vector.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response Summary

Dear Reviewers, ACs, and PCs,

We sincerely thank all four reviewers for their careful and constructive feedback. We are glad that the reviewers recognized SecRespond’s contribution to evaluating forensic investigation and remediation on actual compromised environments (cRrh,QDrV), highlighted its well-structured hierarchical evaluation and large-scale experiments (Mj3G), and values the distinction between alert-driven and proactive investigation (QDrV). We also appreciate that the reviewers found that our training experiments provide convincing evidence of the benefits of the proposed approach (6VDJ). Below, we summarize the additional analyses, experiments, and clarifications addressing the reviewers’ main concerns.

  1. Checkpoint reliability and investigation quality. (Reviewers Mj3G, 6VDJ, cRrh, QDrV)
  • On the benchmark side, the checkpoints are derived from the blueprint, the post-attack snapshot, and the security-product findings and are audited by experts for completeness. Expert and judge scores show strong agreement across 600 checkpoint–trajectory assessments.
  • On the training side, we also examine investigation quality through trajectory evidence and silent-artifact detection. RETrace agents retrieve 19.0 required evidence items on held-out ranges, compared with 17.9 for SFT and 16.8 for GRPO, without more tool calls or longer reports.
  1. Benchmark scale. (Reviewers Mj3G, QDrV)
  • Each range presents a host-wide investigation space, with artifacts scattered across system, service, and user paths, while the evaluation within each range is much richer than the range count suggests.
  • The 10 ranges span 4 entry-point types and over 15 persistence mechanisms, with 280 checkpoints covering 52 capability items and 21 ATT&CK techniques.
  1. Training robustness across model scales, families, and seeds. (Reviewers Mj3G, 6VDJ, cRrh)
  • RETrace improves held-out detection and planning over SFT on both folds at both Qwen scales (9B and 27B).
  • We also conducted a three-seed experiment on Gemma-4, providing effectiveness on a second model family.
  1. Effectiveness of credit assignment in the proposed training method. (Reviewer QDrV)
  • We clarify that the additional credit is assigned to steps whose tool outputs supply an artifact that is subsequently cited as evidence for an accepted finding. Removing such traceback lowers held-out detection by 5.2%.
  • Evidence retrieved by credited actions can also support findings that require synthesis, even when the contribution of each intermediate reasoning step is not explicitly measured. We further analyze improvements on checkpoints that assess investigation scope, interpretation, or synthesis beyond direct artifact grounding and find that RETrace achieves 58.1% detection, compared with 46.1% for SFT and 45.0% for GRPO.
  1. Generalization to unseen attack components. (Reviewers Mj3G, 6VDJ, QDrV)
  • We used two-fold cross-validation within SecRespond to demonstrate the generalization of our proposed training method to unseen attack components given the substantial effort required to construct each range and the lack of an independently constructed public range for this evaluation.
  • We separately analyzed 114 held-out checkpoints involving components absent from the corresponding training fold, on which RETrace outperforms both SFT and GRPO on detection and planning in both folds.
  1. Scope of remediation evaluation. (Reviewers Mj3G, QDrV)
  • We clarify that each agent investigates a read-only snapshot and submits a written remediation plan. The planning scores assess the proposed actions and verification steps. Recovery after execution is outside the scope of this evaluation.
  1. Novelty relative to prior work. (Reviewer QDrV)
  • We clarify how SecRespond differs from previous benchmarks by allowing agents to execute investigative commands autonomously on a full disk snapshot of the compromised host.

Overall, SecRespond's contribution lies in benchmarking autonomous command-line investigation of a full host disk snapshot with capability-oriented evaluation, and proposing a training recipe that credits steps with key evidence. Our evaluation of 23 models shows that discovering intrusions beyond the alerts is a critical gap. Our training study offers an initial exploration of how to improve agent capabilities in realistic post-compromise incident-response scenarios. While RETrace improves model performance on SecRespond, the benchmark remains challenging for current agents. By releasing the benchmark, we hope to encourage further research on training methods that improve model performance on these tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.