acceptodds
Under review as a conference paper at ICLR 2027

CyFER-Bench: Cybersecurity Fault-trace Execution Reasoning Benchmark

Abstract

Large language model (LLM)-based agents are increasingly used in cybersecurity, where they must interpret threat evidence, reason about relevant weaknesses and vulnerabilities, and translate those conclusions into actions in a target environment. Evaluating these agents therefore requires assessing both their stage-specific capabilities and whether those capabilities compose across the full workflow, while also evaluating whether LLMs can reason about their own failures. More broadly, cybersecurity provides a verifiable setting for studying a fundamental question about LLM agents: can models apply available knowledge correctly in context, translate it into effective action, and diagnose failures in their own reasoning? We present CyFER-Bench, a linked benchmark spanning (1) malware-report tactics, techniques, and procedures (TTP) extraction, (2) Common Weakness Enumeration (CWE) identification and explanation, (3) executable weakness exploitation, (4) multi-stage attack planning and execution, and (5) attacker-intent inference. CyFER-Bench provides validated intermediate targets and sandboxed execution, enabling stage-wise evaluation of knowledge, contextual reasoning, planning, action, and failure localization. Our findings highlight recurring challenges for current LLMs on complex multi-stage tasks: strong performance at one stage does not reliably carry to the next, local knowledge does not always translate into coherent reasoning, planning, and action, and models struggle to identify where their own reasoning fails.

Primary area:
datasets and benchmarks
Keywords:
datasets and benchmarks, large language models, multi-step reasoning, execution-based evaluation, cybersecurity, vulnerability detection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9pTSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CyFER-Bench introduces five cybersecurity corpora and an evaluation framework covering malware-report interpretation, weakness and vulnerability reasoning, executable exploitation, attack planning, and attacker-intent inference. It provides intermediate evaluation targets and executable checks to examine relationships between identification, explanation, planning, action, and failure localization. The paper reports that strong performance at an individual stage does not consistently transfer to downstream execution, and that models struggle to diagnose their own reasoning failures.

Critical strengths
  1. The evaluation framework addresses a useful problem. Separating identification, explanation, planning, execution, and failure localization provides a way to study where model capabilities fail to compose. Intermediate targets can support a more informative analysis than final task success alone. This is a meaningful contribution for a dataset-and-benchmark paper.

  2. The resource contribution is substantial and inspectable. The release includes 20,07020{,}070 labeled malware reports, 311311 executable CWE instances, and 196196 scenario–attack-graph pairs, together with preprocessing code, evaluation scripts, and saved experimental records. Local checks could execute reference tests and reproduce stored aggregates from cached per-item scores. These artifacts provide a concrete basis for reuse and further analysis, although the checks do not constitute a full reproduction of the model and agent experiments.

Critical weaknesses
  1. The released malware inputs do not match the stated CAPA-removal protocol. The paper states that all CAPA-derived content is removed from model-visible inputs. However, 20,06720{,}067 of the 20,07020{,}070 released preprocessed reports contain CAPA rule-match sections. The released preprocessing code explicitly includes CAPA-derived descriptions and triggers, and a local prompt-assembly check preserves this information. Explicit ATT&CK identifiers were not found, so the observation does not establish direct copying of label IDs. Nevertheless, descriptions produced by the same tool that supplies the labels may provide substantial predictive cues. The authors need to establish which input version was used and quantify the effect of removing this information before the results can be interpreted as inference from behavioral evidence alone.

  2. The planning and exploitation measures need stronger baseline comparisons and validation against false positives. On all 196196 released scenarios, a baseline simply lists hosts with nonempty vulnerability fields, V(h)≠∅V(h)\neq\varnothing, preserving their existing XML order. Here, V(h)V(h) denotes the vulnerabilities listed for host hh. Under the released scorer, the baseline's mean precision PP, mean recall RR, and exact-match rate EM\mathrm{EM} compare with the respective best reported Table 8 values as follows:

    \text{Heuristic:}\quad (P,R,\mathrm{EM}) &= (1.000,\\;0.717,\\;0.173),\\\\ \text{Table 8 (best per metric):}\quad (P,R,\mathrm{EM}) &= (0.995,\\;0.710,\\;0.138). \end{aligned}$$ The heuristic recovers the exact reference path in $34$ of $196$ scenarios, giving $\mathrm{EM}=34/196\approx0.173$. This comparison needs confirmation against the exact experimental version, but it already raises a question about what additional planning capability the model scores demonstrate. Separately, the CWE stdout-marker criterion can count ordinary output as successful exploitation. For example, with unchanged source, compile flags, and grader, `CWE78_v24` accepts the ordinary input `CWE78_PWNED`, which does not inject an additional command. The effect of such cases on exploitation scores and cross-stage associations has not been established.

These concerns affect the interpretation of the benchmark's measurements, rather than only its presentation. I therefore currently lean toward weak rejection despite the value of the resources. I would raise my rating to weak acceptance if the authors resolve the input discrepancy, establish the planning results relative to the simple baseline, and validate or correct the exploitation scoring, with evidence that the main conclusions remain supported.

Questions for authors
  1. Does the released malware preprocessing and report snapshot correspond to the experiments in the paper? If so, how do TTP results change after removing CAPA-derived rule descriptions and triggers, rather than only explicit ATT&CK identifiers? If a different version was used, can the authors provide the exact preprocessing and model-visible input version?

  2. Can the authors verify the XML-order baseline above using the exact Table 8 evaluation version and include the comparison? What do the model results demonstrate about planning beyond this heuristic, and should the interpretation of the planning results be revised?

  3. Are CWE exploitation outcomes validated independently of the presence of a stdout marker? Please explain the CWE78_v24 example and quantify whether correcting or excluding affected cases changes the exploitation results or the reported cross-stage associations.

2 comments

We thank the reviewer for their thorough evaluation of our work and constructive suggestions.

W1/Q1: CAPA removal Protocol: The repository contained an outdated preprocessing script and report snapshot, which we have now corrected. The pipeline used for all reported results excludes CAPA-derived content from the model-visible input, as described in the paper. To quantify the effect, we ran both conditions on the same subsample of n = 100 reports, with the same prompt and the same decoding settings, so the only difference is whether CAPA content is present in the input. We ran this test with Qwen3.8-27B, we can see that if we expose the CAPA content the TTP F1 beats the CAPA excluded content by 24.7 percent which indicates our result did not get exposed to the CAPA content. In the github we added samples which are used as the input (link: https://anonymous.4open.science/r/CyFERBench-5E12/malware_analysis/dataset/0011d0a5216886d96a23ad206d4e4badf1dbcad6c8050ff778d4e24537089422.txt)

ConditionTTP F1 (mean ± std)
CAPA content retained44.9 ± 2.42
CAPA content excluded (as reported)20.2 ± 1.35

W2/Q2: XML-order baseline clarification: Listing vulnerable hosts in XML order does not necessarily yield a valid attack plan, as planning must account for network connectivity, reachability, and exploitation dependencies. We retained vulnerability information because the planning task assumes an attacker has some prior knowledge of the target environment and must use that knowledge to construct a feasible attack path. Our preprocessing removes ground-truth attack-chain information (metadata.flow and metadata.flow_flag). To investigate the reviewer's concern, we conducted an ablation removing host vulnerability details from the XML. The exact-match (EM) scores remain comparable with and without these details, suggesting that the reported planning performance is not primarily driven by the explicit vulnerability annotations.

ModelEM, With Vuln. FieldsEM, Vuln. Fields Removed
GPT-OSS-20B0.0790.051
Qwen3.8-27B0.0560.066
DeepSeek-v4-pro0.0360.087
GLM-5.3-Flash0.1380.143
Claude-Sonnet-4.50.0660.082
GPT-50.0410.046

W2/Q3. CWE exploitation/ablation: In our evaluation (Line 1148), successful exploitation required the attacker to introduce shell operators (e.g., || or ;) to execute an additional command. Therefore, providing CWE78_PWNED as ordinary input would not satisfy the intended exploitation condition. For every CWE in the suite it verifies the case-specific bypass condition, such as an injected shell operator here. No run can therefore be counted as a success from ordinary program output. To test our grader we created a negative control test where we provided ordinary inputs containing the expected success marker. This establishes the experimental grader distinguishes genuine exploitation from ordinary program output.

Input typeExpected resultVerified result
Ordinary input0/250/25
Plain CWE78_PWNED0/250/25
payload with injected (e.g., || or ; or &&)25/2525/25

We thank the reviewer again for their valuable feedback. We have made every effort to address all comments and concerns and would be happy to provide further clarification if needed.

Reviewer_7hGDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CyFER-Bench, a benchmark for testing LLMs on different cybersecurity tasks, such as threat analysis, finding vulnerabilities, exploiting vulnerabilities, attack planning, and understanding attacker intent.

The benchmark combines five datasets and evaluates several LLMs and agents. The main finding is that models can often identify and explain vulnerabilities, but they struggle to exploit them successfully. Models also have difficulty finding mistakes in their own reasoning.

Critical strengths
  • The paper addresses an important problem. Understanding where LLM-based cybersecurity agents fail is useful.
  • The benchmark covers several cybersecurity tasks and includes a good amount of data.
  • Using real execution results to check whether an exploit works is a strong part of the evaluation.
  • The results are interesting, especially the gap between understanding a vulnerability and actually exploiting it.
Critical weaknesses
  1. The full end-to-end workflow is not tested: One of the main goals of the paper is to test whether LLMs can go from a threat report to successfully exploiting a vulnerability. However, in Section 5.1, the authors use the correct CVEs for the downstream agent experiments instead of the CVEs predicted by the models.

I understand why the authors made this choice, but it means the paper does not fully test its proposed workflow. It would be useful to see the results when the models' actual predictions are used, even if the success rate is low.

  1. Some datasets may not reflect real-world cybersecurity tasks well: A large part of the benchmark uses synthetic examples from existing datasets like Juliet. Also, two vulnerability types make up almost 80% of the CWE explanation dataset. Another concern is that only 49 examples were reviewed by human experts across all five datasets. The agreement between experts on task difficulty was also low. More testing on real-world examples and a larger human review would help show that the benchmark is reliable.

  2. The link between planning and execution is not clearly shown: In Section 5.4, the authors compare attack-planning scores with execution performance. However, the agents do not actually use the plans generated by the models. Because of this, the results do not show whether better plans lead to better execution. It would be helpful to test whether agents perform better when they are given these plans.

  3. The novelty is somewhat limited: The idea of combining different cybersecurity tasks into one benchmark is useful. However, most of the tasks, datasets, and evaluation methods are based on existing work. The paper would be stronger if the authors showed more clearly how their benchmark differs from earlier benchmarks and what new findings it makes possible.

Questions for authors

(in addition to the comments above)

  • Can the authors report end-to-end results using the CVEs predicted by the models instead of the correct CVEs?
  • How would the results change if the benchmark used more balanced datasets and more real-world examples?
  • Does giving the generated attack plans directly to the agents improve their execution success?
3 comments

We appreciate the reviewer’s careful assessment and helpful feedback.

W1 / Q1. End-to-end composition and predicted-CVE exploitation

CyFER-Bench is a sequential pipeline, and in a fully propagated workflow, the CVE identified by the LLM reasoning stage is passed directly to the downstream agent. From our findings in Table 4, correct CVE identification at the end of the pipeline is sparse; so evaluating only these cases would result in a small and uneven sample. However, we acknowledge that the full end-to-end workflow provides a useful complementary perspective. We report here results for the fully propagated pipeline where the final stage is prompted with the LLM-identified CVE (both correct and incorrect predictions) for Qwen3.8-27B and GPT-5.

Exploitation score (0–100)
ConditionQwen3.8-27BGPT-5
Unguided47.0 ± 40.715.0 ± 9.4
Guided (ground-truth CVE)78.0 ± 17.940.0 ± 13.7
Guided (CVE predicted by model)62.5 ± 21.721.7 ± 4.7

W2 / Q2. Dataset realism and human validation

Dataset Realism: Juliet is used only for CWE-level tasks; the benchmark also includes real malware and CVEs evaluated in executable environments. More importantly, we directly tested generalizability against real-world AIxCC vulnerabilities (Appendix A.3). Model rankings remained strongly correlated with Juliet-based results (Spearman ρ = 0.672 for identification; 0.800 for explanation), suggesting that our findings are not artifacts of synthetic data.

Class Imbalance: CWE-78 and CWE-134 dominate (80.1%) because static analyzers produced more consistent and verifiable explanations for these classes (Table 19). For other CWEs, static analyzers often disagreed on weakness descriptions, making reliable ground-truth construction difficult. We therefore prioritized label reliability over class balancing.

Human Validation: The 49-item stratified sample was evaluated by two external PhD-level experts for answerability and difficulty. Agreement was moderate for answerability (Gwet’s AC1 = 0.44) and lower for difficulty (AC1 = 0.23). We believe these values reflect meaningful consensus on a domain-specific task where expert interpretations naturally vary. However, expert disagreement on difficulty does not imply unreliable ground truth. CyFER-Bench uses automated, validated labeling pipelines and executable grading, independent of human difficulty ratings. Additional spot-checks of 50 CWE findings confirmed 100% label accuracy (lines 1050–1054). Importantly, difficulty is subjective and may differ between humans and LLMs. To illustrate this, we additionally report Claude-Sonnet-4.5 ratings on the same 49 items:

CorpusRaterToo EasyCalibratedToo Hard
Overall (n=49)Human Experts24.3%62.0%13.7%
Overall (n=49)Claude Sonnet36.7%61.2%2.0%
Malware (n=10)Human Experts10.1%81.0%9.0%
Malware (n=10)Claude Sonnet20.0%80.0%0%

W3 / Q3. Planning–execution relationship and plan-conditioned exploitation

As suggested by the reviewer, we ran a plan-conditioned exploitation experiment using our best-performing PentestAgent and best-performing model (Qwen3.8-27B).

Exploitation score (0–100)
ConditionScore
Guided (ground-truth CVE)78.0 ± 17.9
Guided (model-predicted CVE)62.5 ± 21.7
Unguided47.0 ± 40.7
LLM-generated Path55.0 ± 18.7
GT Path68.75 ± 24.2

W4. Novelty relative to existing cybersecurity benchmarks

CyFER-Bench changes what constitutes a correct cybersecurity outcome. Existing CTI benchmarks assess whether a model understands reported threats; CWE/CVE benchmarks assess whether it identifies or explains vulnerabilities; and offensive benchmarks assess whether it successfully exploits a target. CyFER-Bench introduces a different criterion: whether the vulnerability identified and exploited is the one justified by the original threat evidence. As motivated in the Introduction, this reflects a critical real-world challenge: determining whether a newly reported threat applies to an organization's own systems, even when the underlying vulnerability is not yet fully understood. This setup produces qualitatively new findings about model competence. By linking threat evidence, vulnerability identification, and execution outcomes, we show that strong performance on individual cybersecurity tasks does not necessarily translate into coherent reasoning or successful execution across the workflow (Tables 3–7). This is supported by the five corpora which are all original contributions, including executable multi-stage attack scenarios and real attacker execution traces paired with ground-truth intent. These enable evaluation at a level of operational fidelity not supported by existing benchmarks.

We thank the reviewer again for their valuable feedback. We have made every effort to address all comments and concerns and would be happy to provide further clarification if needed.

The experiments further validate the gaps that I am highlighting.

  1. Table 3 shows that the exploitation accuracy of GPT5 (33.3) is higher than Qwen (14.8). However, in the end to end experiments shown in the rebuttal CVE exploitation accuracy of GPT 5 (21.7) is much lower than Qwen (62.5). No explanation is provided for this.

  2. The authors explain why there is a class imbalance (difficulty in obtaining labels for other classes). However, that is not justified enough that the benchmark is still skewed and we may not be able to conclude LLM's exploitation capability with this benchmark

  3. The novelty is still not explained sufficiently - this is an empirical evaluation paper by combining existing approaches on existing combination of vulnerability datasets. While some of the experimental analysis is good and useful for the community, it doesnt qualify as novel contribution.

  4. The plan conditioned experiments are helpful. However, the improvement in accuracy from 47% to 55% is not significant, especially when there is ~18-40% deviation in accuracy numbers across experiments

  5. I thank and appreciate the authors for the comparison against AIxCC and the human labellers. However, the human validation reveals that there is moderate to low agreement, further bolstering that the benchmark may not have real-world difficulty as well as real-word distribution of vulnerabilities.

I thanks the authors for the experiments and detailed rebuttal. I retain my original score.

Reviewer_FXueOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CyFER-Bench, a benchmark suite for evaluating LLM and agent capabilities across a linked "threat evidence workflow": malware report → MITRE ATT&CK TTPs → CWEs → CVEs → executable actions. The contribution consists of five interconnected corpora, plus an evaluation of ~6–10 frontier and open-weight models and three pentesting agents across five research questions. The headline findings are: (i) standalone knowledge is strong but does not compose: stage-conditional precision collapses to near zero at CWE→CVE in the linked chain; (ii) explanation ability correlates with identification but rarely with exploitation; (iii) self-diagnosis of the first point of failure is weak and execution feedback does not reliably help; (iv) attack-planning recall, but not precision, correlates with downstream execution; (v) attacker-intent inference correlates with planning but not consistently with exploitation. The benchmark is positioned as a stage-wise, verifiable complement to end-to-end CTF-style evaluations.

Critical strengths
  1. The linked, stage-wise design fills a real and increasingly important gap. Existing evaluations measure either static security knowledge or end-to-end exploit success, from which intermediate reasoning is invisible. By linking TTP→CWE→CVE→execution with validated intermediate targets and scoring each stage conditionally on the preceding one, the paper can attribute failures to specific capability breakdowns rather than reporting a single opaque success rate. This is exactly the kind of evaluation the field needs as LLM agents are deployed in security operations.
  2. Unusually rigorous dataset construction and validation for an offensive-security benchmark. Each corpus has verifiable ground truth: TTP labels from CAPA with explicit leakage removal; Flawfinder-filtered Juliet cases with 100% label spot-checks; exploit instances that must compile, reproducibly trigger, and satisfy a runtime verification condition; scenario–attack-graph pairs validated through ScenarioForge/CORE preflight and runtime checks; and intent labels from instrumented real pentest sessions. The external validation on AIxCC and the expert answerability/difficulty assessments go beyond typical practice and substantially increase my confidence that measured effects reflect model capability rather than benchmark artifacts.
Critical weaknesses
  1. The headline linked-chain statistics appear to rest on very small, unreported stage-conditional samples, making the central quantitative claims fragile.
  2. The flagship "linked workflow" is never evaluated end-to-end. The paper's central claim is that it evaluates a composed threat-to-action pipeline, but the agentic exploitation stage feeds agents the ground-truth CVE rather than the CVE the model's own reasoning chain produced. The paper therefore demonstrates that stages fail to compose, but does not actually measure the composed system. Even a small-scale fully-composed result (model-predicted CVE → agent, on the surviving chains) would substantially strengthen or qualify the central narrative.
  3. Self-diagnosis results lack a trivial-baseline comparison. The paper itself observes that models repeatedly attribute failures to the same stage (GLM-5.3-Flash → CWE, Claude-Sonnet-4.5 → TTP). A majority-class baseline ("always answer CWE") would likely score comparably to several reported models, but no such baseline is reported. Without it, "self-diagnosis is weak (F1 < 0.30)" and "execution feedback helps/hurts" cannot be cleanly attributed to diagnostic capability versus guessing strategy.
Questions for authors
  1. Sample sizes: For Tables 4, 7, and 10, how many chains survive to be scored at each stage, per model? With near-zero rates and ±3–8 point intervals, are the stage-level differences between models statistically meaningful at all?
  2. Model identity and count: Section 5 states 12 models, but Table 28 lists 10.
  3. Statistical procedure consistency: Section 5.5 states Spearman correlation for Table 9, but Appendix C.6.3 describes Pearson.
  4. End-to-end composition: The agentic evaluation uses the ground-truth CVE in the guided condition rather than the CVE produced by the model's own reasoning chain. Do you have any results for the fully composed pipeline (model-predicted CVE → agent), even on the small surviving sample?
  5. Contamination: Juliet, Vulhub, and VirusTotal reports are public and likely in pretraining corpora. Did you run any contamination/near-duplicate check against the evaluated models' training data, or evaluate recency-stratified subsets?
2 comments

We sincerely thank the reviewer for their thoughtful comments and careful evaluation of our work.

W1 / Q1. Stage-conditional sample sizes and uncertainty: All evaluations begin with 30 complete workflows. The table below reports the number of chains surviving each stage in Tables 4, 7, and 10. The low survival counts highlight the difficulty of maintaining correctness across successive stages.

ModelTable 4: Report→TTPTable 4: TTP→CWETable 4: CWE→CVETable 7: Evaluated / EligibleTable 10: 2-step TTP→CVETable 10: 3-step TTP→CWETable 10: 3-step CWE→CVE
GPT-OSS-20B3/300/3-30/300/30/3-
Qwen3.8-27B3/300/3-30/300/30/3-
DeepSeek-v4-pro9/301/90/130/300/91/90/1
GLM-5.3-Flash6/300/6-30/300/60/6-
Claude-Sonnet-4.514/301/140/130/300/141/140/1
GPT-56/301/60/130/300/61/60/1

We wish to clarify that our main finding in Table 4 is the consistently low chain completion rate across all models, which remains meaningful despite the small number of surviving chains. For comparing individual task accuracy across models, we report results on the much larger full corpus in Table 2, providing a stronger statistical basis for those comparisons.

W2 / Q4. End-to-end composition: In Table 4, we report results for the full linked workflow, except the final exploitation stage. This is because very few samples are correctly identified in the penultimate stage of the workflow. Crucially, the “surviving chains” are cases where the model predicted the correct CVE, which by definition match the ground-truth CVE, making them equivalent to the ground-truth condition we used. We acknowledge, however, that the full end-to-end workflow remains meaningful, and we therefore additionally report results for the fully propagated pipeline where the final stage is prompted with the LLM-identified CVE (both correct and incorrect predictions) for Qwen3.8-27B and GPT-5.

Exploitation score (0–100)
ConditionQwen3.8-27BGPT-5
Unguided47.0 ± 40.715.0 ± 9.4
Guided (ground-truth CVE)78.0 ± 17.940.0 ± 13.7
Guided (CVE predicted by model)62.5 ± 21.721.7 ± 4.7

W3. Self-diagnosis and the majority-class baseline: We would like to clarify that we do not attribute the results in Table 7 to LLMs’ self-diagnostic capabilities. Rather, as the reviewer notes, our findings suggest that models often default to preferred-stage predictions, even when provided with the correct CVE or execution logs. To quantify this tendency, we computed two constant-prediction baselines (Always-TTP and Always-CWE) and compared their macro-F1 scores against model performance.

ModelCorrect-CVE (Macro F1)Execution-Log (Macro F1)
Model F1Always-TTPAlways-CWEModel F1Always-TTPAlways-CWE
GPT-OSS-20B0.036 ± 0.0330.316 ± 0.0100.061 ± 0.0290.000 ± 0.0000.333 ± 0.0000.000 ± 0.000
Qwen3.8-27B0.118 ± 0.0690.314 ± 0.0110.067 ± 0.0320.000 ± 0.0000.296 ± 0.0420.111 ± 0.081
DeepSeek-v4-pro0.241 ± 0.1270.284 ± 0.0190.137 ± 0.0360.162 ± 0.0830.284 ± 0.0190.137 ± 0.036
GLM-5.3-Flash0.274 ± 0.0890.298 ± 0.0160.108 ± 0.0360.250 ± 0.1680.296 ± 0.0420.111 ± 0.080
Claude-Sonnet-4.50.133 ± 0.1110.281 ± 0.0190.141 ± 0.0350.364 ± 0.0450.270 ± 0.0210.162 ± 0.034
GPT-50.290 ± 0.1090.313 ± 0.0120.069 ± 0.0330.295 ± 0.0900.315 ± 0.0110.062 ± 0.031

Q2–Q3: The revised PDF now correctly states 10 models in Section 5 (line 259; Table 28) and Spearman correlation for Table 9 in Appendix C.6.3 (lines 2428–2434).

Q5. Potential contamination: We used three probes[1]. For the VirusTotal report and CWE code completion probes, we selected 15 samples each and asked models to generate the second half from the first half. For CVE knowledge, models answered three questions about each CVE: the affected software, vulnerability mechanism, and CVSS score (22 CVEs; 66 questions). These results show varying levels of factual familiarity, however, table 6 of the paper shows that knowing a vulnerability’s description does not guarantee successful exploitation in a running environment.

ModelExact completions/ 30Software recall/22Full recall/22
GPT-OSS-20B0/3000
Qwen3.8-27B1/3000
DeepSeek-v4-pro2/30168
GLM-5.3-Flash0/30177
Claude Sonnet 4.50/301611
GPT-52/30107

[1] Deng et al. “Investigating Data Contamination in Modern Benchmarks for Large Language Models.” NAACL, 2024.

We thank the reviewer again for their valuable feedback. We have made every effort to address all comments and concerns and would be happy to provide any further clarification if needed.

Comments on the paper

1 comment

Dear PC/SAC/AC/Reviewers, We have carefully addressed all concerns raised by the reviewers and incorporated the corresponding changes into our revised manuscript. Below, we provide a summary of each reviewer's concerns alongside the resolutions taken in our updated rebuttal:

End-to-end workflow evaluation: CVEs are rarely correctly identified (Table 4). Hence, to have statistically meaningful results we completed the last part of the workflow using ground truth CVEs. However, to demonstrate the results from the end-to-end workflow, we reported the results from additional experiments using model-predicted CVEs.  For Qwen3.8-27B, the exploitation score is 62.5 with predicted CVEs, compared with 78.0 using ground-truth CVEs and 47.0 without guidance. More details in our responses to Reviewers FXue and 7hGD.

Benchmark novelty and research value: CyFER-Bench's main contribution is not the sum of combining smaller cybersecurity tasks (for which benchmarks exist). The novelty introducing a distinct evaluation objective: whether an agent can interpret a threat report, identify the vulnerability implicated by that threat, successfully exploit it and correspondingly detect if another agent is attempting to exploit the same. This is non-trivial and can not directly be evaluated from prior benchmarks that focus on open-ended vulnerability discovery and exploitation. Additionally, our experiments demonstrate how knowledge, contextual reasoning, planning, and execution fail to compose, highlighting research questions that isolated task benchmarks cannot adequately investigate. (Clarifications: response to Reviewer 7hGD.)

Dataset realism, class distribution, and human validation: We clarify that CyFER-Bench uses real malware reports, real vulnerabilities, and Docker-based networks with exploitable vulnerabilities throughout its pipeline. Juliet serves as a source-code exploitability benchmark, and we additionally evaluate on real-world AIxCC vulnerabilities, showing strong correlations in model rankings (Spearman ρ = 0.672 for identification and 0.800 for explanation). We also report expert agreement statistics to contextualize differences in human judgments. (Response to Reviewer 7hGD; Appendix A.3.)

Stage-conditional sample sizes and statistical reporting: We provide explicit per-model survivor counts across all stages, starting from 30 complete workflows. The small number of surviving chains demonstrates the difficulty of maintaining correctness across successive reasoning stages. (Detailed statistics: response to Reviewer FXue.)

Self-diagnosis baseline comparison: We added Always-TTP and Always-CWE constant-prediction baselines and observed that these simple baselines often match or outperform model predictions, reinforcing our observation that LLMs struggle to localize their reasoning failures, even with additional execution feedback. (Additional experiments: response to Reviewer FXue.)

Planning and execution relationship: We conducted additional plan-conditioned exploitation experiments. LLM-generated attack paths achieve an exploitation score of 55.0, compared with 47.0 for unguided execution and 68.75 using ground-truth paths. (Additional experiments: response to Reviewer 7hGD.)

CAPA information leakage: We corrected an outdated preprocessing script and report snapshot in the repository and clarified the CAPA-removal protocol used in the reported experiments. An additional controlled experiment on 100 reports shows that retaining CAPA-derived information increases TTP F1 from 20.2 to 44.9, a 24.7-percentage-point difference. (Additional analysis and corrected artifacts: response to Reviewer 9pTS.)

Attack-planning baseline and input leakage: We removed host vulnerability annotations and repeated the planning experiments. Exact-match scores remain broadly comparable, suggesting that the reported results are not primarily driven by vulnerability annotations. (Additional experiment: response to Reviewer 9pTS.)

Exploitation scoring reliability: We conducted additional negative-control tests to verify that ordinary inputs containing success markers are not counted as successful exploitation. Ordinary inputs and plain success-marker inputs resulted in 0/25 successes, while payloads with injected shell operators achieved 25/25 successes. These checks validate the intended exploitation condition for the tested cases. (Additional validation: response to Reviewer 9pTS.)

Potential benchmark contamination: We conducted additional memorization and vulnerability-knowledge probes, including report/code completion tests and questions covering 22 CVEs. Exact completions were limited to 0–2 out of 30 across models, while factual CVE recall varied. (Additional experiments: response to Reviewer FXue.)

We look forward to any further feedback and any other clarification we can provide regarding our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.