- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CyFER-Bench introduces five cybersecurity corpora and an evaluation framework covering malware-report interpretation, weakness and vulnerability reasoning, executable exploitation, attack planning, and attacker-intent inference. It provides intermediate evaluation targets and executable checks to examine relationships between identification, explanation, planning, action, and failure localization. The paper reports that strong performance at an individual stage does not consistently transfer to downstream execution, and that models struggle to diagnose their own reasoning failures.
Critical strengths
-
The evaluation framework addresses a useful problem. Separating identification, explanation, planning, execution, and failure localization provides a way to study where model capabilities fail to compose. Intermediate targets can support a more informative analysis than final task success alone. This is a meaningful contribution for a dataset-and-benchmark paper.
-
The resource contribution is substantial and inspectable. The release includes labeled malware reports, executable CWE instances, and scenario–attack-graph pairs, together with preprocessing code, evaluation scripts, and saved experimental records. Local checks could execute reference tests and reproduce stored aggregates from cached per-item scores. These artifacts provide a concrete basis for reuse and further analysis, although the checks do not constitute a full reproduction of the model and agent experiments.
Critical weaknesses
-
The released malware inputs do not match the stated CAPA-removal protocol. The paper states that all CAPA-derived content is removed from model-visible inputs. However, of the released preprocessed reports contain CAPA rule-match sections. The released preprocessing code explicitly includes CAPA-derived descriptions and triggers, and a local prompt-assembly check preserves this information. Explicit ATT&CK identifiers were not found, so the observation does not establish direct copying of label IDs. Nevertheless, descriptions produced by the same tool that supplies the labels may provide substantial predictive cues. The authors need to establish which input version was used and quantify the effect of removing this information before the results can be interpreted as inference from behavioral evidence alone.
-
The planning and exploitation measures need stronger baseline comparisons and validation against false positives. On all released scenarios, a baseline simply lists hosts with nonempty vulnerability fields, , preserving their existing XML order. Here, denotes the vulnerabilities listed for host . Under the released scorer, the baseline's mean precision , mean recall , and exact-match rate compare with the respective best reported Table 8 values as follows:
\text{Heuristic:}\quad (P,R,\mathrm{EM}) &= (1.000,\\;0.717,\\;0.173),\\\\ \text{Table 8 (best per metric):}\quad (P,R,\mathrm{EM}) &= (0.995,\\;0.710,\\;0.138). \end{aligned}$$ The heuristic recovers the exact reference path in $34$ of $196$ scenarios, giving $\mathrm{EM}=34/196\approx0.173$. This comparison needs confirmation against the exact experimental version, but it already raises a question about what additional planning capability the model scores demonstrate. Separately, the CWE stdout-marker criterion can count ordinary output as successful exploitation. For example, with unchanged source, compile flags, and grader, `CWE78_v24` accepts the ordinary input `CWE78_PWNED`, which does not inject an additional command. The effect of such cases on exploitation scores and cross-stage associations has not been established.
These concerns affect the interpretation of the benchmark's measurements, rather than only its presentation. I therefore currently lean toward weak rejection despite the value of the resources. I would raise my rating to weak acceptance if the authors resolve the input discrepancy, establish the planning results relative to the simple baseline, and validate or correct the exploitation scoring, with evidence that the main conclusions remain supported.
Questions for authors
-
Does the released malware preprocessing and report snapshot correspond to the experiments in the paper? If so, how do TTP results change after removing CAPA-derived rule descriptions and triggers, rather than only explicit ATT&CK identifiers? If a different version was used, can the authors provide the exact preprocessing and model-visible input version?
-
Can the authors verify the XML-order baseline above using the exact Table 8 evaluation version and include the comparison? What do the model results demonstrate about planning beyond this heuristic, and should the interpretation of the planning results be revised?
-
Are CWE exploitation outcomes validated independently of the presence of a stdout marker? Please explain the
CWE78_v24example and quantify whether correcting or excluding affected cases changes the exploitation results or the reported cross-stage associations.
2 comments
We thank the reviewer again for their valuable feedback. We have made every effort to address all comments and concerns and would be happy to provide further clarification if needed.
We thank the reviewer for their thorough evaluation of our work and constructive suggestions.
W1/Q1: CAPA removal Protocol: The repository contained an outdated preprocessing script and report snapshot, which we have now corrected. The pipeline used for all reported results excludes CAPA-derived content from the model-visible input, as described in the paper. To quantify the effect, we ran both conditions on the same subsample of
n = 100reports, with the same prompt and the same decoding settings, so the only difference is whether CAPA content is present in the input. We ran this test withQwen3.8-27B, we can see that if we expose the CAPA content the TTP F1 beats the CAPA excluded content by24.7percent which indicates our result did not get exposed to the CAPA content. In the github we added samples which are used as the input (link: https://anonymous.4open.science/r/CyFERBench-5E12/malware_analysis/dataset/0011d0a5216886d96a23ad206d4e4badf1dbcad6c8050ff778d4e24537089422.txt)W2/Q2: XML-order baseline clarification: Listing vulnerable hosts in XML order does not necessarily yield a valid attack plan, as planning must account for network connectivity, reachability, and exploitation dependencies. We retained vulnerability information because the planning task assumes an attacker has some prior knowledge of the target environment and must use that knowledge to construct a feasible attack path. Our preprocessing removes ground-truth attack-chain information (metadata.flow and metadata.flow_flag). To investigate the reviewer's concern, we conducted an ablation removing host vulnerability details from the XML. The exact-match (EM) scores remain comparable with and without these details, suggesting that the reported planning performance is not primarily driven by the explicit vulnerability annotations.
W2/Q3. CWE exploitation/ablation: In our evaluation (
Line 1148), successful exploitation required the attacker to introduce shell operators (e.g.,||or;) to execute an additional command. Therefore, providingCWE78_PWNEDas ordinary input would not satisfy the intended exploitation condition. For every CWE in the suite it verifies the case-specific bypass condition, such as an injected shell operator here. No run can therefore be counted as a success from ordinary program output. To test our grader we created a negative control test where we provided ordinary inputs containing the expected success marker. This establishes the experimental grader distinguishes genuine exploitation from ordinary program output.CWE78_PWNED