- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines eight cybersecurity benchmarks and systematically investigates how benchmark results are influenced by factors other than LLM performance, across four stages: dataset, prompts, inference procedures, and aggregation. It identifies 15 failure modes and confirms that all benchmarks have at least two failures, and uses 10 models to demonstrate that these failures cause changes in ranking. Based on this, it proposes a framework called cyeval-DB7B for measuring the cybersecurity performance of LLMs while controlling for these factors.
Critical strengths
S1. While there have been previous studies on factors that compromise benchmark reliability, this attempt to solve the problem by handling a wide range of failure modes with a common harness is pioneering.
S2. The fact that they have actually published a framework for evaluating a set of benchmarks with a common harness is commendable.
Critical weaknesses
W1. Many of the failure modes discussed are already known, and there is little new insight gained from systematically evaluating multiple failure patterns. While some aspects, such as the denominator issue, are novel, this contribution to ICLR is weak. We would like to see a claim beyond simply stating that "all benchmarks have multiple failure patterns" by looking at multiple failure patterns together. However, there is also the expectation that this can be claimed based on existing experimental results.
W2: While it is clear that cyeval-DB7B is more reliable than existing benchmark pipelines, the claims in the paper alone do not sufficiently support the validity of this pipeline. Currently, it only presents one possible direction for reliability improvement. We would like to see verification of whether the reliability improvement achieved by cyeval-DB7B truly aligns with the performance that should be measured in cybersecurity.
Questions for authors
Related to my accessment:
Q1. The distinction between Extraction and Aggregation seems somewhat ambiguous. Is a reasonable explanation possible for the fact that the denominator problem is included in both? Simply put, since classification has been done, wouldn't it be preferable to clearly separate them into one or the other?
Q2. Figure 2 appears to be a logarithmic graph, but this is not explicitly stated, and the reason why the scale is not evenly divided is unclear. Wouldn't it be better to explicitly state that it is a logarithmic graph?
Q3. As a question related to W1, have you analyzed the relative comparison of the impact of failure modes on rank variation, and their interactions?
Q4. As a question related to W2, could the authors provide some kind of answer by limiting the discussion to cybersecurity regarding "Some failures cannot be fixed automatically because benchmark intent is underspecified, such as whether partial credit applies or which aliases count as correct" in Section6?
Q5: When I ran SecEval with cyeval-DB7B, I encountered another issue with the token length in Claude Sonnet-4.6, which prevented it from being evaluated correctly. Has this issue been fixed in cyeval-DB7B?
Suggestion for future work:
Q1. Are there any other claims that can only be made by limiting the discussion to cybersecurity? For example, is the high frequency of failure modes specific to cybersecurity, or is it a constant in the benchmark?
8 comments
Thank you for your detailed response.
comment on W1
While I understand the authors' arguments, given that the evaluation process is modeled as a construct function, the fact that initial formatting and inference settings deficiencies will cascade and impact the final score is a behavior that can be logically derived without conducting empirical experiments. Therefore, simply demonstrating that scores are distorted in a cascading manner is not sufficient as an academic contribution to this conference.
If the end-to-end audit approach is to be positioned as the main contribution, then I believe the paper should focus on verifying the superiority and validity of cyeval compared to existing pipelines, specifically how it can improve reliability. Please consider a significant restructuring of the paper to center around cyeval, such as formulating evaluation criteria for the entire pipeline using construct functions and verifying cyeval's evaluation capabilities based on those criteria.
Personally, I find it difficult to argue for a comprehensive end-to-end audit of cyeval based on the current paper's content. I feel that adding experiments that carefully analyze the chain of failure patterns and interactions would require fewer modifications to the current structure. That's also why I wrote weakness 1.
comment on W2
I understand that cyeval reduces noise and provides a highly practical and reliable evaluation pipeline.
However, my concern is that the current results are merely the result of "correcting individual implementation failure patterns," and the academic value and novelty of the theoretical framework of "composed pipeline" proposed in the paper are not being fully utilized. It's natural that correcting individual flaws will reduce evaluation fluctuations and improve the numerical results, and while its usefulness as a practical tool is acknowledged, I must say that its novelty as an academic paper is weak.
If the authors position "end-to-end auditing and modeling using composed pipelines" as the core of this paper, as stated in their W1 response, then I believe they should consider restructuring the paper to go beyond simply correcting individual failures and comprehensively and systematically verify the validity and superiority of cyeval's evaluation through the framework of composed pipelines.
Thank you for your continued engagement and constructive suggestions. We will add the interaction analysis to the main text and incorporate the clarifications below in a revision uploaded before the discussion period ends.
1. Our primary contribution is a systematic, end-to-end empirical audit of benchmark evaluation pipelines, not a theoretical framework or a new evaluation harness.
Our research question (§1) is: To what extent do evaluation-pipeline design choices affect the reliability of cybersecurity LLM benchmark outcomes? We audit eight benchmarks (23 tasks, 48,662 questions) across 10 LLMs, identify 14 failure modes across five stages, and quantify their effects on scores and rankings.
We agree that upstream failures can propagate through a composed pipeline (§2). However, whether they occur in practice, their magnitude, their interactions, and their consequences for model comparisons are empirical questions the formulation alone cannot answer; they motivate our audit.
2. Controlled experiments reveal dependencies and interactions between pipeline stages.
Our controlled perturbations isolate individual settings while holding others fixed (see responses to KXCU and 26jq):
- Inference-stage dependency (RedSage-Bench): The newline stop sequence empties Qwen3.6's responses (0%); once removed, raising the token budget from 100 to 2,048 lifts accuracy from 58.8% to 85.9%, so the budget's effect depends on the stop sequence.
- Extraction dependency (CTI-RCM): Gemma-4 yields only two parseable predictions out of 1,000. Excluding unparseable responses reports 100% accuracy; counting them as incorrect yields 0.2%.
As you and reviewer 26jq suggested, we analyzed interactions directly (see our reply to 26jq): a 2×2×2×2 factorial experiment (chat formatting; stop sequence or instruction; token budget; decoding), crossed with the extractor, on RedSage-Bench and SecEval across eight open-weight models. It reveals three findings that cannot be inferred from pipeline composition alone:
- A seemingly correct fix can lower scores until a downstream stage is also fixed. Raising SecEval's five-token budget lowers mean accuracy by 12.8 percentage points under the released scorer but raises it by 5.9 points under a final-answer extractor, so fixes cannot necessarily be validated independently.
- Large interactions occur between adjacent stages. Formatting × budget (P × I) interactions reach 25.3 points and budget × extractor (I × E) 18.6, versus at most 2.6 for formatting × extractor (P × E), suggesting that auditing adjacent-stage interactions can be particularly informative, rather than exhaustively testing all 2⁹ configurations.
- Without a fixed pipeline, model rankings are not identified. Across the 32 SecEval configurations, seven of the eight models span ranks 1–8 or 2–8.
Even excluding Gemma-4 and GPT-OSS (extreme response failures), 11 rank shifts of at least three positions remain among the eight normally responding models (KXCU, W1c).
3. cyeval is an experimental instrument supporting the audit, not the primary contribution.
The audited benchmarks use heterogeneous implementations; 44 of 72 pipeline fields cannot be reconstructed from documentation alone (Appendix C). cyeval standardizes nine fields with established components where benchmark semantics permit, making the audit's fixes executable, reproducing original procedures, and enabling controlled perturbations. We do not claim it is optimal, universally superior, or able to establish an LLM's true cybersecurity competence. We distinguish identifiable implementation failures from judgment-dependent conventions, documenting defensible alternatives and quantifying their effects (Appendix D.2, Table 12).
4. Evidence of measurement reliability and its limits.
- Independent checks: We confirmed 238 gold-label errors against authoritative sources; an independent judge agrees on 99.6% of extraction verdicts.
- Reproduction: The harness reproduces Primus-Merged's published AthenaBench scores within 2 points on all six tasks; for Foundation-Sec's CVSS score we obtain 0.813 versus the published 0.802.
- Cross-benchmark agreement: Kendall's τ-b rises on three of four shared tasks (TAA: 0.20 → 0.54; RCM: 0.16 → 0.64; ATE: 0.05 → 0.24) and falls on VSP (0.58 → 0.31).
These checks do not establish that cyeval or the underlying datasets capture real-world cybersecurity competence; as clarified in revised §7, measurement reliability is necessary but not sufficient for construct validity. Restructuring the paper around cyeval's general superiority would change its research question from an empirical reliability audit to validating a new framework, a claim we neither make nor seek to establish.
To help us address any remaining concerns, could you identify a concrete evaluation criterion or experimental comparison that you consider necessary but missing from the current study?
Thank you for the review and for running the harness.
W1.
Our main contribution is the end-to-end audit. We treat a benchmark score as the output of a composed pipeline, Sb(m)=Ab(Eb(Ib(m,Pb(Db)))) (§2), break every benchmark into these stages, and find that each stage has its own failure modes. Because the stages compose, a failure early in the pipeline cascades to the score:
The pipeline returns a number, not an error, so these failures are silent. The harness results from finding these modes systematically and fixing them in one implementation on standard components (LiteLLM, lighteval-style tasks). This matters: industry models such as Cisco's Foundation-Sec and Trend Micro's Primus report capability on these benchmarks.
W2.
Three kinds of evidence that the harness measures what the benchmarks intend:
What remains, how well the questions cover security competence, is measured at the dataset stage (F1(D)).
Q1.
F2(E) acts within a task (CTI-RCM: 100.0% correct-over-valid vs. 0.2%); F3(A) when tasks scored over different populations are averaged (AthenaBench's Combined lifts Gemma-4 from 3.9 to 17.5).
Q2. Yes: the axis is symmetric-log; we add the label to axis and caption.
Q3.
Rank shifts per mode (one setting changed, all else fixed):
F2(E)′: CVSS unparseable rule (formerly F3(E)). –: n/a for APIs; *: tie.
For a thorough Interaction analysis please refer to 26jq 's response number 3.
Q4.
Partly: ATT&CK alias lists ground alias credit and CVSS distance makes MAD partial credit; "related actor" credit and unparseable outputs have no reference.
Q5.
Thank you. This was a bug in cyeval, the package sent both temperature and top_p on every call. Claude Sonnet 4.6 rejects that combination, so every request failed and was scored as an empty answer. The paper's Sonnet 4.6 results are not affected: they were produced with a harness that sends temperature only. The updated release omits top_p whenever temperature is set.
Future-work Q1.
We chose cybersecurity for three reasons: (1) a rich ecosystem of benchmarks per task and authoritative sources with fixed conventions (CVE, CWE, CVSS, ATT&CK); (2) real-world stakes: specialized models are marketed through these benchmarks; (3) auditing labels and coverage needs domain expertise. The method should transfer, though we cannot show it without domain experts: the decomposition makes mechanical failures easy to find and fix, while expertise is needed for the dataset stage. Frequency cannot be settled from eight benchmarks.