acceptodds
Under review as a conference paper at ICLR 2027

Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks

Abstract

Large language model (LLM) benchmarks are often treated as fixed datasets with stable scores, yet their outcomes depend on configurable evaluation pipelines. We audit eight cybersecurity benchmarks across 10 proprietary, open-weight, and cybersecurity-specialized LLMs. By modeling benchmarks as measurement pipelines, we identify 15 systematic failure modes and show that a single pipeline choice can change a model's score by more than 80 percentage points and substantially alter model rankings. At the cross-benchmark level, two semantically similar task pairs rank the same models differently because of incompatible evaluation conventions. Under an evaluation harness that standardizes pipeline choices while preserving task semantics, nine of 10 models shift by at least three ranks on at least one benchmark. These results show that cybersecurity LLM benchmark scores are pipeline-dependent and motivate pipeline-aware auditing as a core requirement for reliable model evaluation.

Primary area:
datasets and benchmarks
Keywords:
cybersecurity, benchmarks, auditing, meta-review

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aX91OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines eight cybersecurity benchmarks and systematically investigates how benchmark results are influenced by factors other than LLM performance, across four stages: dataset, prompts, inference procedures, and aggregation. It identifies 15 failure modes and confirms that all benchmarks have at least two failures, and uses 10 models to demonstrate that these failures cause changes in ranking. Based on this, it proposes a framework called cyeval-DB7B for measuring the cybersecurity performance of LLMs while controlling for these factors.

Critical strengths

S1. While there have been previous studies on factors that compromise benchmark reliability, this attempt to solve the problem by handling a wide range of failure modes with a common harness is pioneering.

S2. The fact that they have actually published a framework for evaluating a set of benchmarks with a common harness is commendable.

Critical weaknesses

W1. Many of the failure modes discussed are already known, and there is little new insight gained from systematically evaluating multiple failure patterns. While some aspects, such as the denominator issue, are novel, this contribution to ICLR is weak. We would like to see a claim beyond simply stating that "all benchmarks have multiple failure patterns" by looking at multiple failure patterns together. However, there is also the expectation that this can be claimed based on existing experimental results.

W2: While it is clear that cyeval-DB7B is more reliable than existing benchmark pipelines, the claims in the paper alone do not sufficiently support the validity of this pipeline. Currently, it only presents one possible direction for reliability improvement. We would like to see verification of whether the reliability improvement achieved by cyeval-DB7B truly aligns with the performance that should be measured in cybersecurity.

Questions for authors

Related to my accessment:

Q1. The distinction between Extraction and Aggregation seems somewhat ambiguous. Is a reasonable explanation possible for the fact that the denominator problem is included in both? Simply put, since classification has been done, wouldn't it be preferable to clearly separate them into one or the other?

Q2. Figure 2 appears to be a logarithmic graph, but this is not explicitly stated, and the reason why the scale is not evenly divided is unclear. Wouldn't it be better to explicitly state that it is a logarithmic graph?

Q3. As a question related to W1, have you analyzed the relative comparison of the impact of failure modes on rank variation, and their interactions?

Q4. As a question related to W2, could the authors provide some kind of answer by limiting the discussion to cybersecurity regarding "Some failures cannot be fixed automatically because benchmark intent is underspecified, such as whether partial credit applies or which aliases count as correct" in Section6?

Q5: When I ran SecEval with cyeval-DB7B, I encountered another issue with the token length in Claude Sonnet-4.6, which prevented it from being evaluated correctly. Has this issue been fixed in cyeval-DB7B?

Suggestion for future work:

Q1. Are there any other claims that can only be made by limiting the discussion to cybersecurity? For example, is the high frequency of failure modes specific to cybersecurity, or is it a constant in the benchmark?

8 comments

Thank you for the review and for running the harness.

W1.

a claim beyond simply stating that "all benchmarks have multiple failure patterns"

Our main contribution is the end-to-end audit. We treat a benchmark score as the output of a composed pipeline, Sb(m)=Ab(Eb(Ib(m,Pb(Db))))\mathcal{S}_b(m)=\mathcal{A}_b(\mathcal{E}_b(\mathcal{I}_b(m,\mathcal{P}_b(\mathcal{D}_b)))) (§2), break every benchmark into these stages, and find that each stage has its own failure modes. Because the stages compose, a failure early in the pipeline cascades to the score:

  • raw-text prompting (P) yields unusable outputs, extraction (E) finds 2 parseable answers, and the denominator rule turns them into 100% on CTI-RCM;
  • a newline stop (I) empties Qwen3.6's outputs before any answer exists, and the token budget matters only once the stop is fixed (27 of 85.9 points).

The pipeline returns a number, not an error, so these failures are silent. The harness results from finding these modes systematically and fixing them in one implementation on standard components (LiteLLM, lighteval-style tasks). This matters: industry models such as Cisco's Foundation-Sec and Trend Micro's Primus report capability on these benchmarks.

W2.

verification of whether the reliability improvement ... truly aligns

Three kinds of evidence that the harness measures what the benchmarks intend:

  • Construct-irrelevant variance removed. Each fix removes variance unrelated to security knowledge (formatting, truncation, parsing, denominators); formatting alone moves Gemma-4 by 36–90 pp.
  • Correct ground truth. Gold labels are checked against authoritative sources (CVE, CWE, ATT&CK); 238 label errors are confirmed (App. G.1.2).
  • Fidelity and convergence. It reproduces published scores (Primus-Merged on AthenaBench within 2 pp on all six tasks; Foundation-Sec on CVSS, 0.813 vs. 0.802), and benchmarks sharing a task agree more under it (three of four shared tasks: τ-b 0.20 → 0.54, 0.16 → 0.64, 0.05 → 0.24).

What remains, how well the questions cover security competence, is measured at the dataset stage (F1(D)).

Q1.

the denominator problem is included in both?

F2(E) acts within a task (CTI-RCM: 100.0% correct-over-valid vs. 0.2%); F3(A) when tasks scored over different populations are averaged (AthenaBench's Combined lifts Gemma-4 from 3.9 to 17.5).

Q2. Yes: the axis is symmetric-log; we add the label to axis and caption.

Q3.

the impact of failure modes on rank variation, and their interactions?

Rank shifts per mode (one setting changed, all else fixed):

ModelF1(P)F2(P)F3(P)F1(I)F2(I)F3(I)F1(E)F2(E)F2(E)′F4(E)F1(A)F2(A)F3(A)
GPT-5.400+1–+90−1+10+1–00
Sonnet 4.60+1+1–−10+1+10−1–00
Gemma-4+2+100−10−1−9−5*0−6+2*0
Qwen3.60+100−1+10+30+6+6*00
Llama-3.30−100−1−10+4+4+1−20+1
GPT-OSS00−10−100+1−3−100−1
Primus-Nem.−2*−1−20−1−10−4−2−3+2*−10
Primus-Merged0−100−1+1+2+2+2−1+100
Foundation-Sec0+100−10−30+5−1−1−1*0
RedSage0−1+10−10+2+1−1−1000
ρ0.940.950.951.000.450.980.880.210.520.680.060.980.99

F2(E)′: CVSS unparseable rule (formerly F3(E)). –: n/a for APIs; *: tie.

For a thorough Interaction analysis please refer to 26jq 's response number 3.

Q4.

limiting the discussion to cybersecurity ...

Partly: ATT&CK alias lists ground alias credit and CVSS distance makes MAD partial credit; "related actor" credit and unparseable outputs have no reference.

Q5.

Has this issue been fixed?

Thank you. This was a bug in cyeval, the package sent both temperature and top_p on every call. Claude Sonnet 4.6 rejects that combination, so every request failed and was scored as an empty answer. The paper's Sonnet 4.6 results are not affected: they were produced with a harness that sends temperature only. The updated release omits top_p whenever temperature is set.

Future-work Q1.

is the high frequency of failure modes specific to cybersecurity?

We chose cybersecurity for three reasons: (1) a rich ecosystem of benchmarks per task and authoritative sources with fixed conventions (CVE, CWE, CVSS, ATT&CK); (2) real-world stakes: specialized models are marketed through these benchmarks; (3) auditing labels and coverage needs domain expertise. The method should transfer, though we cannot show it without domain experts: the decomposition makes mechanical failures easy to find and fix, while expertise is needed for the dataset stage. Frequency cannot be settled from eight benchmarks.

Thank you for your detailed response.

comment on W1

While I understand the authors' arguments, given that the evaluation process is modeled as a construct function, the fact that initial formatting and inference settings deficiencies will cascade and impact the final score is a behavior that can be logically derived without conducting empirical experiments. Therefore, simply demonstrating that scores are distorted in a cascading manner is not sufficient as an academic contribution to this conference.

If the end-to-end audit approach is to be positioned as the main contribution, then I believe the paper should focus on verifying the superiority and validity of cyeval compared to existing pipelines, specifically how it can improve reliability. Please consider a significant restructuring of the paper to center around cyeval, such as formulating evaluation criteria for the entire pipeline using construct functions and verifying cyeval's evaluation capabilities based on those criteria.

Personally, I find it difficult to argue for a comprehensive end-to-end audit of cyeval based on the current paper's content. I feel that adding experiments that carefully analyze the chain of failure patterns and interactions would require fewer modifications to the current structure. That's also why I wrote weakness 1.

comment on W2

I understand that cyeval reduces noise and provides a highly practical and reliable evaluation pipeline.

However, my concern is that the current results are merely the result of "correcting individual implementation failure patterns," and the academic value and novelty of the theoretical framework of "composed pipeline" proposed in the paper are not being fully utilized. It's natural that correcting individual flaws will reduce evaluation fluctuations and improve the numerical results, and while its usefulness as a practical tool is acknowledged, I must say that its novelty as an academic paper is weak.

If the authors position "end-to-end auditing and modeling using composed pipelines" as the core of this paper, as stated in their W1 response, then I believe they should consider restructuring the paper to go beyond simply correcting individual failures and comprehensively and systematically verify the validity and superiority of cyeval's evaluation through the framework of composed pipelines.

Thank you for your continued engagement and constructive suggestions. We will add the interaction analysis to the main text and incorporate the clarifications below in a revision uploaded before the discussion period ends.

1. Our primary contribution is a systematic, end-to-end empirical audit of benchmark evaluation pipelines, not a theoretical framework or a new evaluation harness.

Our research question (§1) is: To what extent do evaluation-pipeline design choices affect the reliability of cybersecurity LLM benchmark outcomes? We audit eight benchmarks (23 tasks, 48,662 questions) across 10 LLMs, identify 14 failure modes across five stages, and quantify their effects on scores and rankings.

We agree that upstream failures can propagate through a composed pipeline (§2). However, whether they occur in practice, their magnitude, their interactions, and their consequences for model comparisons are empirical questions the formulation alone cannot answer; they motivate our audit.

2. Controlled experiments reveal dependencies and interactions between pipeline stages.

Our controlled perturbations isolate individual settings while holding others fixed (see responses to KXCU and 26jq):

  • Inference-stage dependency (RedSage-Bench): The newline stop sequence empties Qwen3.6's responses (0%); once removed, raising the token budget from 100 to 2,048 lifts accuracy from 58.8% to 85.9%, so the budget's effect depends on the stop sequence.
  • Extraction dependency (CTI-RCM): Gemma-4 yields only two parseable predictions out of 1,000. Excluding unparseable responses reports 100% accuracy; counting them as incorrect yields 0.2%.

As you and reviewer 26jq suggested, we analyzed interactions directly (see our reply to 26jq): a 2×2×2×2 factorial experiment (chat formatting; stop sequence or instruction; token budget; decoding), crossed with the extractor, on RedSage-Bench and SecEval across eight open-weight models. It reveals three findings that cannot be inferred from pipeline composition alone:

  • A seemingly correct fix can lower scores until a downstream stage is also fixed. Raising SecEval's five-token budget lowers mean accuracy by 12.8 percentage points under the released scorer but raises it by 5.9 points under a final-answer extractor, so fixes cannot necessarily be validated independently.
  • Large interactions occur between adjacent stages. Formatting × budget (P × I) interactions reach 25.3 points and budget × extractor (I × E) 18.6, versus at most 2.6 for formatting × extractor (P × E), suggesting that auditing adjacent-stage interactions can be particularly informative, rather than exhaustively testing all 2⁹ configurations.
  • Without a fixed pipeline, model rankings are not identified. Across the 32 SecEval configurations, seven of the eight models span ranks 1–8 or 2–8.

Even excluding Gemma-4 and GPT-OSS (extreme response failures), 11 rank shifts of at least three positions remain among the eight normally responding models (KXCU, W1c).

3. cyeval is an experimental instrument supporting the audit, not the primary contribution.

The audited benchmarks use heterogeneous implementations; 44 of 72 pipeline fields cannot be reconstructed from documentation alone (Appendix C). cyeval standardizes nine fields with established components where benchmark semantics permit, making the audit's fixes executable, reproducing original procedures, and enabling controlled perturbations. We do not claim it is optimal, universally superior, or able to establish an LLM's true cybersecurity competence. We distinguish identifiable implementation failures from judgment-dependent conventions, documenting defensible alternatives and quantifying their effects (Appendix D.2, Table 12).

4. Evidence of measurement reliability and its limits.

  • Independent checks: We confirmed 238 gold-label errors against authoritative sources; an independent judge agrees on 99.6% of extraction verdicts.
  • Reproduction: The harness reproduces Primus-Merged's published AthenaBench scores within 2 points on all six tasks; for Foundation-Sec's CVSS score we obtain 0.813 versus the published 0.802.
  • Cross-benchmark agreement: Kendall's τ-b rises on three of four shared tasks (TAA: 0.20 → 0.54; RCM: 0.16 → 0.64; ATE: 0.05 → 0.24) and falls on VSP (0.58 → 0.31).

These checks do not establish that cyeval or the underlying datasets capture real-world cybersecurity competence; as clarified in revised §7, measurement reliability is necessary but not sufficient for construct validity. Restructuring the paper around cyeval's general superiority would change its research question from an empirical reliability audit to validating a new framework, a claim we neither make nor seek to establish.

To help us address any remaining concerns, could you identify a concrete evaluation criterion or experimental comparison that you consider necessary but missing from the current study?

Reviewer_26jqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that cybersecurity LLM benchmark scores are not intrinsic model properties; they depend heavily on the full evaluation pipeline—prompts, inference settings, output extraction, scoring, and aggregation. Auditing eight benchmarks across ten models, the authors find pipeline choices that can change scores by over 80 percentage points and substantially reorder models.

Critical strengths
  • The audit is broad and well documented.
  • The paper is clear and potentially useful to benchmark maintainers.
Critical weaknesses
  • The failure-mode analysis is insufficiently systematic. The taxonomy is developed and evaluated on the same eight benchmarks and combines implementation bugs, underspecification, defensible evaluation choices, and construct-validity concerns. Moreover, the evidence mixes controlled re-scoring, bundled interventions, peer-score gaps, and cross-dataset comparisons. These analyses have different evidential strength, so the 34 interactions are neither directly comparable nor an estimate of failure prevalence or severity. The paper would benefit from clearer operational criteria, mutually exclusive categories, and systematic analyses such as factorial ablations or a pipeline multiverse.
  • The metric-direction mismatch claim is unsupported. CTI-Bench reports lower-is-better MAD, whereas AthenaBench applies the monotonic transformation 100max(0,1-MAD/7.7), where higher is better. When direction is handled correctly, the two metrics should induce identical model rankings on the same predictions, except for ties introduced by clipping. The reported five-position rank change therefore does not establish an effect of metric direction. The authors should disclose the ranking procedure and apply both metrics to identical predictions.
  • Several headline effects bundle multiple interventions. The reported 85.9-point RedSage improvement jointly removes the newline stop sequence, increases the token budget, and changes reasoning-span handling. Similarly, the CyberMetric analysis simultaneously changes temperature, top-p, and top-k, with only one sampled run. These experiments cannot identify which pipeline choice causes the observed effect. Factorial ablations and repeated runs are needed to estimate individual effects and interactions.
Questions for authors
  1. How were the 15 failure modes derived, and were operational criteria fixed before auditing all eight benchmarks? How many of the 34 interactions are implementation bugs, underspecification, sensitivity to defensible choices, or construct-validity concerns?
  2. Is the reported five-position metric-direction rank change computed from identical predictions? If so, what ranking and tie-handling procedure allows MAD and its monotonic transformation to produce different orderings?
  3. For the 85.9-point RedSage result, which settings differ between the compared runs? Are there results that isolate the stop sequence while holding token budget and reasoning handling fixed? Likewise, are individual decoding parameters isolated in the CyberMetric analysis?
5 comments

Thank you for the review. It asks for operational criteria, mutually exclusive categories, and systematic evidence such as factorial ablations. We now provide all three.

1. Operational criteria (W1, Q1).

were operational criteria fixed before auditing all eight benchmarks?

The frame was fixed; the modes were found within it. Every benchmark is decomposed into the same five stages and nine fields (§2), and each of the 72 field–benchmark pairs is checked against both the documentation and the released code (App. C). A failure mode is a field that changes scores without a change in capability; it belongs to the stage that owns the field. All benchmark–mode pairs were then re-checked against the final list, including nulls (Table 1). The grid is benchmark-independent: a new benchmark fills the same cells, and new modes extend the list, not the procedure.

2. Mutually exclusive categories (W1, Q1).

combines implementation bugs, underspecification, defensible evaluation choices, and construct-validity concerns

Each incident's type follows mechanically from its field's App. C status, recorded independently of score effects:

Field status in App. CTypeIncidents
Documentation and code contradictImplementation defect4
UndefinedUnderspecification10
Specified and followedDefensible choice6
Dataset stage: gold labelsLabel defect7
Dataset stage: coverageConstruct validity5

(After the merge in point 4.) Defensible alternatives and their effects are in Table 12. The 32 incidents measure incidence, not prevalence.

3. Factorial evidence (W1, W3).

systematic analyses such as factorial ablations

Beyond the single-field ablations (Tables 20–27), a full factorial over the nine fields is prohibitive (2⁹ = 512 configurations per benchmark and model), so we crossed the fields that move scores most: a 2×2×2×2 factorial on RedSage-Bench (1,000 questions) and SecEval (2,189) for the eight open-weight models, with the extractor crossed on the outputs. Main effects, released → alternative: mean Δ pp [largest single-model Δ], rank ρ, models moving ≥3 places:

FieldRedSage-BenchSecEval
Chat formatting+11.0 [+38.4], ρ 0.69, 2+24.9 [+68.8], ρ −0.45, 4
Stop (RedSage) / instruction (SecEval)+6.2 [+63.8], ρ 0.67, 1−1.4 [−3.8], ρ 0.95, 0
Token budget+1.5 [+18.6], ρ 0.93, 0−3.4 [+48.9], ρ −0.05, 4
Decoding−1.3 [−4.9], ρ 0.98, 0+0.3 [+7.3], ρ 0.90, 0
Extractor+0.6 [+5.3], ρ 1.00, 0+2.9 [+21.8], ρ 0.64, 1

Fields interact: formatting × budget adds 25 pp on SecEval. For comparability, every mode is also measured as a single-field rank shift (reply to aX91 Q3), with its evidence type in Table 16.

4. Metric direction (W2, Q2).

Is the reported five-position metric-direction rank change computed from identical predictions?

No; it compared disjoint CVE sets. Even on one pipeline, the two VSP tasks rank models differently (τ-b = 0.31), so the gap comes from the question pools, not direction. On identical predictions, MAD and 100·(1 − MAD/7.7) rank all 10 models identically (τ-b = 1.000; max MAD < 3.6, so clipping never binds; ties by stable sort, App. I.3). What does reorder them is the rule for unparseable outputs (skipped in CTI-Bench and Primus; maximal error in arXiv:2508.01059):

Identical predictionsCTI itemsAthena items
MAD vs. 100·(1 − MAD/7.7)00
MAD vs. 1 − MAD/10, unparseable = max56
Same, parseable only00

Primus-Merged scores 0.83 or 0.38 depending only on this rule, yet published work pools such scores: DeepHat-V1-7B is reported at 0.808 and 0.045 by one group (arXiv:2508.01059; 2601.21051), and Primus sums benchmarks with CTI-VSP negated. Since this is F2(E)'s mechanism, F3(E) merges into F2(E): 14 modes, 32 incidents.

5. The bundled analyses, separated (W3, Q3).

which settings differ between the compared runs? ... are individual decoding parameters isolated?

RedSage-Bench, Qwen3.6, stored greedy outputs:

Newline stopBudget 100Budget 2,048
Backend (released)0.00.0
After the reasoning span58.885.9

The stop causes the failure (every response opens with <think> and a newline); the budget adds 27.1 pp only once the stop is moved.

CyberMetric, five runs per configuration (mean ± s.d., %):

ModelGreedyT 1.0+ top-p 0.9+ top-k 50
Primus-Merged17.3 ± 0.260.8 ± 12.060.1 ± 18.060.2 ± 20.3
Primus-Nemotron19.2 ± 0.041.1 ± 15.538.7 ± 19.138.8 ± 19.1

The other eight models move ≤3 pp in every arm (s.d. ≤ 4.5). Temperature alone produces the whole gain; top-p and top-k add nothing. Single sampled runs span 33–78% and 14–66%, so one sampled score is not a stable estimate.

The revision adds the procedure to §2, the merge and the factorial tables (App. [X]), and replaces Table 22 with these means.

Thanks for the responses. These are really helpful.

The response resolves my metric-direction objection and substantially addresses the bundled-intervention concern.

My remaining concern is the taxonomy: documentation status alone does not distinguish implementation defects from defensible choices, and common rank-shift units do not make heterogeneous evidence causally comparable. Please clarify these classification boundaries, confirm fixed reasoning extraction in the RedSage comparisons, and report uncertainty for the factorial effects.

Thank you. We address the three remaining points in turn.

1. Classification boundaries.

documentation status alone does not distinguish implementation defects from defensible choices

We agree. Documentation status records where a behaviour comes from, not whether it is defensible. The revision types each incident by its behaviour first, independently of effect size:

StepTestType
(a)The released behaviour makes the score independent of the model's answer on the affected items: it removes the answer before it exists, scores an error payload as a value, or makes the denominator depend on the response.Defect, whatever the documentation says
(b)Otherwise, documentation and code disagree, or neither defines the field.Specification gap
(c)Otherwise, the field is specified and followed, and a published alternative changes the score on identical inputs.Defensible choice
—Dataset stageLabel defect / construct validity (unchanged)

Example: RedSage-Bench's newline stop cuts every Qwen3.6 response at <think> (0.0% at any budget), so (a) types it a defect. CTI-Bench's partial credit for related actors contradicts its documentation but passes (a), so it is a specification gap, not a defect. Skipping unparseable outputs (previous reply, point 4) also fails (a). Re-typed, the 32 incidents are 5 defects (the RedSage-Bench stop, SecEval's 5-token budget, which hosted APIs reject, and three response-dependent denominators), 9 specification gaps, 6 defensible choices, 7 label defects and 5 construct-validity concerns.

common rank-shift units do not make heterogeneous evidence causally comparable

Also agreed. A rank shift is a common descriptive scale, not a common causal estimand. The revision limits causal comparison to modes whose evidence varies one field and holds everything else fixed:

  • re-scoring of identical outputs: F1(E), F2(E), F2(A), F3(A);
  • single-field re-generation: F1(I)–F3(I), and prompt mode, which is F3(E) after the merge (formerly F4(E)).

Output-audit and peer-gap evidence (F1(P)–F3(P)) and the paired logprob-vs-generative comparison (F1(A)) are reported as incidence with an associational magnitude and are not ranked against the controlled modes. The evidence table (App. F.2) gains an evidence-tier column.

2. Fixed reasoning extraction in the RedSage-Bench comparison.

Confirmed. All four cells come from one stored greedy run of Qwen3.6 at 2,048 tokens (30,000 questions, five subsets); under greedy decoding, the 100-token output is the first 100 tokens of that run. Only two things vary between cells: where the newline stop is applied and the token budget. Every cell uses the same extractor: remove the <think>…</think> span, take the first non-empty line, and score its option letter.

Budget 100Budget 2,048
Newline stop on raw stream (released)0.00.0
Newline stop after the reasoning span58.885.9

Cells are means over the five subsets, each weighted equally.

The budget effect is +27.0 pp (95% CI [26.5, 27.6], paired over questions; 27.1 earlier came from rounded cells). It comes entirely from reasoning responses: 10,241 of 30,000 contain a non-empty reasoning span, and at 100 tokens 10,237 of these have not closed it, so no answer is emitted. Raising the budget turns 8,736 answers from wrong to right and none from right to wrong.

3. Uncertainty for the factorial effects.

The factorial uses a 1,000-question RedSage-Bench subset and all 2,189 SecEval questions; the 2×2 above uses all 30,000 RedSage-Bench questions. We report two 95% intervals: Q, a paired question-level bootstrap (2,000 resamples, the same resample for every model and cell), and M, a bootstrap over the eight models. Mean Δ (pp) and Spearman ρ for selected effects:

EffectMean Δ [Q] [M]ρ [Q] [M]
SecEval, chat formatting+24.9 [24.3, 25.5] [1.5, 46.5]−0.45 [−0.52, −0.45] [−0.92, 0.52]
SecEval, token budget−3.4 [−3.7, −3.1] [−17.6, 13.5]−0.05 [−0.10, −0.02] [−0.92, 0.79]
RedSage, chat formatting+11.0 [10.2, 11.8] [0.2, 23.6]0.69 [0.43, 0.74] [−0.23, 1.00]
RedSage, stop sequence+6.2 [6.0, 6.5] [−6.1, 23.9]0.67 [0.64, 0.81] [−0.23, 1.00]
SecEval, formatting × budget+25.3 [24.5, 26.2] [6.9, 42.9]
SecEval, budget × extractor+18.6 [18.2, 19.1] [7.1, 30.2]
SecEval, formatting × extractor−2.6 [−2.8, −2.4] [−14.5, 8.3]

Every Δ [Q] interval is within ±1 pp. Across models, the two largest interactions stay positive, but most main effects and all ρ intervals are wide, because the effects are concentrated in a few models (e.g., Qwen3.6 for the stop sequence). The revision's appendix reports all main effects and interactions with both intervals, and states model-level conclusions only where the M interval excludes zero. The CyberMetric decoding arms report mean ± s.d. over five runs.

Reviewer_KXCUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper treats a benchmark score as the output of a five-stage pipeline and audits eight cybersecurity QA benchmarks with 10 LLMs. It catalogues 15 failure modes (Table 1), analyzes agreement across 23 tasks, and reports rank changes under a standardized harness with a GPT-5.4 judge (Table 5). The harness and audit code are in two populated anonymized repositories.

Critical strengths
  1. Each failure is tied to a released artifact and a question-level witness, and Table 10's finding that 44 of 72 pipeline fields can't be reproduced from documentation is useful to benchmark maintainers.

  2. The judge is validated more carefully than usual, with an independent judge (99.6% agreement, Table 15) and a generation-versus-extraction decomposition (Table 29).

Critical weaknesses
  1. The headline effects mostly measure runs that failed outright, and the "original" baseline is defined inconsistently. Every effect above 80 pp comes from a model emitting almost no answers (Qwen3.6 at 0.0% under the newline stop, GPT-5.4 rejected by an Azure token minimum, Gemma-4 with two parseable CTI-RCM outputs), whereas the extractor and logprob medians are 1.3 and 2.8 pp (Table 4). Table 5 matches Table 30 only if GPT-5.4's original SecEval score is the repaired 81.4% rather than the listed 0.3%, yet chance-level originals stay in (Gemma-4 at 8.4% on five-option AthenaBench CKT, GPT-OSS near 25% on four-option RedSage-Bench). Without these two models, shifts of at least three ranks fall from 35 to 11, nine of them on MMLU-CS and SecEval.

  2. Section 7 acknowledges in one sentence that scores depend on prompts and extraction, but it doesn't cite the closest work (Sclar et al., 2024; Alzahrani et al., 2024; Biderman et al., 2024) or say what the audit adds beyond carrying that sensitivity into one domain, and its cross-benchmark claims are weak. If models are the standardized columns of the 23×1023 \times 10 matrix, as written, the 95.25% first component is shared task difficulty; Table 30's standardized scores give 95.7% this way but 71% with tasks as variables. With VSP direction corrected, Table 30 also gives τ\tau-b ≈0.31\approx 0.31 on standardized and 0.58 on original scores, so aligned conventions lower agreement rather than explain it.

  3. It would be useful to discuss the critical recent work (e.g., arXiv:2609.15939).

Questions for authors
  1. What serving setup produced Gemma-4's and GPT-OSS's original runs? Near-chance scores caused by serving aren't benchmark failures.
  2. Which matrix orientation does the PCA use, and are the §5.2 τ\tau-b values computed on original or standardized scores?
  3. Suggestion for future work: Test whether the specialized models' published advantages survive the standardized harness.
7 comments

Thank you for the review. We address each point below.

W1a.

The headline effects mostly measure runs that failed outright ... whereas the extractor and logprob medians are 1.3 and 2.8 pp (Table 4).

The >80 pp cases are failed runs, and typical extractor and logprob effects are small. But the failure modes are not confined to failed runs. On normal runs, a single pipeline choice moves scores by 8–27 pp and can change ranks:

ModeBenchmark, modelSingle changeScore
F2(E) denominatorCTI-MCQ, Qwen3.6 (90% parseable)unparseable dropped → counted wrong74.9 → 67.2 (3rd → 4th)
F2(A) metricCTI-TAA, GPT-5.4strict → partial credit74 → 86
F2(I) budgetRedSage-Bench, Qwen3.6 (stop fixed)100 → 2,048 tokens58.8 → 85.9
F3(P) templateSECURE-MAET, RedSage / GPT-OSS+ "Answer:" cue55.4 → 74.2 / 63.2 → 51.8

The large cases matter because they are silent: the RedSage-Bench stop empties only Qwen3.6's outputs, and GPT-5.4's 0.3% on SecEval comes from letters matched inside API error messages, yet both are reported as ordinary scores.

W1b.

Table 5 matches Table 30 only if GPT-5.4's original SecEval score is the repaired 81.4% rather than the listed 0.3%

Correct. With 0.3% throughout, GPT-5.4 moves from last to first on SecEval, ≥3-rank shifts go from 35 to 34, and all 10 models (not nine) shift by ≥3 ranks on at least one benchmark.

W1c.

Without these two models, shifts of at least three ranks fall from 35 to 11, nine of them on MMLU-CS and SecEval.

We reproduce your count:

MMLU-CSSecEvalSECURECTIAthenaCyberMetricRedSageSecBenchTotal
All 10 models9744322435
Without Gemma-4, GPT-OSS4510000111

The 11 remaining shifts involve eight models that answer normally, including 3–7-place drops of all four specialized models on MMLU-CS; with log-probability originals for the open-weight models, MMLU-CS still shows 9 such shifts (5 without the two). The two models' chance-level originals come from prompt delivery (Q1).

W2a, W3.

it doesn't cite the closest work ... or say what the audit adds ... (e.g., arXiv:2609.15939).

Sclar et al. and Alzahrani et al. vary prompt format, answer format, few-shot examples and scoring; Biderman et al. argue that setups are underdetermined without released code. We measure that gap field by field for released cybersecurity benchmarks (44 of 72 fields not reproducible, App. C), across all five stages, with question-level witnesses and rank effects. arXiv:2609.15939 (14 Sep 2026, after our registration) is concurrent work introducing a new agentic benchmark, not an audit.

W2b.

the 95.25% first component is shared task difficulty

Correct: models were the variables. With tasks as variables, the first component explains 71.2% (72.9% on the full matrix), and only it exceeds Horn's null (PC2 13.9% vs. 21.0%): one model-strength axis dominates.

W2c.

With VSP direction corrected ... aligned conventions lower agreement rather than explain it.

Correct: τ-b is 0.58 (original) and 0.31 (standardized), and τ-b is invariant to monotone transforms, so conventions cannot explain the disagreement, which remains (disjoint CVE sets; corrected §5.2 τ-b 0.67 and 0.54). Across the four tasks two benchmarks share, standardization raises agreement on three (TAA 0.20 → 0.54, RCM 0.16 → 0.64, ATE 0.05 → 0.24). Within a benchmark, CTI-Bench's published CVSS conventions reorder identical predictions by up to five positions (reply to 26jq).

Q1.

What serving setup produced Gemma-4's and GPT-OSS's original runs?

vLLM, bfloat16 (App. B.3, Table 9). In the original runs, open-weight models received raw text without their chat template. Changing only this on Gemma-4 (200 questions per task): AthenaBench CKT 8.0 → 88.5, CyberMetric 6.0 → 96.5, CTI-RCM 0.0 → 66.0, SecEval 10.0 → 78.0, SECURE-MAET 50.0 → 86.5; the raw arm reproduces the originals. SecEval's evaluator itself sends raw text; CTI-Bench, CyberMetric, SECURE and SecBench leave formatting unspecified; AthenaBench and RedSage-Bench apply the template, so there the near-chance originals reflect our runs (with it, GPT-OSS scores 83.9% on RedSage CLI, and our harness reproduces AthenaBench's published scores within 2 pp).

Q2.

Which matrix orientation does the PCA use, and are the §5.2 τ-b values computed on original or standardized scores?

Models as variables. The §5.2 τ-b uses the 23×10 standardized matrix with exact-vector CVSS scoring; the published 0.29 and 0.24 come from an earlier version of it; current values are 0.67 and 0.54.

Q3.

Test whether the specialized models' published advantages survive the standardized harness.

Partly: the four specialized models drop 3–7 places on MMLU-CS under standardization (§5.3). A direct test needs their base models, which we did not run.

Thank you for the response. The corrections settle most of the factual items.

W1: Partially resolved. The sensitivity table shows 8–27 pp effects on normal runs, but it has four cases against medians of 1.3 and 2.8 pp, and two of them (partial credit, token budget) are metric or budget choices rather than specification gaps. The SecEval concession and the reproduced 35→1135 \rightarrow 11 count are accepted. The Q1 answer shows the near-chance Gemma-4 and GPT-OSS originals on AthenaBench and RedSage-Bench come from omitting the chat template in the authors' own runs, so they aren't benchmark failures. The headline should rest on the 11 remaining shifts, with failed and misconfigured runs reported separately.

Related work: Partially resolved. The differentiation from Sclar et al., Alzahrani et al. and Biderman et al. is convincing, but it needs to appear in Section 7 along with the citations.

PCA: Settled, given the corrected 71.2%71.2\% with tasks as variables and the Horn's-null result.

τ\tau-b: Partially resolved. The concession is accepted, but the invariance argument doesn't cover a harness that re-runs predictions. The per-task gains cover only three of the four shared tasks. The claim that standardization aligns benchmarks isn't established, and the revision should report original and standardized values side by side.

Specialized models: Still open, but acceptable as future work.

Outstanding are the revised headline framing, Section 7, and the corrected Section 5.2.

Thank you for your constructive follow-up and for acknowledging our clarifications. We agree that the remaining concerns primarily relate to the framing and presentation of our findings, and we will address them in the revision (to be uploaded before the end of the discussion period).

W1: Headline framing and reliability effects.

We agree that failed evaluations, misconfigured runs, and sensitivity to defensible evaluation choices should be clearly distinguished. In particular, the near-chance original runs for Gemma-4 and GPT-OSS on AthenaBench and RedSage-Bench reflect our omission of the chat template, rather than failures in those benchmarks. We will correct this attribution and report these cases separately.

We will also revise the headline claims to distinguish extreme failure cases from effects observed under normally functioning evaluations. The 11 rank shifts of at least three positions remaining after excluding Gemma-4 and GPT-OSS provide a more conservative characterization of ranking sensitivity. We will present these alongside the full results and clarify that the sensitivity examples illustrate possible effects rather than their typical magnitude.

We maintain that both implementation defects and defensible configuration choices can materially affect benchmark outcomes, but agree that their interpretation and attribution must be explicit.

Related work (§7).

Agreed. We will incorporate Sclar et al., Alzahrani et al., and Biderman et al., explicitly distinguishing our contribution: a systematic, field-by-field empirical audit of released cybersecurity benchmark pipelines, spanning five evaluation stages, 15 failure modes, and their consequences for model comparisons.

Cross-benchmark agreement (§5.2).

We agree that standardization does not necessarily improve agreement, particularly when it involves re-generating model predictions. Our previous invariance argument applies only to monotonic transformations of fixed scores and does not establish agreement under re-generation.

We will report original and standardized Kendall's τ-b values side by side for all four shared tasks, including the CVSS/VSP decrease (0.58 → 0.31), and revise any claims suggesting that standardization universally improves cross-benchmark agreement. The supported conclusion is that evaluation-pipeline choices can materially affect cross-benchmark agreement, with improvements or deteriorations depending on the task and configuration.

Specialized models.

Thank you for accepting the base-model comparison as future work. We will retain this limitation explicitly.

We appreciate your careful review and constructive suggestions. These revisions will clarify the distinction between benchmark defects, evaluation misconfigurations, and configuration sensitivity, while preserving our central empirical finding: benchmark scores and model comparisons can depend substantially on the evaluation procedures used to produce them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.