acceptodds
Withdrawn from ICLR 2027

SecEmbedEval: A Comprehensive Benchmark for Evaluating Text Embeddings in Cybersecurity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce SecEmbedEval, a benchmark for evaluating cybersecurity text embeddings. SecEmbedEval contains over 17,000 task instances in Chinese and English, built from public cybersecurity sources through source-aware preprocessing, large language model (LLM)-assisted construction, and blind verification. It covers nine tasks in five categories: semantic similarity, retrieval, classification, clustering, and efficiency. To support aggregated evaluation, we propose two metrics. SEE-Score computes a min-max normalized average across five task measures. Transfer Discordance (TD) quantifies the share of model pairs whose ranking order differs between public leaderboards and our in-domain benchmark. We evaluate 12 representative embedding models. General-domain standing does not transfer (TD ). Frozen cybersecurity-pretrained encoders trail contrastively trained general-purpose models even under English-only controls. No evaluated model, including the strongest commercial system, can ground bare CVE identifiers to their advisories, and clustering remains challenging for every model.

Primary area:
datasets and benchmarks
Keywords:
Cybersecurity, Text embedding, Benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_izriOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SecEmbedEval is a Chinese/English benchmark for cybersecurity text embeddings, with nine subtasks over similarity, retrieval, classification, clustering, and efficiency, built from public sources with heavy LLM assistance. It evaluates twelve models and proposes two aggregates, SEE-Score and TD. The paper refers to released code and data but doesn't link to either.

Critical strengths
  • Security text is a real gap in embedding evaluation, and the reference points are unusually careful (TF-IDF and random probes in Table 10, a floor, ceiling, and degeneracy audit for clustering in Table 25).

  • Holm-corrected leader tests (Table 14) are reported candidly, even though they show most top-1 margins aren't significant.

Critical weaknesses
  • The headline findings are either known or stated beyond the evidence. FinMTEB (Tang & Yang, 2025) already showed on bilingual finance data that general benchmark scores predict domain performance poorly, and it isn't discussed. Here the claim rests on seven models with ρ=+0.43\rho = +0.43, and since TD equals (1−τ)/2(1-\tau)/2, TD=7/21\mathrm{TD} = 7/21 means two thirds of pairs agree. That is weak transfer measured with little power, not its absence. The "objective, not corpus" conclusion pits frozen 110M MLM encoders against 0.6–8B contrastive models, a long-documented gap, without a contrastive security encoder (e.g., ATT&CK-BERT) to isolate the cause. The claim that cost barely tracks score contradicts Tables 5 and 19, where batch-25 throughput and SEE-Score give Spearman ρ≈−0.85\rho \approx -0.85 over all twelve models and −0.70-0.70 over the nine general-purpose ones.

  • Much of the ground truth is produced or checked by LLMs and never validated by humans on the final data (Table 6 covers pilot samples only). The 12,964 retrieval relevance labels come from an unnamed judge whose only reported check is 99.0% agreement with itself (Table 21), over a pool with no BM25 run. STS-1 pairs are admitted through relation-specific bigram-Jaccard bands, which may leave a surface-overlap shortcut, yet no lexical baseline is reported for similarity or retrieval. Appendix F even maps CWE-79 to "输入验证和表示", which isn't a View-1000 Pillar.

Questions for authors

Clarification requests:

  1. Which LLMs generated candidates and judged relevance, and how well does a human-annotated sample of the final qrels agree with the judge? A Qwen-family judge could favor aliyun-qwen3.7.
  2. What AUC do Jaccard and BM25 reach on STS-1 and STS-1-en, and what NDCG@10 does BM25 reach on retrieval?

Suggestions for future work:

  1. Contrastively fine-tuning SecureBERT, next to a general-domain BERT-base, would test the objective-versus-domain claim directly.
Reviewer_8wjTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper introduces a benchmark designated to evaluate text embeddings models specifically on cybersecurity tasks. The benchmark contains more than 17K task instances in both English and Chinese basically coming from sources such. as vulnerability DBs , security advisories and threat intelligence reports this result in nine sub tasks across five categories

the authors introduced two evaluation metrics; SEE-score, basically jus normalized the combination of six task metrics, and. Transfer Discordance , which assess ranking disagreements between "public "embedding leaderboard and the proposed domain-specific benchmark ,

given the above, the paper present result for 12 embedding models from both general purpose and cybersecurity categories .

the main finding is that public benchmarks usually do not predict cybersecurity performance.

Critical strengths
  • cybersecurity data contains specialized terminology, vocabs, identifiers, and semantic distinctions that may not be well represented in general embedding benchmarks. in that sense a more unified approach is potentially valuable

  • one of the elements i consider interesting is the wide range of embedding applications / use cases of embeddings , beyond what we usually see in this type of datasets (mostly retrieval and semantic similarity). This is specially useful in the case of tasks related to CVE identifiers and CEW taxonomies

  • Language centric analysis is particularly interesting. we know most embedding models are English centric. in this paper the authors introduce parallel subsets between English and Chinese to evaluate confounding

Critical weaknesses
  • Limited novelty beyond constructing and adapting a domain specific benchmark. In this case we can say the proposed benchmark mainly follow the structure of existing embedding eval frameworks, for example MTEB . the same we could say about task families (categorizations) and metrics

in that sense the core value in this paper is the construction of cybersecurity specific eval data . in that sense it would be useful to distinguish tasks that genuinely introduce new eval challenges from those that primarily "substitute" cybersecurity text into existing task formulations

  • i think the public leaderboard transfer conclusion is stronger that the statistical evidence supports: as we can see on the paper , only seven models are included in this comparison with the MTED .and i can we we got statistics \rho = 0.43 and p = 0.34 for n =7 but the TD value = 0.33 which shows that ~21 models pairs reverse their relative ranking -> This shows some ranking instability but the correlation seems inconclusive. I ask the authors to correct me if my assessment is wrong in this case

  • it seems to me the aggregate SEE score is model dependent, like, 5/6 components of SEE use a min max normalization. this is known to be a problem in some cases as we are introducing a methodological limitation -> the normalized score of a model can change when additional models are included.

Questions for authors
  • given that the comparison with MTEB includes only seven models and yields a statistically non significant correlation, what support the strong conclusion that "general purpose" leaderboard ranking fail to transfer to cyber tasks?

  • how stable are SEE scores ranking when the evaluated model pool changes? like, could the authors consider normalization against fixed reference points instead of the observed model min / max?

Reviewer_qHsyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

SecEmbedEval provides a comprehensive framework for measuring cybersecurity text embedding quality and demonstrates why cybersecurity-specific evaluation is necessary for selecting and developing effective embedding models.

Critical strengths
  1. The paper combines across security texts across two different languages. Such cross-lingual datasets are limited in cybersecurity and is potentially useful to the field.
Critical weaknesses
  1. Given the abundance of current LLM and agent benchmarks for cybersecurity tasks, the paper fails to adequately demonstrate the need for a cybersecurity-specific embedding benchmark. A potential use could be in distinguishing failures caused by retrieval, reasoning, or domain knowledge. However, the paper does not sufficiently motivate its usefulness.
  2. Section 3 specifies that the dataset was constructed from public sources such as, "vendor advisories, threat reports and blogs, security news and social media, and exploit repositories". But the specific sources are not detailed or cited, making it hard to judge the soundness.
  3. The cybersecurity tasks seem relatively trivial because they primarily involve matching text similarity or predicting predefined labels, rather than requiring deeper cybersecurity understanding. While CWE classification has practical applications, mapping vulnerability descriptions to CWE classes or assigning reports to six broad threat categories largely tests conventional text classification capabilities. It is unclear whether these tasks require cybersecurity-specific representations or offer meaningful challenges beyond standard NLP benchmarks.
  4. The paper provides limited justification for their selection of models, including for classification and clustering. There is no systematic hyperparameter sensitivity analysis. This matters specifically for clustering because performance can be highly sensitive to these choices.
  5. The limited number of human annotation samples and the lack of regular comparisons between synthetic and real cases raise concerns regarding dataset authenticity. A stronger data validation is needed.
  6. The figures are insufficiently descriptive, and the paper overall lacks key methodological and experimental details necessary for clarity and reproducibility.
Questions for authors
  1. What additional insights does SecEmbedEval provide beyond existing cybersecurity LLM and agent benchmarks?
  2. How do the authors justify the cybersecurity-specific nature and difficulty of the selected tasks?
  3. Could authors please clarify key methodological details as stated in the weakness, including: data sources, model and hyperparameter selection and dataset validation?
1 comment

Thank you for the careful summary and for noting the value of the bilingual data, which remains scarce in cybersecurity. We answer the six concerns in turn and tie each one to a concrete revision.

1. Why an embedding benchmark, given the existing LLM and agent benchmarks

CyberMetric, CyberSecEval, CTIBench and ExCyTIn-Bench test reasoning, generation and knowledge recall over a supplied context. None of them measures the representation layer that supplies that context. In a retrieval-augmented security pipeline a wrong answer can come from retrieval, reasoning or domain knowledge, and each calls for a different fix. SecEmbedEval isolates the first. It also settles a model-selection question those benchmarks cannot: on our data the three security-pretrained encoders hold the last three places despite being pretrained on this literature, and public standing does not predict in-domain order. We will state this positioning in the Introduction.

2. Data sources

NVD feeds supply STS-3 and C1 text and labels. The MITRE CWE catalogue supplies C2 labels, read directly off the View-1000 hierarchy. CNNVD advisories and the crawled stream of vendor advisories, threat reports, blogs, security news and exploit-repository pages supply STS-1, STS-2, C3, clustering and retrieval. We will add a per-task provenance table to the appendix with source, counts and access dates. The NVD and CWE fetch scripts are part of the released code.

3. Are the tasks trivial classification

The tasks mirror how security pipelines consume embeddings: deduplication, term and alias resolution, identifier lookup, label triage, campaign grouping and corpus search. Several are far from solved. All twelve models sit in the chance band on CVE identifier grounding, AUC 0.504 to 0.549, including the strongest commercial system. The best clustering score is BCubed F1 0.365, read against a random floor of 0.228 and a trivial ceiling of 0.524, on ground truth where 41.9% of instances carry more than one category. Only one model clears 0.65 AUC on term-level similarity. A TF-IDF probe under the same protocol is last of thirteen on C3 at 0.410 against 0.718 for the leader, so the label tasks are not solvable by lexical matching either. We will add lexical baselines for similarity: bigram-Jaccard reaches AUC 0.624 on STS-1, below eleven of the twelve models, and content-word Jaccard reaches 0.479 on STS-1-en, below all twelve.

4. Model selection and hyperparameter sensitivity

Appendix Table 24 documents the selection rationale across scale, architecture family, instruction tuning, multilinguality, domain specialization and deployment setting. The read-out is fixed rather than tuned per model: logistic regression with 5-fold cross-validated C from 0.001 to 100, and clustering over three paradigms, KMeans at k equal to the number of categories over 10 random states, DBSCAN at eps 0.3, 0.5 and 0.7, and agglomerative under two metrics, with a degeneracy audit and seed-stability tables. We will add a k-sensitivity check for k from 4 to 8.

5. Dataset validation

Verification is per task and stated in Appendix A. A blind second pass admits 90.7% of STS-2 and 89.2% of STS-2-en candidates. Pilot inter-annotator agreement was kappa 0.851 on phrase pairs and 0.939 on sentence pairs. C1 mixes exactly 40 real NVD records with 10 generated ones per class, so provenance cannot predict the label. We have since extended human checking to the final retrieval judgments: on 300 pooled pairs sampled adversarially, binary agreement between the judge and a human reviewer is 93.7%, with Cohen's kappa 0.871. The revision adds a human-validated sample for each LLM-assisted task and a real-versus-synthetic score breakdown for C1.

Thank You

We thank the reviewer again for the time spent on our submission. Each point above maps to a concrete change, and we are glad to clarify any of them further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.