- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SecEmbedEval is a Chinese/English benchmark for cybersecurity text embeddings, with nine subtasks over similarity, retrieval, classification, clustering, and efficiency, built from public sources with heavy LLM assistance. It evaluates twelve models and proposes two aggregates, SEE-Score and TD. The paper refers to released code and data but doesn't link to either.
Critical strengths
-
Security text is a real gap in embedding evaluation, and the reference points are unusually careful (TF-IDF and random probes in Table 10, a floor, ceiling, and degeneracy audit for clustering in Table 25).
-
Holm-corrected leader tests (Table 14) are reported candidly, even though they show most top-1 margins aren't significant.
Critical weaknesses
-
The headline findings are either known or stated beyond the evidence. FinMTEB (Tang & Yang, 2025) already showed on bilingual finance data that general benchmark scores predict domain performance poorly, and it isn't discussed. Here the claim rests on seven models with , and since TD equals , means two thirds of pairs agree. That is weak transfer measured with little power, not its absence. The "objective, not corpus" conclusion pits frozen 110M MLM encoders against 0.6–8B contrastive models, a long-documented gap, without a contrastive security encoder (e.g., ATT&CK-BERT) to isolate the cause. The claim that cost barely tracks score contradicts Tables 5 and 19, where batch-25 throughput and SEE-Score give Spearman over all twelve models and over the nine general-purpose ones.
-
Much of the ground truth is produced or checked by LLMs and never validated by humans on the final data (Table 6 covers pilot samples only). The 12,964 retrieval relevance labels come from an unnamed judge whose only reported check is 99.0% agreement with itself (Table 21), over a pool with no BM25 run. STS-1 pairs are admitted through relation-specific bigram-Jaccard bands, which may leave a surface-overlap shortcut, yet no lexical baseline is reported for similarity or retrieval. Appendix F even maps CWE-79 to "输入验证和表示", which isn't a View-1000 Pillar.
Questions for authors
Clarification requests:
- Which LLMs generated candidates and judged relevance, and how well does a human-annotated sample of the final qrels agree with the judge? A Qwen-family judge could favor aliyun-qwen3.7.
- What AUC do Jaccard and BM25 reach on STS-1 and STS-1-en, and what NDCG@10 does BM25 reach on retrieval?
Suggestions for future work:
- Contrastively fine-tuning SecureBERT, next to a general-domain BERT-base, would test the objective-versus-domain claim directly.
Thank you for the careful summary and for noting the value of the bilingual data, which remains scarce in cybersecurity. We answer the six concerns in turn and tie each one to a concrete revision.
1. Why an embedding benchmark, given the existing LLM and agent benchmarks
CyberMetric, CyberSecEval, CTIBench and ExCyTIn-Bench test reasoning, generation and knowledge recall over a supplied context. None of them measures the representation layer that supplies that context. In a retrieval-augmented security pipeline a wrong answer can come from retrieval, reasoning or domain knowledge, and each calls for a different fix. SecEmbedEval isolates the first. It also settles a model-selection question those benchmarks cannot: on our data the three security-pretrained encoders hold the last three places despite being pretrained on this literature, and public standing does not predict in-domain order. We will state this positioning in the Introduction.
2. Data sources
NVD feeds supply STS-3 and C1 text and labels. The MITRE CWE catalogue supplies C2 labels, read directly off the View-1000 hierarchy. CNNVD advisories and the crawled stream of vendor advisories, threat reports, blogs, security news and exploit-repository pages supply STS-1, STS-2, C3, clustering and retrieval. We will add a per-task provenance table to the appendix with source, counts and access dates. The NVD and CWE fetch scripts are part of the released code.
3. Are the tasks trivial classification
The tasks mirror how security pipelines consume embeddings: deduplication, term and alias resolution, identifier lookup, label triage, campaign grouping and corpus search. Several are far from solved. All twelve models sit in the chance band on CVE identifier grounding, AUC 0.504 to 0.549, including the strongest commercial system. The best clustering score is BCubed F1 0.365, read against a random floor of 0.228 and a trivial ceiling of 0.524, on ground truth where 41.9% of instances carry more than one category. Only one model clears 0.65 AUC on term-level similarity. A TF-IDF probe under the same protocol is last of thirteen on C3 at 0.410 against 0.718 for the leader, so the label tasks are not solvable by lexical matching either. We will add lexical baselines for similarity: bigram-Jaccard reaches AUC 0.624 on STS-1, below eleven of the twelve models, and content-word Jaccard reaches 0.479 on STS-1-en, below all twelve.
4. Model selection and hyperparameter sensitivity
Appendix Table 24 documents the selection rationale across scale, architecture family, instruction tuning, multilinguality, domain specialization and deployment setting. The read-out is fixed rather than tuned per model: logistic regression with 5-fold cross-validated C from 0.001 to 100, and clustering over three paradigms, KMeans at k equal to the number of categories over 10 random states, DBSCAN at eps 0.3, 0.5 and 0.7, and agglomerative under two metrics, with a degeneracy audit and seed-stability tables. We will add a k-sensitivity check for k from 4 to 8.
5. Dataset validation
Verification is per task and stated in Appendix A. A blind second pass admits 90.7% of STS-2 and 89.2% of STS-2-en candidates. Pilot inter-annotator agreement was kappa 0.851 on phrase pairs and 0.939 on sentence pairs. C1 mixes exactly 40 real NVD records with 10 generated ones per class, so provenance cannot predict the label. We have since extended human checking to the final retrieval judgments: on 300 pooled pairs sampled adversarially, binary agreement between the judge and a human reviewer is 93.7%, with Cohen's kappa 0.871. The revision adds a human-validated sample for each LLM-assisted task and a real-versus-synthetic score breakdown for C1.
Thank You
We thank the reviewer again for the time spent on our submission. Each point above maps to a concrete change, and we are glad to clarify any of them further.