acceptodds
Under review as a conference paper at ICLR 2027

Phishing Email Clustering Using Quasi-Ground-Truth Machine Learning

Abstract

Fast and precise detection of new and unknown phishing campaigns is essential for incident response teams and, more broadly, for overall cybersecurity. However, no reliable automated phishing detection methods exist, largely due to varying incident clustering criteria and the scarcity of ground-truth datasets. We address the problem by providing methods for creating quasi-ground truth, enabling the development of two machine-learning clustering models with a Siamese architecture. The models, trained on a new dataset of phishing email, prove transferable onto a test dataset of a different origin. While the generalization of the model's application is not fully satisfactory, we find that it may be improved through cooperation between the two models. The new phishing dataset has been released publicly.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
phishing, clustering, campaign, Siamese network, ground-truth

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_v3dfOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This article studies the phishing detection method, particularly about the training dataset. It proposes a Siamese structure for the quasi GT collection while also claims that a new evaluation method is proposed.

Critical strengths

The strength of this article perhaps lies in the Siamese model for the quasi GTs. Taken texts and URGs as two types of input, this model helps prepare the GT automatically.

Critical weaknesses

This article is hard for me from different domain. I do have some concerns.

  1. The main Siamese structure is rather rough without enough details. I cannot clearly see the network structure even though the texts themselves try to explain it. In addition, I don't know how to train it. In addition, what is the loss?. More details and visual explanation can be better for me.
  2. The article is very boring because there is no enough graphical interpretations on the proposed principles. For example, what's the difference between a total GT and a quasi GT? What is the main pipeline of this work? What is the target of this study? Those missing details make the article very different to follow.
  3. The conclusion is too long. If you want to discuss something, it is better to have an independent (sub-)section on it.
  4. There is no enough theoretical analysis on the proposed method. It is more like a engineering work rather than a scientific study.
  5. The experiments are rough and simple. Only three experiments are difficult to demonstrate the particular merits proposed by the article. In addtion, ablation study are generally required to show the advantages of your methodology.
Questions for authors

See my comments on the weakness for my questions. I am not from this particular domain but your writing and presentation at least make the article difficult to accept.

Reviewer_Jg8KOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This study proposes a novel approach to phishing-email campaign clustering when reliable ground-truth labels are unavailable. The authors develop a cross-modal supervision strategy to generate quasi-ground-truth labels. Text features (BERT and PCA) and URL features (FastText and path entropy) are used to train two Siamese networks for message similarity, alongside a late-fusion method combining their outputs. Results show high pairwise AUC on the internal quasi-ground-truth data but notable degradation on an external dataset, with late fusion providing only slight performance gains.

Critical strengths
  1. The paper addresses a vital operational issue for CERTs by proposing a solution to the real-world scarcity of reliable campaign labels, aiming to reduce the need for manual annotation.

  2. The authors commendably evaluate their method on an external dataset (openly reporting performance drops) and explore the fusion of text and URL features.

Critical weaknesses
  1. The core assumption (that URL and text similarities reliably map to true campaign identities) is flawed, as different campaigns often share infrastructure or narratives. High internal AUC merely reflects the model's agreement with its own pseudo-labels. The lack of human validation or systematic error analysis constitutes a critical methodological gap.

  2. The external dataset is too small (111 messages, 6 campaigns) to establish real-world applicability. Moreover, optimizing fusion parameters directly on the test set introduces severe model-selection bias. Without proper comparisons to established baselines, the claimed clustering improvements remain unproven.

Questions for authors

Can the authors provide a human-annotated evaluation of the pseudo-label quality, including false merges, false splits, and inter-annotator agreement?

Reviewer_yzFbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the complex challenge of phishing email clustering by proposing two Siamese-based machine learning models (one focused on textual analysis, the other on URL characteristics). To address the chronic lack of labelled datasets, the authors introduce a methodology to generate a "pseudo-ground-truth" by cross-referencing textual and URL similarities. The authors then evaluate the models individually and in combination (via distance aggregation) on an external reference dataset.

Critical strengths

The paper's primary innovation is the asymmetric method it develops for generating pseudo-ground truth. Leveraging URL-based similarities to supervise and cluster text (and vice versa) is an ingenious, highly effective approach to overcoming label scarcity in real-world operational scenarios (such as CERT environments). Furthermore, using late fusion for the two models provides strong architectural flexibility, allowing the system to operate effectively even if attackers deliberately obfuscate one modality (text or URL).

Critical weaknesses

The primary methodological weakness emerges from the drastic performance degradation (an estimated drop of 20-30 percentage points in AUC) when the models are transferred from the proprietary training dataset (spamtrap) to the external reference dataset. Although the authors report this issue, the paper lacks a rigorous discussion of the causes of this distribution shift or of domain adaptation techniques to mitigate its impact.

Second, compressing the textual representations (BERT embeddings) to just 50 principal components via PCA seems like an overly aggressive information bottleneck. This drastic dimensionality reduction risks obliterating the semantic micro-variations that, in advanced social engineering contexts, constitute the only real differentiating factor between distinct yet thematically overlapping phishing campaigns.

Questions for authors

Regarding the marked performance drop during validation on the external dataset, have you considered integrating unsupervised domain adaptation techniques to improve the Siamese architectures' generalisation?

Reducing the BERT embeddings to just 50 PCA components is particularly severe. Could you provide additional details or a formal ablation study demonstrating that this specific compression does not compromise the capture of essential semantic signals required for campaign differentiation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.