- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes CDetect, a detector of LLM-driven social bots that operates on a user’s text history without training. It encodes content words with DeBERTa, averages their contextual representations per word type, fits a mixture of von Mises Fisher distributions, and uses the weighted inverse concentration as a bot score. The authors provide a finite-sample bound for this score and report an average AUROC of 0.94 on BotSim-24 and two new benchmarks built from TwiBot-22.
Critical strengths
Bot detection is an account-level task, and aggregating passage-level detector scores across short, topically scattered posts is fragile. Modeling the full history as a set of word types is a reasonable response. It also makes the method invariant to post order by construction, which Table 8 confirms. The method is cheaper than detectors that require 7B language models. These points support the motivation.
Critical weaknesses
-
The geometric score is not compared against simple lexical baselines, and its sign is unstable. Theorem 1 shows that the score depends on the number of retained word types and on their repetition rate. Both are direct lexical quantities. Lexical diversity, burstiness, repetition, and simplicity features are well-known signals for separating human from synthetic text. They can be computed at almost no cost or estimated with an LLM judge, as in G-Eval. None of these baselines is reported, and the correlation between the score and these quantities is not analyzed. The static embedding ablation does not answer this concern, since it removes frequency information. In addition, Figure 6 shows that the signal reverses with the encoder or layer. BERTweet at layer 24 yields an AUROC of 0.10 on BotSim-24, and static embeddings yield 0.41. A score whose direction depends on the representation is unlikely to measure a stable property of the text. This matters because it questions both the novelty and the interpretability claims.
-
The evaluation favors the method and does not reflect real conditions. All bots are synthetic, while all humans are real. In TwiLLM-R, bots are obtained by rewriting rule-based bots' posts, which are already topically repetitive. In TwiLLM-I, each bot produces exactly 30 texts conditioned on a fixed persona and its own recent posts, which induces concentration by design. Human histories vary in length, and the paper does not show that history length is matched across classes. When humans are also topically concentrated, the advantage disappears. On the politics subset (Table 11), Fast-DetectGPT obtains a higher AUPRC than CDetect. We exclude accounts with fewer than 3 or 5 posts, so we never evaluate the cold-start regime. With short histories, the concentration estimate is biased upward, which would flag new human accounts as bots. There is no adversarial evaluation, no second generator, and no real LLM bot. TwiLLM contains only 131 humans and 89 bots. The method also requires labeled data to set the threshold and to select the encoder and layer, so the training-free claim is limited. This matters because the paper presents the method as a practical detector.
Questions for authors
How does CDetect compare with a simple baseline built on lexical diversity, burstiness, and repetition features, and how strongly does the score correlate with the number of word types and history length?
What is the detection performance when humans and bots are matched by history length, including accounts with fewer than 5 posts?
4 comments
Response to Reviewer UCs9 [2/2]
Q4: Dataset clarification and additional evaluation on public social platforms (Weakness 2)
A4: Our main evaluation is BotSim-24 [1], an independently developed large-scale benchmark with 2,394 accounts. In this benchmark (Table 1), DNA-DetectLLM is the strongest baseline, with 0.8898 AUROC and 0.8150 AUPRC. CDetect achieves 0.9964 and 0.9839 respectively, outperforming all baselines. We constructed two TwiLLM benchmarks primarily to complement the evaluation by examining how different methods perform across interaction scenarios, broadening platforms, dataset construction, and generation models.
To evaluate CDetect in more diverse social settings, we extended our evaluation to two additional benchmarks.
(a) Weibo platform [2] with 2,131 human accounts and 339 agent accounts after filtering empty text records.The result is shown below. Since the benchmark is class-imbalanced, AUPRC shows the trade-off between precision and recall for the minority class across decision thresholds. CDetect achieves the highest AUPRC of 0.6410, exceeding the strongest baseline by 21.1%.
| Method | AUROC | AUPRC |
|---|---|---|
| OpenAI RoBERTa | 0.5439 | 0.1378 |
| RADAR | 0.2872 | 0.0896 |
| Log Likelihood | 0.7366 | 0.2664 |
| Fast-DetectGPT | 0.7897 | 0.4300 |
| DNA-DetectLLM | 0.6730 | 0.2450 |
| Binoculars | 0.7969 | 0.4098 |
| CDetect | 0.7942 | 0.6410 |
(b) MoltGraph [3], which links Moltbook posts and comments to their authors. We use these histories for the agent class and TwiBot-22 for the human class. We randomly sample 2,000 accounts per class with seed 42. As shown in the table below, CDetect outperforms all baselines. Together with the Weibo results, this evaluation shows the applicability of CDetect across diverse social media and generation settings.
| Method | AUROC | AUPRC |
|---|---|---|
| OpenAI RoBERTa | 0.2502 | 0.4127 |
| RADAR | 0.1404 | 0.3447 |
| Log Likelihood | 0.5670 | 0.6027 |
| Fast-DetectGPT | 0.5861 | 0.6432 |
| Binoculars | 0.5956 | 0.6556 |
| DNA-DetectLLM | 0.6783 | 0.7020 |
| CDetect | 0.9795 | 0.9730 |
[1] Qiao, Boyu, et al. "Botsim: Llm-powered malicious social botnet simulation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 13. 2025.
[2] Han, Yi, et al. "Toward Detecting Modern Social Media Agents: Dataset, Experimental Analysis, and an Agent-Based Solution." Findings of the Association for Computational Linguistics: EMNLP 2026. 2026.
[3] Mukherjee, Kunal, Cuneyt Akcora, and Murat Kantarcioglu. "Moltgraph: A longitudinal temporal graph dataset of moltbook for coordinated-agent detection." Proceedings of the ACM Conference on AI and Agentic Systems. 2026.
Q5: Evaluation with matched history lengths (Weakness 2 and Question 2)
A5: Thank you for your comment. Figure 3 already reports results with exactly matched history lengths for human and bot accounts at each point on the horizontal axis. Across the evaluated text lengths from 64 to 768, CDetect outperforms all baselines on both BotSim-24 and TwiLLM.
Q6: The politics result and performance on topically concentrated human accounts (Weakness 2)
A6: The Politics evaluation in Appendix H.1 shows that CDetect achieves a lower AUPRC than Fast-DetectGPT (0.6823 vs. 0.7492) and a higher AUROC (0.9240 vs. 0.8308). To further assess how much topical concentration explains CDetect’s performance, we use BotSim-24’s original subreddit labels to count the number of topics. As shown in the table below, CDetect can distinguish the two classes even when human accounts cover as many or fewer subreddits than bot accounts.
| Setting | Human | Bot | AUROC | AUPRC |
|---|---|---|---|---|
| 1-topic human vs. 1-topic bot | 569 | 229 | 0.9937 | 0.9552 |
| 1-topic human vs. ≥3-topic bot | 569 | 282 | 0.9932 | 0.9597 |
Q7: About training-free (Weakness 2)
A7: CDetect is a training-free scoring method, consistent with prior training-free detectors such as Binoculars and Fast-DetectGPT. This terminology follows the common convention in the field. All main conclusions in the paper and these prior papers are based on AUROC and AUPRC and are independent of the decision threshold.
Q8: Detection with fewer than five posts (Weakness 2 and Question 2)
A8: In practice, social bots detection typically focuses on accounts that actively post, to mitigate harms such as opinion manipulation, scams, and harassment. Therefore, we focused our main evaluation on accounts with at least five posts. We also extended our evaluation to accounts with fewer than five posts. The result show that CDetect still achieves the best performance.
| Method | AUROC | AUPRC |
|---|---|---|
OpenAI RoBERTa | 0.5811 | 0.3923 |
| RADAR | 0.6328 | 0.3685 |
| Log Likelihood | 0.5578 | 0.3142 |
| Fast-DetectGPT | 0.5269 | 0.3323 |
| Binoculars | 0.5539 | 0.3275 |
| DNA-DetectLLM | 0.5669 | 0.3387 |
| CDetect | 0.7615 | 0.4480 |
Looking forward to your response
Dear Reviewer,
Thank you again for reviewing our paper! We have posted our responses to your questions and concerns. We are wondering if you have a new evaluation of our paper after reading our responses. If you have other questions, we are very willing to discuss them with you.
Best,
Authors
I read the responses, but it is not clear to me how they address the critical weaknesses identified.
Response to Reviewer UCs9 [1/2]
We sincerely thank you for the constructive comments. Below is our point-by-point response.
A1: On BotSim-24, TwiLLM-R, and TwiLLM-I, we compare CDetect with the type–token ratio (TTR), a random forest using 13 text features (Text-feature RF), and logistic regression using word unigram and bigram TF-IDF features (TF-IDF + LR). TTR measures lexical diversity. Text-feature RF includes TTR and the proportion of distinct bigrams, which captures bigram repetition. All methods use the same account histories within each evaluation setting.
(b) Conventional text baselines on BotSim-24 and TwiLLM. For LLM-driven bot detection, we compare CDetect with the type–token ratio (TTR), a random forest using 13 text features (Text-feature RF), and logistic regression using word unigram and bigram TF-IDF features (TF-IDF + LR). All methods use the same account histories within each evaluation setting.
In-domain evaluation. Each dataset is stratified at the account level into training, validation, and test splits (60% / 20% / 20%, seed = 42). We fit RF and TF-IDF + LR on the in-domain training split and select the LR regularization parameter on the validation split. All methods are evaluated on the same test split. The results are shown in the table below as AUROC / AUPRC.
Cross-dataset transfer of supervised baselines. We fit RF and TF-IDF + LR using the source dataset, then apply the resulting models to the full target dataset. The arrows indicate the training and evaluation datasets for these two supervised baselines. CDetect and TTR score target accounts directly without dataset-specific training. The results are shown in the table below as AUROC / AUPRC. Both supervised baselines score much lower in the cross-dataset evaluations than in their in-domain evaluations. Overall, generalization is a big problem for lexical supervised methods, whereas training-free method TTR just does not work.
A2: We computed Spearman correlations with the number of retained word types and the total word count. The table below reports Spearman rank correlations between the CDetect score and the number of retained word types or the total word count of the account history. Across all three datasets, both correlations are weak and their signs vary across datasets.
A3: We agree that the choice of representation matters, but this observation does not warrant the conclusion that “the score is unlikely to measure a stable property of the text.” CDetect uses contextualized word representations to measure the directional concentration of an account’s contextualized word vectors.
The overall trend in Figure 6 supports this premise. Performance improves at intermediate and higher layers across all four encoders, despite differences in their pretraining objectives and corpora. The signal strengthens as the representations become more contextualized, consistent with the static embedding ablation. These results indicate that intermediate to deep contextual layers are the intended operating range for CDetect. Final-layer representations may be affected by pretraining objectives, domain-specific corpora, or anisotropy, which is not that contextual.