acceptodds
Under review as a conference paper at ICLR 2027

The Geometry of Word Usage: Training-Free Detection of LLM-Driven Social Bots

Abstract

The proliferation of large language models (LLMs) has enabled a new generation of highly human-like social bots, posing significant challenges for reliable detection. Social bot detection is a user-level problem from many fragmented and multi-topic user histories. However, existing AI-text detectors operate at the passage level and degrade under such conditions. We propose a fundamentally different perspective: instead of modeling text likelihood, we characterize the geometry of repeated contextual word usage across a user’s history. Building on this insight, we introduce CDetect, a training-free framework that captures user-level semantic concentration. Specifically, CDetect encodes historical posts, aggregates contextualized representations of content words into representative directions, and models their distribution via a mixture of von Mises–Fisher distributions on the unit hypersphere. The resulting mixture-weighted diversity score quantifies whether a user exhibits semantically dispersed (human-like) or concentrated (LLM-like) language usage patterns. On the theoretical side, we establish finite-sample guarantees for the estimated diversity score, explicitly characterizing how vocabulary coverage and repeated contextual occurrences govern its statistical reliability. Extensive experiments on BotSim-24 and two TwiLLM benchmarks demonstrate that CDetect consistently outperforms both training-based and training-free baselines, achieving an average AUROC of 0.9416 and AUPRC of 0.9203. Further analyses confirm its robustness across varying history lengths, text budgets, and modeling choices.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LLM-Driven Social Bots Detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UCs9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes CDetect, a detector of LLM-driven social bots that operates on a user’s text history without training. It encodes content words with DeBERTa, averages their contextual representations per word type, fits a mixture of von Mises Fisher distributions, and uses the weighted inverse concentration as a bot score. The authors provide a finite-sample bound for this score and report an average AUROC of 0.94 on BotSim-24 and two new benchmarks built from TwiBot-22.

Critical strengths

Bot detection is an account-level task, and aggregating passage-level detector scores across short, topically scattered posts is fragile. Modeling the full history as a set of word types is a reasonable response. It also makes the method invariant to post order by construction, which Table 8 confirms. The method is cheaper than detectors that require 7B language models. These points support the motivation.

Critical weaknesses
  1. The geometric score is not compared against simple lexical baselines, and its sign is unstable. Theorem 1 shows that the score depends on the number of retained word types and on their repetition rate. Both are direct lexical quantities. Lexical diversity, burstiness, repetition, and simplicity features are well-known signals for separating human from synthetic text. They can be computed at almost no cost or estimated with an LLM judge, as in G-Eval. None of these baselines is reported, and the correlation between the score and these quantities is not analyzed. The static embedding ablation does not answer this concern, since it removes frequency information. In addition, Figure 6 shows that the signal reverses with the encoder or layer. BERTweet at layer 24 yields an AUROC of 0.10 on BotSim-24, and static embeddings yield 0.41. A score whose direction depends on the representation is unlikely to measure a stable property of the text. This matters because it questions both the novelty and the interpretability claims.

  2. The evaluation favors the method and does not reflect real conditions. All bots are synthetic, while all humans are real. In TwiLLM-R, bots are obtained by rewriting rule-based bots' posts, which are already topically repetitive. In TwiLLM-I, each bot produces exactly 30 texts conditioned on a fixed persona and its own recent posts, which induces concentration by design. Human histories vary in length, and the paper does not show that history length is matched across classes. When humans are also topically concentrated, the advantage disappears. On the politics subset (Table 11), Fast-DetectGPT obtains a higher AUPRC than CDetect. We exclude accounts with fewer than 3 or 5 posts, so we never evaluate the cold-start regime. With short histories, the concentration estimate is biased upward, which would flag new human accounts as bots. There is no adversarial evaluation, no second generator, and no real LLM bot. TwiLLM contains only 131 humans and 89 bots. The method also requires labeled data to set the threshold and to select the encoder and layer, so the training-free claim is limited. This matters because the paper presents the method as a practical detector.

Questions for authors

How does CDetect compare with a simple baseline built on lexical diversity, burstiness, and repetition features, and how strongly does the score correlate with the number of word types and history length?

What is the detection performance when humans and bots are matched by history length, including accounts with fewer than 5 posts?

4 comments
AuthorsOpenReview ↗

Response to Reviewer UCs9 [1/2]

We sincerely thank you for the constructive comments. Below is our point-by-point response.

Q1: Comparison with lexical baselines (Weakness 1 and Question 1)

A1: On BotSim-24, TwiLLM-R, and TwiLLM-I, we compare CDetect with the type–token ratio (TTR), a random forest using 13 text features (Text-feature RF), and logistic regression using word unigram and bigram TF-IDF features (TF-IDF + LR). TTR measures lexical diversity. Text-feature RF includes TTR and the proportion of distinct bigrams, which captures bigram repetition. All methods use the same account histories within each evaluation setting.

(b) Conventional text baselines on BotSim-24 and TwiLLM. For LLM-driven bot detection, we compare CDetect with the type–token ratio (TTR), a random forest using 13 text features (Text-feature RF), and logistic regression using word unigram and bigram TF-IDF features (TF-IDF + LR). All methods use the same account histories within each evaluation setting.

In-domain evaluation. Each dataset is stratified at the account level into training, validation, and test splits (60% / 20% / 20%, seed = 42). We fit RF and TF-IDF + LR on the in-domain training split and select the LR regularization parameter on the validation split. All methods are evaluated on the same test split. The results are shown in the table below as AUROC / AUPRC.

MethodSupervisionBotSim-24TwiLLM-RTwiLLM-I
Text-feature RFIn-domain supervision0.9993 / 0.99870.8333 / 0.84361.0000 / 1.0000
TF-IDF + LRIn-domain supervision1.0000 / 1.00000.9466 / 0.92681.0000 / 1.0000
TTRTraining-free0.5942 / 0.41790.3803 / 0.34220.4402 / 0.3440
CDetectTraining-free0.9990 / 0.99810.7863 / 0.63370.9701 / 0.9600

Cross-dataset transfer of supervised baselines. We fit RF and TF-IDF + LR using the source dataset, then apply the resulting models to the full target dataset. The arrows indicate the training and evaluation datasets for these two supervised baselines. CDetect and TTR score target accounts directly without dataset-specific training. The results are shown in the table below as AUROC / AUPRC. Both supervised baselines score much lower in the cross-dataset evaluations than in their in-domain evaluations. Overall, generalization is a big problem for lexical supervised methods, whereas training-free method TTR just does not work.

MethodBotSim-24 → TwiLLM-RBotSim-24 → TwiLLM-ITwiLLM-R → BotSim-24TwiLLM-I → BotSim-24
Text-feature RF0.5085 / 0.45740.7724 / 0.78060.4935 / 0.31660.7970 / 0.5485
TF-IDF + LR0.7001 / 0.62580.8307 / 0.74490.7779 / 0.57470.8060 / 0.6474
TTR0.4512 / 0.42070.4375 / 0.33640.5944 / 0.41840.5944 / 0.4184
CDetect0.8543 / 0.80540.9741 / 0.97170.9964 / 0.98390.9964 / 0.9839

Q2: Correlation with word-type count and history length (Question 1)

A2: We computed Spearman correlations with the number of retained word types and the total word count. The table below reports Spearman rank correlations between the CDetect score and the number of retained word types or the total word count of the account history. Across all three datasets, both correlations are weak and their signs vary across datasets.

DatasetSpearman ρ with retained word-type countSpearman ρ with total word count
BotSim-24−0.2059−0.2162
TwiLLM-R−0.0679−0.1684
TwiLLM-I0.14690.1541

Q3: Score direction across encoders and layers (Weakness 1)

A3: We agree that the choice of representation matters, but this observation does not warrant the conclusion that “the score is unlikely to measure a stable property of the text.” CDetect uses contextualized word representations to measure the directional concentration of an account’s contextualized word vectors.

The overall trend in Figure 6 supports this premise. Performance improves at intermediate and higher layers across all four encoders, despite differences in their pretraining objectives and corpora. The signal strengthens as the representations become more contextualized, consistent with the static embedding ablation. These results indicate that intermediate to deep contextual layers are the intended operating range for CDetect. Final-layer representations may be affected by pretraining objectives, domain-specific corpora, or anisotropy, which is not that contextual.

AuthorsOpenReview ↗

Response to Reviewer UCs9 [2/2]

Q4: Dataset clarification and additional evaluation on public social platforms (Weakness 2)

A4: Our main evaluation is BotSim-24 [1], an independently developed large-scale benchmark with 2,394 accounts. In this benchmark (Table 1), DNA-DetectLLM is the strongest baseline, with 0.8898 AUROC and 0.8150 AUPRC. CDetect achieves 0.9964 and 0.9839 respectively, outperforming all baselines. We constructed two TwiLLM benchmarks primarily to complement the evaluation by examining how different methods perform across interaction scenarios, broadening platforms, dataset construction, and generation models.

To evaluate CDetect in more diverse social settings, we extended our evaluation to two additional benchmarks.

(a) Weibo platform [2] with 2,131 human accounts and 339 agent accounts after filtering empty text records.The result is shown below. Since the benchmark is class-imbalanced, AUPRC shows the trade-off between precision and recall for the minority class across decision thresholds. CDetect achieves the highest AUPRC of 0.6410, exceeding the strongest baseline by 21.1%.

MethodAUROCAUPRC
OpenAI RoBERTa0.54390.1378
RADAR0.28720.0896
Log Likelihood0.73660.2664
Fast-DetectGPT0.78970.4300
DNA-DetectLLM0.67300.2450
Binoculars0.79690.4098
CDetect0.79420.6410

(b) MoltGraph [3], which links Moltbook posts and comments to their authors. We use these histories for the agent class and TwiBot-22 for the human class. We randomly sample 2,000 accounts per class with seed 42. As shown in the table below, CDetect outperforms all baselines. Together with the Weibo results, this evaluation shows the applicability of CDetect across diverse social media and generation settings.

MethodAUROCAUPRC
OpenAI RoBERTa0.25020.4127
RADAR0.14040.3447
Log Likelihood0.56700.6027
Fast-DetectGPT0.58610.6432
Binoculars0.59560.6556
DNA-DetectLLM0.67830.7020
CDetect0.97950.9730

[1] Qiao, Boyu, et al. "Botsim: Llm-powered malicious social botnet simulation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 13. 2025.

[2] Han, Yi, et al. "Toward Detecting Modern Social Media Agents: Dataset, Experimental Analysis, and an Agent-Based Solution." Findings of the Association for Computational Linguistics: EMNLP 2026. 2026.

[3] Mukherjee, Kunal, Cuneyt Akcora, and Murat Kantarcioglu. "Moltgraph: A longitudinal temporal graph dataset of moltbook for coordinated-agent detection." Proceedings of the ACM Conference on AI and Agentic Systems. 2026.

Q5: Evaluation with matched history lengths (Weakness 2 and Question 2)

A5: Thank you for your comment. Figure 3 already reports results with exactly matched history lengths for human and bot accounts at each point on the horizontal axis. Across the evaluated text lengths from 64 to 768, CDetect outperforms all baselines on both BotSim-24 and TwiLLM.

Q6: The politics result and performance on topically concentrated human accounts (Weakness 2)

A6: The Politics evaluation in Appendix H.1 shows that CDetect achieves a lower AUPRC than Fast-DetectGPT (0.6823 vs. 0.7492) and a higher AUROC (0.9240 vs. 0.8308). To further assess how much topical concentration explains CDetect’s performance, we use BotSim-24’s original subreddit labels to count the number of topics. As shown in the table below, CDetect can distinguish the two classes even when human accounts cover as many or fewer subreddits than bot accounts.

SettingHumanBotAUROCAUPRC
1-topic human vs. 1-topic bot5692290.99370.9552
1-topic human vs. ≥3-topic bot5692820.99320.9597

Q7: About training-free (Weakness 2)

A7: CDetect is a training-free scoring method, consistent with prior training-free detectors such as Binoculars and Fast-DetectGPT. This terminology follows the common convention in the field. All main conclusions in the paper and these prior papers are based on AUROC and AUPRC and are independent of the decision threshold.

Q8: Detection with fewer than five posts (Weakness 2 and Question 2)

A8: In practice, social bots detection typically focuses on accounts that actively post, to mitigate harms such as opinion manipulation, scams, and harassment. Therefore, we focused our main evaluation on accounts with at least five posts. We also extended our evaluation to accounts with fewer than five posts. The result show that CDetect still achieves the best performance.

MethodAUROCAUPRC
OpenAI RoBERTa0.58110.3923
RADAR0.63280.3685
Log Likelihood0.55780.3142
Fast-DetectGPT0.52690.3323
Binoculars0.55390.3275
DNA-DetectLLM0.56690.3387
CDetect0.76150.4480
AuthorsOpenReview ↗

Looking forward to your response

Dear Reviewer,

Thank you again for reviewing our paper! We have posted our responses to your questions and concerns. We are wondering if you have a new evaluation of our paper after reading our responses. If you have other questions, we are very willing to discuss them with you.

Best,

Authors

I read the responses, but it is not clear to me how they address the critical weaknesses identified.

Reviewer_DyYBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CDetect, a text-only, training-free detector of LLM-driven social-media accounts. It encodes each account's history with a pretrained contextual encoder, averages normalized occurrences of each content-word type, normalizes these averages, and fits a mixture of von Mises-Fisher distributions to the resulting word-type directions. A mixture-weighted inverse-concentration score is used to rank accounts. The paper provides a conditional finite-sample analysis of score estimation, introduces rewriting and interaction variants of a Twitter benchmark, and evaluates detection on these variants and BotSim-24. It reports an average AUROC of 0.9416 and AUPRC of 0.9203, with additional history-budget, representation, topic-subset, and efficiency analyses.

Critical strengths
  1. A simple, potentially useful account-level signal with substantial reported gains. The method needs only account text and a fixed encoder, and Table 1 reports sizable improvements over the strongest evaluated training-free detector on all three benchmarks. The inductive, shared-validation comparison with graph-based detectors in Appendix F is also favorable. This combination of accessibility and reported predictive performance is a meaningful argument for acceptance.

  2. The evaluation includes several relevant controls rather than only a headline benchmark. History and word budgets, mixture size, encoder layers, contextual versus static embeddings, and topic-restricted subsets are examined. In particular, Table 9 shows that contextual representations contribute substantially, while Appendix H partially addresses the concern that the result is only a single-topic versus multi-topic distinction. These analyses make the proposed signal more credible, even though they do not fully identify its mechanism.

Critical weaknesses
  1. The benchmark construction does not yet establish that the signal generalizes beyond the particular human/synthetic provenance contrast. Both TwiLLM variants share only 131 human accounts and 89 bot-source accounts (Table 6), use GPT-5.4 Mini, and retain original human histories while generating the bot class from a different set of source accounts (Appendix D). Thus source-account characteristics, generation protocol, and the target label remain coupled. The two variants are useful but are not independent evidence across generators or human cohorts. The topic and history-budget controls help, but do not resolve this provenance confound. A matched original-versus-generated-history evaluation and a held-out generator/prompt evaluation would materially strengthen the central detection claim. This matters especially because semantic diversity itself can vary with account persona and posting behavior.

  2. The main interpretation in terms of repeated contextual word usage is not sufficiently isolated by the score or experiments. Equation (3) discards each word's mean-resultant length, so the fitted model measures concentration across normalized word-type mean directions, rather than directly measuring within-word contextual dispersion. Two occurrence sets can have the same normalized mean direction and different dispersion, yet be indistinguishable to the detector. The authors explicitly acknowledge discarding the norm, and the contextual/static ablation is valuable, but it does not establish that repeated-use modeling is responsible for the gain. A targeted same-encoder control, such as one occurrence per type or a score retaining within-word resultant lengths, would substantiate this interpretation. The current results support a promising contextual-geometry heuristic more strongly than the proposed account of why it works.

  3. The evaluation would benefit from broader dataset coverage and comparisons with conventional bot detection approaches. Although TwiLLM is derived from TwiBot-22, its bot histories are rewritten or generated by an LLM; this does not establish performance on the original, unmodified benchmark. I would therefore encourage evaluation on the original TwiBot-22, and potentially another established dataset such as TwiBot-20, to clarify the applicability and limitations of the semantic concentration signal beyond the constructed LLM-driven settings. I acknowledge the comparisons with BotRGCN, BotMoE, and BotDMM in Appendix F, but these are limited to BotSim-24. Extending these comparisons and including conventional baselines, such as feature-based random forests and text-only TF-IDF classifiers, would better establish the practical value of CDetect. Comparisons should clearly distinguish the available input modalities and supervision, with matched account histories for text-only methods. Since conventional bot benchmarks do not specifically target LLM-driven accounts, these experiments should be interpreted as tests of broader applicability rather than direct validation of the central LLM-detection claim.

Questions for authors

See the weakness part.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DyYB [1/2]

We sincerely thank you for the constructive comments. Below is our point-by-point response.

Q1: Dataset clarification (Weakness 1)

A1: Thank you for your comment! We constructed the two TwiLLM benchmarks primarily to complement our evaluation by examining how different methods perform across interaction scenarios. Our main evaluation includes 2,394 accounts from BotSim-24 [1], an independently developed, large-scale account-level benchmark.

[1] Qiao, Boyu, et al. "Botsim: Llm-powered malicious social botnet simulation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 13. 2025.

Q2: Additional evaluation of generalization beyond TwiLLM’s generator and prompt (Weakness 1)

A2: To evaluate CDetect in more realistic open-world settings where prompts and LLM generators can vary across agents, we extended our evaluation to Weibo and MoltGraph, two benchmarks based on actual agent activity on public social platforms.

(a) Weibo platform [2] with 2,131 human accounts and 339 agent accounts after filtering empty text records.The result is shown below. Since the benchmark is class-imbalanced, AUPRC shows the trade-off between precision and recall for the minority class across decision thresholds. CDetect achieves the highest AUPRC of 0.6410, exceeding the strongest baseline by 21.1%.

MethodAUROCAUPRC
OpenAI RoBERTa0.54390.1378
RADAR0.28720.0896
Log Likelihood0.73660.2664
Fast-DetectGPT0.78970.4300
DNA-DetectLLM0.67300.2450
Binoculars0.79690.4098
CDetect0.79420.6410

(b) MoltGraph [3], which links Moltbook posts and comments to their authors. We use these histories for the agent class and TwiBot-22 for the human class. We randomly sample 2,000 accounts per class with seed 42. As shown in the table below, CDetect outperforms all baselines. Together with the Weibo results, this evaluation shows the applicability of CDetect across diverse social media and generation settings.

MethodAUROCAUPRC
OpenAI RoBERTa0.25020.4127
RADAR0.14040.3447
Log Likelihood0.56700.6027
Fast-DetectGPT0.58610.6432
Binoculars0.59560.6556
DNA-DetectLLM0.67830.7020
CDetect0.97950.9730

[2] Han, Yi, et al. "Toward Detecting Modern Social Media Agents: Dataset, Experimental Analysis, and an Agent-Based Solution." Findings of the Association for Computational Linguistics: EMNLP 2026. 2026.

[3] Mukherjee, Kunal, Cuneyt Akcora, and Murat Kantarcioglu. "Moltgraph: A longitudinal temporal graph dataset of moltbook for coordinated-agent detection." Proceedings of the ACM Conference on AI and Agentic Systems. 2026.

Q3: The role of cross-context word-type aggregation in CDetect (Weakness 2)

A3: Good question! CDetect uses repeated occurrences of the same word type across contexts to estimate its representative direction. Equation (3) normalizes the mean vector to a unit direction for input to movMF on the unit hypersphere. Our core observation is that the representative directions of different word types tend to cluster more tightly in LLM-driven bot accounts.

To quantify the contribution of cross-context aggregation to detection performance, we varied only the input and evaulated in BotSim-24.

  • Single-occurrence. Following the reviewer’s suggestion to use one occurrence per type, we sampled one occurrence uniformly at random for each word type and used its unit vector as the input point. This lets us directly compare how estimating a word type’s representative direction from a single occurrence or from repeated occurrences across contexts affects detection performance.

  • All-occurrence. The reviewer points out that two sets of occurrences of the same word type can have the same mean direction but different degrees of dispersion. Equation (3) maps both sets to the same unit direction. To address this case, we used each occurrence’s unit vector as a separate input point for movMF fitting, preserving the difference in dispersion in the input. The result is shown in the table below.

MethodAUROCAUPRC
CDetect0.99640.9839
Single-occurrence0.99500.9781
All-occurrence0.99620.9832
AuthorsOpenReview ↗

Response to Reviewer DyYB [2/2]

Q4: Evaluation on original TwiBot-22 and comparison with conventional text baselines (Weakness 3)

A4: Thank you for suggesting a broader evaluation and additional conventional baselines.

(a) Original TwiBot-22. CDetect is designed to detect LLM-driven social bots using semantic concentration in account histories. The original TwiBot-22 benchmark targets conventional bot detection. In practice, we often can use different methods for different goals. We randomly sampled 500 human and 500 bot accounts from the official test split. The table below shows that all LLM-generated text detection methods have failed.

MethodAUROCAUPRC
OpenAI RoBERTa0.50550.4882
RADAR0.51940.5199
Log Likelihood0.51520.5236
Fast-DetectGPT0.46100.4683
Binoculars0.48410.4916
DNA-DetectLLM0.49200.5019
CDetect0.46870.4917

(b) Conventional text baselines on BotSim-24 and TwiLLM. For LLM-driven bot detection, we compare CDetect with the type–token ratio (TTR), a random forest using 13 text features (Text-feature RF), and logistic regression using word unigram and bigram TF-IDF features (TF-IDF + LR). All methods use the same account histories within each evaluation setting.

In-domain evaluation. Each dataset is stratified at the account level into training, validation, and test splits (60% / 20% / 20%, seed = 42). We fit RF and TF-IDF + LR on the in-domain training split and select the LR regularization parameter on the validation split. All methods are evaluated on the same test split. The results are shown in the table below as AUROC / AUPRC.

MethodSupervisionBotSim-24TwiLLM-RTwiLLM-I
Text-feature RFIn-domain supervision0.9993 / 0.99870.8333 / 0.84361.0000 / 1.0000
TF-IDF + LRIn-domain supervision1.0000 / 1.00000.9466 / 0.92681.0000 / 1.0000
TTRTraining-free0.5942 / 0.41790.3803 / 0.34220.4402 / 0.3440
CDetectTraining-free0.9990 / 0.99810.7863 / 0.63370.9701 / 0.9600

Cross-dataset transfer of supervised baselines. We fit RF and TF-IDF + LR using the source dataset, then apply the resulting models to the full target dataset. The arrows indicate the training and evaluation datasets for these two supervised baselines. CDetect and TTR score target accounts directly without dataset-specific training. The results are shown in the table below as AUROC / AUPRC. Both supervised baselines score much lower in the cross-dataset evaluations than in their in-domain evaluations. Overall, generalization is a big problem for lexical supervised methods, whereas training-free method TTR just does not work.

MethodBotSim-24 → TwiLLM-RBotSim-24 → TwiLLM-ITwiLLM-R → BotSim-24TwiLLM-I → BotSim-24
Text-feature RF0.5085 / 0.45740.7724 / 0.78060.4935 / 0.31660.7970 / 0.5485
TF-IDF + LR0.7001 / 0.62580.8307 / 0.74490.7779 / 0.57470.8060 / 0.6474
TTR0.4512 / 0.42070.4375 / 0.33640.5944 / 0.41840.5944 / 0.4184
CDetect0.8543 / 0.80540.9741 / 0.97170.9964 / 0.98390.9964 / 0.9839
AuthorsOpenReview ↗

Looking forward to your response

Dear Reviewer,

Thank you again for reviewing our paper! We have posted our responses to your questions and concerns. We are wondering if you have a new evaluation of our paper after reading our responses. If you have other questions, we are very willing to discuss them with you.

Best,

Authors

Reviewer_CYK4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies account-level detection of LLM-driven social bots. The authors point out that existing AI text detectors such as Fast-DetectGPT and Binoculars are designed for passage-level, semantically coherent long texts, while the historical content of a social account is fragmented and spans multiple topics, so naively aggregating passage-level scores performs poorly. To address this, the paper proposes CDetect, a training-free detection framework whose core idea is to move away from text likelihood and instead characterize the geometry of word usage. CDetect encodes the full history of a user with DeBERTa-v3-large, averages and normalizes the contextual occurrences of each content word after stopword removal to obtain a representative direction for that word, and fits a mixture of von Mises-Fisher distributions on the unit hypersphere to these directions, yielding a mixture-weighted inverse concentration score. The underlying assumption is that human accounts use words in a more semantically dispersed way, which gives a larger score, while LLM accounts are more concentrated, which gives a smaller score, and thresholding the score yields the decision. The paper also gives a finite-sample error bound for the score (Theorem 1) that decomposes the error into a term depending on the number of word types N and a term depending on the effective number of repeated occurrences n_eff, characterizing when the estimate is reliable.

Critical strengths
  1. The problem setting is novel and well motivated, and the method matches the problem. Existing training-free AI text detectors rely on token likelihood and are indeed unstable on fragmented, multi-topic social histories, and evidence in the paper supports this, such as the concatenation-order experiment in Table 8 and the near-random performance of Fast-DetectGPT on TwiLLM. CDetect extracts a user-level signal from the contextual usage geometry of words, naturally avoids the dependence on coherent long texts, and is training-free, interpretable, and cheap, with 64 to 69 percent lower inference time and about 91 percent lower peak GPU memory. This is a detection perspective that is orthogonal to existing ones and of reference value to the community.

  2. The analysis is fairly complete, with theory, method, and experiments echoing each other. Beyond the main experiments, the paper provides a finite-sample error bound, a static-versus-contextual ablation (Table 9, where the average AUROC drops from 0.9416 to 0.5819 with static embeddings, strongly supporting contextual geometry as the key component), and sensitivity analyses over the encoder, layer, number of mixture components, stopword list, and subword pooling. This completeness, where every design choice is validated, increases the credibility of the conclusions.

Critical weaknesses
  1. The evaluation relies heavily on small self-constructed benchmarks, and the external validity of the core assumption is not adequately verified (concern). Two of the three benchmarks (TwiLLM-R and TwiLLM-I) were constructed by the authors, with only 131 human and 89 bot accounts, and the two variants share the same human accounts, so they are not independent evaluations. The bot-side texts were all generated by a single model (GPT-5.4 Mini) with a single prompt template at a fixed temperature of 0.7. Under such a construction, the semantic concentration on the bot side is to a large extent an artifact of the construction pipeline itself, since one template plus one model naturally produces homogeneous word usage, rather than a general property of LLM-driven bots. Notably, this core empirical assumption is never independently validated anywhere in the paper. Neither the main text nor the appendix directly shows a controlled comparison demonstrating that word-direction distributions are more dispersed for human accounts and more concentrated for LLM accounts, and Appendix G only shows that the scores separate the two classes. In other words, the only evidence supporting the assumption is that detection works well, which carries a risk of circular reasoning. On TwiLLM the regularity is not discovered but built in, since generation with a single model and a unified template almost necessarily produces homogeneous word usage, and detection success there cannot prove that the regularity holds for real bots. The only semi-independent source of evidence, BotSim-24 with real Reddit users versus LLM agents, is already near saturation (AUROC 0.9964) and cannot probe the boundary conditions of the assumption. Moreover, in TwiLLM the label bot is fully equivalent to LLM-generated, so the experiments cannot distinguish whether CDetect detects bot behavior or LLM generation style. For normal human users who use LLM assistance for writing posts or polishing text, the score would likewise decrease, and this false positive risk is not analyzed at all, yet it is the most realistic failure mode in deployment. Without tests on multiple generator models, multiple prompt styles, and human accounts assisted by LLMs, the main claim that CDetect detects LLM-driven bots holds only under a restricted synthetic setting.

  2. The fairness of the baseline comparison and the verification of confounders of the diversity score are insufficient (concern). (a) The training-free baselines all use Falcon-7B as the proxy model, while the bot texts were generated by GPT-5.4 Mini. The performance of likelihood-based methods depends heavily on the match between the generator and the proxy, and the OpenAI Detector reaches only 0.0467 AUROC on TwiLLM-I, far below the random level of 0.5, which indicates a severe domain mismatch. The paper does not state whether the proxy model was selected or tuned, and reports no sensitivity to the choice of proxy, so the gaps to the baselines may be systematically amplified. (b) The paper states that no model is trained on the evaluation datasets (Appendix C.2), so the supervised text baselines, namely the OpenAI Detector and RADAR, both trained around 2023 against early LLMs, are entirely out of domain. A supervised in-domain reference upper bound, for example a fine-tuned RoBERTa with user-level aggregation on a training split of the same benchmark, is missing, so the claim that a training-free method outperforms supervised ones lacks a frame of reference. (c) Appendix H.1 of the paper itself already reveals signs of confounding. On the single-topic politics subset the AUPRC of CDetect (0.6823) is lower than that of Fast-DetectGPT (0.7492), suggesting that the score partly conflates topical concentration with machine generation, and topically concentrated human accounts such as vertical-domain bloggers may be misclassified. The paper does not directly analyze this failure condition.

Questions for authors
  1. On the construction of TwiLLM (Appendix D). If the bot texts were generated by multiple different LLMs with diverse prompt styles instead of a unified template, how much of the margin of CDetect would remain? Please report at least one setting with a different generator model or reduced template homogeneity, to rule out the explanation that semantic concentration is a construction artifact of a single template and a single model. In addition, on what kind of data was the regularity that humans are dispersed and LLMs are concentrated first observed? Could the authors provide direct evidence that does not rely on detection AUROC, such as a statistical comparison of word-direction dispersion between human and LLM accounts?

  2. On false positives. How does CDetect perform on real human accounts that use LLM assistance for writing, for example polishing or drafting some of their posts? Even a simulation on existing data, such as rewriting part of a human history with an LLM, would partially answer this. This directly determines the practical usability of the method.

  3. In Appendix H.1 the AUPRC of CDetect on the politics subset is lower than that of Fast-DetectGPT. Does this indicate that the score partly reflects topical concentration rather than generation mode? What is the false positive rate of CDetect on topically concentrated human accounts such as vertical-domain bloggers?

  4. On the baseline protocol (Appendix C.2). Was the proxy model (Falcon-7B) for the training-free baselines selected? Is the gap in Table 1 stable when a proxy better matched to GPT-5.4 Mini is used, or when results over two or three proxies are reported? In addition, could a supervised user-level classifier trained in-domain be added as a reference upper bound?

  5. The threshold in Eq. (6) has to be selected on a labeled validation set. How much labeled data is needed for selecting it? Is the threshold transferable across datasets or time periods? How does this reconcile with the training-free positioning?

4 comments
AuthorsOpenReview ↗

Response to Reviewer CYK4 [1/2]

We sincerely thank you for the constructive comments. Below is our point-by-point response.

Q1: Dataset clarification and additional evaluation datasets. (Weakness 1 and Question 1)

A1: Thank you for your comment! Our main evaluation is BotSim-24 [1], an independently developed large-scale benchmark with 2,394 accounts. In this benchmark (Table 1), DNA-DetectLLM is the strongest baseline, with 0.8898 AUROC and 0.8150 AUPRC. CDetect achieves 0.9964 and 0.9839 respectively, outperforming all baselines. We constructed two TwiLLM benchmarks primarily to complement the evaluation by examining how different methods perform across interaction scenarios, broadening platforms, dataset construction, and generation models.

To evaluate CDetect in more diverse social settings, we extended our evaluation to two additional benchmarks.

(a) Weibo platform [2] with 2,131 human accounts and 339 agent accounts after filtering empty text records.The result is shown below. Since the benchmark is class-imbalanced, AUPRC shows the trade-off between precision and recall for the minority class across decision thresholds. CDetect achieves the highest AUPRC of 0.6410, exceeding the strongest baseline by 21.1%.

MethodAUROCAUPRC
OpenAI RoBERTa0.54390.1378
RADAR0.28720.0896
Log Likelihood0.73660.2664
Fast-DetectGPT0.78970.4300
DNA-DetectLLM0.67300.2450
Binoculars0.79690.4098
CDetect0.79420.6410

(b) MoltGraph [3], which links Moltbook posts and comments to their authors. We use these histories for the agent class and TwiBot-22 for the human class. We randomly sample 2,000 accounts per class with seed 42. As shown in the table below, CDetect outperforms all baselines. Together with the Weibo results, this evaluation shows the applicability of CDetect across diverse social media and generation settings.

MethodAUROCAUPRC
OpenAI RoBERTa0.25020.4127
RADAR0.14040.3447
Log Likelihood0.56700.6027
Fast-DetectGPT0.58610.6432
Binoculars0.59560.6556
DNA-DetectLLM0.67830.7020
CDetect0.97950.9730

[1] Qiao, Boyu, et al. "Botsim: Llm-powered malicious social botnet simulation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 13. 2025.

[2] Han, Yi, et al. "Toward Detecting Modern Social Media Agents: Dataset, Experimental Analysis, and an Agent-Based Solution." Findings of the Association for Computational Linguistics: EMNLP 2026. 2026.

[3] Mukherjee, Kunal, Cuneyt Akcora, and Murat Kantarcioglu. "Moltgraph: A longitudinal temporal graph dataset of moltbook for coordinated-agent detection." Proceedings of the ACM Conference on AI and Agentic Systems. 2026.

Q2: Direct evidence for differences in word-direction dispersion beyond detection AUROC (Weakness 1 and Question 1)

Good idea! We further compute statistics from contextual embeddings without fitting a mixture of von Mises–Fisher distributions or CDetect scores. We report Hedges’ (g) as the standardized mean difference (bot minus human). Bots have lower mean distances on all three datasets, indicating that their word-type directions are more concentrated overall. The result is shown below.

DatasetHuman mean distanceBot mean distanceHedges’ (g)95% bootstrap CI
BotSim-240.29750.2283−3.365[−3.523, −3.206]
TwiLLM-R0.28880.2657−1.009[−1.347, −0.734]
TwiLLM-I0.28880.2403−2.407[−2.981, −1.989]

Q3: Baseline evaluation protocol and in-domain supervised comparison (Weakness 2(a–b) and Question 4)

A3: We follow the default configuration of proxy models from the baseline authors. In black-box evaluations, it is common to use a shared scoring model across baselines. We will clarify this choice in Appendix C.2 and discuss the potential impact of different proxy models on likelihood-based baselines. Appendix F reports results for supervised methods trained on the BotSim-24 training split. CDetect outperforms these methods even when they are trained on the same benchmark.

AuthorsOpenReview ↗

Response to Reviewer CYK4 [2/2]

Q4: The politics result and performance on topically concentrated human accounts (Weakness 2(c) and Question 3)

A4: The Politics evaluation in Appendix H.1 shows that CDetect achieves a lower AUPRC than Fast-DetectGPT (0.6823 vs. 0.7492) and a higher AUROC (0.9240 vs. 0.8308). To further assess how much topical concentration explains CDetect’s performance, we use BotSim-24’s original subreddit labels to count the number of topics. As shown in the table below, CDetect can distinguish the two classes even when human accounts cover as many or fewer subreddits than bot accounts.

SettingHumanBotAUROCAUPRC
1-topic human vs. 1-topic bot5692290.99370.9552
1-topic human vs. ≥3-topic bot5692820.99320.9597

To directly estimate the false positive rate on topically concentrated human accounts, we hold out all 569 single-subreddit human accounts for testing. We randomly sample 20 of the remaining human accounts and 20 bot accounts without replacement (seed = 42). We select the threshold by maximizing Youden’s ($J$) on these 40 calibration accounts, then apply it to the held-out accounts.

Evaluation setHuman accountsMisclassified as botsFPR
Single-subreddit human accounts569193.34%

Q5: Human account histories containing LLM-rewritten posts (Weakness 1 and Question 2)

A5: To examine how partial use of LLM-rewritten content affects detection, we randomly replaced approximately 5%, 10% or 15% of the posts in humans with LLM-rewritten. We evaluated these modified histories against bots from TwiLLM-I. The result is shown below. CDetect maintained stable discrimination between human and bot accounts.

Fraction of LLM-rewrittenAUROCAUPRC
5%0.97810.9777
10%0.97730.9765
15%0.97760.9776

Q6: Threshold calibration and cross-dataset transfer (Question 5)

A6: All main conclusions in the paper are based on AUROC and AUPRC and are independent of the threshold $tau$. This threshold is a scalar calibration parameter used only when a binary decision is required. Same with training-free detectors such as Binoculars and Fast-DetectGPT, CDetect’s deployment threshold can be calibrated using a small labeled sample.

Cross-dataset transfer. We use 40 labeled accounts from BotSim-24 (20 human and 20 bot accounts) for all methods. We select each method’s threshold separately by maximizing Youden’s $J$ on this set. We then apply each threshold directly to a target set of 2,000 MoltGraph agent accounts and 2,000 TwiBot-22 human accounts. The result is shown below. Without using target-set labels for calibration, CDetect achieves the best performance.

MethodF1Acc.Prec.Rec.FPR
OpenAI RoBERTa0.12250.50950.58050.06850.0495
RADAR0.15960.23920.17830.14450.6660
Log Likelihood0.57940.52620.52100.65250.6000
Fast-DetectGPT0.54310.57850.59290.50100.3440
Binoculars0.53040.61580.68190.43400.2025
DNA-DetectLLM0.53800.64580.77320.41250.1210
CDetect0.88180.89300.98520.79800.0120
AuthorsOpenReview ↗

Looking forward to your response

Dear Reviewer,

Thank you again for reviewing our paper! We have posted our responses to your questions and concerns. We are wondering if you have any question of our paper after reading our responses. If you have other questions, we are very willing to discuss them with you.

Best,

Authors

Thank you for your responses, these have largely addressed my concerns. I would like to raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.