- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The main idea is to detect LLM text by checking intermediate-layer vocabulary predictions against human and LLM signature vocabularies. The paper introduces VisScore and a LightGBM extension, tested across eight languages, unseen generators, and text attacks. The LightGBM extension improves detection, but performance still drops on paraphrased text.
Critical strengths
VisScore’s vocabulary-based scoring offers a simple detection signal with readable evidence.
Critical weaknesses
- I have concerns about the baseline implementations, which seem different from the original methods. Appendix C describes class-mean differences instead of PCA for RepreGuard, a difference instead of a ratio for Binoculars, and no standard-deviation normalization for Fast-DetectGPT.
- Several important related works are missing. “Text Fluoroscopy” (EMNLP 2024) uses vocabulary-space projections to select intermediate-layer detection features, while “Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders” (Findings of ACL 2025) studies interpretable internal features for detection. The advance over these methods needs to be clearer. DNA-GPT (ICLR 2024) also provides readable textual evidence, so describing existing detectors as returning only scores is too broad. 3.The interpretability claim needs stronger evidence. Readable decoded words do not by themselves explain why an input span was flagged. The evidence is mostly vocabulary lists and individual examples; it remains unclear how well these explain the final LGBM predictions.
- The F1 scores reported in Section 5.1 and Table 1 are inconsistent: 96.1 versus 94.4.
Questions for authors
- With topic and length controlled, how do detection performance and human false-positive rates change when human and LLM writing styles are matched or reversed? Does the signal go beyond the difference between “flowery prose” and conversational writing?
- Do the concept-hit visualizations show what actually drives VisScore+LGBM’s decisions, including the effects of hit clustering and position?
- Keeping the threshold calibrated to FPR≤1% on training data, what human false-positive rates and LLM-text detection rates do you get on test and OOD data?
4 comments
Official Comment by Authors [2/2]
W4: Section 5.1 and Table 1 report inconsistent F1 scores: 96.1 versus 94.4.
A4: Thank you for catching this inconsistency. Table 1 is correct: VisScore+LGBM's eight-language mean is 94.4. The 96.1 in §5.1 was a stale value from an earlier aggregation; we corrected it to 94.4 (p. 7, lines 365–369).
Q1: Controlling topic and length, how does matching or reversing human/LLM writing styles affect detection performance and human FPR? Does the signal extend beyond “flowery prose” versus conversational writing?
A5: We evaluated VisScore on DetectRL's [6] official prompt_SICO split. This prompt attack uses SICO's ICO component [7] to guide LLMs to generate human-style text with the aim of evading detection. Its prompts retain the topic input of the corresponding human sample and request the same sentence count, while asking the LLM to adopt human vocabulary and sentence structure inferred from paired examples (DetectRL, Appendix D.4.1). Thus, topic and requested length are controlled in the generation protocol. The results are as follows:
| Test split | AUC | Human FPR | LLM TPR |
|---|---|---|---|
| Standard direct-prompt (reference) | 95.52 | 0.92% | 77.62% |
| Human-style prompt_SICO | 94.31 | 1.29% | 69.15% |
Under this prompt attack, VisScore still achieves 94.31 AUC and detects 69.15% of LLM texts at 1.29% human FPR. Retaining this separation when LLMs are explicitly instructed to mimic human vocabulary and sentence structure suggests that the detection signal is not limited to the obvious contrast between “flowery prose” and conversational writing.
Q2: Do concept-hit visualizations faithfully reflect VisScore+LGBM’s decisions, including the contributions of hit clustering and position?
A6: The head receives only deterministic summaries of the two concept-hit sequences shown in the visualization, not text, embeddings, hidden states, or metadata. Clustering (autocorrelation, burstiness, longest run) and position (center of mass, trend) describe the arrangement of these same hits, not additional hidden signals. Thus, identical hit counts with different arrangements can yield different decisions.
To test the trained head's dependence on these summaries, we replaced each feature group with its training median at prediction time, keeping the head and clean low-FPR threshold fixed. Spatial/run and positional occlusion cause 8.20% and 4.09% decision flips, respectively, supporting the head's use of clustering and position beyond hit counts:
| Occluded group | AUC | Mean |Δp(AI)| | Decision flips at clean low-FPR threshold |
|---|---|---|---|
| Density | 98.55 | 0.0069 | 0.51% |
| Spatial/run | 95.99 | 0.0910 | 8.20% |
| Positional | 98.17 | 0.0407 | 4.09% |
| Exclusivity | 98.11 | 0.0330 | 2.44% |
The small density effect does not imply unused hit rates, since other summaries may retain overlapping information. Concept-input occlusion provides complementary faithfulness evidence in W3.
In the revised manuscript, Figure 7 adds two-color hit-position strips and clustering/position readouts, exposing head inputs rather than calibrated prediction contributions (p. 17, lines 889–900). Appendix B clarifies this scope (p. 15, lines 792–803); Appendix D reports the occlusion results (pp. 18–19, lines 924–1001).
Q3: With the training-calibrated FPR≤1% threshold fixed, what human FPR and LLM-text detection rates are achieved on test and OOD data?
A7: We calibrate thresholds to FPR 1% on source training data and freeze them, together with score orientation, for testing across 18 DRX settings: eight in-distribution languages, four cross-generator folds, and six cross-domain folds. Entries are mean (range) over settings, in %.
| Settings | VisScore FPR | VisScore TPR | VisScore+LGBM FPR | VisScore+LGBM TPR |
|---|---|---|---|---|
| In-distribution (8 languages) | 1.21 (0.28–2.82) | 69.60 (46.70–96.22) | 4.80 (2.73–6.98) | 92.94 (84.98–99.35) |
| Cross-generator (4 folds) | 0.90 (0.53–1.27) | 95.77 (91.80–98.20) | 2.68 (2.13–3.33) | 99.12 (97.87–100.00) |
| Cross-domain (6 folds) | 1.45 (0.00–5.60) | 95.62 (88.70–99.90) | 5.50 (0.20–9.50) | 99.05 (96.30–100.00) |
VisScore yields lower mean FPR in all three groups, whereas VisScore+LGBM attains higher recall at the cost of more false positives. Training calibration at FPR 1% does not guarantee the same bound on test or OOD data. In the revised manuscript, §5.2 summarizes this trade-off (p. 8, lines 407–410); Appendix G, Table 14 reports the group means, ranges, and frozen-threshold protocol (p. 21, lines 1121–1132).
Reference
Reference
[1] Text Fluoroscopy: Detecting LLM-Generated Text through Intrinsic Features. EMNLP 2024.
[2] Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders. Findings of ACL 2025.
[3] DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text. ICLR 2024.
[4] Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore. COLING 2025.
[5] Beat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection via Querying ChatGPT. EMNLP 2023.
[6] DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios. NeurIPS 2024.
[7] Large Language Models can be Guided to Evade AI-Generated Text Detection. TMLR 2024.
Thank you for taking the time to consider our responses and revisions. If they address your concerns and improve your assessment of the paper, we would be grateful if you would consider strengthening your recommendation. Thank you again for your support!
Official Comment by Authors
Dear Reviewer Tq4Z,
Thank you again for your time and constructive feedback. We have put great care and effort into going through each of your comments, revising the manuscript and carrying out additional experiments to respond to your concerns. We hope our response and extra experiments have addressed your concerns. As the discussion period is coming to an end, we would kindly invite you to share any updated thoughts. 😊
Best,
The Authors
Official Comment by Authors [1/2]
We sincerely thank you for the time and valuable feedback, which helped improve our manuscript. We have revised it in response to your comments, with changes marked in blue. Below, we address your questions point by point.
W1: Appendix C suggests baseline deviations
A1: Thank you for identifying these discrepancies. We confirm a RepreGuard implementation error: using a normalized class-mean difference instead of one-component PCA over AI/human paired differences. We adopted the official PCA procedure and reran all RepreGuard experiments, the new main results are as follow. The rerun raises eight-language mean AUC by 1.1 points but lowers F1@1% by 16.6. Cross-language RepreGuard retains 90.25 AUC but achieves only 5.57 F1 under direct source-threshold transfer.
For Binoculars and Fast-DetectGPT, our misunderstanding of the methodological details led to inaccurate appendix descriptions; the scoring code follows the official implementations. Thank you for flagging this. Appendix E now clarifies Binoculars' official ratio score and Fast-DetectGPT's analytic sampling discrepancy normalized by the square root of summed token-level variances (p. 19, lines 1007–1015).
RepreGuard's implementation is in Appendix E (p. 19, lines 1017–1022); corrected results are in Tables 1–2 (pp. 6–7, lines 271–287 and 325–351), §5.1 (p. 7, lines 361–369), and Figures 5–6 (pp. 9–10, lines 476–524). §5.2 clarifies ranking versus threshold transfer (p. 8, lines 405–407).
W2: Related work is missing
A2: Thank you for identifying these omissions and the overly broad claim. We have cited Text Fluoroscopy, SAE-based detector, and DNA-GPT, alongside GECScore and ReviseDetect, and removed the claim that prior detectors return only scores. Text Fluoroscopy [1] uses vocabulary-space decoding for layer selection, then detects from the selected hidden states; the SAE study [2] detects from latent features and analyzes their interpretability. DNA-GPT [3] provides regeneration-based evidence, while GECScore [4] and ReviseDetect [5] compare the input with model-produced revisions. VisScore instead directly uses two FDR-controlled concept vocabularies decoded through a frozen unembedding as detector features: their hit-rate difference is the score, which can be recomputed from the displayed hit rates without regenerated reference text.
Our contribution is this direct alignment between detection features and readable evidence, not priority for readable output, intermediate-feature detection, or vocabulary-space analysis. We clarified this scope in the Abstract and contributions (p. 1, lines 011–025; p. 2, §1, lines 058–096), added these methods to Related Work (pp. 2–3, §2, lines 101–154), and contrasted concept inputs with Text Fluoroscopy's layer selection in §3.1 (p. 4, lines 176–183)
W3: Stronger evidence is needed that decoded concepts explain why spans flagged and what drives the predictions.
A3: Readable vocabularies and examples alone do not establish explanation fidelity. VisScore scores documents. At each position, the logit lens decodes the reader's prefix-conditioned hidden state into next-token expectations. A highlight summarizes concept activations in the decoded top-K, reflecting the preceding context rather than judging the marked word. Its intensity reflects the difference between LLM- and human-signature hit rates, not LLM-concept presence alone. Decoded concepts need not occur in the text, nor must the marked word be LLM-typical.
Retraining VisScore+LGBM on the two displayed hit-rate means yields 97.26 eight-language mean AUC versus 98.56 with all features, showing retained signal but not how the full head uses these means. Prediction-time group occlusion probes input dependence, with spatial/run occlusion causing the largest change (8.20% decision flips; See Response to Q2). We also masked concept channels on all English test pairs (AUC 99.81), recomputing all dependent features with the trained head and low-FPR threshold fixed:
AI-concept masking largely preserves ranking (AUC 99.71) but flips 97.92% of AI decisions; human-concept masking lowers AUC to 64.88, and masking both yields chance AUC 50.00. This supports detector-input faithfulness, not token-level causality: we mask concept channels rather than delete words and re-run the reader. Results are in Appendix D, Tables 8, 10–11 (pp. 18–19, lines 924–1001); Appendix B clarifies input transparency and its limits (p. 15, lines 792–803)