acceptodds
Under review as a conference paper at ICLR 2027

VisScore: Explainable LLM-Generated Text Detection via Workspace Concept Visibility

Abstract

Detecting large language model (LLM) generated text is central to trustworthy AI, yet existing detectors, whether training-based classifiers, or statistics-based detectors, return a bare probability or score: they tell the user that a text is LLM-generated, but not which words make it look so. We start from an observation about the scoring model itself: it is an LLM, trained on text distributions similar to those of the generators it must detect, so its internal computation should resonate with the stylistic signature of LLM-generated text. Decoding this computation into the model's own vocabulary space exposes the reader's workspace and confirms the concept-visibility hypothesis: LLM-generated text saturates a small, human-readable “flowery prose” vocabulary, while human text saturates a casual, conversational register. We build on this signal: decode a single golden layer of a frozen reader, discover LLM- and human-signature vocabularies from train-split logit differentials, and score each document by how strongly its workspace activates these words. Since the score aggregates named, readable words, the evidence behind every verdict can be directly inspected and audited by a human reviewer. On DetectRL-X, spanning eight languages, four generators, and six domains, reaches document-level AUC of on English and on the 8-language mean with a Qwen3-8B reader, outperforming all 11 baselines. To further explore the potential of this signal, we fit a -parameter classifier over simple statistics of the concept-hit sequences; by recovering the structure that the scalar score discards, it lifts the mean AUC to and, at the deployment-relevant false-positive rate, the from to . also holds up in the wild: it generalizes to unseen languages, generators, and domains, needs only a few labeled examples to calibrate, and stays robust to varying text lengths, LLM-assisted editing, and paraphrasing or perturbation attacks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM-Generated Text Detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Tq4ZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The main idea is to detect LLM text by checking intermediate-layer vocabulary predictions against human and LLM signature vocabularies. The paper introduces VisScore and a LightGBM extension, tested across eight languages, unseen generators, and text attacks. The LightGBM extension improves detection, but performance still drops on paraphrased text.

Critical strengths

VisScore’s vocabulary-based scoring offers a simple detection signal with readable evidence.

Critical weaknesses
  1. I have concerns about the baseline implementations, which seem different from the original methods. Appendix C describes class-mean differences instead of PCA for RepreGuard, a difference instead of a ratio for Binoculars, and no standard-deviation normalization for Fast-DetectGPT.
  2. Several important related works are missing. “Text Fluoroscopy” (EMNLP 2024) uses vocabulary-space projections to select intermediate-layer detection features, while “Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders” (Findings of ACL 2025) studies interpretable internal features for detection. The advance over these methods needs to be clearer. DNA-GPT (ICLR 2024) also provides readable textual evidence, so describing existing detectors as returning only scores is too broad. 3.The interpretability claim needs stronger evidence. Readable decoded words do not by themselves explain why an input span was flagged. The evidence is mostly vocabulary lists and individual examples; it remains unclear how well these explain the final LGBM predictions.
  3. The F1 scores reported in Section 5.1 and Table 1 are inconsistent: 96.1 versus 94.4.
Questions for authors
  1. With topic and length controlled, how do detection performance and human false-positive rates change when human and LLM writing styles are matched or reversed? Does the signal go beyond the difference between “flowery prose” and conversational writing?
  2. Do the concept-hit visualizations show what actually drives VisScore+LGBM’s decisions, including the effects of hit clustering and position?
  3. Keeping the threshold calibrated to FPR≤1% on training data, what human false-positive rates and LLM-text detection rates do you get on test and OOD data?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors [1/2]

We sincerely thank you for the time and valuable feedback, which helped improve our manuscript. We have revised it in response to your comments, with changes marked in blue. Below, we address your questions point by point.


W1: Appendix C suggests baseline deviations

A1: Thank you for identifying these discrepancies. We confirm a RepreGuard implementation error: using a normalized class-mean difference instead of one-component PCA over AI/human paired differences. We adopted the official PCA procedure and reran all RepreGuard experiments, the new main results are as follow. The rerun raises eight-language mean AUC by 1.1 points but lowers F1@1%F_1@1\% by 16.6. Cross-language RepreGuard retains 90.25 AUC but achieves only 5.57 F1F_1 under direct source-threshold transfer.

MethodMean AUCMean F1F_1Mean F1@1%F_1@1\%
RepreGuard92.3 →\rightarrow 93.485.3 →\rightarrow 86.070.2 →\rightarrow 53.6

For Binoculars and Fast-DetectGPT, our misunderstanding of the methodological details led to inaccurate appendix descriptions; the scoring code follows the official implementations. Thank you for flagging this. Appendix E now clarifies Binoculars' official ratio score and Fast-DetectGPT's analytic sampling discrepancy normalized by the square root of summed token-level variances (p. 19, lines 1007–1015).

RepreGuard's implementation is in Appendix E (p. 19, lines 1017–1022); corrected results are in Tables 1–2 (pp. 6–7, lines 271–287 and 325–351), §5.1 (p. 7, lines 361–369), and Figures 5–6 (pp. 9–10, lines 476–524). §5.2 clarifies ranking versus threshold transfer (p. 8, lines 405–407).


W2: Related work is missing

A2: Thank you for identifying these omissions and the overly broad claim. We have cited Text Fluoroscopy, SAE-based detector, and DNA-GPT, alongside GECScore and ReviseDetect, and removed the claim that prior detectors return only scores. Text Fluoroscopy [1] uses vocabulary-space decoding for layer selection, then detects from the selected hidden states; the SAE study [2] detects from latent features and analyzes their interpretability. DNA-GPT [3] provides regeneration-based evidence, while GECScore [4] and ReviseDetect [5] compare the input with model-produced revisions. VisScore instead directly uses two FDR-controlled concept vocabularies decoded through a frozen unembedding as detector features: their hit-rate difference is the score, which can be recomputed from the displayed hit rates without regenerated reference text.

Our contribution is this direct alignment between detection features and readable evidence, not priority for readable output, intermediate-feature detection, or vocabulary-space analysis. We clarified this scope in the Abstract and contributions (p. 1, lines 011–025; p. 2, §1, lines 058–096), added these methods to Related Work (pp. 2–3, §2, lines 101–154), and contrasted concept inputs with Text Fluoroscopy's layer selection in §3.1 (p. 4, lines 176–183)


W3: Stronger evidence is needed that decoded concepts explain why spans flagged and what drives the predictions.

A3: Readable vocabularies and examples alone do not establish explanation fidelity. VisScore scores documents. At each position, the logit lens decodes the reader's prefix-conditioned hidden state into next-token expectations. A highlight summarizes concept activations in the decoded top-KK, reflecting the preceding context rather than judging the marked word. Its intensity reflects the difference between LLM- and human-signature hit rates, not LLM-concept presence alone. Decoded concepts need not occur in the text, nor must the marked word be LLM-typical.

Retraining VisScore+LGBM on the two displayed hit-rate means yields 97.26 eight-language mean AUC versus 98.56 with all features, showing retained signal but not how the full head uses these means. Prediction-time group occlusion probes input dependence, with spatial/run occlusion causing the largest change (8.20% decision flips; See Response to Q2). We also masked concept channels on all English test pairs (AUC 99.81), recomputing all dependent features with the trained head and low-FPR threshold fixed:

AUCMean |Δp(AI)|Human decision flipsAI decision flips
AI concepts99.710.49661.12%97.92%
Human concepts64.880.501098.88%2.08%
Both50.000.49821.12%97.92%

AI-concept masking largely preserves ranking (AUC 99.71) but flips 97.92% of AI decisions; human-concept masking lowers AUC to 64.88, and masking both yields chance AUC 50.00. This supports detector-input faithfulness, not token-level causality: we mask concept channels rather than delete words and re-run the reader. Results are in Appendix D, Tables 8, 10–11 (pp. 18–19, lines 924–1001); Appendix B clarifies input transparency and its limits (p. 15, lines 792–803)

AuthorsOpenReview ↗

Official Comment by Authors [2/2]

W4: Section 5.1 and Table 1 report inconsistent F1 scores: 96.1 versus 94.4.

A4: Thank you for catching this inconsistency. Table 1 is correct: VisScore+LGBM's eight-language mean F1F_1 is 94.4. The 96.1 in §5.1 was a stale value from an earlier aggregation; we corrected it to 94.4 (p. 7, lines 365–369).


Q1: Controlling topic and length, how does matching or reversing human/LLM writing styles affect detection performance and human FPR? Does the signal extend beyond “flowery prose” versus conversational writing?

A5: We evaluated VisScore on DetectRL's [6] official prompt_SICO split. This prompt attack uses SICO's ICO component [7] to guide LLMs to generate human-style text with the aim of evading detection. Its prompts retain the topic input of the corresponding human sample and request the same sentence count, while asking the LLM to adopt human vocabulary and sentence structure inferred from paired examples (DetectRL, Appendix D.4.1). Thus, topic and requested length are controlled in the generation protocol. The results are as follows:

Test splitAUCHuman FPRLLM TPR
Standard direct-prompt (reference)95.520.92%77.62%
Human-style prompt_SICO94.311.29%69.15%

Under this prompt attack, VisScore still achieves 94.31 AUC and detects 69.15% of LLM texts at 1.29% human FPR. Retaining this separation when LLMs are explicitly instructed to mimic human vocabulary and sentence structure suggests that the detection signal is not limited to the obvious contrast between “flowery prose” and conversational writing.


Q2: Do concept-hit visualizations faithfully reflect VisScore+LGBM’s decisions, including the contributions of hit clustering and position?

A6: The head receives only deterministic summaries of the two concept-hit sequences shown in the visualization, not text, embeddings, hidden states, or metadata. Clustering (autocorrelation, burstiness, longest run) and position (center of mass, trend) describe the arrangement of these same hits, not additional hidden signals. Thus, identical hit counts with different arrangements can yield different decisions.

To test the trained head's dependence on these summaries, we replaced each feature group with its training median at prediction time, keeping the head and clean low-FPR threshold fixed. Spatial/run and positional occlusion cause 8.20% and 4.09% decision flips, respectively, supporting the head's use of clustering and position beyond hit counts:

Occluded groupAUCMean |Δp(AI)|Decision flips at clean low-FPR threshold
Density98.550.00690.51%
Spatial/run95.990.09108.20%
Positional98.170.04074.09%
Exclusivity98.110.03302.44%

The small density effect does not imply unused hit rates, since other summaries may retain overlapping information. Concept-input occlusion provides complementary faithfulness evidence in W3.

In the revised manuscript, Figure 7 adds two-color hit-position strips and clustering/position readouts, exposing head inputs rather than calibrated prediction contributions (p. 17, lines 889–900). Appendix B clarifies this scope (p. 15, lines 792–803); Appendix D reports the occlusion results (pp. 18–19, lines 924–1001).


Q3: With the training-calibrated FPR≤1% threshold fixed, what human FPR and LLM-text detection rates are achieved on test and OOD data?

A7: We calibrate thresholds to FPR ≤\le 1% on source training data and freeze them, together with score orientation, for testing across 18 DRX settings: eight in-distribution languages, four cross-generator folds, and six cross-domain folds. Entries are mean (range) over settings, in %.

SettingsVisScore FPRVisScore TPRVisScore+LGBM FPRVisScore+LGBM TPR
In-distribution (8 languages)1.21 (0.28–2.82)69.60 (46.70–96.22)4.80 (2.73–6.98)92.94 (84.98–99.35)
Cross-generator (4 folds)0.90 (0.53–1.27)95.77 (91.80–98.20)2.68 (2.13–3.33)99.12 (97.87–100.00)
Cross-domain (6 folds)1.45 (0.00–5.60)95.62 (88.70–99.90)5.50 (0.20–9.50)99.05 (96.30–100.00)

VisScore yields lower mean FPR in all three groups, whereas VisScore+LGBM attains higher recall at the cost of more false positives. Training calibration at FPR ≤\le 1% does not guarantee the same bound on test or OOD data. In the revised manuscript, §5.2 summarizes this trade-off (p. 8, lines 407–410); Appendix G, Table 14 reports the group means, ranges, and frozen-threshold protocol (p. 21, lines 1121–1132).

Reference

[1] Text Fluoroscopy: Detecting LLM-Generated Text through Intrinsic Features. EMNLP 2024.

[2] Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders. Findings of ACL 2025.

[3] DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text. ICLR 2024.

[4] Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore. COLING 2025.

[5] Beat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection via Querying ChatGPT. EMNLP 2023.

[6] DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios. NeurIPS 2024.

[7] Large Language Models can be Guided to Evade AI-Generated Text Detection. TMLR 2024.


Thank you for taking the time to consider our responses and revisions. If they address your concerns and improve your assessment of the paper, we would be grateful if you would consider strengthening your recommendation. Thank you again for your support!

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer Tq4Z,

Thank you again for your time and constructive feedback. We have put great care and effort into going through each of your comments, revising the manuscript and carrying out additional experiments to respond to your concerns. We hope our response and extra experiments have addressed your concerns. As the discussion period is coming to an end, we would kindly invite you to share any updated thoughts. 😊

Best,

The Authors

Reviewer_8DKKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Proposes a scoring mechanism (VisScore) based on word project from residual representations. Shows good results on a single benchmark, even on edits.

Critical strengths

Method is very well described and easy to follow with clear motivation behind the work.

Results are strong, particularly ones after the LGT has been altered.

Critical weaknesses

I would request the authors to consider adding results of a few more benchmarks. This eases the mind of the reader.

Questions for authors

would it be possible to do an inter-benchmark eval, train on one and test on another?

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank you for your time and valuable feedback, which have helped improve the quality of our manuscript. We have revised the manuscript in response to your comments and marked the revisions in blue to facilitate your review. Below, we address your concerns and questions point by point.


W1: Results on additional benchmarks would strengthen readers' confidence in the evaluation.

A1: Thank you for this valuable suggestion. To address it, we have added a new benchmark, DetectRL [1], to further strengthen confidence in the effectiveness of our method. We follow the same cross-generator protocol as the RepreGuard paper, training on one generator and testing on the other three. Each generator column gives the mean AUC when training on that generator; the last three columns average all transfer pairs. Bold marks the best result within each group.

MethodChatGPTClaude-instantGoogle-PaLMLlama-2-70bAUCF1F_1F1@1%F_1@1\%
Statistics-based
Log-Likelihood93.0998.9093.7493.0394.6975.9469.61
Entropy91.0392.6493.4191.0592.0382.9647.31
LRR88.6297.3689.3088.4390.9372.9349.03
LogRank78.5878.2183.8678.0679.6870.1449.73
Binoculars78.8292.1680.4980.1682.9168.1158.32
Fast-DetectGPT76.5588.9777.0677.1079.9265.4541.34
RepreGuard96.7994.9896.9596.7096.3688.8483.31
TF-IDF Score84.7387.1482.9586.8785.4272.9052.05
VisScore (ours)96.3395.9299.4297.2097.2289.1881.02
Training-based
XLM-RoBERTa-base98.4897.9798.1598.0698.1789.6285.76
BiScope97.0698.5997.8896.9797.6288.0793.48
SeqXGPT94.8393.6093.6394.1494.0566.693.81
DivEye90.4987.7591.2091.7690.3074.1274.04
TF-IDF Score + LGBM83.6177.8783.5483.7182.1868.5759.74
VisScore + LGBM (ours)97.6598.4199.5598.7298.5890.5589.07

As shown above, VisScore+LGBM achieves the highest mean AUC and F1F_1 among all methods (98.58 / 90.55), and VisScore achieves the best mean AUC and F1F_1 among statistics-based methods (97.22 / 89.18), surpassing RepreGuard, which relies on opaque hidden-state projections, while keeping its evidence readable as named concepts. Detailed results are provided in Appendix A, Table 5 (p. 14, lines 702–742).


Q1: Can you evaluate cross-benchmark transfer by training on one benchmark and testing on another?

A2: Thank you for this valuable suggestion. To address it, we have added bidirectional cross-benchmark evaluation between DetectRL-X (DRX) and DetectRL, directly testing generalization beyond the source benchmark.

MethodSource →\rightarrow targetAUCF1F_1F1@1%F_1@1\%
XLM-RoBERTa-baseDRX →\rightarrow DetectRL91.2275.3851.28
RepreGuardDRX →\rightarrow DetectRL89.8283.0646.77
TF-IDF ScoreDRX →\rightarrow DetectRL85.0668.6230.63
VisScoreDRX →\rightarrow DetectRL94.2586.9387.62
VisScore+LGBMDRX →\rightarrow DetectRL91.3384.1381.31
XLM-RoBERTa-baseDetectRL →\rightarrow DRX91.3282.0183.29
RepreGuardDetectRL →\rightarrow DRX95.5584.3489.28
TF-IDF ScoreDetectRL →\rightarrow DRX78.7970.9428.35
VisScoreDetectRL →\rightarrow DRX97.2887.7690.93
VisScore+LGBMDetectRL →\rightarrow DRX96.6187.2185.69

VisScore achieves the highest AUC, F1F_1, and F1@1%F_1@1\% in both directions, while VisScore+LGBM ranks second in AUC and F1F_1. Together, these results highlight the robustness of our methods across benchmarks, supporting their potential for more complex data and application settings. Detailed results and analysis are provided in Appendix A, Table 6 (p. 15, lines 757–778).


Reference

[1] DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios. NeurlPS 2024.


Thank you for taking the time to consider our responses and revisions. If they address your concerns and improve your assessment of the paper, we would be grateful if you would consider strengthening your recommendation. Thank you again for your support!

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 8DKK,

Thank you again for your time and constructive feedback. We hope our response and extra experiments have addressed your concerns. As the discussion period is coming to an end, we would kindly invite you to share any updated thoughts. 😊

Best,

The Authors

Reviewer_ucAxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method for detecting text generated by LLMs. The proposed method extracts residual stream activations from a frozen observer LLM and projects them into vocabulary space using the logit lens. From labeled training data, a pattern distinguishing LLM from human text is learned and applied during detection. It mostly outperforms 11 baselines and enables signal localization, while many previous methods only enable per-document binary classification.

Critical strengths

S1. The method is genuinely interesting and demonstrates strong empirical results over a comprehensive evaluation benchmark. S2. The proposed scoring mechanism is relatively simple and significantly more interpretable than various existing methods. S3. The manuscript is written quite well and is easy to follow once you understand the main idea.

Critical weaknesses

W1. The method can be seen as somewhat incremental compared to prior work like RepreGuard [1] and Text Fluoroscopy [2]. The core idea of using intermediate activations from observer LLMs for downstream tasks has been researched extensively, including for AI-generated text detection. Text Fluoroscopy also projects such activations into vocabulary space, albeit with a different layer selection process (divergence between distributions rather than purely based on detection performance). Even the localization of the detection signal to specific text spans isn’t fully novel, with SeqXGPT [3], one of the baselines evaluated in this work, being able to localize it at sentence (rather than token-level) granularity. Therefore, the main contribution of VisScore lies in its interpretability through concept vocabularies and their hit rate scoring mechanism, and the manuscript doesn’t make this sufficiently clear. W2. While vocabulary-space projection is framed as being beneficial for detection performance, a concern I have is that reducing the resulting logits to concept-hits may discard some useful information contained in the original hidden representation. Assumably, some patterns associated with AI-generated text lie in how tokens are used in combination, such as syntax (e.g. sentence length) or verbosity. Since the detector still works well, this raises questions on whether the claimed interpretability accurately reflects the underlying signal, or whether the token scores are merely proxies for more complex internal features. W3. All experiments use DetectRL-X, which seems relatively saturated, particularly by the training-based approaches described in Table 1. Since the benchmark is quite extensive, this is not a major issue, but it also limits the evidence for real-world generalization. W4. Although VisScore outperforms detectors that use hidden representations without projecting back into the vocabulary space, it would be interesting to have a direct comparison to a simple baseline that operates in vocabulary space without using hidden representations as well. Even a simple lexical classifier (e.g. TF-IDF with logistic regression) could be an insightful ablation. This could isolate how much of the improvement comes specifically from vocabulary-space decoding. W5. Figure 3 compares detection performance across layers, as measured by the AUC on the test set. If the layer (and potentially other hyperparameters) were indeed tuned against the test set rather than a held-out validation set, the claimed performance might be inflated. W6. The authors state that their LightGBM classification head only needs ~10K parameters, and they directly compare this number to XLM-RoBERTa-base, which has ~270M parameters, in Section 5.1. However, LightGBM still needs a forward pass on the main model (e.g. Qwen3 with ~8B parameters), which XLM-RoBERTa-base doesn’t, so this comparison isn’t completely fair. To this end, it would also make sense to include an experiment comparing the computational efficiency of all detection schemes.

Questions for authors

Q1. Could you elaborate on the motivation for projecting the hidden state into vocabulary space rather than using the raw representations directly? In particular, do you have any concerns about the potential loss of information discussed in W2?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors [1/2]

We sincerely thank you for your time and valuable feedback, which helped improve our manuscript. We have revised the manuscript in response to comments, with changes marked in blue. Below, we address your questions point by point.


W1: Clarify the novelty of concept vocabularies and hit-rate scoring relative to RepreGuard, Text Fluoroscopy, and SeqXGPT

A1: Thank you for this thoughtful comment and for identifying where our contribution needs clearer emphasis. We agree that observer-LLM activations and signal localization have been explored in prior work. Text Fluoroscopy [2] uses vocabulary-space decoding and distributional divergence for layer selection, but detects using raw hidden states rather than decoded concepts; RepreGuard [1] scores a discriminative PCA direction in hidden-state space; and SeqXGPT [3] labels log-probability sequences for sentence-level localization rather than expressing the signal through named concepts. Our contribution lies in the concept-visibility hypothesis and a detection mechanism in which decoded concepts are simultaneously the features and the readable evidence. Specifically, we link class-dependent activations in a frozen reader to two readable signature vocabularies, discovered from workspace-logit differences under FDR control. VisScore scores their hit-rate difference, capturing both LLM-signature activation and reduced human-signature activation. The displayed hit rates let readers inspect the evidence and recompute the score without a separate post-hoc explanation.

We have clarified the contribution and prior-work distinctions in the Abstract (p. 1, lines 11–33), Introduction (p. 2, lines 58–96), Related Work (pp. 2–3, lines 101–154), and §3.1 (p. 4, lines 176–183), avoiding priority claims for interpretable detection or span localization.


W2 & Q1: Why vocabulary-space projection? Does binarization lose token-combination, syntax, or verbosity cues? Are scores faithful or proxies for complex internal features?

A2: Thank you for valuable questions.

Motivation for vocabulary-space decoding

We project hidden states into vocabulary space to build a detector whose activation-based features can also be read as evidence. The projection expresses the reader's activations through named vocabulary channels, allowing class-dependent concepts to be selected and used directly for scoring. Choosing these concept channels over raw hidden representations reflects a trade-off between auditability and representational richness.

Information loss from binary concept hits

We understand the concern about information loss and agree that reducing logits to binary concept hits discards logit magnitudes and information about unselected vocabulary. To examine retained structure, we retrained VisScore+LGBM with different hit-sequence feature groups on DRX. We found that spatial/run features alone approach full-model AUC (98.07 versus 98.56), while removing spatial/run or positional features lowers it, suggesting useful structure beyond mean hit rates.

ConfigurationMean AUC
VisScore97.38
+LGBM: all98.56
+LGBM: hit-rate means only97.26
+LGBM: spatial/run only98.07
+LGBM: no spatial/run98.11
+LGBM: no positional98.19

To probe your token-combination, syntax, and verbosity concerns, we correlated VisScore with mean sentence length and word-level n-gram repetition on the DRX English test split. We found weak correlation with sentence length but stronger negative correlations with both repetition measures. Together with the ablations, this suggests useful cues remain after binarization. While linguistic information loss is difficult to quantify or attribute, strong detection results support the practical trade-off between representational richness and auditability.

SignalAUCSpearman ρ\rho with VisScore
Sent. length60.040.209
Trigram repetition rate81.41−0.540
Bigram repetition rate85.12−0.604
VisScore99.77—

Faithfulness of the interpretable scores

VisScore is computed directly from displayed hit rates. For VisScore+LGBM, we occluded either or both concept sides and recomputed all dependent features. On the English test, AI-side occlusion largely preserves ranking (AUC 99.71) but flips 97.92% of AI decisions at the clean low-FPR threshold; human-side occlusion lowers AUC to 64.88. This supports detector-input faithfulness, compatible with concepts reflecting complex contextual patterns within the reader.

AUCHuman decision flipsAI decision flips
None99.81——
AI concepts99.711.12%97.92%
Human concepts64.8898.88%2.08%
Both50.001.12%97.92%

The information-loss discussion is in §3.2 (p. 6, lines 289–294); feature-group ablations, sentence-length control, and concept-input occlusion are in Appendix D (pp. 18–19, lines 918–1001).

AuthorsOpenReview ↗

Official Comment by Authors [2/2]

W3: Relying exclusively on DetectRL-X limits the evidence for real-world generalization.

A3: Thank you for highlighting this concern. In our added DetectRL [4] evaluation, VisScore+LGBM leads all evaluated methods in mean AUC and F1F_1, while VisScore leads the statistics-based group. VisScore also leads both directions of the added DRX ↔\leftrightarrow DetectRL transfer in AUC, F1F_1, and F1@1%F_1@1\%, extending the evidence beyond DRX. See our responses to Reviewer 8DKK's W1 and Q1, respectively, for details.


W4: A direct comparison with a simple lexical baseline.

A4: Thank you for this suggestion. We added two surface-word baselines: TF-IDF Score builds class-specific word lists from training TF-IDF weights and scores their normalized weight difference; TF-IDF Score+LGBM learns from surface-word hit-sequence statistics.

TF-IDF Score performs similarly to VisScore in-distribution, but both lexical baselines degrade sharply across languages. Despite high English transfer AUC, it trails VisScore by 19.97 and 31.61 F1@1%F_1@1\% points across generators and domains, respectively.

Cells report AUC / F1F_1 / F1@1%F_1@1\% in percentages. In-dist averages eight languages; cross-lang averages off-diagonal transfers; cross-gen/cross-dom evaluate English transfer to held-out generators/domains.

MethodIn-distCross-langCross-genCross-dom
TF-IDF Score97.48 / 91.40 / 81.4768.51 / 11.23 / 0.6997.74 / 92.10 / 77.4396.25 / 83.20 / 65.39
VisScore97.38 / 91.90 / 80.8096.45 / 86.57 / 82.5299.70 / 98.10 / 97.4099.80 / 97.40 / 97.00
TF-IDF Score+LGBM96.73 / 89.72 / 89.5553.66 / 0.49 / 0.3196.63 / 87.15 / 86.8294.09 / 78.80 / 77.78
VisScore+LGBM98.60 / 94.40 / 94.1096.84 / 88.58 / 87.8599.80 / 97.70 / 98.3099.70 / 97.20 / 96.90

The comparison is in §5.5 and Table 4 (p. 9, lines 450–471), with detailed domain analyses in Appendix D (p. 18, lines 961–969).


W5: Were the layer or other hyperparameters tuned on test rather than held-out validation data, potentially inflating performance?

A5: Thank you for flagging this leakage risk, which we overlooked when selecting a shared layer using test-set AUC. We now use 9:1 training splits for concept discovery and validation, respectively, selecting the shared layer by equally weighted eight-language validation AUC and reserving test data for reporting.

The revised Figure 3 scan retains the same K=50K{=}50 and top-100 concept settings and again selects L22, with eight-language validation/test macro AUCs of 96.30/96.45.

See §3.2 (p. 5, lines 237–243) and Figure 3 (p. 8, lines 392–395) for the revised protocol, and §5.4 (p. 9, lines 432–442) for validation-selected L22 and held-out test results.


W6: Please clarify total cost and compare efficiency across detection methods.

A6: Thank you. The approximately 10K parameters cover only the trainable LightGBM head, not the frozen 8B reader. We have clarified that this does not imply lower total model size or inference cost.

On one NVIDIA L40S, we timed 100 documents with maximum input length 512, 10 warmups, and 5 repeats. XLM-R takes 2.435 ms/doc versus VisScore's 50.510. VisScore's latency is comparable to other single-reader methods and roughly half that of Fast-DetectGPT and Binoculars; VisScore+LGBM adds about 1.6 ms for features and the three-seed CPU head. The relative latency uses VisScore as reference.

MethodMean latency (ms/doc)Relative latency
Fast-DetectGPT103.5472.050$\times$
Binoculars103.0372.039$\times$
LogRank49.5260.980$\times$
LRR51.2821.015$\times$
Entropy51.7991.026$\times$
Log-Likelihood49.0920.972$\times$
RepreGuard50.1260.992$\times$
XLM-RoBERTa-base2.4350.048$\times$
BiScope53.0381.050$\times$
SeqXGPT145.5132.880$\times$
DivEye51.4601.019$\times$
TF-IDF Score (CPU scope)0.4420.009$\times$
VisScore50.5101.000$\times$
VisScore + LGBM52.1341.032$\times$

See §5.1 (p. 7, lines 361–369) for the parameter distinction; Appendix F, Table 12 (p. 20, lines 1046–1079) for timings; and Table 13 (p. 21, lines 1081–1106) for memory, batched throughput, and adaptation.


Reference

[1] RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns. TACL 2025.

[2] Text Fluoroscopy: Detecting LLM-Generated Text through Intrinsic Features. EMNLP 2024.

[3] SeqXGPT: Sentence-Level AI-Generated Text Detection. EMNLP 2023.

[4] DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios. NeurIPS 2024.


Thank you for taking the time to consider our responses and revisions. If they address your concerns and improve your assessment of the paper, we would be grateful if you would consider strengthening your recommendation. Thank you again for your support!

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer ucAx,

Thank you again for your time and constructive feedback. We have put great care and effort into going through each of your comments, revising the manuscript and carrying out additional experiments to respond to your concerns. We hope our response and extra experiments have addressed your concerns. As the discussion period is coming to an end, we would kindly invite you to share any updated thoughts. 😊

Best,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Overview and Clarified Contributions

Dear SACs, ACs, and Reviewers,

Thank you for your time and thoughtful feedback, and for recognizing the strengths of our work. Reviewer ucAx described the method as "genuinely interesting" with "strong empirical results over a comprehensive evaluation benchmark." Reviewer 8DKK highlighted the clear motivation and strong results, particularly after LLM-generated text is altered. Reviewer Tq4Z noted that VisScore offers "a simple detection signal with readable evidence." We appreciate this recognition of the method’s empirical strength, simplicity, and readable evidence.

We summarize below our clarified contribution and substantive rebuttal revisions. Manuscript changes are marked in blue; detailed results and locations are provided in the individual responses.

Our central contribution is the direct alignment between detection features and readable evidence. Building on prior work on observer-LLM activations and vocabulary-space decoding, VisScore identifies two readable signature vocabularies from a frozen reader and directly scores their hit-rate difference. These concepts constitute both detection features and displayed evidence. We clarified this distinction and expanded the discussion of Text Fluoroscopy, SAE-based detection, DNA-GPT, and related methods. (Reviewer ucAx: W1; Reviewer Tq4Z: W2.)

In addition to these clarifications, we made four substantive revisions:

  1. Corrected implementations and evaluation protocol. We corrected RepreGuard’s implementation, adopted official PCA, and reran its evaluations. We corrected the Binoculars and Fast-DetectGPT descriptions and the inconsistent F1 value. Training-split validation replaced test-based layer selection and again selected L22, with test data reserved for reporting. (Reviewer Tq4Z: W1, W4; Reviewer ucAx: W5.)

  2. Additional benchmark and lexical controls. We added DetectRL cross-generator evaluation and bidirectional DRX–DetectRL transfer. VisScore+LGBM leads evaluated methods in mean AUC and F1 on DetectRL generator transfer; VisScore leads both cross-benchmark directions in AUC, F1, and F1@1%, with AUCs of 94.25 and 97.28. Two surface-word baselines further distinguish contextual concept decoding from lexical discrimination under distribution shift. (Reviewer 8DKK: W1, Q1; Reviewer ucAx: W3, W4.)

  3. Retained information and explanation faithfulness. We added feature ablations, sentence-length/repetition analyses, and prediction-time feature-group and concept-input occlusion. On the English test, masking human-signature concepts reduces VisScore+LGBM’s AUC from 99.81 to 64.88; masking both sides yields chance performance. New two-color hit-position strips and clustering/position readouts expose the head’s inputs. We explicitly distinguish this transparency from token-level causality or calibrated attribution. (Reviewer ucAx: W2, Q1; Reviewer Tq4Z: W3, Q2.)

  4. Robustness and deployment costs. On the official human-style prompt_SICO split, VisScore retains 94.31 AUC and 69.15% TPR at 1.29% human FPR. We added frozen-threshold FPR/TPR across 18 DRX settings and end-to-end efficiency measurements, distinguishing the small trainable head from the frozen 8B reader and acknowledging that training-calibrated FPR need not hold on test or OOD data. (Reviewer Tq4Z: Q1, Q3; Reviewer ucAx: W6.)

The importance of this alignment becomes concrete when an editor reviews a passage flagged as AI-generated. The author asks, "What evidence supports this judgment?" A confidence score alone cannot answer that question or give the editor an inspectable basis for a confident judgment. RepreGuard (TACL 2025) reveals class-dependent patterns in an observer model’s hidden representations. VisScore takes this perspective further by expressing such differences through named concepts activated differently on human- and LLM-generated text. These concepts need not appear literally in the passage; they reflect the reader’s processing of its context. Their hit-rate difference directly constitutes the VisScore score, allowing the editor to inspect the evidence and recompute the score rather than simply accept a label. This connection between internal activations and readable scoring evidence is the contribution we hope readers recognize.

We thank everyone again for the time and care given to our work. 😊 In particular, we are sincerely grateful to the Area Chairs for their kind reminders. We understand that the substantial reviewing workload may have limited opportunities for reviewers to respond, and thank the reviewers for their (really!) thoughtful and valuable suggestions. We believe this work makes an important and useful contribution to the community by advancing the field’s exploration of interpretable evidence for LLM-generated text detection. As the discussion draws to a close, we respectfully ask that our revised manuscript and substantive responses receive careful consideration in the final assessment.

Bests,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.