- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work discusses how to create a zero-shot detector of LLM-generated text using a text's token sequence and its reversed copy with a frozen LLM. The hypothesis that the authors state is about how LLM text produces more unpredictability under reversal than human text.
Critical strengths
- The method itself is zero-shot, very simple and cheap to run. The experiments conducted were also especially extensive and useful.
Critical weaknesses
- I'm not entirely sure if reversal is the only thing that is useful here. According to Table 9, it seems that any other form of corruption is sufficient.
- The paper's novelty is also limited. Several past works [1, 2] discuss using a reverse loss / cross-entropy for detection.
- I also feel like the attacks are not realistic to evaluate this method. Maybe using RAID [3] as a benchmark could be useful here.
[1] Yu et al. Reverse Modeling in Large Language Models. NAACL 2025. [2] Guo et al. BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens. NeurIPS 2024. [3] Dugan et al. RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors. ACL 2024.
Questions for authors
- Does this hypothesis still hold for reverse-style LLMs (like LEDOM)?
- What is the FPR on polished human prose (law articles, academic abstracts) and on texts under 150 words? Limited texts is also an interesting ablation to run.
- How was chosen in the first place (apart from the empirical study in Table 10)?
5 comments
Responses to Reviewer bdtD (2/2)
W3 (realism of the attacks, RAID [3]).
As suggested by the reviewer, we evaluated on RAID, which attacks human-written and machine-written text alike. We ran TIDE and the baselines of Table 1 with Qwen3.5-4B on the eight domains of RAID, 500 human-written documents per domain and 500 from its six chat generators, each document under each attack in the table, and report the AUROC per attack averaged over the domains in the following table (best per column in bold):
| Method | None | Alt. spell. | Art. del. | Homogl. | Ins. par. | Number | Misspell. | Synonym | Up/low | Whitesp. | ZWSP | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Entropy | 0.916 | 0.912 | 0.903 | 0.891 | 0.913 | 0.919 | 0.912 | 0.867 | 0.910 | 0.890 | 0.867 | 0.900 |
| Log-Rank | 0.919 | 0.916 | 0.917 | 0.889 | 0.919 | 0.922 | 0.917 | 0.865 | 0.915 | 0.904 | 0.897 | 0.907 |
| LRR | 0.913 | 0.911 | 0.919 | 0.822 | 0.914 | 0.915 | 0.914 | 0.859 | 0.915 | 0.909 | 0.825 | 0.892 |
| Fast-DetectGPT | 0.780 | 0.768 | 0.826 | 0.723 | 0.745 | 0.780 | 0.772 | 0.715 | 0.732 | 0.704 | 0.804 | 0.759 |
| Binoculars | 0.793 | 0.776 | 0.827 | 0.752 | 0.762 | 0.786 | 0.777 | 0.654 | 0.713 | 0.711 | 0.841 | 0.763 |
| DNA-DetectLLM | 0.753 | 0.736 | 0.801 | 0.773 | 0.687 | 0.751 | 0.736 | 0.690 | 0.666 | 0.567 | 0.863 | 0.729 |
| Lastde | 0.901 | 0.895 | 0.901 | 0.778 | 0.896 | 0.891 | 0.898 | 0.676 | 0.883 | 0.903 | 0.859 | 0.862 |
| SpecDetect | 0.911 | 0.900 | 0.906 | 0.854 | 0.911 | 0.911 | 0.902 | 0.774 | 0.893 | 0.895 | 0.866 | 0.884 |
| TIDE | 0.935 | 0.934 | 0.919 | 0.942 | 0.928 | 0.932 | 0.933 | 0.802 | 0.928 | 0.897 | 0.914 | 0.915 |
The results show that TIDE is the best method on nine of the eleven attacks and on average. We have added this table to Appendix N of the revised paper, marked in blue.
Q1 (Does the hypothesis still hold for reverse-style LLMs, such as LEDOM).
Yes, with the two directions exchanged. We use the public preliminary checkpoint of LEDOM-7B on the 25 English cells of Table 1 with the framework of TIDE unchanged. The following table gives the class means of the two losses (human-written / LLM-generated) and the AUROC of TIDE, the last column with the two losses exchanged:
| Surrogate | TIDE | TIDE, losses exchanged | ||
|---|---|---|---|---|
| Qwen3.5-4B (forward) | 2.73 / 1.82 | 6.05 / 6.25 | 0.995 | – |
| Llama-2-7B-chat (forward) | 2.50 / 1.86 | 6.63 / 7.24 | 0.992 | – |
| LEDOM-7B (reverse-trained) | 6.27 / 6.79 | 2.47 / 2.17 | 0.548 | 0.947 |
The results show that TIDE works with a surrogate trained in either direction, because it checks the difference between the two readings. For the reverse-trained model the reversed text is the native reading, LLM-generated text is again the class that costs more to read against the direction of generation, and TIDE with the two losses exchanged reaches 0.947 AUROC. The 0.548 of the original order is expected, since for a reverse-trained model the original text is the reading against its training direction, so the roles of the two losses are swapped.
Q2 (FPR on polished human prose and on texts under 150 words).
We test on the academic-abstracts domain of RAID and report the FPR of TIDE and the baselines of Table 1. The threshold of each method is set on the human-written documents of our English training split and applied unchanged, for TIDE and the average of the baselines, as shown in the following table:
| Method | FPR, all abstracts | FPR, under 150 words |
|---|---|---|
| Baselines | 0.069 | 0.071 |
| TIDE | 0.023 | 0.031 |
The results show that TIDE falsely flags 2.3% of the abstracts and 3.1% of the short ones against 6.9% and 7.1% for the baselines on average, and at the same thresholds detects 87% of the machine-written abstracts of RAID against 75%. The reason is that TIDE scores the difference between the two readings of the same document, so what polished or short text changes in the forward loss it changes in the reverse loss as well, and the threshold holds. We have added this table to Appendix N of the revised paper, marked in blue.
Q3 (how was chosen).
We chose the score in two steps. First, we observed that LLM-generated text loses more predictability than human writing when the surrogate reads it in reverse (Section 3). Second, to measure this directional asymmetry, we tried several candidates. Among them, the difference and the ratio are the most intuitive and the best-performing ones. We evaluated both and found them complementary, e.g., the ratio is stronger on news and the difference on peer reviews in Table 10, and their product combines them without a weight and adjusts itself to the scale of the losses (see W1 in our reply to Reviewer msKX). Therefore, is the observed asymmetry, written in the form that the empirical comparison singled out.
Gentle Follow-Up
Dear Reviewer bdtD,
Thank you for your detailed and constructive review. We have posted our response and uploaded a revised PDF with the new experiments and clarifications marked in blue. As the author-reviewer discussion period is nearing its end, we would like to kindly check whether there are any remaining concerns that we can help address. We would be more than happy to provide any further results or discussion that would be useful.
Best regards,
Authors
Responses to Reviewer bdtD (1/2)
We thank the reviewer for the constructive suggestions and the pointers to related work. Here are our point-to-point responses:
W1 (is reversal the only useful corruption, Table 9).
We agree that on English with Qwen3.5-4B, the first column of Table 9, the existing corruptions seem sufficient. For example, the 4-token block shuffle reaches 0.994, compared to 0.995 for TIDE. However, the other three columns of Table 9 show that they fail once the surrogate or the language changes. For example, in the other languages, the best corruption reaches 0.960 with Qwen3.5-4B and 0.860 with Llama-2-7B-chat against 0.970 and 0.925 for TIDE, and the best corruption changes from column to column. The reason is that the existing corruptions either keep most of the forward order, changing the direction of reading only in part, or destroy the structure of the text, which raises the loss of both classes alike, whereas reverse reading in TIDE maintains every adjacency of the text and flips only the direction of reading. Therefore, the signal is the direction of reading rather than the amount of disruption, and TIDE's reverse reading is the only one that measures it in full.
W2 (novelty relative to Yu et al. [1] and BiScope [2]).
Yu et al. [1] train a model on both reading orders and use the gap between its forward and reverse losses, which tracks text quality, to select pretraining data. TIDE trains nothing and reads the reversed text with a frozen forward-only surrogate, so its reverse loss is the cost of reading against the direction of generation. The plain difference of the two losses is the D row of Table 10, and the product with the ratio that TIDE uses improves on it, for example from 0.953 to 0.970 on non-English languages with Qwen3.5-4B.
As for BiScope, it does not change the reading direction when the surrogate reads the text. Its backward cross-entropy only looks back one position when the loss is computed. As suggested by the reviewer, we added BiScope [2] as a baseline on every cell of Table 1, trained once on the English training split and applied unchanged, as shown in the following table (the better one per column is in bold):
The results show that BiScope matches or exceeds TIDE on the English cells that share a similar distribution with its training set but falls behind on the other languages, and overall TIDE is 0.035 and 0.045 better than BiScope on the two surrogates. We have added BiScope to Table 1 of the revised paper, marked in blue.