- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes ZS-MTR-RR, a zero-shot AI-generated text detection method that adaptively combines global text features with trajectory features based on the input text length. The proposed method also uses a model pair, one base model and one instruction-tuned model, to strengthen the feature set through comparative analysis. Evaluation on existing benchmarks demonstrates that the proposed method outperforms several existing baselines.
Critical strengths
- This paper proposes an adaptive fusion method that smoothly routes the score from the global score to the trajectory score based on text length, making detection performance more stable across different text lengths.
- Evaluation on existing datasets demonstrates the better performance of the proposed method against existing methods.
Critical weaknesses
- The novelty of this paper needs more clarification. Several important components and insights have already been well explored by existing studies. For example, the base and instruction-tuned model pair is directly adopted from Binoculars [1]; the token-reuse features and , which evaluate the output distribution at a position against the token just read, are essentially the backward loss of BiScope [2]; the multi-lag recurrence is very similar to the context window in Profiler [3]; and the surprisal-volatility feature together with its information-density interpretation originates from GPT-who [4]. I highly suggest the authors investigate related work more thoroughly and re-position the paper correctly within the existing literature.
- Several key designs of the paper need more justification; they appear empirical or arbitrary rather than principled. For example, why does use the top-100 candidates rather than a ratio of the vocabulary size? Why does use a uniform estimate for tokens outside the top-100? The method only ever uses the probability and rank of the token just read, so the authors could store these two numbers directly instead of caching all top-100 probabilities and approximating the tail. Additionally, why does use token recurrence rather than word recurrence? No ablation or theory justifies these choices.
- The global score performs much worse than the Binoculars or Fast-DetectGPT score, although all of them are computed from the same proxy outputs. Why not replace with one of these better-performing global scores? This would not affect the paper's claimed advantage of adding the trajectory score to the final score.
- Many key hyper-parameters of the proposed method are selected empirically and need calibration for different domains or models, so out-of-distribution evaluation is essential for testing the generalizability of the proposed method when deployed. Appendix B.2 further admits that the feature set, coefficients and gate were developed with reference to EvoBench results and that the scalers are estimated on the evaluation pool, so the main results are optimistically biased relative to the baselines. Thus, instead of only showing AUROC, the authors should report the F1 score of the proposed method against baselines with a fixed threshold.
- According to Table S11, the proposed method shows more vulnerability to adversarial attacks than several baselines, which may expose more problems when deployed.
[1] Hans, Abhimanyu, et al. "Spotting llms with binoculars: Zero-shot detection of machine-generated text." ICML. 2024.
[2] Guo, Hanxi, et al. "Biscope: Ai-generated text detection by checking memorization of preceding tokens." NeurIPS. 2024.
[3] Guo, Hanxi, et al. "Profiler: Black-box AI-generated Text Origin Detection via Context-aware Inference Pattern Analysis." EMNLP. 2025.
[4] Venkatraman, Saranya, Adaku Uchendu, and Dongwon Lee. "Gpt-who: An information density-based machine-generated text detector." Findings of NAACL. 2024.
Questions for authors
Addition to the weaknesses, here are more detailed questions:
- In the complementary evidence, both and are computed from the difference of the two log-probability sequences ( on the full sequence, on the latter half). Why is there no corresponding counterpart for ?
- Why does the proposed method only consider the latter half, rather than combining it with the former half or other finer-grained ranges? The experiments show that the latter half performs best on its own, but other ranges may be complementary.