- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes EchoPrompt, a training-free zero-shot detector for LLM-generated text. Based on the hypothesis that machine text implicitly retains dependency on upstream instructions, EchoPrompt prepends a generic assistant prefix () and scores text by comparing the likelihood gap between an instruction-tuned model and a base model: . The method demonstrates performance gains across multiple benchmarks, particularly in the low-FPR regime (TPR@1%FPR).
Critical strengths
- Clear Writing and Good Presentation: The paper is well-written, well-organized, and easy to follow. The overall methodology and workflow are clearly illustrated with high-quality figures (e.g., Figure 1).
- Comprehensive Experimental Evaluation: The authors conduct extensive experiments spanning multiple public benchmarks, diverse proxy model architectures/scales, and challenging adversarial attack settings, providing thorough empirical results.
- Practical Low-FPR Performance: The improvement in macro-average TPR@1%FPR from 73.11% (IRM) to 81.86% is practically meaningful for real-world scenarios where false accusations must be strictly bounded.
Critical weaknesses
- Incremental Technical Novelty over IRM: Mathematically, the proposed score is essentially the existing IRM baseline plus a prefix-induced conditional probability shift: . Framing the addition of a hardcoded generic prompt as a new paradigm of "Latent Prompt Restoration" leaves a noticeable gap between the conceptual pitch and the actual technical delta.
- Generality of Prefix Selection and the "Restoration" Hypothesis: The core claim that performance gains stem from "latent prompt restoration" requires scrutiny. In Appendix Table 4, candidate prefixes exhibit considerable AUROC variance across prompt families, while random noise yields 0.7992. If the mechanism truly reactivated an intrinsic prompt-response dependency, semantically related assistant prompts should yield stable improvements. Since many candidate variants fail to outperform IRM, it is difficult to confirm a universal "restoration" phenomenon rather than empirical distribution fitting via a dev-set tuned template.
- Unexplained Positive Gain on Human Text for Qwen Models: In Section 3.3 (Figure 2 and Table 7), human text exhibits negative gains under Llama proxies as expected by the authors' hypothesis. However, on Qwen2.5 (1.5B and 3B Multi-Domain), human text also obtains positive likelihood gains after prefix injection. If the prefix selectively reactivates machine-specific prompt dependencies, why does human text benefit similarly under Qwen? This contradiction suggests the gain may arise from general prefix priming or attention smoothing rather than true prompt restoration.
- Domain and Task Boundaries: AUROC drops markedly to 82.50% on RAID non-chat/completion data. This indicates the method heavily relies on text conforming to conversational assistant conventions and struggles to generalize to open continuation, code, or creative writing.
Questions for authors
- In the candidate prefix evaluation (Appendix Table 4), what are the exact performance figures for the unprompted baseline (IRM)? How many prefix candidates actually beat the unprompted baseline?
- Under the "latent prompt restoration" hypothesis, how do the authors explain the positive context gains observed on human-written text for Qwen base models?
- What is the actual classification F1 performance across benchmarks if evaluated with a fixed decision threshold pre-calibrated on validation data rather than an oracle post-hoc Best-F1?
- Experimental Suggestion for Mechanistic Verification: If the authors consider "latent prompt restoration" to be the primary mechanistic contribution of this work, I suggest conducting a more direct mechanistic verification experiment. An intuitive idea would be: on benchmarks where upstream generation prompts are available, evaluate the detector using the original ground-truth prompt for each text. In theory, the true prompt should serve as the oracle upper bound for prompt restoration. If using the ground-truth prompt does not yield a noticeably larger likelihood gain or better class separability than the generic prefix (or even underperforms it), it would be difficult to attribute the empirical improvements to the "restoration of latent prompt dependency," as opposed to generic stylistic steering from an assistant template. Could the authors provide experimental results or discussion on this intuitive check?
W1: The technical novelty over IRM is limited.
EchoPrompt's contribution is to expose a detection signal not explicitly measured by unconditioned scoring: machine text retains a measurable dependency on an assistant–response context even when its original prompt is unavailable. We restore this shared context in the instruction-tuned proxy and use the base model to calibrate the likelihood. Sec. 3.3 establishes this contextual response; Fig. 3 identifies the prompt–response clause as the strongest individual component; and Table 3 validates the calibrated score (0.9864 AUROC / 0.9519 Best-F1). The full prefix improves AUROC by 5.33–14.73 points. Across five main settings, EchoPrompt improves Best-F1 by 2.64 points and TPR@1%FPR from 0.7311 to 0.8186. These results demonstrate the value of our training-free detector.
W2 & Q1: Prefix generality and its performance relative to IRM are unclear.
On the independent HC3 development set, IRM achieves 0.6737 AUROC; all 17 prefix candidates exceed it (0.7019–0.9003). We tested the same candidates and one fixed EchoPrompt prefix using three proxy pairs across DetectRL, RealDet, and RAID, without dataset-specific tuning. EchoPrompt exceeds IRM in all nine settings. The same prefix is used throughout, rather than selected separately for each proxy or dataset. Six comparisons against an unprompted score are shown below:
W3 & Q2: Positive human-text gains under Qwen challenge the hypothesis.
Positive human gain does not contradict our hypothesis: machine text should gain more. Table 7 reports Qwen2.5-1.5B human/machine gains of 0.0209/0.0486 and 0.0040/0.0461; for Qwen2.5-3B, 0.0056/0.0377 and −0.0062/0.0362 (Multi-Domain/Multi-LLM). The machine-minus-human differences are +0.0277, +0.0421, +0.0321, and +0.0424, consistently supporting the predicted relative effect.
W4: Performance on RAID non-chat raises generalization concerns.
DetectRL, RealDet, and RAID contain diverse non-dialogue texts; EchoPrompt does not require conversational formatting. It achieves 0.9998 AUROC on Yelp continuation, 0.9696 on few-shot human-style writing, 0.9960 on ICO/SICO, and 0.9846 on back-translation. On RAID Non-chat, EchoPrompt leads at 0.8249, versus IRM (0.7948), Binoculars (0.7966), Fast-DetectGPT (0.7972), and DNA-DetectLLM (0.7562). This subset challenges other detectors too, while EchoPrompt remains strongest by AUROC.
Q3: Fixed-threshold F1 performance is not reported.
We calibrated method-specific thresholds using 795 examples per class from DetectRL Multi-Domain and transferred them unchanged to five evaluations. We removed exact text overlaps with calibration data and used no test labels to select thresholds.
EchoPrompt leads in every setting, with a 3.01-point macro-average F1 gain.
Q4: The original generation prompt is proposed as an oracle upper bound.
The original prompt is not an upper bound on detection AUROC. The detection proxy can also differ from the actual generator, so the source prompt need not maximize class separation under that proxy. In RAID, machine passages are generated from prompts linked to human reference texts. We apply each original prompt to both classes. Machine-minus-human mean likelihood gains are +0.0958, +0.0687, +0.0257, and +0.0418 across four settings, confirming stronger machine-side contextual responses. However, increased likelihood need not improve class separation:
The original prompt is worse in three settings and comparable in one. Because it is also relevant to the human source, it can raise both classes' likelihoods without increasing AUROC. EchoPrompt instead restores a shared response context that improves detection.