- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PEC-Detector for document-level machine-generated text detection, which models relations between remote discourse history, local context and current sentence. Its key observation is that conditioned on local context, the contribution of remote history to semantic prediction tends to grow as the machine-generated documents unfold, but remains fairly constant in human-written documents. Motivated by this observation, the proposed method learns position-wise “planning – execution coupling” representations and encodes their sequence to classify the source with an auxiliary semantic prediction objective. Experiments are performed across domains, generators, attack settings, and languages.
Critical strengths
- The research question is sound and testable. Investigating how the conditional predictive contribution of remote context changes over the course of a document is an approach to view document-level detection from the most useful perspective. The use of an independent predictor trained without source labels for the first observation also avoids circular explanations based on supervised classification features directly.
- Detections results reported are competitive. PEC-Detector is superior to the compared baselines in several evaluation settings, indicating that the designed contextual relation representations are applicable in real detection.
- Experiment coverage is relatively wide. Besides the main benchmarks, there are component ablations, analyses of sentence order, text length, training data scale and multilingual performance in this paper.
Critical weaknesses
- The core interpretation as “planning – execution coupling” is not sufficiently established. Operationally, the planning state is a representation that is encoded from remote history. The fact that it enhances current-sentence prediction does not automatically prove that it represents discourse planning: topic-continuity, entity-recurrence, semantic-repetition, and formulaic-organization could all yield the same effect. The authors correctly note that this state does not directly reconstruct the author’s original plan. Still, additional justification and evidence should be provided to validate why planning – execution coupling is a more mechanistic explanation than general long-range semantic dependence. The present results more directly bolster the detection value of remote-context relations rather than the postulated mechanistic interpretation.
- The link between the central observation, the architecture, and the performance gain is still not complete. Fig. 1 addresses the position-wise evolution of a scalar predictive gain, while the final classifier is fed with high-dimensional representations comprising current-sentence semantics, contextual states, and the multiple interaction features. Good performance of the full model thus does not directly prove that the observed gain trajectory accounts for the gain. The ablations do verify that the individual components are useful, but do not allow isolating the contribution of this specific observation in isolation from richer semantic representations, context modeling, and model capacity. A simple detector on only predictive-gain trajectories with an similarly matched ordinary sentence-sequence classifier in terms of encoder and context coverage and number of parameters would directly help.
- Alternative explanations for the original observation are not well controlled. The amount of available remote history naturally increases with document position. Human and machine texts can also differ in the number of sentences, sentence length, topic concentration, and semantic repeats. The progress normalization and document-level aggregation in the appendix are intuitive, but do not entirely control these factors. Further, the central tendency is mainly observed on CMV without reliable evidence of consistent replication across domains and generators. As a result, it is not clear whether the trajectory indicates a stable source-related difference or characteristics of the given dataset and its text distributions.
- Some experiments do not entirely support the related mechanistic and generalization claims. Performance degradation under test-time sentence shuffling indicates sensitivity to ordered discourse information, but not uniquely to planning – execution relationships; a general coherence model may indicate a similar sensitivity. Additionally, Multi-Attack experiments are trained on training data from the same settings, thus these results should not be taken to directly indicate robustness to unseen attacks. MAGE experiments should similarly specify which generators or domains are not seen during training, to separate effectiveness on heterogeneous data from transfer to truly unseen data.
Questions for authors
- Does the trajectory difference in Figure 1 continue given remote-history length fixed and document length, topic and semantic repetition are equal? Does it reproduce itself across other domains and generators?
- How well can a detector do with only the mean or the slope or the full sequence of predictive gains? What added value does PEC have over a capacity matched ordinary sentence-sequence classifier?
- During joint training, does the classification loss also update the predictor through the predictive-gain branch? If yes, how is the desired semantic-prediction interpretation of this gain confirmed?
General Response
For clarity and simplicity, we refer to Reviewers XQw9, 7p6J, and 2Pxv as R1, R2, and R3, respectively.
We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged by their recognition of the strengths of our work.
We have carefully addressed each individual comment raised by the reviewers and believe that we have successfully responded to most of their concerns. In the revised manuscript, we have incorporated the suggested experiments, expanded the related discussions, and made the corresponding revisions to further strengthen our work. Below, we summarize the core contributions of our study, the updates to our experiments, and the clarifications included in our revision.
Core Contributions
We revisit document-level machine-generated text detection from a dynamic writing perspective. Given local context, the conditional contribution of remote discourse history to the current writing follows distinct trajectories as documents unfold: it generally increases in MGT, but remains relatively stable in HWT.
Based on this observation, We propose PEC-Detector, a framework that formulates source prediction as identifying ordered planning-execution coupling trajectories to distinguish MGT from HWT.
Extensive experiments on public benchmarks demonstrate that our method delivers superior performance across a variety of complex scenarios, including multi-domain, multi-LLM, cross-setting, out-of-distribution transfer, and adversarial attacks.
Experimental Updates During Rebuttal
Main Text (Section 4.2 and Table 3): Added out-of-distribution evaluation on MAGE Unseen Domains and Unseen LLMs, and distinguished these held-out settings from the original MAGE cross-setting evaluation (R1, R2, R3).
Main Text (Robustness Against Attacks) and Appendix C.6: Added an evaluation that trains PEC-Detector on clean Multi-Domain data and tests it on unseen attack distributions (R1, R3).
Main Text (Section 4.3 and Table 5): Added the AvgPool State ablation to evaluate whether average pooling can replace the planning and local-context encoders (R1).
Appendix A.1: Added predictive gain controls using gain mean, gain slope, the full predictive gain trajectory, and the sentence semantic trajectory (R3).
Appendix C.2: Added results at TPR@1%FPR and TPR@5%FPR (R1).
Appendix C.4: Added detailed F1 results for MAGE Unseen Domains and Unseen LLMs (R1, R2, R3).
Appendix C.5: Added a strict unseen newer generator protocol, where the detector is trained and selected using GPT-3.5 only, then evaluated on GPT-4-Turbo, Gemini-2.0-Flash, and Claude-3.7-Sonnet without adaptation (R1, R2).
Appendix C.7: Added the Late-Trajectory ablation, which evaluates whether the latter half of the position-wise coupling trajectory is sufficient for detection (R1).
Appendix C.8: Added a capacity matched generic long-context control to evaluate the additional value of the explicit planning-execution relation construction beyond general long-range semantic information (R2, R3).
Clarification and Manuscript Updates During Rebuttal
Main Text (Related Work): Added discussion of Lastde and Profiler and clarified their relation to PEC-Detector (R1).
Main Text (Sections 3.2, Figure 2): Clarified the operational meaning of the planning state, removed the unimplemented mappings Ap and Ae, distinguished the two conditional prediction heads, explained the role of local context in the relation mapping, and aligned Figure 2 with the implementation (R1, R2, R3).
Main Text (Section 4.1): Clarified that the original MAGE Fixed-domain & Arbitrary-models and Arbitrary-domains & Model-specific settings are cross-setting evaluations rather than strict out-of-distribution transfer settings (R2).
Main Text (Section 4.2) and Appendix C.5: Clarified the checkpoint selection protocol for newer generator evaluation and explicitly stated that the target generators are excluded from model development (R1, R2, R3).
Appendix C.10: Corrected the encoder robustness description and explicitly identified all-mpnet-base-v2 as the frozen default encoder and all-MiniLM-L6-v2 as the frozen replacement encoder (R2).
Reference List: Corrected the omitted HLD author and checked the complete reference list for remaining bibliographic inconsistencies (R1).
We believe that these additions and clarifications comprehensively address the reviewers' concerns and strengthen the manuscript. For ease of review, all revisions are highlighted in blue in the uploaded revised manuscript.
We look forward to the reviewers' favorable consideration and remain grateful for their valuable feedback.