- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on discrepancies in text distribution between humans and large language models (LLMs) and proposes a framework called Sequential-Structural Discrepancy (SSD) to distinguish between human-written and LLM-generated text without relying on provenance supervision, which indicates the text's origin. As the ability to distinguish between human-written and LLM-generated text becomes more important, this line of research becomes more valuable. From a structural perspective, the SSD framework is intriguing because it distinguishes between the two through a two-stage process. The thorough experiments conducted are also commendable. However, this study raises several concerns.
Critical strengths
- S1: This paper is written in an easy-to-understand manner.
- S2: An anonymized implementation and experimental configurations are included in the supplementary material and are also available at https://anonymous.4open.science/r/SSD-BC84/.
- S3: The effect of the unlabeled pretraining scale is presented in Table 4.
Critical weaknesses
-
W1: Kim et al.'s study highlights the structural differences between human- and LLM-generated text. However, it is not included in the references.
- Kim, Z. M., Lee, K., Zhu, P., Raheja, V., & Kang, D. (2024, August). Threads of subtlety: Detecting machine-generated texts through discourse motifs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5449-5474).
-
W2: The paper contains several unclear points.
- How is context C_v, which is described on page 4, created?
- Which function is used as the activation function, σ, in equation (8)?
- What is s_j in the equation to define p_i?
- Is MHSA MultiHeadSelfAttention?
- In the experiment shown in Table 1, shouldn’t the authors have discussed what information was missing based on a comparison with Lastde, and how to address those gaps?
- As shown in Table 5, the "label-free pretraining, then encoder fine-tuning" approach in view learning outperforms the full model in L2R(ID). However, the reason for this is unclear.
-
W3: The technical novelty is not particularly high.
-
W4: The limitations of this research are not clearly stated.
Questions for authors
Please see the "Weaknesses" section.
5 comments
We thank the reviewer for identifying several points that require clarification.
We thank the reviewer for identifying several points that require clarification.
Limitations (W4)
We will add a dedicated limitations section covering four aspects. First, SSD depends on automatically predicted dependency trees. Although the spaCy/Stanza and corruption experiments demonstrate some tolerance to parsing errors, performance may still be affected by inaccurate parses. Second, L2R, the evaluated M4 subset, and the C4 pretraining corpus are English, so cross-lingual transfer remains unverified. Third, SSD is less effective in some specialized domains: on MedicalText, it obtains 87.01% AUROC, below Lastde's 91.15%, because formulaic human clinical records may also exhibit strong sequential-structural consistency. Finally, dependency parsing, dual-view encoding, and proxy-model inference introduce additional cost. SSD requires 1.03 seconds per text under our reported setting, making it faster than generation-intensive detectors but slower than lightweight probability-based methods.
Methodological clarifications (W2)
Context .
is not an additional tensor; it denotes the information available to view when reconstructing masked word . contains the masked sequence , word order, and positional information. contains the same masked node features together with the directed, relation-typed dependency graph ; after RGCN layers, it covers the corresponding -hop syntactic neighborhood. The two views therefore share the same masked node features and reconstruction targets, but use different contextual structures and propagation paths. The conditional-expectation expression describes the squared-loss optimum rather than an additional implementation step.
Equation (8).
The outer activation is the logistic sigmoid, which constrains the gate to ; the inner activation is .
Definitions of and .
is the -th proxy-model subword and denotes its preceding subwords. For word , contains its aligned subword indices, and their log-probabilities are averaged:
MHSA.
Yes. MHSA denotes Multi-Head Self-Attention.
Additional experimental analysis (W2)
Comparison with Lastde.
We examined representative MedicalText errors. A formulaic human clinical record has a low mean discrepancy of 0.5847 and is misclassified as machine-generated, whereas a syntactically elaborate LLM text has a higher discrepancy of 0.6748 and is misclassified as human-written. This illustrates that standardized human writing can exhibit strong cross-view consistency, while complex machine writing can produce larger discrepancy. Lastde additionally models local fluctuations and multiscale dynamics in token-probability sequences, providing complementary evidence in this domain. We will add this failure analysis and discuss combining these signals as future work.
Encoder fine-tuning in Table 5.
Fine-tuning the pretrained encoders raises L2R AUROC by only 0.61 points (96.59% to 97.20%) but reduces M4 by 8.39 points (78.39% to 70.00%). Provenance supervision improves in-domain fitting but weakens transfer. Freezing the encoders therefore deliberately trades a small ID gain for a substantially more transferable discrepancy.
Dear Reviewer FBqH,
Thank you for your valuable feedback and for taking the time to review our rebuttal. As the discussion period is nearing its end, we would kindly appreciate it if you could take another look at our responses. We hope our clarifications have addressed your concerns and would be grateful if you could consider them in your final assessment. Thank you again for your time and consideration.
Best regards, The Authors
Clarifying SSD’s Novelty and Relation to Prior Structural Detection Work
Relation to Kim et al. and technical novelty (W1 and W3)
We thank the reviewer for pointing out Kim et al. (2024), which is an important antecedent to our motivation and should have been discussed. We will add this reference in the revision. Kim et al. demonstrate that human- and machine-generated texts exhibit different discourse-level structures; we do not claim that the existence of structural differences itself is new.
The two works nevertheless address different technical questions. Kim et al. transform RST discourse trees into recursive hypergraphs and compare motif frequencies and depths across human and machine corpora. These discourse motifs are subsequently used as structural features in supervised classifiers. SSD neither extracts discourse motifs nor defines discrepancy by comparing two authorship groups. Instead, it computes a token-aligned discrepancy within each individual text, between representations of the same content learned from sequence-conditioned and dependency-conditioned contexts.
The contribution of SSD is therefore not simply combining a Transformer and an RGCN. Both view encoders independently reconstruct the same representations produced by a frozen language model, providing a common reference space for their outputs. They are pretrained without human/LLM provenance labels and frozen before downstream detection. Thus, provenance supervision can learn how to use the discrepancy but cannot create or reshape the encoders that produce it. The resulting discrepancy then controls adaptive view fusion and is retained as an explicit token-level feature for classification.
Our results directly test whether this formulation contributes beyond standard structural features or heterogeneous architectures:
The frozen sentence-level discrepancy alone obtains 74.88%/72.14% AUROC on L2R/M4 without fitting a classifier (Table 3), showing that the discrepancy already carries transferable detection information.
To control for architectural differences, we replaced the sequential Transformer with an RGCN over a bidirectional word-order chain while retaining an RGCN over the dependency graph for the structural view. This matched-architecture variant obtains 96.40%/78.14% on L2R/M4, close to the full model's 96.59%/78.39%. The effectiveness of SSD therefore does not primarily arise from combining different encoder architectures.
Removing D reduces M4 AUROC from 78.39% to 66.89%; removing discrepancy-guided gating reduces it to 69.26%; and using D only for gating yields 75.61% (Table 5). These results verify its complementary roles in fusion and classification.
Under the same protocol, the syntax-based PRDetect obtains 89.95%/65.12% on L2R/M4, compared with 96.59%/78.39% for SSD, further distinguishing cross-view discrepancy modeling from using syntactic structure alone.
In short, Kim et al. study structural patterns across authorship groups, whereas SSD studies cross-view disagreement within each text and isolates its formation from provenance supervision. We will revise the introduction and related work to make this distinction explicit.