- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses the performance degradation of machine-generated text detectors in personalized domains, where model-generated text imitates an individual author’s writing style. It proposes PERSFIX, a post-processing method that augments an existing detector score with an auxiliary score obtained by projecting the final-token residual-stream representation from GPT-2 layer 10 onto an extracted direction.
Motivated by the hypothesis that discriminative features may include inverted components, whose machine-versus-human directions flip across domains, and consistent components, whose directions remain stable, the authors extract a consistent linear direction from the final-token residual-stream representation at GPT-2 layer 10. This direction is obtained by aligning machine-minus-human class contrasts across general and personalized domains, and its projection is added to the original detector score.
The correction produces large improvements across multiple detectors on the evaluated personalized benchmarks and transfers across authors and generator models within the personalized literary setting.
The use of a final-token representation is not supported by controls for token aggregation, sequence length, suffix conventions, or other surface-level artifacts. The extraction data also vary simultaneously in genre, task, author style, and historical period, making it difficult to isolate a machine-generation signal from these confounds. The method adds an auxiliary linear projection score rather than directly identifying and suppressing inverted components within the original detector.
The paper shows utility as a post-hoc score correction, but its broader claims concerning representational feature duality and the suppression of detector-internal inverted features are not yet sufficiently supported.
Critical strengths
- Practical utility and detector-agnostic applicability.
PERSFIX provides a simple post-hoc correction for personalized style imitation, a setting in which existing detectors can degrade severely or produce below-chance rankings. It requires neither retraining nor internal modification of the underlying detector and yields large empirical gains, particularly in the fine-tuned literary setting. Its transfer across authors and generator models within this setting suggests that the method may be useful as a lightweight correction rather than merely as a detector-specific adjustment.
- An actionable response to a known failure mode, supported by extensive empirical analyses.
This paper moves beyond documenting feature inversion by extracting a cross-domain class-contrast direction and using its sample-level projection to adjust detector scores. This method is evaluated across both training-based and training-free detectors, multiple personalization settings, proxy models, layers, and probe-set sizes. Comparisons with random, domain, and generic MGT directions indicate that the proposed direction contains predictive information beyond an arbitrary score perturbation. The Appendix E direction comparisons are explicitly upper-bound analyses because α\alphaα is selected on the test set; they should not be read as held-out performance comparisons.
Critical weaknesses
- The representational interpretation is not adequately validated.
The claim that PERSFIX identifies a consistent machine-generation feature relies on a final-token residual-stream representation from GPT-2 layer 10. This state is not trained as a document representation and may be sensitive to the suffix, termination or truncation convention, sequence length, and other local properties. The probing domains differ simultaneously in genre, author style, historical period, and generation task, including rewrite versus continuation. This paper compares layers, modules, proxy models, and extraction datasets, but does not compare token aggregation schemes or control for sequence length, suffix effects, or simple stylometric features. The projections show same-direction HWT–MGT separation on the two probe datasets used to define the construction. They do not isolate whether the direction is stable with respect to machine generation itself, as opposed to the particular combination of source corpus, generation task, author, period, length, and terminal-token convention in those probes. This weakens the causal interpretation attached to the score correction.
- The mathematical formulation and experimental specification contain consequential inconsistencies or ambiguities.
Equations 5–6 use cross-domain difference vectors paired by a common index , whereas Algorithm 1 constructs each within-domain difference set using all MGT–HWT combinations and then takes the Cartesian product . These define materially different estimators. If Algorithm 1 is implemented literally, depends only on the sums, and hence on the class-mean contrasts up to scale, of the two domain-wise difference sets. It does not retain the sample-level cross-domain contrast structure suggested by Equations (5)–(6). The matrix has rank at most two, and all nonzero eigenvectors lie in the span of the two domain-level mean contrast vectors. Section 4.2 restricts the search for to , yet Table 2 reports mean values as high as 3.00; Table 2 is described as using development-set selection, whereas the separate oracle-tuned experiments are introduced only in Appendix E. The current text therefore does not explain how the Table 2 values were obtained. These discrepancies affect the effective estimator, the strength of the correction, and the composition of the headline results.
Questions for authors
- Estimator Construction:
Which exact formulation of matrix was used in the reported experiments: the sample-paired formulation in Equations 5–6, the all-pairs Cartesian-product formulation in Algorithm 1, or another implementation? Please describe both the within-domain MGT–HWT pairing and the cross-domain aggregation procedure. If Algorithm 1 was used literally, our reading is that the matrix factorizes through the two domain-wise sums of MGT–HWT difference vectors and has rank at most two. How should the resulting eigenspace and the claimed feature-duality interpretation be understood?
- Hyperparameter and evaluation setup:
How can the mean values of 2.90–3.00 in Table 2 arise from the candidate range specified in Section 4.2? Was a different search range, score normalization, or oracle-tuning procedure used for Table 2? Please report the exact range and selection data used for this table, and clarify whether the headline averages include nine or eleven detectors and identify the included detectors.
- Representation robustness:
Are there results showing whether the main improvements persist under alternative decoder-compatible extraction methods, such as mean pooling over later token positions or EOS pooling with a consistently appended EOS token, and after controlling for sequence length, suffix conventions, and simple stylometric baselines? If not, what evidence supports interpreting the extracted direction as a domain-consistent machine-generation feature rather than as a benchmark-specific auxiliary detector?
4 comments
Response 2/3 — W2 Part 1 + Q1
W2, Part 1 + Q1: Exact form of the matrix and the geometry of feature duality
Our experiments use the index-paired form in Equations 5–6 with . Appendix D.1 describes the pairs. In the general domain, each pair is a human XSum text and its LLaMA-3-8B rewrite . In the personalized domain, each pair is a human Jane Austen text and its LLaMA-3-8B continuation . Thus, and each pair a human text with the machine text generated from it. Across the two domains, we pair these difference vectors by the index and compute . Algorithm 1 was mistakenly written in all-pairs form. We have corrected it to match Equations 5–6.
We agree with the reviewer that Equations 5–6 and the all-pairs form in Algorithm 1 give different matrices for a finite sample. The rank-two derivation given by the reviewer shows how they are related. Let and be the mean difference vectors of the two domains. The all-pairs sum equals . It has rank two when and are nonzero and not collinear.
The XSum and Jane Austen probe samples have no correspondence, so the index across the two domains is arbitrary. The cross-domain pairing can therefore be treated as random. Averaged over uniformly random pairings, the index-paired equals . This is the same matrix up to a scale factor, so it has the same eigenvectors. The index-paired differs from it by a noise term with zero mean, which comes from the finite sample.
When and are nonzero and not collinear, the two nonzero eigendirections of have closed forms. The positive eigenvalue belongs to . This is the bisector of the two mean directions, and it corresponds to . The negative eigenvalue belongs to . This is their difference, and it corresponds to . In other words, the mean "machine minus human" directions of the two domains form an angle. Their bisector is the part shared by both domains, and their difference is the part that reverses between domains.
We checked this on our probe data. The angle between and is . The index-paired has cosine similarity 0.89 with the bisector, and has cosine similarity 0.95 with the difference vector. On the representative Fine-tuned subset, which is Jane Austen with Qwen3-4B and Lastde++, PersFix reaches 0.815 AUROC with the index-paired and 0.833 with the bisector. On the representative General subset, both reach 0.842. The two forms are therefore close in direction on the probe data and close in effect on these two subsets. In the revision, we fix Algorithm 1 and add this derivation to Appendix F.4.
Response 3/3 — W2 Part 2 + Q2
W2, Part 2 + Q2: Range of in Table 2, selection data, and detector counts
As Section 6.1 states, for the combined variants in Table 2 is chosen on the development set. The oracle upper-bound analysis is a separate experiment in Appendix E.
The values in Table 2 are standardized weights. When we built Table 2, we z-scored the detector score and the projection separately before combining them with . As a result, is in units of standard deviation and is not limited to . We did this so that the adjustment strength could be averaged across detectors whose raw scores have very different scales, such as log-likelihoods and probabilities. Equation 7 applies to the raw scores. The two are different scalings of the same linear combination. The caption of Table 2 did not state this, and we apologize for the omission. In the revision, the caption says that the last column is the mean selected standardized strength, chosen on the development set.
We also checked how the raw-score setup of Section 4.2 behaves in each range of baseline AUROC. We computed the five ranges on all 405 main-experiment subsets with Equation 7 exactly as written, with chosen on 200 development pairs. We compared this with a wider range, :
| Unadjusted detector AUROC | Subsets | Unadjusted detector | Sign-flipped on dev | alone | PersFix, | PersFix, |
|---|---|---|---|---|---|---|
| 155 | 0.292 | 0.705 | 0.811 | 0.792 | 0.802 | |
| 69 | 0.564 | 0.557 | 0.790 | 0.790 | 0.793 | |
| 62 | 0.723 | 0.723 | 0.799 | 0.844 | 0.843 | |
| 33 | 0.863 | 0.863 | 0.754 | 0.901 | 0.901 | |
| 86 | 0.974 | 0.974 | 0.755 | 0.978 | 0.978 | |
| All | 405 | 0.596 | 0.753 | 0.789 | 0.848 | 0.852 |
Where the baseline is below 0.5, the projection alone is the strongest variant. It reaches 0.811, against 0.792 for PersFix with and 0.802 with . From 0.5 to 0.65, PersFix equals alone at 0.790. From 0.65 to 0.8 and from 0.8 to 0.9, combining the detector with is better than either part alone. The values are 0.844 against 0.723 and 0.799, and 0.901 against 0.863 and 0.754. Above 0.9, PersFix stays at the level of the detector, with 0.978 against 0.974. Widening the range from to changes the overall mean from 0.848 to 0.852. In the revision, Table 2 stays in the main text with the corrected caption. The 405-subset table is added as Appendix E.4 and referred to in Section 6.1.
Regarding the number and names of the detectors, we clarify as follows. The paper evaluates eleven detectors, as listed in Table 1. They are RoBERTa, DeTeCtive, Entropy, Log-Likelihood, LogRank, Fast-DetectGPT, DetectLRR, Lastde, Lastde++, DNA-DetectLLM, and SpecDetect++. The text said nine because we added DNA-DetectLLM and SpecDetect++ later and did not update it. In the revision, the text says eleven detectors, and the mean values in the abstract and introduction are computed over all eleven. In the Fine-tuned setting, the mean AUROC rises from 0.466 to 0.839. Some of the additional analyses in this response cover a subset of these detectors.
Thank you for the detailed response.
Authors' response resolves the implementation and hyperparameter ambiguities, but only partially addresses the representational concern. Although the empirical utility of PERSFIX becomes clearer, the feature-duality interpretation remains insufficiently validated.
Response 1/3 — W1 + Q3
We thank the reviewer for the careful reading and the technical questions.
W1 + Q3: What Vcon means, and controls for length and surface features
Our definition of a consistent feature is operational. It is a direction in representation space along which human-written and machine-generated texts separate with the same sign in both the general and the personalized domain. Section 3.1 gives this definition and Section 4.1 gives the extraction objective. We do not intend a causal claim, and we do not claim that Vcon represents "machine generation itself."
We acknowledge that some sentences in the paper read more strongly than this. For example, the introduction says that the inverted feature "causes" the trap and that the adjustment "suppresses" inversion. We reword these sentences in the revision. What the paper claims is empirical: adjusting detector scores along this direction repairs detection in personalized settings. This holds across 11 detectors and 45 groups, and under cross-model and cross-scenario transfer.
Ablations already in the paper argue against simple surface explanations. Figure 5a varies the GPT-2 layer and module. Figure 5c replaces the reference model and extracts Vcon again from 19 other probe pairs with different authors and styles. Appendix E replaces Vcon with a domain direction, an MGT direction, or random directions. If the improvement came only from surface cues such as length or style, random and domain directions would not perform so much worse than Vcon. The 19 different probe datasets would also not give similar improvements.
We also tested sequence length directly. We used the representative Fine-tuned subset from Appendix E.1, which is Jane Austen with Qwen3-4B and Lastde++. We measured the correlation between the Vcon projection and the token count. It is −0.01 for human-written texts and −0.07 for machine-generated texts, so both are near zero. We then split the samples of this subset into short, medium, and long groups. Detection with the Vcon projection alone reaches 0.90, 0.88, and 0.87 AUROC in the three groups. On this subset, the projection does not depend on length.
Two further facts argue against seeing Vcon as an extra detector tuned to one benchmark. First, in all main and transfer experiments, Vcon is extracted once from 300 probe pairs, 150 per domain. It is then applied unchanged to 3 benchmarks, 45 groups, and 11 detectors, with no tuning per benchmark. Second, when Figure 5c extracts Vcon again from 19 other probe datasets, the improvement stays similar. In the revision, we add the length control as Appendix E.5.