- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies AI-generated text detection through discourse reconstructibility. A frozen language model reconstructs removed sentence blocks, and Hungarian matching aligns the reconstructed sentences with the originals regardless of sentence order. The initial method achieves 82.53% AUROC on a 70-document benchmark but drops to 61.55% on a 1,000-document stress test. A revised V3 method adds context discounting and similarity features, reporting 90.32% AUROC on 784 eligible documents. The authors also report low false-positive rates on non-native English essays and claim a linguistic fairness guarantee.
Critical strengths
Hungarian matching addresses a concrete problem: an infiller can reconstruct the relevant content in a different sentence order. Equations (4)–(5) avoid penalizing these reorderings and provide a clear formulation of the alignment step.
Critical weaknesses
-
The V3 results are difficult to assess without a clear separation between development and evaluation. Section 5.2 introduces V3 after inspecting failures, but does not establish whether its rules were fixed before testing on untouched data. Filtering alone does not explain the improvement: V2 reaches 68.92% AUROC on the same 784 documents, compared with V3’s 90.32%. However, Table 1 also contains inconsistencies that need resolution. The frontier GBDT reports 0% human FPR but 91.20% precision. For the balanced 1,000-document GBDT row, 65.20% recall and 12.40% FPR imply approximately 84.02% precision, not 67.50%, if calculated from the same predictions.
-
The paper does not establish its claimed linguistic fairness guarantee. Theorem 1 concerns sentence reordering, not false-positive rates across writer populations. Section 4 reports zero false positives among 12 ESL essays with a 95% confidence interval of [0%, 0%]; a standard two-sided exact binomial interval has an upper endpoint of approximately 26.5%. The 2.88% rate on 208 essays is an empirical observation, not a guaranteed bound. The uncertainty estimates need correction, and the fairness claim should be restricted to what the evaluated data support.
Questions for authors
-
Could you provide the confusion counts for the two GBDT rows identified above? Are precision, recall, and human FPR computed from the same predictions at the same threshold, or are different averaging procedures used?
-
What is the exact V3 scoring function, and which data informed its rules and the 28% threshold? Were any reported results obtained on documents untouched during development? In the five-fold evaluation, were feature and rule choices made within each training fold?
-
How was the [0%, 0%] confidence interval for the 12 ESL essays calculated? Is linguistic invariance intended as a formal guarantee or an observation limited to these datasets? If formal, which result establishes it?
Response to Reviewer qhzS — Exact Confusion Counts, Table 1 Arithmetic Reconciliation, and Empirical Bounds
Dear Reviewer qhzS,
Thank you for your expert review and for directly auditing our calculations. You identified genuine arithmetic discrepancies, statistical errors, and overclaimed bounds. We concede these errors unreservedly and provide exact, data-grounded corrections below.
1. Confusion Counts & Arithmetic Reconciliation (Table 1)
You are completely correct that Table 1 contained arithmetic contradictions:
Row 5 (Frontier GBDT, N=70) -- 0.00% FPR vs. 91.20% Precision:
Row 9 (Balanced N=1,000 GBDT) -- Recall 65.20%, FPR 12.40%, Reported Precision 67.50%:
2. V3 Scoring Formulation, Thresholding & Validation Separation
3. Exact Clopper-Pearson Intervals & Formal Scoping of Theorem 1
Thank you again for holding this work to the highest standards of scientific rigor.