- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1(−1)
- Soundness
- 2
Read review
Paper summary
This paper questions the use of fractal scaling exponents as features for detecting AI-generated text at the level of individual short documents. It distinguishes population-level differences between human and machine text from the reliability with which such exponents can be measured in a single document. The results suggest that the former does not imply the latter, as single-document estimates are dominated by estimation noise at practical lengths. The paper therefore argues that the apparent value of fractal features mainly reflects generator- or population-level information, and proposes reliability testing as a prerequisite for their use.
Critical strengths
-
The paper identifies an important and often overlooked inferential gap in this literature. A distributional difference between classes is frequently treated as evidence that a statistic can characterize individual documents, but these are fundamentally different claims. By reformulating detector performance as a measurement-validity problem, the paper provides a useful perspective for evaluating document representations more generally.
-
The empirical study is broad and generally careful. The authors examine their central measurement from several angles and include controls against plausible artifacts. They also report results that complicate their own narrative rather than presenting only supportive evidence. This transparency strengthens the credibility of the work.
Critical weaknesses
-
The central conclusion is broader than the evidence currently supports. The experiments convincingly show that the examined estimators are unreliable for short documents, but they do not establish that fractal information is generally not a property of documents. Some alternative estimators approach or satisfy the authors’ own reliability criterion, yet their downstream value is not tested. The claims should therefore be narrowed to the estimators, document lengths, and settings actually evaluated, or supported with additional experiments.
-
The mechanistic interpretation is not yet fully established. The decomposition compares with , but classification separability depends on relative to the total per-document dispersion ; this standardized gap is never reported for any individual exponent or (domain, generator) cell, and non-overlapping pooled intervals only reflect mean precision at large . Some analyses also do not fully separate document-specific information from finer-grained population factors, e.g. prompt type is left in rather than in . The group-mean results admit other explanations: replacement also removes estimation noise, while subtraction retains 59% of the transfer gain. The evidence thus does not uniquely show that the benefit comes only from population means rather than document residuals.
-
The writing is verbose and repetitive, which weakens an otherwise interesting argument. The same decomposition, headline numbers, and thermometer analogy are restated across the abstract, introduction, methods, figure, discussion, and conclusion. The figurative labels for the four predictions are also reused long after their mapping to Eq. 2 has been established, without adding a definition, prediction, or evidence, so their purpose beyond illustration is unclear. The paper would benefit from substantial compression, consistent terminology, and more direct exposition.
Questions for authors
-
DFA and local Whittle approach or clear the stated reliability bar for . Do they also show no downstream value when used as detector features, or should the central claim be narrowed to the and Hölder estimators actually used in the detector?
-
Could you report the standardized population gap, relative to , for each exponent in each (domain, generator) cell, so that the reliability results can be compared directly with single-feature separability?
I thank the authors for their detailed response and supplementary results. However, I remain concerned about the presentation. The writing is still unnecessarily obscure and the manuscript contains defensive claims far beyond standard writing conventions. For instance, I performed a full-text search and located 208 strings of "not" in the revision provided by the authors. I note that the authors stated they used AI for writing assistance, and I have no objection to this practice. However, I recommend that the authors revise the AI-generated drafts more thoroughly to improve readability. Due to this reason, I will retain my score, but I still appreciate the authors' substantial effort made during the rebuttal.