acceptodds
Under review as a conference paper at ICLR 2027

Fractal Signature Is a Property of the Generator, Not the Document: Split-Half Reliability Bounds for Scaling-Exponent AI-Text Detectors

Abstract

Scaling exponents that summarise the fractal structure of language, the Hurst and H\"older exponents, separate human from machine-generated text at the corpus level, and a growing literature uses them as per-document detector features. Each exponent is defined by an asymptotic limit, so a per-document value is an estimate computed from roughly tokens, yet no reliability coefficient has been reported for such an estimate at document length. We decompose the per-document estimate into a population term, a document term, and estimator error, and measure the ratio of the latter two with a pre-registered interleaved split-half check on GAGLE documents scored under Gemma-2B and Mistral-7B. The decomposition yields four falsifiable predictions, which we test with a fixed gradient-boosted detector, a within-population classification task, an edit battery that separates token-order from content perturbations, and zero-shot transfer to RAID. At the population level, all cells (domain, generator, exponent) separate with non-overlapping intervals. At the document level, the H\"older estimate reaches and under the two scorers, the Hurst estimate at most , and none of the twenty candidate quantities clears the bar. All four predictions hold. Deleting the fractal features from an -feature detector leaves AUROC unchanged to four decimals; the family receives of split gain but of permutation importance; within machine text, where population factors are balanced by design, it reaches macro-F1 against for all features (chance ); and its AUROC gain on RAID is recovered () by replacing each document's fractal features with their group (generator, domain) mean. Under token-order edits, deleting the family raises recall by and points. Independently, a curvature detector's apparent robustness to paraphrase is an artifact of scorer–attacker lineage: its recall of – falls to – under attackers outside that lineage. The fractal signature is a property of the generating process, not of the document. We propose an admissibility condition: a scaling statistic should be used as a document-level representation only once its split-half reliability at the deployment length clears a declared bar, a check that costs one split and one correlation and does not depend on the downstream model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fractal complexity, measurement reliability, split-half reliability, scaling exponents, feature importance, adversarial robustness, LLM detection benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_c2svOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1(−1)
Soundness
2
Read review
Paper summary

This paper questions the use of fractal scaling exponents as features for detecting AI-generated text at the level of individual short documents. It distinguishes population-level differences between human and machine text from the reliability with which such exponents can be measured in a single document. The results suggest that the former does not imply the latter, as single-document estimates are dominated by estimation noise at practical lengths. The paper therefore argues that the apparent value of fractal features mainly reflects generator- or population-level information, and proposes reliability testing as a prerequisite for their use.

Critical strengths
  1. The paper identifies an important and often overlooked inferential gap in this literature. A distributional difference between classes is frequently treated as evidence that a statistic can characterize individual documents, but these are fundamentally different claims. By reformulating detector performance as a measurement-validity problem, the paper provides a useful perspective for evaluating document representations more generally.

  2. The empirical study is broad and generally careful. The authors examine their central measurement from several angles and include controls against plausible artifacts. They also report results that complicate their own narrative rather than presenting only supportive evidence. This transparency strengthens the credibility of the work.

Critical weaknesses
  1. The central conclusion is broader than the evidence currently supports. The experiments convincingly show that the examined estimators are unreliable for short documents, but they do not establish that fractal information is generally not a property of documents. Some alternative estimators approach or satisfy the authors’ own reliability criterion, yet their downstream value is not tested. The claims should therefore be narrowed to the estimators, document lengths, and settings actually evaluated, or supported with additional experiments.

  2. The mechanistic interpretation is not yet fully established. The decomposition compares Var(η)\mathrm{Var}(\eta) with Var(δ)\mathrm{Var}(\delta), but classification separability depends on Δμ\Delta\mu relative to the total per-document dispersion Var(δ)+Var(η)\mathrm{Var}(\delta)+\mathrm{Var}(\eta); this standardized gap is never reported for any individual exponent or (domain, generator) cell, and non-overlapping pooled intervals only reflect mean precision at large NN. Some analyses also do not fully separate document-specific information from finer-grained population factors, e.g. prompt type is left in δ\delta rather than in μ\mu. The group-mean results admit other explanations: replacement also removes estimation noise, while subtraction retains 59% of the transfer gain. The evidence thus does not uniquely show that the benefit comes only from population means rather than document residuals.

  3. The writing is verbose and repetitive, which weakens an otherwise interesting argument. The same decomposition, headline numbers, and thermometer analogy are restated across the abstract, introduction, methods, figure, discussion, and conclusion. The figurative labels for the four predictions are also reused long after their mapping to Eq. 2 has been established, without adding a definition, prediction, or evidence, so their purpose beyond illustration is unclear. The paper would benefit from substantial compression, consistent terminology, and more direct exposition.

Questions for authors
  1. DFA and local Whittle approach or clear the stated reliability bar for H^\hat H. Do they also show no downstream value when used as detector features, or should the central claim be narrowed to the R/SR/S and Hölder estimators actually used in the detector?

  2. Could you report the standardized population gap, Δμ\Delta\mu relative to Var(δ)+Var(η)\mathrm{Var}(\delta)+\mathrm{Var}(\eta), for each exponent in each (domain, generator) cell, so that the reliability results can be compared directly with single-feature separability?

1 comment

I thank the authors for their detailed response and supplementary results. However, I remain concerned about the presentation. The writing is still unnecessarily obscure and the manuscript contains defensive claims far beyond standard writing conventions. For instance, I performed a full-text search and located 208 strings of "not" in the revision provided by the authors. I note that the authors stated they used AI for writing assistance, and I have no objection to this practice. However, I recommend that the authors revise the AI-generated drafts more thoroughly to improve readability. Due to this reason, I will retain my score, but I still appreciate the authors' substantial effort made during the rebuttal.

Reviewer_2S7fOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This article considers 'fractal signature' in the detection of machine generated text, by which the authors mean Hurst exponent and other fractal scaling measures of the sequence of next token probabilities. This has previously been shown to be (moderately) useful in the detection of machine generated text.

The main question considered in the article is whether there exists a reliably detectable document level fractal signature, distinct from the signature of the generation process. That is, given a language model, we could ask whether a particular document generated by that language model has a different fractal signature to typical documents generated by the language model. The authors show that, broadly, differences in fractal signature of a document from the signature of its generation class do not persist in subparts of the document, and conclude that measurement noise due to the low numbers of tokens dwarf anything fundamental one might want to say about the particular document.

Other parts of the document deal with the question of whether fractal signature is useful in distinguishing AI and machine text, the results here are mixed but broadly say that the signal is much less strong than other measures, but still useful.

Critical strengths

The experiments were interesting, and are different to the typical analysis we see of machine generated text detectors. It is good that new techniques are introduced to the community.

Critical weaknesses

I am very experienced in reading both maths and ML papers, but I found it incredibly difficult to understand this paper. After my first full read through of the article, I interpreted the title and main message as saying that differences in fractal signature of AI and human text exist but cannot be detected at the document level, this is not what the article says! Rather, they should be interpreted as saying that document specific deviation of fractal signature from the typical fractal signature of text generated the same way is mainly due to measurement noise rather than anything more fundamental. I did not find the analogy with temperature measurement to be particularly useful.

There are several questions that I am interested, do different human authors write text with different fractal exponents, do language models write with different fractal exponents in different domains. But as to whether there is interesting document level differences in the fractal signature of documents written by the same language model, I would like to see more done to motivate the question. A priori, my guess would be no, and I would like the author to explain more why there is an interesting question here, perhaps by speculating as to why one might hope to find these.

I found experiment P1 rather weak, because the AUROCs are so close to 1 that we are relying on differences in very few documents.

Questions for authors

I do not have any specific questions whose answers would materially change my score.

Reviewer_NRUGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper performs a study of how good two particular statistics are for discriminating between human and machine-generated text when those statistics are estimated over the document level rather than over the population level. They propose and us a "split-half check" over a large set of documents, with the quantities used in the estimation of the two statistics estimated from different language models (and under different settings in the appendix). The authors find that the document level estimates are not reliable AI detectors, concluding that "fractal signature is a property of the generating process, not of the document". They recommend performing this split half check before relying on these estimates for document-level AI detection

Critical strengths
  • The research question is untested and relevant: it seems that the community is using these individual document level statistics even though the validity of estimating the statistics at this level (rather than at the population level) hasnt been tested. The work tries to answer that question through careful tests. It provides a relatively large set of empirical results and a good initial framework for doing this type of analysis
  • Several parts of the presentation are well chosen/exected: The decomposition of the per-document estimate into a population mean, a document term, and estimator error is nice: it makes it easier to see where errors/noise might be coming from. The first few paragraphs of the introduction set up the work nicely: motivating the problem, giving intuitive metaphors and explaining the issue with current practices
Critical weaknesses
  • My main concerns are about validity/justification some of the protocol, and therefore about some of the conclusions drawn from them. These protocol justifications seem to be entirely absent from the current version: why is this split-half reliability check a valid test to run/draw conclusions from? From what I understand, the Spearman--Brown step assumes the two halves are parallel forms of the full-length measurement, which is not the case here given the interleaved nature. What other assumptions are required for these tests to be valid and why does this setting satisfy those assumptions? Aside from this, even accepting the protocol, some of the conclusions seem to strong as the alternative measurements in the appendix reverse the conclusion. The general claim about the scaling statistics is based on the test not passing this 0.6 threshold, but under the measurement settings in the appendix (Appendix F.5), it does. And even under the settings in which the test fails, the observed value is quite close to the threshold. This makes it feel like the conclusions are quite sensitive to seemingly small estimator design choices. Perhaps this is a finding of its own, but that should be stated rather than just a global claim about adequecy of scaling statistics for this task. Similarly, the prediction tests (P1-4) are stated, but I believe the conclusions that can be drawn from them are weaker than what's presented. While the tests can show that the fractal features add almost nothing to detection given the other features, that doesn't mean they don't contain predictive document-level information and can't be used for the prediction (just that the info is redundant with other features).
  • There are several presentation issues: Parts of section 3 is dense and filled with jargon (particularly 3.1 and the latter halves of 3.2 and 3.3). The formal definitions of the concepts used are, of course important, but no intuition about what these concepts mean (e.g., the integrated trajectory) is provided, which makes it incredibly difficult for a reader to understand what it is that the statistics under study are trying to measure and why we might a priori expect them to not work when measured at the document (rather than population) level. The abstract is not accessible. It talks about niche concepts without background (e.g., the half-split reliability) and doesn't give enough context for other concepts (e.g., a fixed gradient-boosted detector, and the non-overlapping CIs of 20 cells). The contributions box has a similar feel: the text very much has an AI-generated flavor. It's just really not understandable
Questions for authors

How do you reconcile the paper's general claims about document-level H and S estimates with the results in Appendix F, where the estimators under these alternative measurement settings pass your test? Sure, the methods may measure somewhat different quantities, but why are those estimators less valid than the ones at the focal point of the paper? To me, this suggests more that conclusions depend on the estimator. Not that we can make global claims about all such estimators

What we report is also not the foreseeable imprecision of a power-law fit over 1.6 decades, which would degrade both exponents alike; instead, the two diverge by nearly an order of magnitude in how well each reproduces itself on the same document, 0.576 against 0.063 (primary sources, Appendix C).

I don't understand what you mean here.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.