acceptodds
Under review as a conference paper at ICLR 2027

WebMix-Collapse: A Rewrite Benchmark for Sizing the Length Confound in Short-Form Synthetic-Text Detection

Abstract

On rewrite-based benchmarks for synthetic-text detection, a detector can score well by counting words. We build such a benchmark, WebMix-Collapse, with 10,668 English web documents and model rewrites spanning six contamination rates, ten generator families (1B to 671B parameters) and five rewrite depths, and we measure how much of a statistical detector's accuracy on it is document length. Word count alone reaches AUROC 0.942. Our 19-feature ensemble reaches 0.987, so a single feature recovers 91% of the ensemble's signal above chance. The strongest test uses a split in the release where human documents are the shorter class. There the ensemble does not drift toward chance. It inverts, to 0.289, and matching on length inside the split brings it back to 0.853. Fast-DetectGPT and Binoculars at GPT-2 scale, scored on the same split, do not invert (0.969 and 0.977). The same artifact explains two results that would otherwise pass as findings: accuracy that rises with rewrite depth, where the ensemble beats word count by at most 0.0026, and an apparent DeepSeek-R1 evasion (0.562) that disappears once reasoning traces are stripped (0.994). Out of domain the ensemble keeps most of its ranking (0.855), yet the threshold that gives 1% false positives in domain flags 85% of held-out human documents. A fine-tuned RoBERTa (0.947) and Binoculars at Falcon-7B scale (0.849) hold up out of domain, so the problem is the way rewrite benchmarks are evaluated, not statistical detection itself. We package the controls as the Confound Attribution Protocol (CAP), run it on fifteen detector configurations, and report the two steps that do not work. Our recommendation is simple: report every rewrite-based detection result next to a length-only baseline on the same rows.

Primary area:
datasets and benchmarks
Keywords:
synthetic text detection, model collapse, benchmark, dataset, spurious correlation, document length confound, out-of-distribution generalization, evaluation methodology, machine-generated text, AI-generated text detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zRYSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces WebMix-Collapse, which is basically a benchmark for synthetic-text detection. Documents are rewritten by ten LLM families under one common prompt with various contamination levels, recursive rewrites and more.

Critical strengths
  • The paper is well-written and has a good motivation to begin with. Most experiments are rigorous and can be understood very easily.
  • I do appreciate the authors detailing their negative results and limitations as well.
Critical weaknesses
  • Most of the studies on the paper, especially about the 2.18x2.18x gap, comes from the single prompt used in the paper. This cannot be made generalized for rewrite-based benchmarks.
  • It seems to me that the only detector which seems to fail is the proposed detector. Every other detector seems to work reasonably well, and none of them are length-matched in domain.
  • There also seem to be some minor contradictions in the paper. I'm a bit confused on what Step 5 exactly means in Appendix C and Section 5.6.
  • Could the authors rigorously test the methods by truncating documents or targeting rewrite length? This would be more robust in comparison to using a split.
Questions for authors

None, most questions are listed in the Weaknesses.

Reviewer_5birOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces WebMix-Collapse, a benchmark of over 10,000 English web documents and model rewrites covering six contamination rates, ten generators, and five rewrite depths. The paper's central question is how much of a statistical detector's accuracy depends on text length. The authors find that much of an ensemble classifier's performance relies on text length and propose a Confound Attribution Protocol to determine whether benchmark performance reflects genuine detection capabilities or dataset artifacts.

Critical strengths
  • The paper addresses the important and well-known issue of length confounding in detection benchmarks.
  • The corpus is extensive and could be useful for studying length confounds.
Critical weaknesses
  • The paper is written very densely, and the abstract in particular does not read smoothly.

  • Critically, the introduction paragraphs preceding the contributions section contain only three citations. For an academic paper, especially at a conference such as ICLR, this raises concerns about academic rigor. For example, where are the references for the statistical detectors mentioned in Line 42?

    • Citations are also missing for the Hugging Face datasets, such as AG News.
    • The main results for detectors such as Binoculars are not presented in the main text (Section 5.5).
    • Overall, the paper does not seem to meet the presentation standards expected at ICLR.
  • The main finding relies on an ensemble classifier with features that are strongly dependent on text length. Essentially, this merely demonstrates that a length-sensitive classifier bases its predictions on length.

  • More interesting detectors, such as Binoculars, are in fact less sensitive to length and do not exhibit the same inversion in performance.

  • Line 136: One model accounts for 87% of the generated data. While it is commendable that the authors acknowledge this limitation, why not address it by generating a more balanced dataset? The paper claims to cover ten models, yet the vast majority of the data comes from a single model.

  • The paper is motivated by model collapse, but what it actually investigates is the more general problem of AI-generated text detection. I would therefore suggest framing it as an AI-text detection paper rather than focusing narrowly on model collapse.

Questions for authors

see above

Reviewer_UjQHOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces WebMix-Collapse, a benchmark for studying document-length confounds in AI-generated text detection, and proposes the Confound Attribution Protocol (CAP) to diagnose detectors’ reliance on length.

Critical strengths
  1. The model-collapse perspective is interesting and offers a novel motivation.

  2. The benchmark’s focus on document length is creative, and the introduction of CAP is a valuable contribution.

Critical weaknesses
  1. The abstract and Introduction are difficult to follow. Numerous results and technical terms obscure the main narrative, while some concepts lack introduction and some expressions have unclear referents. I suggest simplifying the abstract and clarifying the connections among the research question, method, and findings.

  2. The main text has too few figures and tables and relies too heavily on the appendix. The paper should be restructured to better balance explanation and results.

  3. The paper needs to strengthen its comparison with prior work and sharpen how its novelty is positioned. Its systematic benchmark comparison is confined to RAID and HART, which is a narrow basis.

  4. The dataset construction section needs a clearer overview: a consolidated table linking each experiment to its datasets, subsets, and sample sizes would make the setup easier to follow.

Questions for authors
  1. Could you please add an overview figure covering dataset construction and CAP?

  2. Why does the ensemble perform well on AG News, yet show the opposite pattern in Sec 5.6? The differences in reversal magnitude, generators, and generation procedures require further discussion.

Comments on the paper

2 comments
Area_Chair_AzprOpenReview ↗

Discussion

Dear Reviewers,

We are reaching the end of the discussion period! If the authors have responded to your review, please read their response, leave a comment, and update your rating if your assessment has changed.

Thank you,

AC.

Dear Reviewers

We have tried our best to provide the details to the points put forward and we sincerely thank all three reviewers for investing time and energy reviewing our work, the paper being dense comes from the fact that we wanted to have every data backing the results we had throughout the experiments and findings. We started with having a lightweight detector for synthetic data, having all features and eventually finding the insights from it. We also had model collapse as a motivation behind why it's utmost important to detect synthetic data.

We deeply acknowledge the fact that all three reviewers went through our work and provided us with the feedback/clarity needed. In the light of the rebuttal responses if at all our responses were able to throw light on our motivation and results we had, we would like to request all, if revisiting the scores for our work can be looked into.

Again, we sincerely thank you all

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.