- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces WebMix-Collapse, which is basically a benchmark for synthetic-text detection. Documents are rewritten by ten LLM families under one common prompt with various contamination levels, recursive rewrites and more.
Critical strengths
- The paper is well-written and has a good motivation to begin with. Most experiments are rigorous and can be understood very easily.
- I do appreciate the authors detailing their negative results and limitations as well.
Critical weaknesses
- Most of the studies on the paper, especially about the gap, comes from the single prompt used in the paper. This cannot be made generalized for rewrite-based benchmarks.
- It seems to me that the only detector which seems to fail is the proposed detector. Every other detector seems to work reasonably well, and none of them are length-matched in domain.
- There also seem to be some minor contradictions in the paper. I'm a bit confused on what Step 5 exactly means in Appendix C and Section 5.6.
- Could the authors rigorously test the methods by truncating documents or targeting rewrite length? This would be more robust in comparison to using a split.
Questions for authors
None, most questions are listed in the Weaknesses.
Discussion
Dear Reviewers,
We are reaching the end of the discussion period! If the authors have responded to your review, please read their response, leave a comment, and update your rating if your assessment has changed.
Thank you,
AC.