- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper attempts to attack the AI edit score of a detector, the first adversarial robustness technique aimed at suppressing AI edit scores, rather than binary AI detector probabilities. The authors identify regions characteristic of AI-edited writing using pairs of human and AI-edited text, and use that to identify rewriting regions in unseen text at inference time. They accomplish their goal
Critical strengths
The strengths of this paper lie in the metrics used. They have a robust suite of semantic, detail, and grammatical preservation metrics. Additionally, the results that Editlens score while not moving embedding distance closer to human-written text are compelling.
Critical weaknesses
There is fairly low novelty of approach. The difference between this approach and approaches designed to reduce the binary score of fully AI-generated documents is not well-established. Additionally, the conclusion that AI editing scores may be unreliable is not well-founded or discussed prior.
Questions for authors
- Do other adversarial attacks designed for binary AI detection methods accomplish similar goals?
- Please expand on why adversarial attacks not moving texts closer to human-written texts questions the reliability of edit scores.
2 comments
I think a necessary part of this work is establishing if and how binary AI detection adversarial attacks fail at the task of attacking AI editing scores. This concern is not fully addressed by benchmarking a single adversarial attack (StealthRL), so I will maintain my score.
PRESS reduces all four editing-extent scores, while StealthRL increases three of them. PRESS also modifies only 14.7% of the words on average, compared with 85.5% for StealthRL. These results show that attacks designed for binary detection do not necessarily work well for editing-extent estimation, motivating the task-specific design of PRESS.