- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces OpAI-Bench, a benchmark for AI-text detection under progressive human-to-AI revision. Starting from 15,722 human-written documents across four domains (student essays, news, government reports, scientific abstracts), the authors build nine-version trajectories (v0 to v8) in which a fixed, document-seeded sentence ordering determines which sentences are rewritten. Sentence-level AI coverage rises from 0% to 100% under a fixed operation schedule (polish, paraphrase, style, compress, expand, style, paraphrase, polish), and each version is produced by editing the previous one, so earlier-selected sentences are re-edited at every later step. Provenance labels are released at the document, sentence, and token/span levels, with token labels projected from rewritten sentence regions. The authors evaluate 8 document-level detectors, 7 sentence-level detectors, and 2 token/span-level localizers, in zero-shot and fine-tuned settings, with Qwen3-8B held out as an unseen generator. The main empirical claim is that detectability is non-monotonic in AI coverage, with a pronounced dip around v4 where compression is applied at 50% coverage. Three ablations are offered to separate the contributions of coverage, operation, and revision history, and these are repeated with a second editing model.
Critical strengths
- The released data is a useful resource for the community. Each source document comes with all intermediate versions, stable sentence identifiers, and aligned document-, sentence-, and token-level provenance, across four domains and roughly 280K samples. Even though the current paper makes limited use of the trajectory structure, other researchers can build version-aware tasks on top of it, such as change detection between consecutive versions or estimating the extent of AI editing.
- The evaluation is broad and carefully controlled at the data level. The paper covers detectors from several families at three granularities, in both zero-shot and fine-tuned settings, with a held-out generator and splits made at the source-document level to prevent leakage across versions. The fixed-coverage ablation also yields a concrete and actionable finding: at matched coverage, compression is consistently harder to detect than paraphrasing or expansion. This gives detector developers a specific failure mode to target.
Critical weaknesses
- Limited novelty. Controlled AI involvement (APT-Eval, EditLens), word- and sentence-level attribution (HACo-Det, DAMASHA), and operation taxonomies (CUDRT, which covers closely related create, update, delete, and rewrite operations) are all established. The cumulative trajectory is therefore the only distinctive element, yet the paper tests its value only through a single ablation, which in fact shows a similar v4 drop under independent editing. No task makes use of the trajectory itself, such as detecting changes between consecutive versions or identifying the version at which AI involvement begins. The token- and span-level labels are also projected from whole rewritten sentences, so they add little beyond sentence-level provenance.
- Insufficiently justified core design. The entire benchmark rests on a fixed nine-version schedule, but the paper offers only a brief rationale for it. The number of versions, the order of operations, and the position of each operation in the schedule are never ablated. This matters because the headline “critical v4 region” coincides with the only compression step. The paper’s own fixed-coverage ablation suggests that this is an operation effect, possibly confounded by the shorter length of compressed sentences, and not a property of intermediate coverage. The claim that intermediate versions are “harder than both endpoints” also requires a prevalence-independent metric. AUROC is reported only at the document level with v1 to v8 pooled, and F1-AI is undefined at v0, so per-version AUROC or balanced accuracy is needed to support this comparison.
Questions for authors
- Could you clarify how OpAI-Bench differs from prior work on controlled AI involvement and edit operations, such as CUDRT, APT-Eval, and other similar benchmarks, which already cover closely related edit operations and show that detection does not scale consistently with the degree of AI editing? It would also be helpful to know what the cumulative trajectory enables beyond per-version evaluation, given that the independent-editing ablation shows a similar drop at v4. If possible, a task that directly uses the trajectory, such as detecting changes between consecutive versions or identifying the version at which AI involvement begins, would make this contribution much clearer.
- Since compression appears only at v4, would it be possible to run a small ablation that changes the order or position of operations in the schedule, for example placing compression at a lower or higher coverage level?
- The current token- and span-level labels are projected from whole rewritten sentences, so a lightly polished sentence and a sentence rewritten several times receive the same label. Have you considered graded labels based on edit magnitude or the number of rewrites, and how would the fine-grained localization results change under such labels? Similarly, would it be feasible to include a subset with real human edits interleaved with AI edits, so that the benchmark more closely reflects realistic human–AI co-editing?
If the authors can address the weaknesses and questions above, I would be happy to raise my score.
8 comments
Response to Reviewer e5tR [2/7]
Table R1. Cumulative detection rates (%) across revisions. Each value is the percentage of trajectories detected at least once by that version, among trajectories not flagged at v0. Results cover four domains and the three primary generators, using 5%-FPR thresholds calibrated per detector, domain, and generator.
| Detector | By v1 | By v2 | By v3 | By v4 | By v5 | By v6 | By v7 | By v8 |
|---|---|---|---|---|---|---|---|---|
| DeskLib | 2.5 | 8.8 | 19.8 | 22.2 | 38.2 | 48.4 | 53.8 | 58.0 |
| GigaCheck | 2.8 | 13.6 | 35.1 | 41.9 | 56.9 | 69.0 | 72.7 | 74.9 |
| E5-Small | 1.4 | 2.9 | 5.6 | 6.8 | 15.6 | 19.6 | 23.5 | 26.3 |
| OOD-LLM-Detect | 3.8 | 8.5 | 17.8 | 21.1 | 28.2 | 32.1 | 35.1 | 37.8 |
| RADAR | 1.9 | 4.2 | 7.1 | 8.3 | 13.2 | 16.1 | 18.5 | 19.9 |
We further analyze score changes between consecutive versions of the same document. As shown in Table R2, all five detectors fall below 0.5 (0.24–0.48) at v3→v4, even though the average share of AI-edited sentences increases from 42% to 51%. This is the only transition where all five detectors fall below 0.5. At every other transition, most detectors score the later version higher. Within the same document, more AI editing therefore does not always lead to a higher detector score.
Table R2. Consecutive-version ordering of detector AI scores. Each value measures how often the later version receives a higher AI score than the preceding version, with ties counted as half. Results cover four domains and four generators.
| Detector | v0→v1 | v1→v2 | v2→v3 | v3→v4 | v4→v5 | v5→v6 | v6→v7 | v7→v8 |
|---|---|---|---|---|---|---|---|---|
| DeskLib | 0.78 | 0.65 | 0.75 | 0.28 | 0.87 | 0.73 | 0.56 | 0.83 |
| GigaCheck | 0.58 | 0.63 | 0.71 | 0.48 | 0.71 | 0.76 | 0.48 | 0.59 |
| E5-Small | 0.73 | 0.55 | 0.66 | 0.26 | 0.80 | 0.58 | 0.56 | 0.74 |
| OOD-LLM-Detect | 0.63 | 0.55 | 0.64 | 0.35 | 0.81 | 0.61 | 0.62 | 0.63 |
| RADAR | 0.63 | 0.51 | 0.63 | 0.24 | 0.74 | 0.44 | 0.46 | 0.54 |
Together with the existing ablation, these analyses demonstrate the value of OpAI-Bench's cumulative trajectories for studying detection delay and score changes between consecutive revisions, beyond evaluating individual document versions.
W1.4: The token- and span-level labels are also projected from whole rewritten sentences, so they add little beyond sentence-level provenance.
We clarify that OpAI-Bench's token- and span-level annotations capture AI editing provenance rather than the exact words or characters changed during rewriting (Appendix B.3). This lets us evaluate whether detectors can identify AI-edited text, including small revisions that leave much of the original human writing unchanged, which prior work has shown to be difficult to detect (Saha and Feizi, 2025). Although these annotations are derived from sentence-level editing decisions, they support evaluation at the output granularity of token- and span-level detectors, without first aggregating their predictions to sentences, allowing us to examine how accurately these models localize AI-edited regions within documents.
To examine whether edit intensity changes these conclusions, we additionally evaluated token- and span-level detectors with graded labels based on edit magnitude and rewrite count (Q3.1 and Appendix E.8). Recall increases with rewrite count and generally with edit magnitude, revealing differences in localization difficulty among regions with the same binary label, while the trends across versions, including the drop at v4, remain consistent with the binary evaluation.
W2.1:Insufficiently justified core design. The entire benchmark rests on a fixed nine-version schedule, but the paper offers only a brief rationale for it.
Thanks for raising this concern. We clarify that the benchmark is designed to model progressive AI-assisted writing, where different parts of a document are revised over multiple rounds and previously edited content can be revisited. As described in Sections 3.1–3.2, our design combines gradually increasing AI coverage, diverse editing operations, and cumulative revisions, with each version building on the preceding one. Each intermediate version is a valid stopping point, so the benchmark evaluates detection at different levels of AI involvement without assuming that real workflows reach full AI editing, and the fixed schedule keeps comparisons consistent across documents, domains, and generators.
Each component of the schedule is also tested through controlled ablations of AI coverage, editing operations, and cumulative versus independent editing (Section 5.2 and Appendix F). We have further added an ablation on the position of compression (W2.2; Table 6 and Appendix F.5) and per-version AUROC (W2.4; Table 4 and Appendix E.7).
Response to Reviewer e5tR [3/7]
W2.2: The number of versions, the order of operations, and the position of each operation in the schedule are never ablated. This matters because the headline “critical v4 region” coincides with the only compression step.
As described in Section 3.1, nine versions balance coverage resolution with construction cost, with one human source (v0), seven intermediate mixed-authorship checkpoints (v1–v7), and one fully AI-edited endpoint (v8). Regarding the order of operations, the five operations represent different revision intents. Polishing, paraphrasing, and style rewriting are repeated at different coverage levels, while compression and expansion are placed in the intermediate region as contrasting deletion and addition operations (Section 3.1).
Following the reviewer's suggestion in Q2, we have added a compression-position ablation to test whether the drop at v4 moves with the compression step (Section 5.2, Table 6, and Appendix F.5). Compression is moved to v2 (Early) or v6 (Late) by swapping it with the operation originally applied at that version, while the coverage schedule, sentence selection, and cumulative construction remain unchanged. At every matched version, GPT-5.4 and DeskLib perform worst under the schedule that applies compression at that version (Table R5). For example, moving compression to v6 reduces the F1-AI of GPT-5.4 from 88.3% at v5 to 45.3% at v6, compared with 77.8% at v6 under the original schedule. The degradation therefore moves with the compression step, confirming that compression contributes to the drop at v4 in the main schedule. At the same time, performance also varies at versions without compression (Q2), consistent with our main finding that detectability depends on editing operations and cumulative revision history as well as AI coverage.
W2.3: The paper’s own fixed-coverage ablation suggests that this is an operation effect, possibly confounded by the shorter length of compressed sentences, and not a property of intermediate coverage.
Thanks for raising this point. Our results indicate that both factors play a role, in different ways. AI coverage determines the overall level of detectability, which remains low across the early and intermediate versions (e.g., per-version AUROC of 0.51–0.69 at v1–v3 for the five detectors with signal; Table 4). The specific drop at v4, however, follows the compression operation: when compression is moved to v2 or v6, the degradation appears at that version instead (W2.2; Table 6 and Appendix F.5). Accordingly, the revised paper describes the degradation around v4 as associated with compression at intermediate coverage rather than with coverage alone (Section 5). Regarding the possible length confound, the operation effect remains after controlling for length:
- Sentence length. In Ablation 1, comparing compressed sentences with paraphrased sentences of exactly the same word count (matches found for 99.8% of compressed sentences), the three LLM-based sentence detectors still detect compressed sentences 22–36 pp less often.
- Matched length and coverage. We compared compression and paraphrasing on the same source documents, retaining only pairs within ±10% document length and ±5 pp AI-word coverage (Gemini-2.5-Flash, 62 matched pairs; detection rate at 5% FPR). Compression remains harder to detect for DeskLib (−10.48 pp), GigaCheck (−13.98 pp), and OOD-LLM-Detect (−8.06 pp), with 95% confidence intervals excluding zero.
- Fixed coverage. At the same target sentence coverage, compression is harder to detect than paraphrasing or expansion (Appendix F.2).
- Edit magnitude. From v3 to v4, AI coverage increases from 40% to 50%, while cumulative normalized Levenshtein distance remains nearly unchanged (0.414 vs. 0.416) and sentence-level recall decreases from 0.248 to 0.220. The stepwise edit distance at v4 (0.308) is substantially larger than at the polishing stages (0.112 at v1 and 0.045 at v8), so the drop occurs despite substantial textual revision (Table 12, Appendix E.4).
W2.4: The claim that intermediate versions are “harder than both endpoints” also requires a prevalence-independent metric. AUROC is reported only at the document level with v1 to v8 pooled, and F1-AI is undefined at v0, so per-version AUROC or balanced accuracy is needed to support this comparison.
To address the reviewer's concern, we have added per-version document-level AUROC to the main text (Table 4 in Section 5; full results in Appendix E.7, Table 17). Each version is compared with the human v0 documents, and AUROC is averaged over the four domains and the three primary generators.
Response to Reviewer e5tR [1/7]
We thank the reviewer for the constructive and detailed comments, and for recognizing the value of OpAI-Bench as a research resource and its comprehensive evaluation. We have uploaded a revised version of our paper to accommodate all your suggestions. Below, we further provide detailed clarifications for each of your questions, including new experiments and analyses.
Thanks for this comment. We clarify that, while controlled AI involvement, fine-grained attribution, and edit-operation taxonomies have been studied in prior work, OpAI-Bench tracks AI-text detection across cumulative revision trajectories of the same human-written document. Each source document has nine aligned versions (v0–v8), where each version builds on the previous one and earlier AI-edited sentences can be rewritten again, with the editing operation, AI coverage, and provenance known at every step. APT-Eval, EditLens, HACo-Det, DAMASHA, and CUDRT address important aspects of AI editing and attribution, but none of them provides successive versions of the same document as part of the benchmark (Table 1). APT-Eval generates each polished variant independently from the original human text, HACo-Det revises a document over multiple turns but evaluates only the final coauthored text, EditLens focuses on documents edited in a single pass, and CUDRT constructs a separate dataset for each operation. We have added CUDRT to the Related Work (Section 2), and we show below how the trajectories are used in our analyses (W1.2 and W1.3).
Thanks for the comment. We would like to clarify that the cumulative trajectory is not a separate element of OpAI-Bench, but the setting that links its other components. Unlike the benchmarks compared in Table 1, OpAI-Bench provides successive versions of the same document, each building on the previous one, and combines controlled AI coverage, cumulative editing, diverse edit operations, and multi-granularity annotations. Each version records its AI coverage, editing operation, and provenance, so the effects of these factors on detection can be followed along one revision history rather than measured on isolated edits. The value of this design is examined in three ways: the cumulative-versus-independent ablation, the edit-magnitude analysis, and two new trajectory-level analyses (W1.3).
Cumulative-versus-independent ablation. The independent setting preserves the same version schedule while removing accumulated re-editing (Section 5.2 and Appendix F.3). Since compression is applied at v4 in both settings, a drop around v4 is expected in both. The drop remains under independent editing, with magnitude varying across detectors and generators, which shows that the non-monotonic pattern does not depend on accumulated re-editing alone. Revision history matters elsewhere: previously AI-edited sentences are detected more often than newly edited ones (Q1.3), and cumulative editing raises sentence-level balanced accuracy for all three LLM detectors (+2.9 to +5.1 pp, source-paired and adjusted for length and AI-word coverage, 95% CIs excluding zero). The trajectory-level analyses in W1.3 further show the v4 drop within individual documents.
Edit-magnitude analysis. Each version is compared with both its preceding version and the original human source (Section 5.1 and Appendix E.4). The final version (v8) has the smallest stepwise change but the highest sentence-level recall, indicating that detectability depends on accumulated revision history and not only on the change introduced at a single step.
Following the reviewer's suggestions, we have added two analyses that use the trajectory itself (Section 5.2 and Appendix E.6).
For detection delay, we track when each document is first flagged as AI-edited along its revision trajectory. Although AI editing begins at v1, Table R1 shows that only 1.4–3.8% of trajectories are flagged at this version. Detection increases across subsequent revisions, but even by v8, only 19.9–74.9% have been flagged at least once, depending on the detector. For example, DeskLib detects 2.5% by v1, 22.2% by v4, and 58.0% by v8. This analysis shows how long AI involvement can remain undetected across successive revisions of the same document, rather than only measuring detection performance at individual stages.