acceptodds
Under review as a conference paper at ICLR 2027

MixDetect: Word-Level Localization and Quantification of AI Editing

Abstract

Large language models are increasingly used to edit human-written text rather than generate entire texts from scratch. Conventional AI-text detectors mainly distinguish human-written from fully AI-generated text, while recent methods for AI-edited text typically provide only a text-level label or editing-degree score. We introduce MixDetect, a word-level framework for localizing and quantifying AI editing. MixDetect separately predicts whether each word has been edited and, conditional on editing, how substantial the edit is, allowing editing scope and editing intensity to be estimated separately. During training, source–edited pairs are aligned to construct word-level supervision, while inference requires only the input text. Experiments show that MixDetect accurately localizes AI-edited words, reflects differences in editing intensity, and reveals different scope–intensity patterns across editing degrees and operations. The overall AI editing magnitude increases under additional AI editing, decreases when AI-generated text is edited by humans, and remains nearly unchanged under ordinary human-to-human editing. The aggregated text-level predictions also perform well on binary and ternary AI-text classification and remain effective under domain and generator shifts. These results show that AI editing can be analyzed beyond a single authorship label or editing-degree score by identifying both where AI editing occurs and how substantial the edits are.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Text Detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_beXROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MixDetect, an approach for identifying and quantifying AI-edited or AI-generated text. It not only estimates the proportion of text affected by AI editing, but also measures how substantial the edits are. The method uses two separate prediction objectives to model editing occurrence and editing intensity, allowing it to capture both the scope and strength of AI editing.

Critical strengths

A key contribution of this work is that it explicitly models the magnitude of AI editing rather than only determining whether a text contains AI-generated content. By separating editing scope from editing intensity, the proposed formulation provides a more fine-grained characterization of human–AI collaborative writing and may facilitate further studies on the differences between human-written, AI-edited, and AI-generated text.

Critical weaknesses

The main concern is that the proposed editing intensity is supervised using an automatically constructed proxy rather than human-annotated edit magnitude. For substituted words, the target is defined by contextual dissimilarity between source and edited words, and inserted words are assigned maximum intensity. As a result, the current evaluation primarily demonstrates consistency with this proxy, but does not establish that the predicted intensity corresponds to human-perceived editing severity. Independent human annotation or external validation would make this claim more convincing. In addition, the related-work coverage appears incomplete. Several recent works on fine-grained human–AI authorship localization, including word-level and sentence-level detection in collaborative text, are not discussed. Since fine-grained localization itself has prior precedent, the paper should more carefully position its novelty around the explicit decomposition of editing occurrence and editing intensity.

Questions for authors

Why are ρQ\rho_Q and ρM\rho_M in Table~1 evaluated using partial Spearman correlation rather than standard Spearman correlation?

1 comment

W1: Editing intensity lacks independent human validation.

We conducted an independent evaluation with three annotators on 120 word-level editing examples, yielding 360 ratings. The reference intensity agreed with human ratings in 296 cases (82.22%), and MixDetect's predictions agreed in 273 cases (75.83%); inter-annotator agreement was 0.64. MixDetect also showed strong correlations with external editing-degree and editing-proportion measures on EditLens and APT-Eval. On LAMP, which provides human-annotated editing regions, its regional localization AUROC was 0.769. These results support both the word-level supervision and the predicted editing intensities.

W2: Related work and novelty positioning need clarification.

We will expand the related work to cover HACo-Det (ACL 2025) and SenDetEX (EMNLP 2025). The former studies fine-grained authorship attribution in human–AI co-written text; the latter identifies AI-generated sentences in mixed text. EditLens estimates AI editing degree at the document level. MixDetect instead models word-level edit occurrence and conditional editing intensity in AI-edited text, locating edited words while separately quantifying editing scope Q, conditional intensity M, and overall magnitude S. Its central contribution is the scope–intensity decomposition, rather than word-level authorship attribution alone.

Q1: The use of partial Spearman needs justification.

Partial Spearman measures the predictive quality of Q and M separately: it controls for the reference M when evaluating Q, and for the reference Q when evaluating M. We also computed standard Spearman correlations. Under Edit-Only, standard/partial correlations were 0.886/0.878 for Q and 0.658/0.640 for M; under Full, they were 0.879/0.873 for Q and 0.671/0.617 for M. Both metrics show that MixDetect effectively predicts editing scope and conditional intensity.

Reviewer_tgRcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the localization and quantification of AI editing at the word level. It proposes MixDetect, which separately predicts whether each word has been edited and the conditional intensity of that edit. These predictions are aggregated into text-level editing scope, intensity, and overall magnitude. Training supervision is constructed by aligning source-edited text pairs, while inference requires only the input text. The evaluation examines word-level localization, editing intensity, different editing operations and directions, and document-level classification under domain and generator shifts. The central contribution is a decomposition intended to characterize both where AI editing occurs and how substantial it is.

Critical strengths
  1. Meaningful problem formulation. Separating editing scope from conditional editing intensity offers a potentially useful view of AI-assisted writing beyond document-level authorship classification. The detector also requires only the input text at inference time (Sections 3.1-3.3, pp. 4-5), making the proposed analysis practically applicable.

  2. Useful empirical controls. The LAMP evaluation uses human-annotated edit regions and reports a regional AUROC of 0.769 under Edit-Only, while ArgRewrite provides a human-to-human editing control (Section 4.3, p. 8). Together with the domain/generator-shift classification results in Table 3 (p. 9), these experiments provide meaningful evidence that the model captures more than arbitrary textual change.

Critical weaknesses
  1. W1: The validity and evaluation of the word-level editing measurements remain insufficiently established. Equation (4) defines substitution intensity through embedding dissimilarity, assigns maximum intensity to insertions, and excludes deletions from word-level labels (p. 4; Algorithm 2, p. 14). Table 1 and Figure 3 evaluate predictions against these alignment-derived targets (p. 6), supporting prediction of the proxy but not independently validating its interpretation as substantive editing magnitude. In particular, phrase substitutions, reordering, and meaning-changing deletions may not be adequately represented by this construction. The reported substitution-level correlation of 0.384 supports rank agreement with the proxy; whether that ranking corresponds to independently judged editing magnitude remains unresolved. The human-editing controls and correlations with requested editing degree and realized edit distance provide useful complementary evidence (pp. 6-8), but do not directly validate individual AI-edit intensities. The concern is whether the scores support their intended interpretation, rather than a requirement for absolute numerical calibration. Furthermore, Table 1 and Appendix C.4 do not explicitly identify the contributing text categories or the treatment of no-positive texts in APtext (pp. 6 and 16). If homogeneous texts are included, pooled scores may partly reflect between-document discrimination; this is a protocol question, not evidence that the reported scores are inflated.

  2. W2: The interpretation and additional value of the scope-intensity decomposition need stronger support. Figure 6 reports Q=0.005 and M=0.669 for a human-written example and explicitly states that M is not interpreted when Q is near zero (p. 15). Nevertheless, Figure 5 interprets changes in M relative to human-written sources as different editing-intensity patterns (p. 7). Predicted M remains numerically computable when predicted Q is positive, but a difference from a reference value that the paper treats as uninterpretable need not measure a meaningful change in conditional intensity. More broadly, the claimed distinction between texts with similar overall magnitude but different scope-intensity profiles needs validation independent of the same alignment proxy. This matters particularly because Direct-s has higher reported Edit-Only word AUROC and text-level correlation in Table 4 (0.945/0.877 versus 0.885/0.861), while its ternary classification results are omitted (p. 9). Superior scalar accuracy is not required, but the separate outputs need a demonstrated interpretive benefit to justify the trade-off. These concerns affect the decomposition's central claim without invalidating the separate findings about S.

Questions for authors
  1. What evidence independent of the alignment procedure supports interpreting the intensity targets as substantive editing magnitude? Could a small, independently assessed subset of existing source-edited pairs establish whether the target and predicted intensity rankings agree with judgments of how substantial the edits are, thereby supporting the intended interpretive use of these scores (Equation (4), p. 4; Figure 3, p. 6)?

  2. How should the labels and overall score be interpreted for phrase substitutions, reordering, and meaning-changing deletions? Please provide representative existing examples showing what the construction captures or misses, including the rationale for assigning every insertion maximum intensity (Section 3.1, p. 4; Algorithm 2, p. 14).

  3. Which text categories contribute to Table 1, and how are texts without positive edit labels handled in APtext? What localization performance is obtained on texts containing both edited and unchanged words, where within-document localization can be assessed directly (p. 6; Appendix C.4, p. 16)?

  4. How is the paired-change analysis in Figure 5 (p. 7) reconciled with the near-zero-Q caveat in Figure 6 (p. 15)? How should predicted M and its paired changes be interpreted when the human source has very little predicted editing, and do the operation-level conclusions remain supported after appropriately handling such reference values?

  5. Can you provide independently assessed examples from the existing evaluation data with similar S but different Q and M, showing that the predicted profiles correspond to meaningful differences in editing behavior rather than only differences in the alignment-derived targets (Figure 1, p. 2; Figure 5, p. 7)?

  6. Can you complete the Direct-s ternary results in Table 4 using the same validation-based threshold-selection protocol (p. 9; Appendix C.5, p. 17)? In light of its stronger scalar results, what demonstrated benefit of the separate Q and M outputs supports the proposed formulation?

4 comments

W1 & Q1: Word-level editing intensity lacks independent validation.

Three independent annotators rated 120 word-level editing examples by comparing the source and edited texts and assigning one of three severity levels: minor, moderate, or substantial. Across 360 individual ratings, MixDetect's predicted intensity agreed with the human rating in 273 cases (75.83%), while the automatically constructed reference intensity agreed in 296 cases (82.22%). Inter-annotator agreement was 0.64. On independent evaluations, the Spearman correlation between MixDetect's overall editing score and EditLens editing degree reached 0.913/0.939 (Edit-Only/Full). On APT-Eval, correlations with requested editing degree and proportion were 0.578/0.507, respectively. Together, these results support the validity of the intensity supervision and predictions.

W1 & Q2: Editing labels for complex operations need clarification.

Examples from the existing test set illustrate the alignment labels: (1) Synonymous substitutions: nestles solemnly under → rests quietly beneath produces three SUBSTITUTE labels with intensities 0.108/0.080/0.084. (2) Within-sentence reordering: moving in front of her to the end of a sentence preserves the main clause as KEEP, labels the relocated words as INSERT, and records their original positions as DEL. (3) Sentence relocation: 19 words moved across paragraphs are labeled INSERT at their new location, while 381 other target words remain KEEP. (4) Added details: appending descriptions of the soleplate and power cord after It looks brand new yields 19 INSERT words. (5) Added scene description: introducing sound and smell details into a restaurant review preserves 103 original words as KEEP. (6) Deletion only: removing the 10–18% qualification about greenhouse-gas emissions leaves 306 target words labeled KEEP, with the omitted span recorded as source-side DEL. INSERT receives intensity 1 because the target word has no matched source word; SUBSTITUTE intensity is determined by contextual similarity between the matched words.

W1 & Q3: The word-level evaluation protocol needs clarification.

Table 1 evaluates MixDetect's word-level predictions on 2,031 AI-edited texts: 1,999 contain both edited and unchanged words, and 32 contain no positive edit labels. APtext is computed within each text containing both positive and negative labels, then averaged across texts. We also evaluated MixDetect and Direct-s on the same 1,999 partially edited texts. Under Edit-Only, word AUROC was 0.882 vs. 0.873, and APtext was 0.679 vs. 0.650 (MixDetect vs. Direct-s). MixDetect therefore performs better at locating edited words within the same document.

W2 & Q4: The ΔM comparison is unclear for low-Q sources.

Figure 5 originally compared M before and after editing. We additionally evaluated the conditional intensity M directly on the texts produced by each editing operation. Under Edit-Only, Make it more detailed yielded M = 0.756 and Make it more descriptive M = 0.725, compared with 0.592 for Paraphrase and 0.621 for Simplify. The nine-operation rankings based on edited-text M and the original ΔM were highly consistent (Spearman 0.983–1.000). The operation-specific intensity patterns remain clear, and we will present the comparison directly using M on the edited texts.

W2 & Q5: Distinct Q/M profiles need supporting examples.

We identified two Grammarly edits of the same source. The source includes: In the pickling of hams they had an ingenious apparatus ... a machine consisting of a hollow needle attached to a pump. Paraphrase retains the quoted passage and changes it to they utilized an innovative device ... an apparatus made up of a hollow needle linked to a pump. Improve reorganizes the passage into a separate explanation: During the pickling of hams, they used an ingenious machine ... This machine consisted of a hollow needle attached to a pump. Their overall scores are similar: S = 0.1803/0.1861, with independent EditLens scores of 0.3884/0.3702. However, their profiles differ: Q = 0.3949/0.3041 and M = 0.4579/0.6154 (Paraphrase/Improve). MixDetect thus distinguishes editing scope and conditional intensity even when overall editing magnitude is similar.

W2 & Q6: The benefit of Q/M decomposition over Direct-s is unclear.

We completed the missing Direct-s ternary classification results. Under Edit-Only, macro-F1 was 0.848 for MixDetect vs. 0.850 for Direct-s; under Full, both achieved 0.916. The original Table 4 word AUROC of 0.945 for Direct-s included human source texts, whereas MixDetect's 0.885 was evaluated only on AI-edited texts. On the same 1,999 partially edited texts, MixDetect achieved higher word AUROC (0.882 vs. 0.873) and APtext (0.679 vs. 0.650). The decomposition therefore preserves text-level classification performance, improves within-document editing localization, and provides separate estimates of editing scope Q and conditional intensity M.

Thank you for the detailed response and additional experiments. Clarifying that the original Table 4 word AUROC for Direct-s included unedited human source texts resolves a major point of confusion. On the shared set of 1,999 partially edited texts, MixDetect outperforms Direct-s in within-document localization while matching its ternary macro-F1, which addresses my concerns about the localization protocol and baseline comparison. The examples of complex edit labels and distinct Q/M profiles are also helpful and worth summarizing in the revision, especially regarding pure deletions and relocated spans. Finally, the human annotation study adds useful independent evidence that was missing from the initial submission.

Two brief clarifications would help me complete my assessment:

Human evaluation protocol. Please state how the 120 word-level examples were sampled, including the breakdown between substitutions and insertions and the class distribution across the three severity levels. Please also clarify which agreement statistic the reported 0.64 represents, how continuous reference and predicted intensities were discretized into the three levels, and whether those thresholds were fixed independently of the 120 evaluated items. Reporting a threshold-free rank correlation against the aggregated human ratings, particularly on substitutions, would further strengthen this result.

Operation-level intensity rankings. Reporting M directly on the edited texts is a sensible way to avoid subtracting uninterpretable baseline values from near-zero-Q human sources. Still, the reported rank correlation of 0.983 to 1.000 is hard to reconcile with Figure 5. Figure 5 plots a much larger ΔM for Paraphrase it than for Make it more detailed and Make it more descriptive, whereas the response reports a lower edited-text M for Paraphrase (0.592) than for Make it more detailed (0.756) and Make it more descriptive (0.725). Was that rank correlation computed against the signed difference M(edited) - M(source), while Figure 5 plotted the absolute difference |M(edited) - M(source)| relative to a high source baseline around 0.80? Please provide the nine-operation values for both edited-text M and the originally plotted ΔM statistic, specify their exact definitions, and clarify whether the comparison between Paraphrase and expansion operations in Section 4.2 needs to be revised.

Overall, the response has substantially strengthened the paper. I am keeping my rating at 2/4 for now, and would be glad to raise it to 3/4 once these two points are clarified.

Q1: Human evaluation protocol.

Directly assigning continuous word-level intensity scores or precisely identifying edited words and operations is difficult for human annotators, especially under rewriting. We therefore used relative judgments. We randomly sampled 120 source–edited examples from the training set, with texts no longer than 300 words, and organized them into three-example comparisons. Three annotators independently ranked each triplet by perceived editing magnitude. The three ranks describe relative differences within a triplet, not absolute severity levels. Examples were sampled without quotas for substitutions or insertions, and no discretization thresholds were used.

We computed Fleiss' κ over the three relative-rank labels, obtaining 0.64. Across the 360 individual rank assignments (120 examples × three annotators), the alignment-derived reference and MixDetect predictions agreed with human rankings in 296 (82.22%) and 273 (75.83%) cases, respectively. These are threshold-free ranking agreements, providing an independent human assessment of the relative editing magnitude captured by our scores.

Q2: Operation-level intensity rankings.

As stated in the original Figure 5 caption and Section 4.2, ΔM denotes the magnitude of paired score changes. For each operation and seed, we average M(edited) − M(source) over matched pairs; D is the mean signed change across four seeds, and Figure 5 plots |D|. The previously reported Spearman correlations (0.983–1.000) used signed differences, not |D|.

Thus, |D| measures score displacement, whereas edited-text M measures predicted conditional intensity. Under Edit-Only, Paraphrase has a larger |D| than Make it more detailed (0.212 vs. 0.049), but a lower edited-text M (0.592 vs. 0.756). The complete results are:

OperationEO MEO ∣D∣\lvert D\rvertFull MFull ∣D∣\lvert D\rvert
Fix any mistakes0.7210.0830.7100.079
Improve it0.6580.1470.7140.077
Paraphrase it0.5920.2120.6320.157
Sound fluent0.6740.1300.7380.051
Simplify it0.6210.1840.6690.121
Make it more descriptive0.7250.0800.7990.009
Make it more detailed0.7560.0490.8480.058
Shorten it0.6640.1400.7350.054
Summarize it0.6910.1110.7610.026

We will retain |D| as the magnitude of score change and clarify its distinction from edited-text M in Section 4.2. Both expansion operations have higher edited-text M than Paraphrase in Edit-Only (0.756/0.725 vs. 0.592) and Full (0.848/0.799 vs. 0.632). This comparison directly characterizes the edited texts without relying on M for near-zero-Q human sources.

The author addressed my concerns, I will improve my score to 3/4(confidence 3).

Reviewer_pX7AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MixDetect trains a model on aligned human-written and AI-edited texts to predict per-word edit occurrence and intensity, then aggregates these into editing scope and magnitude. This is more fine-grained than whole-text based classification.

Critical strengths

The method uses finer-grained supervision and decomposition, which seems novel.

Critical weaknesses

Unclear that alignment will always work. What if text goes through significant rewriting?

There's no adversarial evaluation -- AI text detection tends to apply in settings such as academic integrity checks, where users have incentives to evade detection. Consider applying RAFT: Realistic Attacks to Fool Text Detectors (EMNLP 2024) or similar to evaluate the adversarial robustness of your approach.

Baseline comparison seems incomplete -- e.g., RAIDAR (ICLR 2024), a black-box approach based on rewriting.

Questions for authors

How would alignment work if there is a significant rewrite?

How would your method perform in adversarial settings?

1 comment

W1 & Q1: Alignment may fail under substantial rewriting.

Source–edited alignment is used only to construct word-level supervision during training; MixDetect requires only the input text at inference time. Our alignment first identifies exact lexical anchors, then matches the remaining words using contextual similarity and order-preserving dynamic programming. In one substantially rewritten example, the source was reduced from 308 to 81 words, yet the alignment identified 26 KEEP, 33 SUBSTITUTE, and 22 INSERT words, including meaningful matches such as transporter → bicycle and human → person. Another heavily rewritten review yielded menu → options. These examples show that the alignment can recover word-level correspondences even after substantial rewriting.

W2 & Q2: Adversarial robustness has not been evaluated.

We evaluated RAFT (EMNLP 2024) on 200 AI-edited and 200 AI-generated texts using targeted word substitutions with a 10% modification budget. For AI-edited texts, after changing 4.47% of words on average, MixDetect Full achieved 0.879 AUROC / 0.832 macro-F1, compared with 0.968/0.910 before attack. For AI-generated texts, after changing 1.79% of words on average, it retained 0.999 AUROC / 0.940 macro-F1, compared with 1.000/0.990 before attack. MixDetect therefore maintains effective text-level detection under score-targeted word-substitution attacks.

W3: The baseline comparison omits RAIDAR.

We added RAIDAR (ICLR 2024) using its seven rewriting instructions and a local Qwen3-4B-Instruct rewriter. On the same test set, the AUROC/macro-F1 results for Human vs. AI-edited were 0.962/0.904 for MixDetect Full, 0.770/0.701 for RAIDAR-MLP, and 0.823/0.745 for RAIDAR-Logistic. For Human vs. AI-generated, the corresponding results were 1.000/0.997, 0.936/0.854, and 0.959/0.897. MixDetect outperformed both RAIDAR variants on both tasks, with a particularly large margin for AI-edited text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.