- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MELD, an AI-text detector trained through class-balanced, generator-family-specific detection tasks sharing a common representation of human writing. An Ettin-1B encoder projects token representations into a low-dimensional style space containing multiple human anchors and generator-family centers. Family-specific token scores are aggregated using top-quantile pooling, and training combines detection, family attribution, rewrite-task prediction, and formatting consistency. The work also introduces a large training mixture containing content-matched human–AI rewrite pairs and a held-out evaluation pool, MELD-eval. The authors report strong results across several benchmarks, including 99.7% TPR at 1% FPR on MELD-eval and 91.9% seven-way family attribution accuracy. A further case study analyzes detector-score trends in approximately 5.9 million scientific texts and peer-review summaries。
Critical strengths
-
The data construction directly addresses important content and formatting shortcuts. The content-matched human–AI rewrite pairs are a substantive contribution. Matching the source content across labels reduces obvious topic and document-content mismatches, while symmetric formatting treatment targets superficial cues that could otherwise dominate detection. The manuscript explicitly diagnoses line-break artifacts in scientific prose and introduces normalization and augmentation to reduce them. This is a meaningful improvement over simply collecting larger but poorly matched human and machine corpora.
-
The reported performance is strong across multiple evaluation settings. The results are not confined to a single favorable benchmark. Under the same MELD-eval calibration protocol, MELD reports 99.7% TPR compared with 95.5% for the strongest listed baseline. The broader benchmark results also indicate substantial potential as a practical detection system. The authors appropriately disclose less decisive results, including overlapping confidence intervals with the strongest competitor on M4GT, rather than presenting every numerical improvement as conclusive superiority
Critical weaknesses
-
The experiments do not directly establish the benefit of the central family-specific detection objective. The central methodological claim is that class-balanced, family-specific AI-versus-human tasks improve detection and generalization relative to pooled binary training. However, comparisons with existing public checkpoints do not control for encoder architecture, training data, compute budget, normalization, or pooling. Consequently, they establish the performance of the overall system more clearly than the contribution of the proposed objective. Crucially, the “no family CE” ablation removes the auxiliary family-attribution loss; it does not replace the core family-specific detection loss with a pooled binary objective. It therefore does not answer the paper’s principal methodological question. A matched pooled-binary baseline using the same encoder, corpus, sampling plan, normalization, and training budget is needed. The anchor analysis has a related limitation. Replacing eight anchors with one in an already trained checkpoint, without reoptimization, demonstrates dependence on the learned representation but does not establish superiority over a single-anchor model trained under comparable conditions.
-
The training–evaluation relationship for several public benchmarks is not sufficiently auditable. Section 5.2 describes MAGE, M4GT, and DetectRL evaluation pools as excluded from training, while Table 5 explicitly lists these datasets among the training sources. These statements could be consistent if disjoint splits are used, but the manuscript needs to distinguish dataset-level holdout, split-level holdout, and deduplicated evaluation within the same source distribution. These settings support different generalization claims. The reported near-duplicate removal against an evaluation registry is a positive safeguard. Nevertheless, datasets containing multiple generations and attacks derived from a common source require clear source-document or prompt-level provenance. The arXiv corpus has an explicit paper-level split, but comparable documentation is needed for the public benchmark components. I am not asserting that leakage occurred; the concern is that the current description does not allow the reader to establish the intended independence of the evaluations.
Questions for authors
- For RAID, MAGE, M4GT, and DetectRL, which exact splits contribute to training, validation, threshold calibration, and final evaluation? Are all generations and attack variants from the same original document or prompt assigned together? What precisely does “excluded from training” mean for each benchmark?
- How does pooled binary training perform with the same Ettin-1B encoder, training corpus, sampling distribution, formatting normalization, pooling rule, and optimization budget? The current auxiliary-loss ablation does not isolate the contribution of .
- Are the human examples used to estimate thresholds independent of those used to measure realized FPR? For MELD-eval, does the bootstrap resample original seeds or prompts as clusters, keeping their generated and attacked variants together? Does it propagate uncertainty from threshold estimation?
- What direct quantitative evidence supports AI-span localization and training-free new-family enrollment? Mixed-document detection AUROC does not establish token- or span-level localization accuracy. Similarly, an enrollment rule and a conditional score-stability bound do not establish new-family recognition performance or preservation of the old population’s FPR. Please identify the corresponding evaluations, including before-and-after enrollment results.
2 comments
Response to Reviewer dh2q (2/2)
3. Independent thresholds and bootstrap intervals (Q3).
For the main benchmark results, we use the same pool of human texts to set the threshold and measure FPR (Appendix C). To test independent calibration, we set the threshold on one half of each pool’s human texts and measure FPR on the other half. Table R5 compares this procedure with the fixed checkpoint threshold. On HC3, MAGE, and DetectRL, TPR remains close to the value obtained with a same-pool threshold.
Table R5. MELD with and without pool-specific calibration. The pool-calibrated threshold is set on one half of the pool’s human texts and evaluated on the other half. Values are %.
| Pool | MELD FPR, fixed | MELD TPR, fixed | MELD FPR, pool-calibrated | MELD TPR, pool-calibrated | ModernBERT-Detect TPR, pool-calibrated |
|---|---|---|---|---|---|
| HC3 | 1.67 | 99.20 | 1.00 | 98.46 | 74.13 |
| MAGE | 1.84 | 91.99 | 1.00 | 89.55 | 87.02 |
| DetectRL | 1.63 | 97.69 | 1.01 | 96.92 | 17.36 |
| Short texts (≤192 words) | 5.47 | 93.60 | 1.07 | 89.93 | 81.74 |
Even with a fixed threshold, MELD keeps the FPR at 1.6–1.8% on all three benchmarks. Pool-specific calibration brings it to 1.0% at a cost of only 0.7–2.4 TPR points, and MELD stays ahead of ModernBERT-Detect on every benchmark under the same protocol. For short texts, the same calibration gives 1.1% FPR at 89.9% TPR. In practice, most detectors would need calibration on human text from the target population to reach a low FPR (§3.4, §5.5).
For MELD-eval, we now further show results at the level of source seeds: calibration and evaluation use disjoint human seeds, and each seed stays together with all of its generated and attacked variants (Table R9). Appendix C did not state the resampling unit of the bootstrap. It now resamples seeds as clusters and re-estimates the threshold in every replicate, so the intervals also reflect the uncertainty of the threshold.
Table R9. MELD-eval with independent calibration seeds. Intervals are 95% percentile intervals from 1,000 seed-cluster bootstrap replicates, with the threshold re-estimated in each. Values are %.
| Threshold | FPR | TPR | 95% interval of TPR |
|---|---|---|---|
| Same pool (Table 3) | 1.00 | 99.70 | [99.16, 99.83] |
| Independent calibration seeds | 1.09 | 99.50 | [99.04, 99.79] |
4. Span localization and new-family enrollment (Q4).
To provide quantitative evidence that supports the AI-span localization and training-free new-family enrollment we have carried out additional experiments.
- For localization, experiments in Table R2 score the predicted spans against word-level labels on the public DAMASHA benchmark (Sai Teja et al., 2026) and two held-out mixed-document sets. Zero-shot MELD exceeds the supervised model on the latter two sets, the fine-tuned version outperforms the supervised model in the paper on DAMASHA. MELD also offers much lower human-word FPR.
Table R2. Span localization on the same documents. Word-level F1 (%) of the AI class on documents containing both human and AI text. Human-word FPR is the percentage of words in fully human documents that are marked as AI. Binoculars is scored on 64-word windows.
| Method | DAMASHA | arXiv interleaved | Continuation | Human-word FPR (lower is better) |
|---|---|---|---|---|
| DAMASHA model (supervised) | 94.72 | 58.45 | 77.53 | 8.47 |
| Binoculars | 39.98 | 3.94 | 16.38 | 0.99 |
| MELD (zero-shot) | 73.41 | 75.89 | 94.87 | 0.06 |
| MELD (fine-tuned on DAMASHA data) | 98.71 | 74.01 | 91.43 | 0.34 |
- Regarding enrollment, we enroll DeepSeek as a new family using 64 labeled documents (32 Flash and 32 Pro) and report the results in Table R10. We report detection and attribution before enrollment, after enrollment with the threshold unchanged, and after enrollment with the threshold re-estimated from human texts only. The results show that the enrolled family is recognized with 92.4% recall, while the attribution accuracy on the existing families is nearly unchanged. With the threshold unchanged, the human FPR rises slightly from 1.09% to 1.25%, and recalibration on human texts brings it back to 1.14%.
Table R10. DeepSeek-family enrollment, before and after. Values are %.
| Metric | Before enrollment | After enrollment, threshold unchanged | After enrollment, threshold recalibrated |
|---|---|---|---|
| Human FPR, MELD-eval | 1.09 | 1.25 | 1.14 |
| AI TPR, MELD-eval | 99.50 | 99.53 | 99.48 |
| Existing-family top-1 accuracy | 91.90 | 91.76 | 91.76 |
| DeepSeek-family recall | — | 92.41 | 92.41 |
We hope these clarifications and additional results address your questions. Your comments on the data splits, the independence of the thresholds and the enrollment evaluation helped us make the evaluation protocol much clearer, and we sincerely appreciate your careful review. We would be grateful if you would consider raising your score, and we are happy to answer any further questions.
Response to Reviewer dh2q (1/2)
Thank you for your positive assessment of our work. We believe MELD has the potential to make a meaningful contribution to the field, both as a strong open-source detection tool and as a way to explore broader patterns of LLM usage. We address each point below. The new tables are added to the revised manuscript: Table R6 is Table 19 (Appendix J), Tables R7 and R8 are Tables 7 and 8 (Appendix B), Table R9 is Table 9 (Appendix C) and Table R10 is Table 11 (Appendix E).
1. Matched pooled-binary baseline and separately trained one-anchor model (W1 and Q2).
Table R1 (please also refer to the response to Reviewer J5SJ) provides the matched comparison. All configurations share the backbone, training corpus, sampling plan, format normalization, pooling rule and training budget. Another two ablations keep MELD’s architecture and auxiliary losses and only replace the family-specific detection loss with pooled binary cross-entropy, with and without equal weighting of the positive families. MELD is ahead of all three on the Table 17 subset, on MELD-eval and on both DeepSeek tests. The ordering of the four matched configurations also holds on Muse Glimmer and Nemotron (Appendix J, Table 16).
As for the anchor, we train a separate one-anchor model from the same pretrained encoder with the same corpus, losses and schedule, and compare it with the full model on the Table 17 subset. The results in Table R6 demonstrate the performance gain of having multiple human anchors.
Table R6. Separately trained one-anchor model. Values are %.
2. Compositions of the public benchmarks (W2 and Q1).
Thank you for your comment, our work indeed combines many different data sources. We want to stress that we have been very rigorous to prevent any data leakage from training to testing, additionally we have also employed the near-duplicate removal strategy to further prevent evaluation on “near-seen” or “similar” data. We depict our training and testing set curation in Table R7 in detail, i.e. listing the exact partitions of the four benchmarks that enter training, validation, threshold calibration and final evaluation.
Table R7. Components of the public benchmarks used at each stage.
To reply to your individual questions: HC3 and Ghostbuster are held out as whole datasets (Appendix B, Table 6). RAID, MAGE, M4GT and DetectRL also contribute training data, and for them “excluded from training” refers to the evaluation set. Table 1 uses the official RAID test set scored through the public leaderboard (see also point 6 of our response to Reviewer wfQo), and Table 2 uses held-out partitions of MAGE, M4GT and DetectRL. Training examples that near-duplicate the evaluation registry were removed as well (104,660 in total, §4).
Regarding whether variants of one source stay together, the public benchmarks of Tables 1 and 2 are evaluated on official test partitions or on datasets held out as a whole, so we did not split them. Our own split exists only inside the training data, where it holds out rows for validation and for the fixed threshold. It is made per row rather than per source, and Table R5 shows that the results also hold when the threshold is set on held-out human texts.
To further ensure that the reported results do not rely on sources shared with training, we removed every test row that matches a training row exactly, shares at least 20% of its word 8-grams with the training data, or has the same opening as a training row, and evaluated the remaining rows again (Table R8). This check is stricter than the near-duplicate removal in §4, which drops a training example when at least half of its 8-word shingles appear in an evaluation set, so it still finds rows to remove. AUROC stays within 1 point on the three benchmarks.
Table R8. AUROC after removing all test rows that could potentially overlap the training data.