- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DepthTriad, a compact representation for detecting AI-generated text content using a frozen, detector-selected surrogate model. It computes three profiles across network depth: the token-wise mean and standard deviation of the turning between consecutive residual updates, and the standard deviation of token-state RMS magnitudes. DepthTriad-Zero combines three prescribed summaries using pooled source median/IQR normalization, while DepthTriad-Linear fits a linear SVM to the full profiles using 500 matched human/AI-polished ImBD pairs. The scores are transferred to MIRAGE and DetectRL-X. The reported results are strong, including macro AUROCs of 0.9557 and 0.9837.
Critical strengths
-
Strong transfer from limited source supervision. With 500 source pairs, Linear achieves macro AUROC 0.9557 on MIRAGE and 0.9837 on DetectRL-X (Tables 1, 2, 4). Coverage spans generation and revision, seventeen generators and eight languages, with a source-size-matched DetectAnyLLM rerun in Appendix J. This breadth strengthens the empirical contribution.
-
Simple features supported by useful controls. The profiles have clear geometric definitions and a small learned head. Exact visible-length matching retains 79.6% of MIRAGE and 95.2% of DetectRL-X with strong performance (Appendix F, Table 12). Table 3 favors the combined Zero score over individual axes and pairs, supporting investigation of the representation.
Critical weaknesses
-
The representation's contribution is not isolated from the surrogate and probe. Baselines use different backbones and preprocessing: Appendix A lists GPT-Neo-2.7B/2,048 tokens for DetectAnyLLM, while DepthTriad uses its own surrogates/256 tokens. Matching source-pair counts does not remove this confound. The central claim concerns computation-path geometry, so performance gains should be distinguished from those due to the chosen surrogate and probe. Single-layer and layer-wise steering-vector probes should share the same backbone, 500 source pairs, preprocessing, and target records. Table 18 scores terms from a joint fit rather than independently refitting feature subsets, so it does not establish how much angular features add to an optimized S-only detector. Table 3 addresses Zero, not this Linear question. These missing controls are the main reason for my recommendation.
-
Transfer is better established for ranking than for a fixed operating point. Appendix F reads TPR at 1%/5% FPR from each target ROC; this does not show that a source-calibrated threshold retains those FPRs after transfer. On DetectRL-X, Linear improves AUROC from 0.9584 to 0.9837 but lowers balanced accuracy at threshold zero from 0.8592 to 0.8355 (Table 4). These averages do not determine the achieved FPR. Reporting target FPR and TPR by language/domain at a source-calibrated low-FPR threshold would strengthen the practical claim. The paper acknowledges population-specific calibration. This is a secondary limitation, not an allegation that target labels were used to fit the scoring function.
Questions for authors
-
Controlled representation comparison. On DiffusionGemma, or another common strong backbone, how do source-selected single-layer and layer-wise steering-vector probes compare under the same 500 source pairs, preprocessing, and target records? Please also independently refit S-only, D-only, and D+T+S full-profile readouts rather than scoring terms from the joint fit. A consistent margin over these controls would directly strengthen the case for the proposed representation and could change my recommendation.
-
Design and stability of the fixed Zero readout. Appendix C says labeled source trajectories informed the coordinates. What source-only procedure selected the direction anchors, turning band, and signs, and were those choices frozen before any target evaluation? Table 6 shows substantial sensitivity: shifting the direction anchors by -2/28 changes DetectRL-X AUROC from 0.958 to 0.876. Could you report stability over independent source-pair selections, separating design/selection from its validation where appropriate? This would clarify the reproducibility and scope of the training-free readout. I am requesting clarification of the selection protocol, not assuming target leakage.
-
Threshold transfer. If a low-FPR threshold is calibrated using only held-out source human examples, what FPR and TPR does that fixed threshold achieve on the target languages and MIRAGE operation groups? Please distinguish this experiment from Table 13's target-ROC operating points. Reliable error control under this protocol would strengthen the practical relevance of the claimed frozen transfer.
4 comments
3. Transfer of a source-calibrated threshold.
With thresholds fixed from held-out source humans at nominal 5% FPR, Linear achieves target macro FPR/TPR of 1.07%/72.23% on MIRAGE and 1.73%/67.50% on DetectRL-X. Zero achieves 2.10%/63.73% and 3.45%/62.80%, respectively. Both variants retain macro FPR below 5% and macro TPR above 60% on both benchmarks.
Source calibration. With detector parameters fixed, we calibrate each scoring configuration's threshold using 2,845 held-out ImBD human texts, disjoint from the original 500 fitting pairs. The achieved source FPR is 4.96% for every configuration, and thresholds are applied unchanged to target groups. Table 13 reports target-ROC operating points, and Table 4 uses the default zero-score cutoff; the evaluation here determines the cutoff entirely from source humans.
Transfer across operations and languages. We compare all six methods on the same sampled target records, reporting actual FPR/TPR (%) by MIRAGE setting and operation and by DetectRL-X language.
Training-free methods.
| Target group | DepthTriad-Zero | SpecDetect | Fast-DetectGPT |
|---|---|---|---|
| DIG Generate | 1.60/90.20 | 2.80/9.00 | 5.80/43.80 |
| DIG Polish | 3.40/50.20 | 3.60/0.80 | 5.80/10.60 |
| DIG Rewrite | 1.80/48.00 | 2.60/1.40 | 6.40/11.00 |
| SIG Generate | 1.60/92.20 | 3.60/10.80 | 6.00/42.40 |
| SIG Polish | 2.20/50.60 | 1.60/1.40 | 4.20/9.00 |
| SIG Rewrite | 2.00/51.20 | 3.60/2.00 | 6.60/11.60 |
| MIRAGE macro | 2.10/63.73 | 2.97/4.23 | 5.80/21.40 |
| Arabic | 2.00/57.80 | 98.60/100.00 | 16.40/25.20 |
| Chinese | 2.20/55.40 | 75.80/98.60 | 11.40/14.20 |
| English | 7.40/86.40 | 0.60/7.80 | 14.80/26.80 |
| French | 2.80/69.60 | 0.40/18.60 | 8.40/14.40 |
| German | 0.80/13.60 | 0.40/12.00 | 5.60/15.20 |
| Portuguese | 5.00/84.20 | 0.20/13.40 | 8.20/15.20 |
| Russian | 1.00/46.00 | 100.00/100.00 | 3.00/3.80 |
| Spanish | 6.40/89.40 | 0.80/12.20 | 5.80/18.80 |
| DetectRL-X macro | 3.45/62.80 | 34.60/45.32 | 9.20/16.70 |
Cross-benchmark learned methods.
| Target group | DepthTriad-Linear | DetectAnyLLM | SV-Detect |
|---|---|---|---|
| DIG Generate | 1.00/97.20 | 3.80/60.40 | 1.00/74.20 |
| DIG Polish | 0.60/59.00 | 4.20/75.40 | 1.40/48.20 |
| DIG Rewrite | 1.40/62.80 | 4.20/75.80 | 0.20/53.20 |
| SIG Generate | 1.00/95.80 | 5.40/60.00 | 1.60/68.60 |
| SIG Polish | 0.80/57.80 | 2.80/73.00 | 1.00/49.60 |
| SIG Rewrite | 1.60/60.80 | 5.40/76.20 | 1.80/52.20 |
| MIRAGE macro | 1.07/72.23 | 4.30/70.13 | 1.17/57.67 |
| Arabic | 0.40/62.40 | 0.00/0.00 | 0.00/0.20 |
| Chinese | 1.20/87.40 | 0.00/0.00 | 0.00/0.20 |
| English | 5.20/91.00 | 3.80/92.20 | 0.80/80.80 |
| French | 1.20/72.60 | 0.00/7.40 | 0.00/0.00 |
| German | 0.00/7.60 | 0.00/0.00 | 0.00/0.00 |
| Portuguese | 1.80/88.80 | 0.20/3.20 | 0.00/0.00 |
| Russian | 0.00/40.00 | 0.80/0.20 | 0.00/0.00 |
| Spanish | 4.00/90.20 | 1.80/15.20 | 0.00/0.20 |
| DetectRL-X macro | 1.73/67.50 | 0.83/14.77 | 0.10/10.17 |
Zero achieves higher macro TPR and lower macro FPR than both training-free baselines on both benchmarks. Linear achieves the highest macro TPR among the learned methods on both benchmarks, together with macro FPRs of 1.07% and 1.73%.
Transfer across domains. We also group the same predictions by domain, using the unchanged thresholds. Entries report actual FPR/TPR (%).
| Benchmark | Domain | DepthTriad-Zero | DepthTriad-Linear |
|---|---|---|---|
| MIRAGE | Academic | 0.22/56.39 | 0.87/74.43 |
| MIRAGE | Comment | 4.91/76.38 | 2.61/79.75 |
| MIRAGE | 2.09/60.28 | 0.35/59.58 | |
| MIRAGE | News | 1.43/61.07 | 0.36/70.18 |
| MIRAGE | Website | 2.56/65.36 | 0.68/68.60 |
| DetectRL-X | Academic | 3.83/70.99 | 1.62/78.79 |
| DetectRL-X | News | 1.64/56.65 | 0.75/54.11 |
| DetectRL-X | Novel | 3.50/54.34 | 0.61/69.10 |
| DetectRL-X | SEO | 3.27/68.05 | 3.12/77.86 |
| DetectRL-X | Webtext | 6.84/65.50 | 3.04/69.45 |
| DetectRL-X | Wiki | 1.66/60.99 | 1.20/55.42 |
Across all reported domains, Linear maintains FPR below 5%. We will add the operation, language, and domain results, together with the source-calibration protocol, alongside Table 13.
I would like to thank the authors for the detailed response and the additional results. The new ablations under matched settings show that the angular features contribute beyond state dispersion. The fixed source-calibrated threshold evaluation also addresses my concern about threshold transfer. To this point, I'm willing to raise my overall recommendation from 2 to 3, given that these new results can be incorporated into the next revision with necessary clarification.
Thank you again for your encouraging follow-up and your willingness to raise your recommendation. We have now incorporated the additional results and requested clarifications into the revised manuscript, with all changes highlighted in blue.
Thank you for the careful review and constructive suggestions. We appreciate your recognition of the strong transfer from limited source supervision and the geometric formulation. Your questions highlight three important aspects of the method: the contribution of the representation, the design and reproducibility of Zero, and the transfer of a source-calibrated threshold. We have conducted additional analyses for each point and present the findings below.
1. Depth integration and layer-wise steering geometry.
Your question concerns the contribution of the representation itself. Our controlled comparisons show that integrating depths improves on every single-depth DTS detector, while D/T steering geometry adds predictive information beyond state magnitudes.
Depth integration. We compare the full profile with three-feature DTS detectors trained separately at each of 29 depths. The comparisons use the same DiffusionGemma backbone, preprocessing, 500 source pairs, and target records, with normalization and the linear SVM fitted independently.
Full DTS exceeds the strongest individual depth, depth 5 on both benchmarks, by 0.0891 AUROC on MIRAGE and 0.0424 on DetectRL-X. We will include the complete depth-wise curves in the appendix.
Steering geometry beyond state magnitudes. We isolate the contribution of steering geometry by independently refitting all seven D/T/S feature combinations. D captures residual-update directional persistence, T captures token-wise variation in turning, and S captures the spread of state magnitudes.
Adding D/T steering geometry to S raises AUROC by 0.1095 on MIRAGE and 0.0703 on DetectRL-X. The corresponding 95% confidence intervals are [0.1066, 0.1123] and [0.0690, 0.0718], from 1,000 paired bootstrap replicates over target human-source clusters.
Each axis also improves AUROC when added to the other two. All six 95% confidence intervals are positive:
The advantage over S-only persists across five draws of 500 source pairs. Full DTS achieves 0.9577 ± 0.0038 AUROC on MIRAGE and 0.9795 ± 0.0024 on DetectRL-X, versus 0.8414 ± 0.0054 and 0.9075 ± 0.0041 for S-only (mean ± SD). Together, these results support complementary contributions from D, T, and S. We will add the independent refits alongside Table 18.
2. Design and stability of Zero.
To assess whether the choice of depths depends on the sampled source texts, we repeat the selection using five independently sampled source sets. All five selections choose the original D anchors and T band. Detection performance remains stable, with AUROC standard deviations below 0.001 on both MIRAGE and DetectRL-X.
Original design. The fixed rule was developed from labeled ImBD trajectories, architecture-aware choices, and exploratory evaluation on earlier MIRAGE development sets. It combines D at indices 11 and 22, T at indices 10–14, and S over 25 sliding-attention layers, with signs (−, +, +). Pooled source median/IQR calibration uses no labels. The three normalized summaries are clipped to [−5, 5] and averaged.
Source-only selection and validation. Each draw uses 500 design pairs and 1,000 disjoint validation pairs. Five-fold pair-grouped cross-validation compares 200 configurations: joint D-anchor shifts and T-band-center shifts in {−4, −2, 0, 2, 4}, T widths in {3, 5, 7, 9}, and the original S mask or all layers. Normalization is fitted within each training fold, and the original signs remain fixed. Selections are frozen before source validation and target evaluation.
All five draws select all layers for S, with the original S mask ranking second. AUROC (mean ± SD) is:
Source resampling consistently recovers the angular coordinates whose perturbation is tested in Table 6. Both readouts retain stable target performance across source draws. We will clarify the design history and add the selection/validation protocol in Appendix C.