acceptodds
Under review as a conference paper at ICLR 2027

LesionProbe: Detecting AI-Generated Text through Module Contribution Patterns

Abstract

The widespread adoption of large language models (LLMs) has increased the need to distinguish LLM-generated text (LGT) from human-written text (HWT). However, existing detectors often struggle under distribution shifts, limited text lengths, and paraphrasing. We investigate whether these two types of text differ in how a fixed proxy LLM's predictions depend on its internal computations, and whether such differences provide transferable detection signals. To this end, we propose LesionProbe, a framework inspired by lesion studies in neuroscience. It selectively ablates the attention and MLP branches, individually and jointly, at each layer of a frozen proxy LLM, and measures the resulting changes in next-token predictive distributions using normalized Jensen–Shannon divergence. These responses form module contribution patterns across layers and token positions. Shapley-based analysis reveals systematic layer-wise differences between HWT and LGT, while a lightweight classifier uses the extracted patterns for detection without fine-tuning the proxy LLM. Experiments on DetectRL demonstrate strong detection performance and generalization. Across generation-source configurations, LesionProbe achieves an average AUROC of and a true-positive rate of at a false-positive rate of . Under leave-one-domain-out evaluation, the corresponding results are and . Evaluations with ten proxy LLMs yield mean AUROC values of -, with further experiments showing strong performance across text lengths. Under DIPPER paraphrasing, [email protected]%FPR decreases from to . These findings support module contribution patterns as transferable detection signals and offer insight into how a proxy model responds differently to human-written and generated text.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI-generated text detection, Module contribution patterns, Mechanistic interpretability, Shapley attribution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ouUTOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LesionProbe, a method for detecting AI-generated text. It removes the attention and MLP modules at different layers of a frozen language model and measures how the output predictions change. These changes are used as features to train a small detector. The paper also uses Shapley values to compare the effects of attention and MLP modules on human and AI-generated text. Experiments show that the method performs well across different source models and writing domains, although its performance is less stable in some cross-benchmark settings.

Critical strengths
  1. The paper is easy to follow. The authors explain the motivation and method clearly.
  2. The main idea is interesting. The paper detects AI text by removing parts of a language model and observing how its predictions change. This is different from most existing methods.
  3. The experiments are fairly complete. The method is tested in several settings and is compared with strong baselines as LTD, and the results generally support the main claim of the paper.
Critical weaknesses
  1. The paper provides a computational cost analysis for LesionProbe but gives limited information about the cost of the baseline methods. The LesionProbe results in Table 1 use the full 32-layer setting, which requires 97 runs per input. The faster five-layer version shows lower performance in a separate experiment. In some settings, strong baselines such as LTD achieve performance close to LesionProbe. Reporting their computational costs would therefore help readers better assess the practical benefits of LesionProbe and its suitability for deployment.

  2. LesionProbe performs well on average, but its performance varies across different benchmark transfers. This suggests that the detected patterns may still contain benchmark-specific information or noise.

Questions for authors

q1. Zeroing a module may create unnatural hidden states. Have the authors tested other intervention methods, such as mean replacement or partial scaling, to verify that the detected patterns are not specific to zero ablation?

q2. What factors do the authors believe cause the large performance differences across benchmark transfers, especially in settings such as MAGE-to-AIGTBench?

2 comments

We thank you for taking the time to review our submission and for your thoughtful feedback. Below, we address your concerns point by point.

W1. Inference Cost and Performance Trade-off

Thank you for this constructive suggestion. We have added a comparison of detection performance and inference cost. LesionProbe achieves the highest AUROC in this setting while requiring less inference time and GPU memory than LTD, and sparse interventions further improve efficiency. Detailed results are presented in Table 1 below.

Table 1. Detection performance and inference cost on DetectRL. Trainable methods are trained on ChatGPT-generated text and corresponding HWT; evaluation covers all four source LLMs.

MethodAUROC (%) ↑Time per Sample (s) ↓Peak GPU Memory (GB) ↓
FastDetect-GPT87.300.0619.90
Binoculars95.100.1929.73
RoBERTa96.970.013.69
RAIDAR78.519.6862.36
RepreGuard92.300.0516.81
LTD95.2412.9249.98
LesionProbe ({32})95.950.0818.97
LesionProbe ({1, 32})97.500.1618.97
LesionProbe (Peak-5)97.770.3318.97
LesionProbe (All)98.862.1119.23

Although full-layer interventions incur higher inference costs than RoBERTa and RepreGuard, sparse interventions offer a practical trade-off between performance and efficiency. Compared with the full configuration, Peak-5 achieves a 6.4× speedup with only a 1.09 percentage point decrease in AUROC. Using layer 32 alone increases the speedup to 26.4× while maintaining an AUROC of 95.95%.

Q1. Are the Results Specific to Zero Ablation?

Thank you for this constructive suggestion, which helps us better assess the robustness of LesionProbe to different intervention choices. LesionProbe consistently achieves strong ID and OOD detection performance across different intervention strategies, showing that its effectiveness does not depend on zero ablation. Detailed results are presented in Table 2 below.

Table 2. ID and OOD performance under alternative interventions on texts generated by ChatGPT and Google-PaLM (mean ± std, %). α denotes the branch-output scaling factor; α = 0 is zero ablation.

InterventionID AUROC ↑ID [email protected]%FPR ↑OOD AUROC ↑OOD [email protected]%FPR ↑
Zero ablation (α = 0)99.86 ± 0.0195.72 ± 1.0399.47 ± 0.1094.10 ± 0.10
Scaling (α = 0.25)99.85 ± 0.0295.85 ± 1.0599.43 ± 0.0294.63 ± 0.45
Scaling (α = 0.50)99.81 ± 0.0294.78 ± 1.7599.27 ± 0.2594.10 ± 0.35
Scaling (α = 0.75)99.75 ± 0.0194.23 ± 0.8599.31 ± 0.2994.17 ± 0.16
Mean replacement99.87 ± 0.0196.32 ± 0.5699.40 ± 0.0491.32 ± 2.24

Partial scaling with α = 0.25, 0.50, and 0.75 maintains stable ID and OOD performance. Mean replacement also performs strongly, with a slight decrease in OOD TPR. These results suggest that LesionProbe’s effectiveness is not specific to zero ablation and that module-intervention responses provide strong detection signals across intervention choices.

W2 and Q2. Understanding Performance Differences Across Benchmark Transfers

Thank you for raising this concern. We conducted a feature analysis to investigate the differences in cross-benchmark performance. The results suggest that stronger consistency in human–AI feature patterns between RAID and AIGTBench contributes to better transfer performance than between MAGE and AIGTBench.

We examine this explanation using two measures: class separation within each benchmark and alignment of class-separation directions across benchmarks. We use token-averaged lesion-response features from the MAGE, RAID, and AIGTBench test sets, with a shared standardization fitted only on MAGE and RAID training data.

For each benchmark DD, let z∈R96z\in\mathbb{R}^{96} denote the standardized lesion-response feature vector of an individual text, and let μD,A\mu_{D,A} and μD,H\mu_{D,H} denote the AI and human feature centroids, respectively. We define:

JD=∥μD,A−μD,H∥2212[EA∥z−μD,A∥22+EH∥z−μD,H∥22].J_D = \frac{\lVert\mu_{D,A}-\mu_{D,H}\rVert_2^2}{\frac{1}{2}\left[\mathbb{E}_{A}\lVert z-\mu_{D,A}\rVert_2^2+\mathbb{E}_{H}\lVert z-\mu_{D,H}\rVert_2^2\right]}.

Larger JDJ_D indicates greater separation between the two centroids relative to within-class variability, suggesting better class discriminability within the benchmark. To compare how the classes differ across benchmarks, we define:

\cos\left( \mu_{S,A}-\mu_{S,H}, \mu_{T,A}-\mu_{T,H} \right).$$ Higher $A_{S,T}$ means that the human-to-AI feature direction is more consistent between the source and target benchmarks, potentially facilitating cross-benchmark generalization. The results are summarized in Table 3. **Table 3.** Feature separation and directional alignment, with AIGTBench as the shared target. Values are mean ± standard deviation across five runs. | Source → Target | Source separation J | Target separation J | Direction alignment A ↑ | OOD AUROC (%) ↑ | |---|---:|---:|---:|---:| | MAGE → AIGTBench | 0.113 ± 0.002 | 0.149 ± 0.006 | 0.322 ± 0.014 | 60.11 ± 5.49 | | RAID → AIGTBench | 0.100 ± 0.003 | 0.149 ± 0.006 | 0.765 ± 0.001 | 91.45 ± 1.30 | AIGTBench exhibits relatively strong within-benchmark class separation ($J=0.149$), consistent with its high ID AUROC (95.48%). However, its class-separation direction aligns more closely with RAID ($A=0.765$) than with MAGE ($A=0.322$), corresponding to substantially different OOD AUROC (91.45% vs. 60.11%). A similar pattern emerges in reverse transfers: AIGTBench → RAID (89.81%) vs. AIGTBench → MAGE (72.87%). These consistent bidirectional trends further support directional misalignment as a plausible explanation for the observed performance differences. Thank you again for your careful review and constructive feedback. Have a great day!
Reviewer_LHeaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper proposes detecting LLM generated text using changes in a frozen proxy LLM’s predictions caused by individually and jointly ablating attention and MLP modules.

Critical strengths
  • This paper present an interesting methodology by extracting valuable features from a frozen LLM.

  • The empirical results seems to be overall competitive in comparison to the other presented methods.

Critical weaknesses
  • There is a substantial body of literature showing that relatively simple probing or representation-based methods can achieve very strong AI-generated-text detection performance. However, the related-work section discusses only a limited number of such approaches (RepreGuard). For example [1] reports near-saturated performance on the DetectRL benchmark and also reports that a simple fine-tuned RoBERTa detector performs strongly on this dataset. The choice of the nearly saturated DetectRL and lack of overview on the modern literature makes it hard to assess the benefits of LesionProbe.

  • The use of Shapley attribution is reasonable, but the paper does not sufficiently discuss the substantial prior literature on circuit discovery, component attribution, and Shapley-based analysis in neural networks and language models. [2] would for example be a relevant precedent. The current presentation may give the impression that the methodology follows primarily from the neuroscience analogy, whereas it should also be situated within the existing mechanistic-interpretability literature.

  • If I understand correctly, in the proposed method each detection sample requires multiple intervention-based forward passes through a proxy LLM, which may make it substantially more expensive than standard probing or representation-based detectors. Although the paper discusses sparse-layer interventions in Section 4.5, it does not provide a sufficiently comprehensive comparison of inference cost, wall-clock latency, memory use, or computational budget against existing methods. This is an important practical limitation that should be addressed more explicitly.

[1] Quaremba, G. et al (2026) Linear Probing Provides Robust and Efficient Detection of Machine-Generated Text

[2] Held & Yang, (2023) Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers

Questions for authors
  • Would using the representations from all proxy-model layers directly as input to the discriminator provide comparable performance? More generally, does the lesion-response features introduce information that is not already captured by the raw layer-wise representations?

  • LLMs have evolved rapidly over the past two years, whereas the evaluation focuses mainly on the older models included in the DetectRL benchmark. How well does the proposed method perform on text generated by more recent and stronger LLMs?

  • The paraphrasing and perturbation experiments appear to use only one fixed parameterization. Since DIPPER-XL allows the paraphrasing strength to be varied, how does the method respond to different levels of paraphrasing aggressiveness, and how does its robustness compare with that of existing detectors?

  • What specific insights does the Shapley analysis provide about the relative importance of different layers or modules in distinguishing human-written from LLM-generated text?

6 comments

We thank you for taking the time to review our submission and for your thoughtful feedback. We first address the concerns raised in the weaknesses section, followed by responses to the specific questions.

W1. Limited Coverage of Representation-Based Methods and the Suitability of DetectRL for Evaluating LesionProbe

1.1 Expanded Discussion of Representation-Based Detection.

Thank you for highlighting this omission, which has helped us improve our overview of representation-based detection. We have expanded the “Internal Representations” paragraph in Section 2.1, lines 131–140, to include additional relevant studies, as follows:

Internal Representations. More recently, detection methods have examined evidence within proxy LLMs. RepreGuard [3] exploits source-related patterns in hidden representations to distinguish HWT from LGT. Quaremba et al. [1] further demonstrate the effectiveness of simple linear probes on frozen LLM representations, introducing layer-averaged linear probes (LLP) and concatenated-layer linear probes (CLP). SV-Detect [4] uses layer-wise projections onto human–AI discriminative directions as detection features. Steer-to-Detect [5] learns a steering vector that modifies hidden activations to improve class separability for detection. These studies demonstrate the value of internal representations for detection and motivate further investigation of the evidence available within proxy models.

We also evaluate both methods (LLP and CLP) on text generated by frontier LLMs to assess their generalization beyond DetectRL; the results are reported in Table 2 (Q2). Both LLP and CLP achieve strong detection performance on the evaluated frontier models. However, their overall performance remains below that of LTD and LesionProbe.

1.2 Suitability of DetectRL and Broader Evaluation of LesionProbe

Upon checking [1], we found that the near-saturated performance on DetectRL is observed in ID AUROC. Despite differences in training-set size, our RoBERTa achieves a similarly high ID AUROC (99.39% versus 99.98% in [1]). LTD (99.94%) and LesionProbe (99.89%) also approach saturation in ID AUROC, indicating that this pattern is not specific to RoBERTa.

We thank the reviewer for raising this concern. We first explain why DetectRL remains informative, then describe our efforts to more comprehensively evaluate LesionProbe.

  • OOD performance remains far from saturated. Evaluating out-of-distribution generalization is a key focus in AI-generated text detection [1, 3]. On DetectRL, [1] reports RoBERTa’s OOD AUROC scores of 91.08% on ArXiv, 87.35% on Reddit, 84.91% on Yelp, and 77.81% on News, indicating substantial room for improvement. In our experiments, OOD AUROC on DetectRL is not saturated in either cross-LLM or cross-domain evaluation. For example, LTD, a strong baseline, achieves only 84.12% AUROC on ChatGPT → Claude-instant and 87.59% on Claude-instant → Google-PaLM. LesionProbe achieves the highest average cross-LLM OOD AUROC of 98.74%, demonstrating that DetectRL remains informative for evaluating generalization.

  • Low-FPR detection remains challenging. Evaluating detection at a low FPR is important for limiting false accusations of HWT. In our cross-domain experiments, LesionProbe achieves the highest mean [email protected]%FPR in both ID (97.65%) and OOD (87.66%) settings, compared with 86.19%/40.52% for RoBERTa and 97.56%/33.45% for LTD. Thus, near-saturated ID AUROC does not imply saturation in low-FPR detection, particularly in OOD settings.

  • Broader evaluation of LesionProbe. Beyond DetectRL, we evaluate cross-benchmark performance on RAID, MAGE, M4GT-Bench, and AIGTBench. Together, these five benchmarks comprise 19 domain subsets and 20 source-LLM subsets, providing five ID and 20 OOD settings for assessing detection performance and generalization. Detailed results are reported in Appendix C, Table 5 (lines 972–1000 and 1005–1008).

W2. Expanded Discussion of LesionProbe’s Position within Mechanistic Interpretability

Thank you for highlighting this omission. We provide the following discussion to situate LesionProbe within the existing mechanistic-interpretability literature, beyond the neuroscience analogy.

Interventions provide a means of assessing component influence through changes in model behavior. Wang et al. [6] use causal interventions to identify attention heads involved in indirect object identification. Conmy et al. [7] automate circuit discovery by pruning connections while preserving the behavior of interest, while Syed et al. [8] use gradient-based attribution to estimate connection importance more efficiently. Related component-level analyses include causal tracing by Meng et al. [9], which restores selected hidden activations after input corruption to locate computations supporting factual recall, and causal mediation analysis by Vig et al. [10], which measures how neurons and attention heads contribute to gender bias.

Shapley-based attribution accounts for interactions among components. Ghorbani and Zou [11] quantify individual neuron contributions to network performance, while Held and Yang [2] assess attention-head contributions to multilingual task performance to identify interfering heads for pruning. LesionProbe measures component influence through prediction changes caused by individual and joint ablations of attention and MLP branch outputs. Shapley attribution quantifies each branch’s contribution across the presence and absence of the other.

We have added this discussion to Section 2.2, lines 155–169, of the revised manuscript.

W3. Inference Cost and Performance Trade-off

Thank you for this constructive suggestion. We have added a comparison of detection performance and inference cost. LesionProbe achieves the highest AUROC in this setting while requiring less inference time and GPU memory than LTD, and sparse interventions further improve efficiency. Detailed results are presented in Table 1 below.

Table 1. Detection performance and inference cost on DetectRL. Trainable methods are trained on ChatGPT-generated text and corresponding HWT; evaluation covers all four source LLMs.

MethodAUROC (%) ↑Time per Sample (s) ↓Peak GPU Memory (GB) ↓
FastDetect-GPT87.300.0619.90
Binoculars95.100.1929.73
RoBERTa96.970.013.69
RAIDAR78.519.6862.36
RepreGuard92.300.0516.81
LTD95.2412.9249.98
LesionProbe ({32})95.950.0818.97
LesionProbe ({1, 32})97.500.1618.97
LesionProbe (Peak-5)97.770.3318.97
LesionProbe (All)98.862.1119.23

Although full-layer interventions incur higher inference costs than RoBERTa and RepreGuard, sparse interventions offer a practical trade-off between performance and efficiency. Compared with the full configuration, Peak-5 achieves a 6.4× speedup with only a 1.09 percentage point decrease in AUROC. Using layer 32 alone increases the speedup to 26.4× while maintaining an AUROC of 95.95%.

Q1. Do Lesion Responses Offer Benefits Beyond Raw Layer-Wise Representations?

Upon checking [1], we found that its appendix reports ablations of LLP and CLP without PCA, directly examining the reviewer’s suggestion of using representations from all proxy-model layers as input to the discriminator. Both methods achieve strong ID performance in these ablations. However, our evaluation of LLP and CLP on frontier LLMs (Table 2, Q2) shows that their OOD performance remains below that of LesionProbe.

The reason for this performance gap is that hidden states also encode information unrelated to text provenance, such as semantics and writing style, which may distract probes from source-related signals and limit generalization to unseen generators. LesionProbe instead summarizes each layer’s intervention responses using three interpretable scalars, measuring prediction changes when disabling the attention branch, the MLP branch, or both. These features explicitly characterize component influence on predictions, and our OOD results support their effectiveness for generalization.

Q2. How Well Does LesionProbe Generalize to Frontier LLMs?

Thank you for this valuable question. Since most training-based methods already achieve strong ID performance, we focus here on OOD detection. Table 2 shows that LesionProbe maintains strong detection performance on frontier LLMs, particularly at low FPR.

Table 2. OOD detection performance of detectors trained on DetectRL and evaluated on nine unseen frontier LLMs from Name That LLM [12]. Each cell reports AUROC / [email protected]%FPR (%).

MethodGPT 6 AstraGPT 5.6 SolClaude Fable 5Claude Sonnet 5Gemini 3.1 Pro PreviewGemini 3.8 FlashGLM 5.3Kimi K3Qwen 3.8 Max
RoBERTa91.30 / 33.1189.41 / 27.3387.25 / 16.7890.16 / 28.4497.90 / 52.8996.25 / 50.8987.19 / 19.1183.16 / 8.6793.20 / 27.55
RAIDAR65.10 / 11.5664.80 / 4.6764.01 / 14.9970.79 / 6.8977.20 / 0.0071.65 / 25.3362.90 / 10.6762.80 / 4.0066.14 / 6.89
RepreGuard75.11 / 24.0070.78 / 10.0070.80 / 33.5675.89 / 31.5690.11 / 74.2285.22 / 30.4465.11 / 13.3356.45 / 8.4472.78 / 16.00
LTD97.17 / 70.2296.93 / 48.0096.94 / 67.5698.71 / 80.8999.83 / 94.2299.76 / 90.2295.98 / 69.7892.67 / 27.7898.24 / 82.89
LLP96.65 / 32.0096.79 / 40.0095.82 / 35.3496.72 / 52.0099.14 / 79.5699.19 / 81.3393.83 / 21.1193.39 / 14.4497.13 / 38.66
CLP96.94 / 49.7896.48 / 20.6795.99 / 38.0396.70 / 57.5599.10 / 77.7898.89 / 78.4493.94 / 15.5693.62 / 10.6797.23 / 46.22
LesionProbe98.83 / 82.0098.96 / 86.8998.18 / 75.8498.61 / 87.5699.74 / 94.6799.32 / 90.2298.77 / 82.6797.07 / 64.2298.02 / 80.44

LesionProbe achieves the highest average AUROC (98.61%) and [email protected]%FPR (82.72%), compared with 97.36% and 70.17% for LTD. These results support its generalization to frontier LLMs, particularly at low FPR.

Q3. How Robust Is LesionProbe to Different Paraphrasing Strengths?

Thank you for this constructive suggestion. Our original experiments use DetectRL’s released paraphrased samples, generated with a fixed DIPPER parameter setting. DIPPER controls paraphrasing through lexical diversity (L) and word-order diversity (O), with higher values requesting stronger changes. Following your suggestion, we evaluate how these methods respond to different levels of paraphrasing aggressiveness. LesionProbe demonstrates strong robustness, achieving the highest [email protected]%FPR at all four tested strengths. Detailed results are presented in Table 3 below.

Table 3. Robustness to DIPPER paraphrasing. DetectRL uses L = O = 40; we jointly vary L and O across 20, 40, 60, and 80. “Before” denotes no paraphrasing. Each cell reports AUROC / [email protected]%FPR (%).

MethodBeforeL20/O20L40/O40L60/O60L80/O80
RoBERTa98.99 / 68.9799.28 / 71.9099.33 / 70.2399.14 / 61.7398.01 / 47.70
RAIDAR79.53 / 12.1075.64 / 6.4371.17 / 5.4359.16 / 1.2342.31 / 0.40
RepreGuard95.77 / 83.4795.83 / 79.7391.47 / 70.4391.20 / 64.0784.17 / 48.10
LTD99.89 / 98.6399.84 / 93.3399.69 / 90.9799.71 / 77.8096.50 / 55.30
LesionProbe99.90 / 95.9399.86 / 95.2399.53 / 91.6399.48 / 89.2399.38 / 87.43

LesionProbe achieves the highest [email protected]%FPR at all four paraphrasing strengths. At L80/O80, it retains an AUROC of 99.38% and a TPR of 87.43%, compared with 96.50% and 55.30% for LTD. Although its TPR decreases as paraphrasing becomes stronger, LesionProbe remains more robust than the evaluated baselines at this low FPR.

Q4. What Does the Shapley Analysis Reveal About Module Contributions?

The Shapley analysis reveals consistent layer-wise attribution patterns across generation sources. MLP contributions differ most strongly in the early and final layers, with additional localized differences in intermediate layers. Attention contributions show smaller but structured differences, particularly in the early layers, while later-layer differences remain closer to zero. Together, these profiles reveal distinct roles for attention and MLP components in the proxy model’s processing of HWT and LGT, rather than a uniform difference across layers.

We have expanded this discussion in Section 3.3, lines 305 – 312.

Thank you again for your careful review. We hope these clarifications and additional results address your concerns about the paper’s soundness. Have a great day!

Reviewer_J4TWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LesionProbe, a novel method for detecting AI-generated text by examining how attention and MLP components across proxy-model layers affect predictions. It achieves strong detection and generalization performance and offers an interpretable perspective on text detection through internal model interventions.

Critical strengths
  1. The idea of building a detector from layer-wise differences in attention and MLP effects between HWT and lgt is highly novel.

  2. Due to the detailed analysis of the underlying principles, this method has a high degree of interpretability; The performances remains stable across different proxy models.

Critical weaknesses
  1. For the cross-benchmark evaluation, tables reporting the absolute ID and OOD results of the different methods would improve readability and provide clearer support for the experimental conclusions.

  2. The main text does not discuss the decreases in TPR observed with some proxy models.

Questions for authors
  1. The Experimental Protocol paragraph would benefit from being divided into shorter paragraphs.

  2. Although the gate convention can be inferred from the description and is provided in Figure 1, a brief sentence in Section 3.2 explicitly stating that 0 disables a branch and 1 retains it would make the formulation easier to follow.

2 comments

Thank you for taking the time to review our submission and provide positive feedback!

We first address the reviewer’s concern (W2), followed by the revisions made to improve the manuscript’s readability (W1, Q1, Q2), for which we sincerely appreciate the reviewer’s constructive suggestions.

1. Analysis of [email protected]%FPR Decreases for Some Proxy Models (W2)

Thank you for pointing out this issue. Proxy models may differ in their representational capabilities, potentially affecting the discriminability of lesion-response features. After switching proxies, most AI texts may remain distinguishable from human texts, while harder samples become less separable. This can reduce TPR under a stringent FPR constraint even when overall AUROC remains high.

To examine this possibility, we compare feature shifts in AI texts that become missed after switching proxies with those that remain correctly detected. We use Llama-3.1-8B as the default proxy and switch to Gemma-7B-IT or Llama-2-13B-Chat, whose TPRs decrease from 94.80% to 73.46% and 79.80%, respectively, at FPR ≤ 0.1%.

To quantify these changes, we first standardize the intervention features using training data and compute the human and AI feature centers for each proxy. For a sample xx, we define

m(x)=dH(x)−dA(x)dH(x)+dA(x)+ϵ,m(x)=\frac{d_H(x)-d_A(x)}{d_H(x)+d_A(x)+\epsilon},

where dH(x)d_H(x) and dA(x)d_A(x) denote the Euclidean distances to the human and AI feature centers, respectively. A positive m(x)m(x) indicates that the sample is closer to the AI center, while a negative value indicates that it is closer to the human center.

We then convert m(x)m(x) into its percentile q(x)q(x) within the training human distribution:

q(x)=1NH∑h∈Htrain[1{m(h)<m(x)}+121{m(h)=m(x)}].q(x)=\frac{1}{N_H}\sum_{h\in H_{\mathrm{train}}}\left[\mathbf{1}\lbrace m(h) < m(x)\rbrace+\frac{1}{2}\mathbf{1}\lbrace m(h)=m(x)\rbrace\right].

Higher q(x)q(x) indicates greater relative AI-likeness, rather than a higher predicted AI probability. We quantify the cross-proxy shift as

Δq(x)=qswitched(x)−qdefault(x),\Delta q(x)=q_{\mathrm{switched}}(x)-q_{\mathrm{default}}(x),

where Δq(x)<0\Delta q(x)<0 indicates reduced relative AI-likeness after switching proxies, while Δq(x)>0\Delta q(x)>0 indicates increased relative AI-likeness.

Analysis of Feature Shifts. At each proxy's low-FPR operating point, we divide AI texts into two groups:

  • Newly missed: Detected by the default proxy but missed after switching.
  • Detected by both: Correctly detected by both proxies.

For each group, we compute the median Δq\Delta q and define their difference as

C=median⁡(Δq∣newly missed)−median⁡(Δq∣detected by both).C = \operatorname{median}\left(\Delta q \mid \text{newly missed}\right) - \operatorname{median}\left(\Delta q \mid \text{detected by both}\right).

where C<0C < 0 indicates a greater reduction in relative AI-likeness among newly missed samples. The results are shown in the table below.

Table 1. Proxy-dependent feature shifts in AI texts at FPR ≤0.1%\leq 0.1\%. Results are averaged over five seeds, with medians and CC computed separately for each seed.

ProxyAUROC (%)TPR (%)Newly missedMedian Δq\Delta q (newly missed)Median Δq\Delta q (detected by both)CC
Llama-3.1-8B (default)99.8894.80————
Gemma-7B-IT99.4073.46227.4−0.043−0.016−0.027
Llama-2-13B-Chat99.5779.80165.0−0.033−0.007−0.027

Findings. Table 1 shows that newly missed samples exhibit larger reductions in relative AI-likeness than samples detected by both proxies: median Δq\Delta q is −0.043 versus −0.016 for Gemma-7B-IT, and −0.033 versus −0.007 for Llama-2-13B-Chat. This pattern holds across all five seeds for both proxies.

These results support the proposed explanation that proxy-dependent representations can affect the discriminability of lesion-response features, particularly for harder samples. Overall discrimination remains strong, as reflected by high AUROC, but reduced separability for these samples may lower TPR under a stringent FPR constraint.

We have added this analysis to Section 4.4 (lines 486–505) and Appendix B.5 (lines 895–930) of the revised manuscript.

2. Improving Clarity and Readability (W1, Q1, Q2)

W1. Following your suggestion, we have replaced the heatmap with tables reporting absolute ID and OOD results to improve readability and provide clearer support for the experimental conclusions. LesionProbe’s AUROC results (%) are shown below, with ID results in bold.

Table 2. LesionProbe’s cross-benchmark AUROC (%).

Train ↓ / Test →RAIDMAGEM4GT-BenchAIGTBenchDetectRL
RAID98.1986.3189.4591.4596.23
MAGE80.2495.7690.6860.1184.62
M4GT-Bench92.2682.6898.2875.0196.91
AIGTBench89.8172.8775.2295.4890.01
DetectRL86.8568.5489.9582.4899.88

Detailed baseline results are provided in Table 5 of Appendix C, lines 972–1000 and 1005–1008, of the revised manuscript.

Q1. We have divided the Experimental Protocol paragraph into two shorter paragraphs, each with a distinct focus, to improve readability. The revised text can be found in Section 4.1, lines 373-412, of the revised manuscript.

Q2. We have explicitly stated that a gate value of 0 disables the branch, whereas 1 retains it, in Section 3.2, lines 205 – 206, of the revised manuscript.

Thank you for taking the time to read through our responses. Have a great day!

Comments on the paper

1 comment

We sincerely thank the reviewers for their time and constructive feedback. We have revised the manuscript, with changes highlighted in blue. The main revisions are summarized below:

  • Expanded the discussion of representation-based detection and mechanistic interpretability in Sections 2.1 and 2.2.
  • Clarified the intervention gates in Section 3.2 and reorganized the experimental protocols in Section 4.1.
  • Expanded the interpretation of layer-wise and module-specific Shapley attributions in Section 3.3.
  • Added a feature analysis of low-FPR performance differences across proxy models in Section 4.4 and Appendix B.5.
  • Replaced the cross-benchmark heatmap with tables reporting absolute detection results in Appendix C.

Our responses also report additional comparisons of inference cost, evaluation on nine frontier LLMs including LLP and CLP baselines, robustness tests across paraphrasing strengths and intervention strategies, and a feature analysis of cross-benchmark transfer differences.

We appreciate the reviewers’ valuable suggestions and welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.