- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates how Chinese AI-text detectors respond to word-level and character-level perturbations. It introduces a “reading axis,” measured through the differential evasion index , a saliency-coverage index , and multiscale sensitivity measurements. The authors also derive a bound on deletion-based probability sensitivity under sigmoid saturation and propose human-exemplar-guided iterative rewriting with semantic and formatting constraints. Experiments use a 5,000-text corpus comprising outputs from four generators and human-written Wikipedia sentences, with six publicly available Chinese detectors and additional controls. In the main rewriting comparison, the proposed method reduces the target detector’s AUROC from 0.968 to 0.606 while achieving a mean semantic cosine similarity of 0.903. The paper interprets these results as evidence that evasion depends on covering the detector’s native reading axis.
Critical strengths
-
The paper identifies a worthwhile diagnostic question and proposes measurable behavioral comparisons. Comparing Chinese detectors under word-level and character-level perturbations is a useful way to examine assumptions about attack granularity. The study considers multiple released detectors and reports uncertainty rather than relying only on selected successful attacks. This provides a potentially valuable starting point for a more carefully controlled investigation of granularity-dependent sensitivity.
-
The rewriting results suggest a potentially useful trade-off between detector-score reduction and semantic similarity. The main comparison evaluates methods on the same AI texts and human reference set while reporting both AUROC reduction and semantic similarity. Within that protocol, the full method improves both reported quantities relative to the exemplar-only variant. This is a promising empirical result, although semantic cosine similarity remains a proxy for meaning preservation, and the implementation and comparison conditions need clarification.
Critical weaknesses
-
The proposed reading axis and axis-coverage principle are not adequately separated from perturbation strength and attack budget. The paper argues that replacing the same proportion of words or characters cancels attack strength and isolates reading granularity. However, equal proportions of substituted units do not equate realized character-edit distance, semantic distortion, fluency degradation, or replacement distributions. A positive $\xi$ may therefore reflect differences between the two attack procedures rather than an intrinsic word-reading property of the detector. The reported sign-change control is interesting, but it does not resolve these unmatched perturbation characteristics. The evidence for the axis-coverage principle has a related confound. Figure 3 increases the attack budget while observing increases in both $\tau$ and AUROC reduction. Under Equation (4), if saliency weights are fixed, total saliency is nonzero, and every unit has marginal inclusion probability , then $\mathbb{E}[\tau\mid x]=r.$ Thus, coverage increases mechanically with budget even without saliency-aware targeting. The observed correlation does not establish that $\tau$ explains evasion independently of budget. Fixed-budget comparisons, randomized-saliency controls, or analyses controlling for realized perturbation strength are needed to substantiate the central mechanism claim.
-
The theoretical interpretation and several evaluation statements contain substantive inconsistencies. Equation (S6) is not a general identity for joint mutual information because it omits higher-order interactions. A direct counterexample is three independent uniform bits with $Y=U_1\oplus U_2\oplus U_3$. Every marginal information term and every pairwise correction appearing in the equation is zero, whereas $I(Y;U_1,U_2,U_3)=1$ bit. The equation therefore cannot support the claimed general information-theoretic explanation without additional assumptions or an explicit approximation argument. The evaluation also requires reconciliation. The manuscript reports TPR@1%FPR equal to 1 while reporting AUROCs substantially below 0.99. If these quantities come from the same examples, scores, and standard ROC definition, they cannot hold simultaneously: reaching TPR $=1$ at FPR $=0.01$ implies AUROC of at least 0.99. Different evaluation sets could explain the discrepancy, but that distinction must be explicit. Separately, Appendix S7 reports mean semantic similarities as low as 0.728 while describing the experiment as using the same semantic guard, whereas Algorithm 1 enforces similarity of at least 0.85 to the original text. These discrepancies affect confidence in the reported quality-constrained evaluation.
Questions for authors
-
Which exact datasets and scoring functions produce the reported TPR@1%FPR and AUROC values? Please explain whether these values correspond to different experiments. Also, why do the Appendix S7 similarities fall below the hard constraint in Algorithm 1? Is the guard applied relative to the original text or the preceding rewrite?
-
Does the word-versus-character conclusion remain after matching realized edit distance and semantic degradation? At a fixed attack budget, does $\tau$ predict evasion better than random coverage or coverage computed with shuffled saliency weights?
-
What precisely implements Sample? Please specify the rewriting model, prompts, decoding settings, exemplar selection, separation between the exemplar library and evaluation references, and detector-query budgets for all methods. What happens when no candidate satisfies the semantic constraint?
-
Is Equation (S6) intended to hold under additional distributional assumptions, or is it a truncated approximation? Which central conclusions remain independently supported after correcting this equation and restricting the saturation theorem to the perturbations covered by its assumptions?