acceptodds
Under review as a conference paper at ICLR 2027

Beyond a Writer Label: Rich Context Shapes AI-Text Detector Scores

12 views

Abstract

AI-text detector evaluations depend on how test passages are prompted. We ask whether a brief writer identity captures the response to a richly developed background. Across 150 Chinese literary-writing topics and five generators (4,500 texts), a one-sentence writer label changes detector scores little, while a detailed fictional background produces much larger shifts. The background also lowers scores more than the tested anti-AI-style directive: by 0.062 on AIGC-ZHv2's classifier scale and 0.098 corpus-rank units for Fast-DetectGPT. Two further biographies preserve the strategy-level separation. In a separate three-generator cohort, an independent second draw preserves the aggregate score-shift directions, and life-history facts carry the larger component response within the complete persona. A detailed description of a fictional town, with no assigned writer identity, then produces a reduction similar to that life-fact bundle across the 12 statistical outputs (median 0.264 versus 0.257 within the same experiment). Rich contextual information can therefore elicit large detector-score shifts without an explicit writer identity. Evaluations should distinguish brief labels from developed contexts and report their generator-specific responses.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Machine-Generated Text Detection, Prompt Sensitivity, Persona Prompting, Author Backgrounds, Detector Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_taSZOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study investigates how the contextual content within a generated prompt influences the scores assigned by Chinese-language AI text detectors. The primary experiment utilizes 150 Chinese literary writing prompts and 5 generative models to produce a corpus of 4,500 texts, comparing several conditions: task-only (N), a single-sentence author identity (S), a detailed fictional author background (P1), and instructions to adopt a style counter to typical AI-generated text (A). Key findings include:

Condition S has a minimal impact on detection scores, whereas condition P1 leads to a substantial decrease—a reduction greater than that observed with condition A. Follow-up experiments (II–V) further deconstruct these effects: The observed effects remain consistent when alternative fictional personas (P2/P3) are used. Factual details regarding life experiences (B) account for the primary effect. The direction of the effect remains consistent following independent resampling. A description of a fictional town (C)—which contains no information about an author’s identity—produces a reduction in scores comparable to that of condition B (median: 0.264 vs. 0.257).

Critical strengths
  1. Statistical design specifications: Paired comparisons are conducted using "topic" as the unit of inference; 5,000-iteration bootstrap confidence intervals are calculated; Wilcoxon tests are applied with Holm correction; effect sizes (dzd_z) are reported; and results are stratified by generative model.

  2. High transparency and reproducibility: Audit trails are maintained for corpus reuse, byte-for-byte prompt consistency, and generation timestamps; scripts to reconstruct all tables are also provided.

  3. The construct validity control in Experiment V is cleverly designed, effectively decoupling "authorship" from "contextual richness." The authors also proactively clarify that equivalence within a ±20% margin cannot be established, avoiding overstatement.

  4. Transparent disclosure of unfavorable findings: The AUROC for LastDE/LRR falls below 0.5, scores for RoBERTa-ZH exhibit saturation, and the AUROC for the ensemble model remains virtually unchanged.

Critical weaknesses
  1. The outcome variable lacks practical significance. The primary metric used is a "shift in relative rank within the corpus" rather than actual detection performance (e.g., TPR at a fixed FPR, or AUROC). The authors themselves acknowledge that a decrease in the score does not necessarily imply that the text has become more "human-like." In Table 6, the AUROC for the ensemble model drops only from 0.990 to 0.981; consequently, the core question—whether the detector would effectively fail—remains unanswered. The suite of detectors evaluated is both limited and outdated.

  2. All 12 statistical metrics rely on a single reference model: gpt2-chinese-cluecorpussmall, which has approximately 100 million parameters. The implementation of "Binoculars" uses the same model to serve as both the "observer" and the "performer," which deviates from the original methodology. RoBERTa-ZH provides virtually no meaningful information in the subsequent experiments. The study lacks comparisons with larger reference language models (LMs) or commercial Chinese-language detection tools.

  3. The human baseline is mismatched. The 150 human-written text samples are drawn from authors of the Republic of China era (including 71 excerpts by Yu Dafu); since these neither correspond to the same task nor originate from the same historical period, the reported AUROC values ​​can only serve as a descriptive reference.

  4. Alternative explanations have not been ruled out. A richer context may simply introduce more specific nouns and details, which naturally inflates perplexity in small language models (LMs). The study fails to measure or control for potential mediating variables, such as named entity density, lexical diversity, topical relevance, text quality, or output length (approximately 654 words for B/C versus 569 words for N).

  5. The claim that "providing context outperforms anti-AI instructions" relies on a single instance; condition A uses only one template. In Experiment V, removing just one requirement from A′ actually led to an increase in most statistical scores, suggesting that the effects of such instructions are highly sensitive to specific wording—making the conclusions difficult to generalize.

  6. The scope is narrow, and the contribution is largely incremental. The study is limited to Chinese literary writing. There is significant heterogeneity among the generative models tested (e.g., Kimi-K2.5 shows almost no effect), meaning the pooled results are driven by a small subset of the models. Compared to the work of Koike et al. (2024), Li & Wan (2025), and Gao et al. (2026), the incremental contribution is limited.

Questions for authors
  1. Could you report the variation in TPR at fixed thresholds (e.g., FPR = 1% or 5%) to clarify whether the shift in scores truly impacts detection performance?

  2. Does the observed effect persist when using a larger reference LM (e.g., on the scale of Qwen 7B), the original Binoculars method (which employs two distinct models), or a commercial Chinese-language detector?

  3. Is it possible to quantify the density of named entities, the level of specific detail, lexical diversity, and text length in the P1/B/C outputs, and then use regression or mediation analysis to determine to what extent these factors account for the observed decline in scores?

  4. Does the generated text remain strictly aligned with the prompt? Is there any change in quality? Could you provide an assessment—conducted by humans or an LLM—regarding the quality and topical relevance of the generated content?

  5. Could you construct multiple variations of "anti-AI" prompts (differing in length or focus) to test the robustness of the "context > instruction" phenomenon?

  6. Could you include human-written samples from the same period and task (e.g., a collection of contemporary Chinese prose) as a baseline for comparison?

  7. Can these findings be replicated in English or across other genres (such as news articles or argumentative essays)?

  8. Why does Kimi-K2.5 show almost no such effect? ​​Does this suggest that the phenomenon stems primarily from the specific generation behavior of certain models, rather than being a universal "context effect"?

Reviewer_mrr9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how different prompting strategies affect AI-generated text detector scores. It compares brief writer labels, detailed fictional backgrounds, anti-AI-style instructions, and non-biographical contextual descriptions. The results suggest that rich contextual information can substantially shift detector scores. The study is interesting, but several concerns remain regarding novelty, experimental controls, and practical implications.

Critical strengths
  • The paper addresses a meaningful problem concerning the sensitivity of AI-text detectors to generation prompts.

  • The experiments cover multiple generators and detectors, with useful analyses involving component decomposition, resampling, and additional controls.

Critical weaknesses
  • Prior studies have already shown that prompting constraints, author attributes, and personalized writing can affect AI-text detection. This paper mainly extends these findings by comparing brief writer labels with detailed backgrounds. The authors should better clarify the contribution beyond [1] and [2].

[1] How You Prompt Matters! Even Task-Oriented Constraints in Instructions Affect LLM-Generated Text Detection

[2] When Personalization Tricks Detectors: The Feature-Inversion Trap in Machine-Generated Text Detection

  • The analysis focuses primarily on relative detector-score changes, which do not necessarily translate into detection failures. For example, Appendix Table 6 shows that the ensemble AUROC decreases only from 0.990 to 0.981. Evaluation at fixed false-positive rates and comparisons with better-matched human-written texts would strengthen the practical significance.

  • Detailed backgrounds may change not only contextual richness but also vocabulary, narrative content, and textual specificity. Although some prompt-length controls are included, it remains unclear which factors drive the observed score changes. More targeted controls or linguistic feature analyses would help explain the results.

Questions for authors

see Weaknesses

Reviewer_wU2DOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how prompt context influences the scores assigned to LLM-generated Chinese literary prose by AI-text detectors. Its main experiment compares task-only prompting, a brief writer-role label, a detailed fictional writer biography, a direct anti-AI-style directive, their combination, and a real-writer biography across 150 topics and five generators (4,500 texts). Follow-up experiments vary persona instances, separate biography facts from narrative/style cues, examine independent generation draws, and introduce a detailed fictional-town description without an assigned writer identity. The authors evaluate classifier scores and multiple statistical detection outputs with paired topic-level analyses. They find that the brief role label changes scores little, while detailed biographies induce substantially larger reductions; the town description produces a similarly large reduction on the statistical outputs in its own experimental cohort. The study argues that the amount and type of context supplied during generation should be treated as an explicit variable in detector evaluations.

Critical strengths

A systematic and relatively transparent empirical comparison. The paper compares multiple prompt strategies within matched topic--generator cells, evaluates several detector families, reports generator-specific variation, and provides full Chinese prompts and implementation details. Its topic-level inference and separate stochastic resampling help establish that the observed score changes are not merely an artifact of a single generation draw. The non-biographical setting control sharpens the central observation. Experiment V shows that a detailed fictional town, despite assigning no writer identity, yields score reductions comparable in magnitude to the life-fact bundle on the tested statistical outputs. This is a useful qualification to interpretations that attribute the earlier effects specifically to persona or author identity.

Critical weaknesses

The methodological contribution is essentially prompt engineering. The core intervention is to prepend hand-written biographies or setting descriptions to an otherwise unchanged literary-writing task and compare detector scores. No new detector, learning algorithm, principled prompt-construction method, general evaluation framework, or mechanistic model is introduced. Existing work cited by the authors already establishes that task constraints, author roles, and personalization can affect detection. The main distinction between a brief label and a long, content-rich prompt is plausible a priori; the town-context comparison refines that observation but does not fundamentally change it. Moreover, the prompt factors are not independently controlled: B and C provide numerous concrete entities, objects, events, and narrative materials that the generator can reuse. Without separating context length, specificity, topic alignment, lexical distribution, and content transfer, the study remains a collection of particular prompt comparisons rather than a generalizable result. These limitations directly undermine the paper's significance and its claimed conceptual contribution (Sections 3--5; Appendix E). The detector evaluation relies on dated or nonstandard components and does not substantiate relevance to modern detection systems. Most of the twelve statistical outputs are computed using the same \texttt{uer/gpt2-chinese-cluecorpussmall} reference LM (Appendix A.2), a small GPT-2-era model that is not representative of contemporary language-model scoring backbones. Several outputs are closely related likelihood/rank transformations, so enumerating many significant scores does not establish robustness across genuinely different detector architectures. The paper also uses only two supervised Chinese detector checkpoints; one (RoBERTa-ZH) is effectively saturated at the score floor in the later experiments (Appendix B.7). Its Binoculars implementation employs identical observer and performer models rather than the usual paired-model configuration. These are important limitations in the \emph{detector/scoring setup}; it would be inaccurate to call all five generation models old, since the paper also uses more recent generators such as GLM-4.5 and Kimi-K2.5. Finally, the manuscript primarily reports relative score shifts, not operational errors. For example, the Experiment I ensemble AUROC only changes from 0.990 (N) to 0.981 (P1) against its historical literary reference (Appendix Table 6). As a result, the paper neither establishes a general weakness of modern AI-text detectors nor demonstrates substantial real-world detection failure.

Questions for authors

What is the substantive research advance beyond a manually designed prompt comparison? How do the findings go beyond the established observation in prior work (e.g., Koike et al., 2024; Li and Wan, 2025; Gao et al., 2026) that generation prompts and author-related attributes change detector behavior? What falsifiable mechanism or general prediction does the proposed interpretation provide? Can the central effects be reproduced with genuinely contemporary and diverse detector backbones? In particular, please re-evaluate representative likelihood/rank/curvature methods using stronger, independently selected Chinese or multilingual reference LMs, and compare against more recent supervised detectors. Why should results based largely on a single small Chinese GPT-2 model be expected to generalize? Can the authors separate the effects of contextual information from direct content transfer and output length? Experiment V reports mean lengths of 568.9 characters for N and 653.8/655.5 for B/C. Do the findings remain after matched-length scoring and controlling for entity density, lexical diversity, and overlap with the supplied setting details? Where is the evidence of practically consequential detection degradation? Please report threshold-based false-negative rates or TPR at fixed FPR, with uncertainty intervals and a suitable matched human corpus. How should the claimed importance be interpreted alongside the high ensemble AUROC under P1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.