- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether authors can make subsequent AI-generated replies easier to detect by rephrasing their own messages while preserving their meaning. REAIR uses detector feedback on trial replies to select source rephrasings. Experiments on discussion posts and research proposals show detection gains.
Critical strengths
The paper presents an interesting intervention that gives authors a way to influence the detectability of subsequent AI replies. The substantial gains under matched settings are supported by separate final-reply evaluation, a manual semantic audit, and analyses controlling source and reply length. The transfer experiments also provide useful evidence about the approach’s limitations.
Critical weaknesses
The human comparison does not establish how well the method distinguishes human from AI replies when both respond to the rephrased sources. AI replies are generated from rephrased sources, but the human references respond only to the original sources. Rephrasing could also change human response style and hence false-positive rates. Although the paper explicitly discloses this design, neither the reported detection rates nor AUROC establish whether human and AI replies become easier to distinguish when both respond to the rephrased sources.
The practical applicability remains limited by dependence on the reply configuration used during optimization. The tested changes of reply model or instruction yield no statistically significant improvement in fixed-threshold detection rate, and some reduce detection, although some instruction-transfer AUROC gains remain. Improvements are also substantially smaller under sampled generation. These limitations matter because authors seeking to detect undisclosed AI use generally do not know the respondent’s model or instructions.
Questions for authors
Is there evidence about how rephrasing affects human replies? In particular, would the detection gains persist when both human and AI replies are collected for rephrased sources, with false-positive rates measured in that condition?
Given the limited transfer gains in Section 4.4, what deployment setting do the authors envision? Is the approach intended primarily for cases where the respondent’s model and instructions are known, or is there evidence that it remains effective when these are unknown during source optimization?
4 comments
Follow-up on Our Rebuttal
Dear Reviewer,
Thank you again for your time and thoughtful feedback on our paper. We have provided detailed responses and additional experimental results in our rebuttal, and would appreciate hearing whether these address your questions and concerns.
If any points remain unclear or require further discussion, please let us know. We would be happy to clarify them and discuss any remaining concerns.
Thank you for your time and consideration.
Best regards, The authors of Paper 48244
I thank the authors for their response. The authors provided a new pilot, which deserves credit. There is improvement from 0.87 to 0.91 in AUROC, however only 10 replies per condition from one person provides rather little evidence about the range of human responses. It does give positive signal but falls short of fully addressing the point. I am not requesting new experiments as this is last day, but this uncertainty remains.
On transfer, the detection-rate improvements are helpful evidence. It’s still unclear whether the improvement is sufficiently reliable across plausible unknown configurations given the small held out test and mixed detection rate resultss across models.
With these concerns I view my original concern as partly addressed. Since score increments jumps directly to weak positive, I think the response is helpful though does not quite justify raising my score.
Response to Reviewer jbeH’s Follow-up
Thank you for your follow-up and for recognizing the positive signal from our human-reply pilot and the helpful evidence provided by the transfer experiments.
Due to the limited rebuttal timeframe, these additional studies were conducted as pilots. We plan to include a larger human-reply study and evaluations across a broader range of reply configurations in the revised manuscript to further validate these encouraging observations.
Thank you again for your time and constructive feedback.
Response to Reviewer jbeH
Thank you for the review. We address the human-reply comparison and the intended deployment setting below.
Our fixed-threshold AI-TPR gains do not depend on human replies: With the detector and threshold fixed, TPR measures the proportion of AI replies classified as AI. Changes in human replies therefore cannot alter our measured AI-TPR improvement.
Changes in human replies do not necessarily increase the false-positive rate: Human replies can differ across people or change when the source is rephrased. These differences in wording or style do not necessarily make the replies more likely to be classified as AI. Detection gains can therefore persist even when human replies change.
Comprehensively studying how rephrasing affects human replies requires substantial time and effort: Reliably measuring changes in the human-reply distribution requires substantive replies to both original and rewritten sources from comparable participants. This separates the effect of rephrasing from differences between people. Designing and conducting such a dedicated study requires substantial additional time and effort.
Human pilot: AUROC improves numerically when both humans and AI reply to rephrased posts: To examine whether source rephrasing affects human reply scores and false positives, we collected replies to 20 CMV posts, randomly assigned to Original or frozen BoN-8 rephrasing (10 each). One author wrote one reply per post, seeing only one version without knowing whether it was original or rephrased. We scored all 20 replies with the same Binoculars detector.
Human replies to rephrased posts have higher mean Binoculars scores, indicating weaker AI evidence on average. The two-sided test finds no significant difference in mean scores. Although the rephrased group has one additional false positive, this small pilot does not establish that rephrasing systematically increases human false positives.
The rephrased group has a higher AUROC. The result is an encouraging signal that detection gains may persist when both humans and AI reply to rephrased posts.
Envisioned deployment: authors use likely reply configurations to guide source rephrasing and improve AI-reply detection: An author preparing a discussion post or research proposal may not know the respondent's exact configuration, but can identify a set of likely models and prompts. We envision authors using REAIR to rephrase their posts or proposals based on likely reply models and prompts, with the aim of making subsequent AI replies easier to detect.
Joint selection improves AUROC numerically across the tested model set: To test the envisioned setting, we used feedback from several likely reply models to select one shared rephrasing, then evaluated it across those models.
For each of 20 CMV posts, we selected among 100 rewrites using trial replies from Qwen, Gemma and Ministral at T=0.7. A candidate had to improve the Binoculars score over Original for all three models. We chose the candidate with the largest average improvement. If none qualified, we kept Original. We then froze the selections and generated three fresh replies per model to both Original and the selected version.
The same selected rewrites improve AUROC numerically for all three models, while TPR changes are mixed. This provides preliminary evidence for selecting against a set of possible models without knowing the respondent’s exact choice.
REAIR shows numerical transfer gains and offers a starting point for methods robust to unknown models and prompts: In Qwen3-32B prompt transfer, both TPR and AUROC improve numerically in seven of eight method–temperature comparisons, and AUROC improves in all eight (Table 24). These results provide an encouraging signal for extending detection gains beyond the optimization prompt. REAIR provides a first step and a common framework for studying author-side detection enhancement. Our next work will focus on stronger methods that are robust to unknown reply models and prompts.