acceptodds
Under review as a conference paper at ICLR 2027

PeerReveal: Learning to Generalize in AI Peer Review Detection

Abstract

Can we distinguish fully AI-generated reviews from those using AI only to rewrite human judgement? Furthermore, can we do so reliably when reviews are generated from diverse AI models and workflows across scientific domains? To this end, we introduce PeerReveal, a new detector that grounds its decisions in learned relationships between a review's claims and reference claims drawn from AI-generated reviews of the same paper, complemented by review-level textual signals. We evaluate PeerReveal on PeerShift, a new, challenging test set of 5,430 reviews spanning multiple domains, including machine learning, life sciences, and social sciences, combining variation in disciplinary review conventions with diverse AI review-production mechanisms. With regular training, PeerReveal achieves 70.52% pooled Macro-F1 on this benchmark, improving over the strongest baseline by 11.48 points. We develop an effective and practical adversarial training framework that further improves the detector's generalization using a small local generator optimized to produce challenging reviews, raising pooled Macro-F1 from 70.52% to 82.40%. These results show that robust AI-review detection benefits from richer review-level evidence and adversarial training on more challenging generation processes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI-Text Detection, Peer Review, Adversarial Training, Reinforcement Learning, Large Language Models (LLMs)

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Hr4oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper builds a detector that tells you whether a human wrote a peer review, whether an AI rewrote it, or whether it was written entirely by an AI(LLM). The paper breaks each review into claims, compares them with AI reviews of the same paper, and trains against a tricky AI generator and claims generalisation.

Critical strengths

The detector beats the earlier detector on that detector’s own test by a big gap. They compare it against many competitors, and they’re honest about where it makes mistakes. Adding the generated reviews boosts several detectors, regardless of the reason. The numbers I checked add up. Also, their effort to create a dataset of tricky AI-generated reviews, and the many methods and criteria they considered for capturing the essence of AI reviews, are commendable.

Critical weaknesses

The evaluation which the paper chose would benefit its own detector and work against other detectors. This is because their search picked writing recipes that were hard for a group of detectors, including the earlier detector and two small claim-based models. The dev data used for model selection uses the same recipes on different papers. The AI-generated and AI-rewritten test reviews are synthetic, not real people's drafts polished by AI. So performance on unseen review workflows isn't really tested.

If you can specify which detector checkpoint supplied the generator's reward, and whether the development writers overlap with the test writers.

One important point is that generalisation across domains is never tested. The detector trains on all three domains and is tested on the same three. The authors also didn't run held-out-style experiments with retraining. Leave-one-domain-out, and holding out one recipe or writer family at a time. The authors describe detector training as cheap, so this looks feasible.

Aslo no evidence has demonstrated that the adversarial component is beneficial. The paper didn't add the same amount of ordinary data from the untrained generator, and the rewrite data comes from an untrained model with the detectors only choosing among candidates. Please run that control. The trained generators also still often evade the final detector, depending on the domain.

Smaller points. The simple text classifier gets no reference reviews or text statistics, so part of the gap is about inputs. Removing the references costs only a few points in a single run. Could you show what happens when each review draws references from a different paper, and report results across several training seeds?

Questions for authors

See weaknesses.

2 comments

Dear Reviewer,

We appreciate your careful reading and the concrete experiments you suggested, all of which we have now run.


The evaluation may favour PeerReveal, since the search picked recipes that are hard for a panel of detectors.

PeerReveal was not in the search panel, but the panel’s two claim-based models are our own earlier architectures (Appendix A.1), so the search also targeted our own method family. If the search only exploited the panel, detectors outside it would find PeerShift easier, but they do not: the eight baselines never used in the search reach 35–87 AUROC, like the panel’s Sem-Detect and RoBERTa (77.9 and 77.6), while PeerReveal reaches 98.0 (Table 1). Finally, on two external benchmarks, PeerReveal is still the best detector (response to Reviewer eQKw, W1).

The test reviews are synthetic, so unseen real review workflows are not tested.

We agree that PeerShift's non-human reviews come from our own pipelines: real AI review drafts cannot be collected at scale. However, every rewrite is reconstructed from a real human review (Section 3.1). To test unseen real workflows, we ran two public frontier reviewing agents (Google Deep Review, Stanford Agentic Reviewer); PeerReveal is the only detector that labels every one of their reviews as AI-generated (response to Reviewer 4RN6). On the two external benchmarks (response to Reviewer eQKw, W1), in which real human reviews are polished by other LLMs, PeerReveal labels 82.5% of these polished reviews as AI-rewritten.

Which checkpoint supplied the reward, and do development and test writers overlap?

  • Reward. The reward came from an earlier PeerReveal checkpoint, trained only on the ML regular corpus. It reaches 96.8 / 94.7 / 89.2 Macro-F1 on Sem-Detect's three test sets, and stayed frozen during generator training.
  • Writers. Yes: the development and test sets use the same 13 writer models. To check whether this matters, we (i) removed each writer's development data and re-ran checkpoint selection, which chose the same checkpoint in 57 of 60 cases, and (ii) held out whole writer families (see below).

Generalisation is not tested by holding out a domain, recipe or writer family.

  • Domains. We retrained PeerReveal and PeerReveal-Base with one domain held out from training and development; three-class Macro-F1 on that domain:

    Held-out domainPeerRevealPeerReveal-BasePeerReveal, all domainsSem-Detect, all domains
    ML68.860.177.851.5
    Nature Communications74.972.480.955.6
    PeerJ76.768.984.860.2

    An unseen domain costs PeerReveal 6–9 points, but adversarial training still helps there (+2.5–8.7 over PeerReveal-Base), and PeerReveal still beats Sem-Detect trained on that domain.

  • Writer families. When one of five writer families is removed from training and development, 98.7–100% of its PeerShift reviews are still recognised as non-human, and overall Macro-F1 stays at 79.4–81.5 (81.2 with full training).

  • Recipes. Recipes generate only development and test reviews, never training data (Section 3.3), so all are already held out from training; removing each from the development set as well keeps the same checkpoint in 40 of 40 cases.

The benefit of the adversarial component is not shown: compare with the same amount of ordinary data from the untrained generator.

We ran this control. The untrained Qwen3.5-9B generated the same amount of ordinary reviews with the regular procedure, replacing the adversarial data in the second training stage. Starting from the same PeerReveal-Base checkpoint (71.0), the detector reaches only 72.4 with this ordinary data, against 81.2 with the adversarial data. The gain therefore comes from the adversarial framework, not from more data from the same model.

The trained generators still often evade the final detector.

We agree; Table 4 and Tables 12–13 report these evasion rates. A fixed detector can always be targeted by an adversary trained against it, so robustness in detection is an eternal goal. We will discuss this in the Limitations section (Section 7).

The text classifier lacks references and text statistics; what if references come from a different paper, and is this stable across seeds?

We ran both experiments with five training seeds:

ModelMacro-F1 (5 seeds)
PeerReveal, all inputs81.5 ± 1.1
Target review only (no references, no text statistics)79.8 ± 1.4
References from a different paper79.9 ± 0.5
RoBERTa, same data73.0

Even with no extra inputs, PeerReveal is about 7 points ahead of RoBERTa, so the gap is not mainly about inputs; paper-matched references add about 1.6 points.


We hope our answers have addressed your concerns. We have also updated our PDF. Please let us know if any further clarification or additional information is needed.

Dear Reviewer,

As the author-discussion period is nearing its end, we want to follow up once more and ask whether there are any remaining concerns or points we could clarify further. We would be happy to address any additional questions.

Thank you again for your time and feedback.

Reviewer_eQKwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PeerReveal, an AI text detector for the peer review domain which aims to distinguish fully AI-generated reviews from those where AI is only used to rewrite a human review. PeerReveal compares a target review's claims to those which are produced by AI-generated reviews of the same paper and also leverages textual features to perform a three-label classification (human, AI-rewritten, or fully AI). A new challenging peer review test set called PeerShift was also constructed using an automated quality-constrained search methodology. Finally, the authors introduce an adversarial training framework to further improve the generalizability of the classifier by optimizing a small LLM generator to produce difficult training examples. Experiments are conducted using Sem-Detect and the newly introduced PeerShift dataset which show that PeerReveal generally outperforms alternative methods.

Critical strengths
  1. The technical contributions of this work are strong and it addresses an important & timely problem: distinguishing different levels of AI use in the peer review process. I expect that the proposed method and dataset will be valuable for future efforts to address this open problem

  2. Experimental results demonstrate that the proposed PeerReveal method outperforms a wide range of existing methods on PeerShift as well as the single baseline for Sem-Detect. Additional results show that the adversarial training framework successfully increases the ability of a small local generator to evade existing methods.

Critical weaknesses
  1. The main experimental results comparing PeerReveal to existing detectors are only conducted on PeerShift, which is the new benchmark that the authors themselves developed. The results would be more convincing if PeerReveal were compared to other detection algorithms on existing benchmarks as well. This is important considering that PeerShift was constructed using an automated search mechanism and appears to be lacking human validation of its quality (how can we be sure that the AI-rewritten reviews are faithful to the original human reviews in content?)

  2. The adversarial training framework was evaluated using only a single generator model (Qwen3.5-9B) so it is unclear how well this approach will generalize to other model families & sizes.

Questions for authors
  1. Have you considered hybrid authorship scenarios where a mix of AI rewriting and AI generation of the review is used? This seems quite likely to occur in practice as human reviewers supplement their own judgment with that produced by an LLM. How do you expect PeerReveal to perform in such cases?

  2. How did you come up with the weights for Equation 4 (lines 322-323)?

  3. Are all of the human-written reviews you collected for PeerShift from before LLMs gained widespread popularity? If not, how did you ensure that human-labeled reviews were not AI written or edited?

4 comments

Dear Reviewer,

Thank you for the detailed and thoughtful questions, which we address below. We have also updated our PDF.


W1(a). Detectors are only compared on PeerShift, the benchmark created by the authors.

We agree that comparisons on existing benchmarks are important. Section 6.3 (Table 3) already compares PeerReveal-Base with Sem-Detect on Sem-Detect's three test sets, with gains of 11.8–21.6 Macro-F1 points.

Following your suggestion, we additionally evaluated PeerReveal and prior detectors on two recent peer-review benchmarks built by other groups, Saha et al. (ICML 2026)[1] and PeerPrism (SIGIR 2026)[2]. Both contain human-written, LLM-polished and fully LLM-generated reviews, matching our three classes:

Three-class Macro-F1PeerRevealPeerReveal-BaseSem-DetectEditLens†TF†Fast-DetectGPT†ANCHOR†
Saha et al.: PeerRead reviews from 2013–2017, hard subset, 158 papers67.862.952.764.155.554.945.8
PeerPrism: ICLR/NeurIPS 2021–24, 153 papers78.275.956.760.947.432.847.6

† These detectors output a single AI-likeness score. For a fair comparison, we split it into three classes with the two cut points that maximize its Macro-F1 on our calibration set (Appendix D.6).

PeerReveal outperforms every prior detector in the table.

[1] https://openreview.net/forum?id=znFIR7WFGt
[2] https://doi.org/10.1145/3805712.3808602

W1(b). "(…) how can we be sure that the AI-rewritten reviews are faithful to the original human reviews in content?"

Every rewrite is reconstructed from a real human review of the same paper (Section 3.1), and that source review is also in the test set, so each pair can be checked directly:

  • Like Sem-Detect’s rewrites, each rewrite starts from one human review and keeps its content: its claims match the source with a mean similarity of 78.4, close to the 81.2 of Sem-Detect’s rewrites.
  • An LLM judge (Claude Sonnet 4.6) found that 99.5% of all 1,810 rewrites keep the reviewer’s judgement.

W2. The adversarial framework is only tested with one generator, Qwen3.5-9B.

To test whether the framework generalizes to another model family, we trained a GLM-4-9B generator on the ML domain with the same framework, rewards and steps. Starting from the same PeerReveal-Base checkpoint, we then repeated the adversarial training stage with the GLM-generated reviews in place of the Qwen-generated ones. The resulting detector reaches 79.9 Macro-F1, close to the 81.2 obtained with the Qwen data and well above PeerReveal-Base (71.0).

Q1. How does PeerReveal perform on hybrid authorship reviews that mix AI rewriting and AI generation?

We agree this is likely in practice and ran a new experiment. For 558 papers, we replaced 25 / 50 / 75% of the claims in the paper's AI-rewritten review (0%) with claims from its fully AI-generated review (100%), keeping the review coherent:

AI-generated share0%25%50%75%100%
Labelled AI-generated by PeerReveal7%18%31%48%94%

PeerReveal labels more reviews as AI-generated as the AI-generated share grows, and labels the rest mostly as AI-rewritten, which is the correct label when most of the review’s judgement still comes from the human reviewer. The other detectors barely respond: RoBERTa, for example, labels almost none of the 0–75% reviews as AI-generated (0–7.5%). Since PeerReveal reasons at the level of individual claims, an interesting next step is to identify which parts of a review were written by the human reviewer and which by the LLM. We will add this experiment to the appendix and discuss this direction as future work.

Q2. "How did you come up with the weights for Equation 4?"

The weights were fixed before generator training (Eq. 4, Appendix B.1). PeerReveal receives the largest weight (.40) because it is the detector the generated data is meant to improve. Sem-Detect receives .30 as the strongest prior three-class detector. RADAR and ANCHOR receive .15 each, adding two further detector families so that the generator does not overfit to one detector family.

Q3. Are all of the human-written reviews from before LLMs gained widespread popularity?

Not all of them. The ML human reviews are from ICLR/NeurIPS 2021–22, before ChatGPT's release. For Nature Communications and PeerJ, too few pre-ChatGPT reviews with matching manuscripts are available, so we also use later ones. AI use is also much lower outside ML: Liang et al. (2024, Figure 4)[3] find no significant increase of AI-modified content in Nature Portfolio reviews after ChatGPT’s release. To be safe, we retrained without any post-ChatGPT human review, and Macro-F1 is unchanged (81.25 vs 81.29 with the same seeds).

[3] https://arxiv.org/pdf/2403.07183


We hope these new results address your concerns, and we would be glad to clarify anything further.

Dear Reviewer,

As the author-discussion period is nearing its end, we want to follow up once more and ask whether there are any remaining concerns or points we could clarify further. We would be happy to address any additional questions.

Thank you again for your time and feedback.

Thanks for your response to my review. While I appreciate the new results, my concerns have not been fully addressed. The rebuttal table omits RoBERTa, which Table 5 shows is the strongest baseline on both benchmarks. With it included, PeerReveal’s lead is 1.2 and 0.3 points. Considering that these are single runs and within seed variation, it appears that much of the advantage seen on PeerShift does not transfer to independent data. PeerReveal also labels half of Saha et al.’s fully AI-generated reviews as AI-rewritten.

Regarding rewrite faithfulness, the only evidence presented is an automated LLM judge so this is still lacking in human validation. The rebuttal cites 99.5% judgement preservation, but the same judge rates only 91.2% of rewrites faithful overall.

Finally, the new hybrid authorship results raise questions about how cases which predominately consist of AI-generated content should be handled. At 75% of AI-generated claims, most of the reviews are not labeled as AI-generated, which is concerning.

Based on the above, I have decided to maintain my original assessment.

Dear Reviewer,

Thank you for your follow-up and for reading our revised PDF closely.

RoBERTa. Thank you for checking Table 5; RoBERTa should also have been in our rebuttal table. Unlike the other baselines, it is not a published detector: we trained it on our data, and its adversarial version on reviews from our adversarial framework, which adds 5.0 points on PeerPrism, so its strength also reflects our framework. The best published detector, EditLens, reaches 64.1 and 60.9, against 67.8 and 78.2 for PeerReveal. The AI-generated reviews PeerReveal misses are labelled AI-rewritten: only 1.7% are mistaken for human, the fewest of all detectors.

Rewrite faithfulness. We agree that a human check would help. Our claim similarity (78.4, close to the 81.2 of Sem-Detect's rewrites) is computed from embeddings, not by an LLM judge. The 91.2% criterion is strict: one "unfaithful" rewrite, for example, merges four line-level citation fixes into one general remark while keeping every main concern and the reject verdict. The reviewer's judgement is kept in 99.5% of rewrites.

Hybrid reviews. At 75% AI-generated claims, PeerReveal still flags 97.7% of reviews as AI-involved (47.6% AI-generated, 50.1% AI-rewritten), and its AI-generated share rises steadily with the AI-generated content. RoBERTa labels only 0–7.5% of the 0–75% reviews as AI-generated. A mixed review has no single correct label among three classes; attributing individual claims, which PeerReveal already represents separately, is the natural next step (Section 7).

We hope these additional results help, and thank you again for engaging with our rebuttal.

Reviewer_4RN6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for detecting the provenance of scientific peer reviews by distinguishing among three categories: human-written reviews, AI-rewritten reviews based on human judgments, and fully AI-generated reviews. The authors address the challenge of generalizing AI-text detection across different language models, reviewing workflows, and scientific domains. The paper introduces a claim-level detection architecture learning interactions between claims extracted from a target review, and claims extracted from AI-generated reference reviews of the same manuscript, combined with review-level textual features. It also proposes an adversarial training framework to generate challenging training examples, and a benchmark containing reviews across machine learning, Nature Communications, and PeerJ Psychology and Social Sciences. The experimental results are promising and better than similar work.

Critical strengths
  1. The paper explicitly distinguishes fully AI-generated evaluations from human evaluations rewritten by an LLM. This distinction is valuable because language polishing and automated scientific judgment represent substantially different forms of AI involvement.
  2. The proposed architecture moves beyond whole-review similarity and predefined aggregate statistics by learning relationships between individual claims in target and reference reviews.
Critical weaknesses

The central claim of the paper concerns generalization beyond known generation procedures. However, there is an important distinction between generalization to new papers or writer models and generalization to entirely unseen mechanisms. Although test papers are disjoint and some writer models are unseen during training, the model-selection process has access to development examples produced using the same mechanisms as the final test set. This is not a major issue, but it weakens the stronger interpretation that the detector is robust to entirely unknown review-production workflows.

Questions for authors

none

2 comments

Dear Reviewer,

Thank you for your positive assessment and for pointing out an important distinction.


"(…) the model-selection process has access to development examples produced using the same mechanisms as the final test set."

We agree with this distinction. PeerShift's generation mechanisms never appear in training; they appear only in the development set used for checkpoint selection (Appendix D.3). We measured how much this exposure matters:

SettingResult
Development-based checkpoint selection (5 seeds)81.5 ± 1.1 Macro-F1
No development data: always take the last epoch (5 seeds)80.1 ± 0.8 Macro-F1
Each mechanism removed from the development set, selection re-run (40 cases)Same checkpoint in 40/40

We have revised the claim in our PDF: generation mechanisms are unseen in training, and their presence in checkpoint selection accounts for at most about 1.4 points.

To further test unseen real generation workflows, we also ran two public frontier LLM reviewing agents on PeerShift ML test papers: Google Deep Review (https://research.google/deepreview/) and the Stanford Agentic Reviewer (https://paperreview.ai/). Both search the literature before writing. Without retraining, PeerReveal labels all of their reviews as AI-generated, outperforming all other baselines (reviews labelled AI-generated / reviews):

DetectorGoogle Deep ReviewStanford Agentic Reviewer
PeerReveal10/1025/25
PeerReveal-Base4/1025/25
Sem-Detect regular0/1014/25
Sem-Detect adversarial2/1020/25
RoBERTa regular1/1025/25
RoBERTa adversarial3/1025/25

We hope this addresses your concerns, and we’re happy to discuss further.

Dear Reviewer,

As the author-discussion period is nearing its end, we want to follow up once more and ask whether there are any remaining concerns or points we could clarify further. We would be happy to address any additional questions.

Thank you again for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.