- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper builds a detector that tells you whether a human wrote a peer review, whether an AI rewrote it, or whether it was written entirely by an AI(LLM). The paper breaks each review into claims, compares them with AI reviews of the same paper, and trains against a tricky AI generator and claims generalisation.
Critical strengths
The detector beats the earlier detector on that detector’s own test by a big gap. They compare it against many competitors, and they’re honest about where it makes mistakes. Adding the generated reviews boosts several detectors, regardless of the reason. The numbers I checked add up. Also, their effort to create a dataset of tricky AI-generated reviews, and the many methods and criteria they considered for capturing the essence of AI reviews, are commendable.
Critical weaknesses
The evaluation which the paper chose would benefit its own detector and work against other detectors. This is because their search picked writing recipes that were hard for a group of detectors, including the earlier detector and two small claim-based models. The dev data used for model selection uses the same recipes on different papers. The AI-generated and AI-rewritten test reviews are synthetic, not real people's drafts polished by AI. So performance on unseen review workflows isn't really tested.
If you can specify which detector checkpoint supplied the generator's reward, and whether the development writers overlap with the test writers.
One important point is that generalisation across domains is never tested. The detector trains on all three domains and is tested on the same three. The authors also didn't run held-out-style experiments with retraining. Leave-one-domain-out, and holding out one recipe or writer family at a time. The authors describe detector training as cheap, so this looks feasible.
Aslo no evidence has demonstrated that the adversarial component is beneficial. The paper didn't add the same amount of ordinary data from the untrained generator, and the rewrite data comes from an untrained model with the detectors only choosing among candidates. Please run that control. The trained generators also still often evade the final detector, depending on the domain.
Smaller points. The simple text classifier gets no reference reviews or text statistics, so part of the gap is about inputs. Removing the references costs only a few points in a single run. Could you show what happens when each review draws references from a different paper, and report results across several training seeds?
Questions for authors
See weaknesses.
2 comments
Dear Reviewer,
As the author-discussion period is nearing its end, we want to follow up once more and ask whether there are any remaining concerns or points we could clarify further. We would be happy to address any additional questions.
Thank you again for your time and feedback.
Dear Reviewer,
We appreciate your careful reading and the concrete experiments you suggested, all of which we have now run.
The evaluation may favour PeerReveal, since the search picked recipes that are hard for a panel of detectors.
PeerReveal was not in the search panel, but the panel’s two claim-based models are our own earlier architectures (Appendix A.1), so the search also targeted our own method family. If the search only exploited the panel, detectors outside it would find PeerShift easier, but they do not: the eight baselines never used in the search reach 35–87 AUROC, like the panel’s Sem-Detect and RoBERTa (77.9 and 77.6), while PeerReveal reaches 98.0 (Table 1). Finally, on two external benchmarks, PeerReveal is still the best detector (response to Reviewer eQKw, W1).
The test reviews are synthetic, so unseen real review workflows are not tested.
We agree that PeerShift's non-human reviews come from our own pipelines: real AI review drafts cannot be collected at scale. However, every rewrite is reconstructed from a real human review (Section 3.1). To test unseen real workflows, we ran two public frontier reviewing agents (Google Deep Review, Stanford Agentic Reviewer); PeerReveal is the only detector that labels every one of their reviews as AI-generated (response to Reviewer 4RN6). On the two external benchmarks (response to Reviewer eQKw, W1), in which real human reviews are polished by other LLMs, PeerReveal labels 82.5% of these polished reviews as AI-rewritten.
Which checkpoint supplied the reward, and do development and test writers overlap?
Generalisation is not tested by holding out a domain, recipe or writer family.
Domains. We retrained PeerReveal and PeerReveal-Base with one domain held out from training and development; three-class Macro-F1 on that domain:
An unseen domain costs PeerReveal 6–9 points, but adversarial training still helps there (+2.5–8.7 over PeerReveal-Base), and PeerReveal still beats Sem-Detect trained on that domain.
Writer families. When one of five writer families is removed from training and development, 98.7–100% of its PeerShift reviews are still recognised as non-human, and overall Macro-F1 stays at 79.4–81.5 (81.2 with full training).
Recipes. Recipes generate only development and test reviews, never training data (Section 3.3), so all are already held out from training; removing each from the development set as well keeps the same checkpoint in 40 of 40 cases.
The benefit of the adversarial component is not shown: compare with the same amount of ordinary data from the untrained generator.
We ran this control. The untrained Qwen3.5-9B generated the same amount of ordinary reviews with the regular procedure, replacing the adversarial data in the second training stage. Starting from the same PeerReveal-Base checkpoint (71.0), the detector reaches only 72.4 with this ordinary data, against 81.2 with the adversarial data. The gain therefore comes from the adversarial framework, not from more data from the same model.
The trained generators still often evade the final detector.
We agree; Table 4 and Tables 12–13 report these evasion rates. A fixed detector can always be targeted by an adversary trained against it, so robustness in detection is an eternal goal. We will discuss this in the Limitations section (Section 7).
The text classifier lacks references and text statistics; what if references come from a different paper, and is this stable across seeds?
We ran both experiments with five training seeds:
Even with no extra inputs, PeerReveal is about 7 points ahead of RoBERTa, so the gap is not mainly about inputs; paper-matched references add about 1.6 points.
We hope our answers have addressed your concerns. We have also updated our PDF. Please let us know if any further clarification or additional information is needed.