- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies document-channel manipulation in AI-assisted peer review, where authors can insert hidden or natural-looking text into a PDF to influence an LLM reviewer. Using a fixed text-extraction and review pipeline on ICLR and NeurIPS papers, the authors show that explicit instructions, naturalized intent, and evidence framing can all significantly shift review scores across multiple models. The paper also evaluates several input designs and finds that separating evidence extraction from the final review step is substantially more robust than directly concatenating extracted paper text into the reviewer prompt.
Critical strengths
-
Careful experimental controls. The paired clean versus modified design, matched-context comparisons, repeated reviews, and polarity reversal help isolate the effect of the inserted directional content.
-
Useful systems comparison. The paper compares multiple reviewer input designs and shows that separating extraction from review generation substantially reduces the measured attack effect.
Critical weaknesses
-
Unjustified truncation. The review pipeline truncates extracted text at 24,000 characters, but the paper does not provide a clear justification for this choice.
-
Unvalidated review system. The paper does not sufficiently validate or calibrate the AI reviewer against the original human review scores or decisions. In particular, it does not provide a direct mapping or correlation analysis between clean AI scores and native reviewer scores, making it difficult to judge whether the reported score shifts and threshold crossings are meaningful in a real peer review setting.
-
Limited novelty. The core idea of manipulating LLM reviewers through document content or hidden prompt injection has already been explored in a lot of works.
-
Clarity. The paper is difficult to follow due to dense terminology and an overly complicated presentation of the experimental setup.
Questions for authors
Please see Critical Weaknesses and answer first two questions.