acceptodds
Under review as a conference paper at ICLR 2027

Document-Channel Manipulation in a Text-Extracted AI Review Pipeline

Abstract

AI-assisted peer review turns a submission PDF into both scientific evidence and an input channel for evaluation control. We connect document interventions, directional content controls, and four input designs in a fixed review pipeline using page-order pypdf extraction and a 24,000-character cap. On a frozen 456-paper holdout spanning ICLR 2024–2025 and NeurIPS 2024–2025, author-side payloads increase GLM-5's 1–10 score by for explicit instructions, for naturalized intent, and for evidence framing. Matched white-text cases reproduce positive effects on GLM-5 and GPT 5.2; Gemini-2.5-flash is inconclusive for explicit instruction. On an independent 84-paper reserve, repeated comparisons and polarity reversal support sensitivity to directional content under matched extracted context. Comparing four input designs, the pooled attack shift is for plain concatenation, with delimiters, with structured summaries, and (95% CI ) with separate extraction and review calls. Organizer-side audit markers also affect the review: natural-language marker reproduction remains after filtering. These results connect controlled content interventions to concrete input designs and distinguish score stability from marker suppression.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Document-Channel Manipulation in a Text-Extracted AI Review Pipeline

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ft8COpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies document-channel manipulation in AI-assisted peer review, where authors can insert hidden or natural-looking text into a PDF to influence an LLM reviewer. Using a fixed text-extraction and review pipeline on ICLR and NeurIPS papers, the authors show that explicit instructions, naturalized intent, and evidence framing can all significantly shift review scores across multiple models. The paper also evaluates several input designs and finds that separating evidence extraction from the final review step is substantially more robust than directly concatenating extracted paper text into the reviewer prompt.

Critical strengths
  1. Careful experimental controls. The paired clean versus modified design, matched-context comparisons, repeated reviews, and polarity reversal help isolate the effect of the inserted directional content.

  2. Useful systems comparison. The paper compares multiple reviewer input designs and shows that separating extraction from review generation substantially reduces the measured attack effect.

Critical weaknesses
  1. Unjustified truncation. The review pipeline truncates extracted text at 24,000 characters, but the paper does not provide a clear justification for this choice.

  2. Unvalidated review system. The paper does not sufficiently validate or calibrate the AI reviewer against the original human review scores or decisions. In particular, it does not provide a direct mapping or correlation analysis between clean AI scores and native reviewer scores, making it difficult to judge whether the reported score shifts and threshold crossings are meaningful in a real peer review setting.

  3. Limited novelty. The core idea of manipulating LLM reviewers through document content or hidden prompt injection has already been explored in a lot of works.

  4. Clarity. The paper is difficult to follow due to dense terminology and an overly complicated presentation of the experimental setup.

Questions for authors

Please see Critical Weaknesses and answer first two questions.

Reviewer_m9ihOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates automatic review pipelines with respect to author manipulation (e.g. prompt injections) and organizer marking. It uses a sample of ~450 original papers form ICLR and NeurIPS (years 24 and 25).

Critical strengths

The paper addresses a critical and timely research topic. While conferences like ICLR are flooded with AI generated content, using AI to streamline the review process might appear to be a solution. The paper gives a good indication why this is not (yet) the case.

Critical weaknesses

The paper suffers from several severe problems. First of all (human) readability. Already the abstract is a loose collection of highly detailed facts that make it very hard to follow. I have published on this topic, but even after reading the abstract and introduction several times I could not understand what the paper is investigate exactly and what the contributions are.

The mathematical formulation of the setup and evaluation appears to be highly over-complicated, providing a pseudo mathematical formulation for rather simple process pipelines.

The reported results in the tables are very hard to decipher interpret for a human - plots would probably work much in favor of the reader.

The novelty of the setup and findings remains unclear. What new insights beyond existing works like Murray (2025) or Keuper (2025) does this work provide?

The number of samples use is also quite low in comparison to related work.

The experimental setup neglects the fact that ICLR and NeurIPS use different sparse review scores in different years, e.g. a score of 7 might mean weak accept or accept or does not exist in some processes. Hence, the review scare have to be treated differently per venue and can not be compared directly.

Questions for authors

Did you use your reviewing pipeline on your own submission? What was the outcome? If the answer is an accept, then the pipeline clearly need to be adjusted towards human readability.

Reviewer_7gcHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the effects of manipulating AI review scores under multiple settings, including author-inserted texts and organizer-inserted audit markers. The experiment spans 456 holdout papers. The outcome is based on model-generated scores after a fixed process.

Critical strengths
  1. The paper used pair control by comparing clean and modified version of the same paper, averaging results across different insertion methods within each paper. On the separate set of 84 papers, the paper also presented controlled experiments and tests.

  2. The paper conducted experiments with multiple models, including GLM-5, GPT-5.2, Gemini-2.5-Flash, using the same papers, interventions and scoring rules. This reduces the finding less dependent on the choice of reviewer models.

Critical weaknesses
  1. I am not convinced by the comparison between full and neutralised document scores. In section 3, the author stated that rating direction, decision cues, unsupported empirical claims are removed together. This changes both the evaluation guidance and the scientific evidence available for the model. Therefore, the full-neutralized difference alone cannot show the cause of the score change, whether it is due to the evaluative cues, or added scientific claims, or other text changes.

  2. The near-zero attack shift cannot show that the review after the defence is accurate and useful. being stable does not mean that the review is better.

  3. The models selected are outdated, more than 6 months ago.

Questions for authors
  1. Can you also include the results from recent models?

  2. Can you further explore what changes after neutralisation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.