acceptodds
Under review as a conference paper at ICLR 2027

Two Asymmetries of LLM Self-Review: Family-Dependent Recall Gaps and Same-Model False-Positive Bias

Abstract

We study self-review as an empirical problem in LLM-as-a-judge settings, using a programmatic injection log as ground truth across a 4×4 writer–verifier design (400 technical reports, 286 flawed and 114 clean, 1,600 reviews) and a 9-verifier pilot (50 instances, 4,345 reviews). We report two asymmetries. First, recall gaps are model-dependent, not universal: in a logistic regression with cluster-robust standard errors clustered by instance, self-review lowers detection odds overall (log-odds -0.60, p=0.00011), but this effect is model-dependent rather than a uniform property: the GPT-family verifiers consistently show a self-review blind spot (odds ratio 0.55), whereas DeepSeek (OR 1.72) and Google (OR 1.70) verifiers are stricter on their own outputs—yet within Google the direction splits (gemini-2.5 verifiers stricter on self, gemini-3-pro-preview blind), so even vendor family is not a reliable unit. Second, error type is asymmetric: at comparable recall (self 73.9% vs. cross 72.6%, sign test p=0.70), self-review produces substantially more false positives on clean reports (FPR 16.0% vs. 0.0%, two-sided exact McNemar p=0.0078), and a same-model fresh-session condition (FPR 18.0%) is consistent with model self-similarity rather than authorship as the source of this bias. A controlled re-parsing study rules out verdict-extraction artifacts. Finally, a counterbalanced probe holds report content fixed and varies only the authorship label (self/other/none); it finds no reliable effect of the label on recall or false-positive rate (paired McNemar p=0.115 and p=0.273, respectively), indicating that the apparent self-review blind spot is a content-ownership effect rather than a label effect. These results support an empirical, non-causal conclusion: self-review reliability must be validated per model, and verifier selection should consider both missed defects and false alarms rather than treating model separation as a proxy for quality.

Primary area:
datasets and benchmarks
Keywords:
self-review, LLM-as-a-judge, cross-model verification, hallucination detection, false positive rate

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_GuHqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether LLMs review their own outputs differently from other models’ outputs. A crossed writer–verifier experiment finds model-dependent recall gaps, while a matched probe reports more false positives in same-model review.

Critical strengths

The crossed evaluation reveals pairing-specific differences hidden by aggregate results. Measuring false positives alongside recall provides a useful basis for assessing whether additional verification improves reliability.

Critical weaknesses
  1. The paper does not apply ICLR official template, and the main paragraph exceeds 9 pages.

  2. For a fixed verifier, self and cross conditions contain different writers’ reports. The regression includes model family, self-review, and their interaction, but no writer or report-difficulty controls. A writer whose defects are harder for every verifier could appear to have a self-review blind spot.

  3. Clean labels are based on the absence of injected defects, so valid criticisms of pre-existing errors could be counted as false positives. Binary FLAWED verdicts also do not identify which defect was detected. In the blinded audit, one annotator achieved 46% accuracy across 50 reports.

  4. The main experiment’s reported recall and balanced accuracy imply approximately 12% self-review FPR versus 19% cross-review FPR, whereas the matched probe reports 16% versus 0%. Clean samples are unevenly distributed across writers, so pooling self and cross conditions assigns different weights to verifier identities.

Questions for authors

Do the model-specific self-review gaps persist after accounting for report or task difficulty and verifier identity?

How were clean reports independently validated, and how were individual defect detections established from binary verdicts? Please also clarify how Section 4.5 obtained 200 clean reports from a main benchmark containing 114.

The main experiment’s reported recall and balanced accuracy imply approximately 12% self-review FPR versus 19% cross-review FPR, opposite to the matched probe’s 16% versus 0%. What differences in model composition, sampling, or review conditions explain this reversal?

Reviewer_zBBtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the reliability of LLM self-review versus cross-model verification. Using a writer–verifier framework with injected errors, the authors find that self-review recall gaps vary across models rather than following a universal pattern. They also observe higher false-positive rates in self-review at comparable recall. The findings suggest that independent verification is not always more reliable and that verifier selection should account for both error detection and false alarms.

Critical strengths

The research question in this paper is worthwhile. Exploring whether LLM self-review reveals systematic mechanisms beyond model-specific performance differences could offer valuable insights into LLM-based verification and help improve the reliability of LLM/agent systems.

Critical weaknesses
  1. Insufficient experimental control. The self-review and cross-review settings involve reports generated by different models, which may differ in difficulty, style, and error distribution. This makes it difficult to isolate the effect of self-review itself. Moreover, self-review is widely used in different tasks, such as code generation, mathematical reasoning, and factual question answering. The paper does not clearly justify why technical reports are chosen as the primary evaluation setting. Although the paper includes a HumanEval experiment, the evaluation still lacks a systematic comparison across task types. These limitations raise concerns about whether the findings can generalize to broader self-review scenarios.

  2. Unclear research gap and limited novelty. The paper does not clearly establish what is missing from existing research. The main observation that independent verification does not always improve reliability is also relatively intuitive, as prior work generally evaluates the effectiveness of additional verifiers experimentally. It remains unclear what substantial new insight this paper contributes beyond these observations.

Questions for authors
  1. Experimental Control: How do the authors ensure that the observed differences between self-review and cross-review are not caused by variations in report difficulty, writing style, or decoding strategy across different writer models?

  2. Task Selection: Why were technical reports chosen as the primary evaluation task? Given that self-review is also widely used in code generation, mathematical reasoning, and other tasks, how representative is this setting?

  3. Research Gap and Novelty: What specific gap in existing research does this paper aim to fill? Since prior work generally evaluates the effectiveness of independent verifiers experimentally, what new insight does this study provide beyond showing that cross-model verification is not always beneficial?

  4. Interpretation of Results: Given that the aggregate recall of self-review and cross-review is nearly identical (31.5% vs. 32.4%), and some model-level trends reverse across experiments, how should these findings be interpreted? Are the observed asymmetries primarily model-dependent, task-dependent, or both?

  5. Reproducibility: The paper acknowledges that exact model snapshots, decoding parameters, complete API logs, and repeated runs were not recorded. How can the authors establish that the observed differences are stable and reproducible, particularly when some reported gaps are relatively small?

  6. Presentation and Readability: Could the authors simplify the terminology and improve the presentation of the results? The paper introduces several terms, such as “content-ownership effect” and “model self-similarity,” without sufficiently clear explanations. Tables 1 and 3 also contain overlapping information, while Figures 1 and 2 have labels and captions that overlap with the plotted elements. These issues collectively make the paper difficult to follow.

Reviewer_VfT7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether LLMs judge their own outputs differently from other models’ outputs. Four models write technical reports, a script injects known errors into some of them, and every model then labels every report as CLEAN or FLAWED. On top of this 4×4 design, the authors add several further experiments.

The claims are as follows. Self-review recall gaps depend on the model: GPT verifiers miss more of their own errors, while DeepSeek and Gemini catch more. Self-review produces more false positives at similar recall. The authorship label alone has no effect.

Critical strengths

The question matters for anyone building self-correction or multi-agent pipelines, and the basic design is the right kind. It is a fully crossed writer × verifier setup with labels that don’t come from an LLM. Reporting false positives alongside recall, and adding fresh-session and label-only controls, are good instincts.

Critical weaknesses
  1. The phenomenon reported in the paper is not new, and the only new finding has not yet been confirmed.
  • LLM reviewers' self-bias is well documented (Zheng et al., 2023; Panickssery et al., 2024). Wataoka et al. (2024) have attributed it to familiarity rather than authorship. This is essentially the paper's "self-similarity/content ownership" explanation.

  • Different models have varying degrees of influence, and reviewers' scores should consider both false negatives and false positives, which is fairly standard practice.

  • A potential new finding is that models are more rigorous with their own outputs, contrary to self-bias. However, this finding is based only on 50 valid reports and compared with an unspecified cross-validator who did not produce any false positives.

  • Aside from "validating for each model," the paper does not provide any mechanisms or mitigation measures.

  1. Some of the paper's evidence is untenable, for example:
  • Label validity. The accuracy of blinded manual review was only 46%, comparable to the random level. Therefore, it remains unclear whether injected errors are detectable, or whether the “clean” reports prepared by LLM are truly clean.

  • Self- and cross-comparisons never controlled for the difficulty of each author's report. When I refitted Table 3 using author and validator fixed effects, the mean self-effect was approximately -0.09 (p ≈ 0.6), and the GPT gap narrowed to approximately 8–10 percentage points.

  • Data inconsistency. Figure 1 gives OR values ​​of 1.54 and 3.70, while the values ​​in the text are 1.72 and 1.70. The -0.60 for the “population” is actually the GPT reference level coefficient.

Also: The paper doesn't correctly follow the official template.

Questions for authors

Please kindly refer to the weaknesses. It is unlikely that I will change my rating during the rebuttal phase in its current form.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.