- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether LLMs review their own outputs differently from other models’ outputs. A crossed writer–verifier experiment finds model-dependent recall gaps, while a matched probe reports more false positives in same-model review.
Critical strengths
The crossed evaluation reveals pairing-specific differences hidden by aggregate results. Measuring false positives alongside recall provides a useful basis for assessing whether additional verification improves reliability.
Critical weaknesses
-
The paper does not apply ICLR official template, and the main paragraph exceeds 9 pages.
-
For a fixed verifier, self and cross conditions contain different writers’ reports. The regression includes model family, self-review, and their interaction, but no writer or report-difficulty controls. A writer whose defects are harder for every verifier could appear to have a self-review blind spot.
-
Clean labels are based on the absence of injected defects, so valid criticisms of pre-existing errors could be counted as false positives. Binary FLAWED verdicts also do not identify which defect was detected. In the blinded audit, one annotator achieved 46% accuracy across 50 reports.
-
The main experiment’s reported recall and balanced accuracy imply approximately 12% self-review FPR versus 19% cross-review FPR, whereas the matched probe reports 16% versus 0%. Clean samples are unevenly distributed across writers, so pooling self and cross conditions assigns different weights to verifier identities.
Questions for authors
Do the model-specific self-review gaps persist after accounting for report or task difficulty and verifier identity?
How were clean reports independently validated, and how were individual defect detections established from binary verdicts? Please also clarify how Section 4.5 obtained 200 clean reports from a main benchmark containing 114.
The main experiment’s reported recall and balanced accuracy imply approximately 12% self-review FPR versus 19% cross-review FPR, opposite to the matched probe’s 16% versus 0%. What differences in model composition, sampling, or review conditions explain this reversal?