- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new explanation for the self-preference of VLM judge models, arguing that this issue arises from self-confirmation on the image input rather than self-recognition. To support this claim, it conducts causal tests utilizing image interventions (swapping or fading images) to separate authorship from image content. Results on VLMs of various families show that self-preference depends on the judge's visual input.
Critical strengths
- The proposed grounded self-confirmation mechanism to account for self-preference in VLM judges, in place of the existing self-recognition hypothesis, is interesting. The predictive capability of this self-confirmation idea (i.e., using VA to predict SF) also further supports the paper's claim.
- The experiments cover a large pool of VLMs of different families and varying parameter sizes. Both open-source and commercial models are evaluated to assess the proposed new mechanism.
Critical weaknesses
- Experiments in this paper seem not to sufficiently separate self-confirmation from self-recognition. In particular, for the experimental design in Section 4, the statement that "recognition predicts regardless of image; grounded self-confirmation predicts $ A_{true} > 0$ for the true-image answers, but $A_{alt} \approx 0 $" requires further justification. Although results indeed show that $ A_{true} > 0$ for the true-image answers and $A_{alt} \approx 0 $ for many VLMs, an alternative explanation could still be self-recognition with visual context. For example, the judge may fail to recognize its response generated based on another wrong image when judging the inputs with the correct image, leading to $A_{alt} \approx 0 $. Additional clarification on the difference between self-confirmation and self-recognition, and experiments that further rule out the self-recognition possibility (e.g., directly measure the self-recognition of VLMs) will further strengthen this paper's claim on the new self-confirmation mechanism.
- The proposed self-confirmation mechanism does not explain the results on larger commercial models, where does not hold. As the authors have also acknowledged this exception, further analyses and discussions could be provided. In addition, although the abstract highlights experiments on commercial models, their detailed results only appear in the appendix. I would recommend that the authors include the key results in the main paper.
Questions for authors
- How well do self-recognition-based measures predict SF, compared with VA in Section 6? This comparison could help distinguish self-recognition from self-confirmation.
- Have the authors investigated why commercial models exhibit different patterns from the open-source models (e.g., authorship-related preference on GPT-4.1-mini)? Are these differences attributable to model size or any other factors?
3 comments
recognition vs. verification accuracy, larger judges, and main-text results (2/2)
(Continued from the previous comment.)
Q1: How well does self-recognition predict self-favoritism, compared with verification accuracy?
Short answer: it predicts it too, somewhat less well, and it does not replace verification accuracy.
We tested recognition exactly the way the paper tests verification accuracy: measured on one half of the items, used to predict self-favoritism on the other half, and checked on held-out judges and families. Recognition of the judge's own true-image answers predicts self-favoritism across the cohort with r = +0.79 when the image is shown and +0.76 from the text alone. Verification accuracy gives r = -0.85, and -0.93 when measured on answers from commercial models outside every cohort family (-0.90 when siblings are also removed from self-favoritism). On held-out judges and families, recognition explains less of the variation (held-out R² 0.54 and 0.50 with the image, 0.46 and 0.42 from text) than verification accuracy (0.61 and 0.67; 0.84 and 0.81 on the commercial answers); we did not test the difference. Together the two explain more than verification accuracy alone (R² from 0.72 to 0.87). So recognition is a real additional predictor, and we now report it. Part of it is style: larger judges spot their own answer even when both answers reach the same final answer (0.67).
Because this is a correlation across judges, recognition's causal role remains open, and two results limit it. First, three of the six most self-favoring judges recognize their own text only at chance (0.53), yet favor themselves by 7.7 to 13.6 points (Table 20). Second, from the text alone the cohort recognizes its donor-image answers as well as its true-image answers (equivalent within 0.02, p = 0.002), yet donor-image answers earn no extra preference on average. Recognizing an answer as one's own does not by itself earn it a preference.
Q2: Why do the larger and commercial judges differ?
The submission already tests size (App. M): within each open model family, self-favoritism rises with scale, and the mixture-of-experts judge behaves like its active, not total, parameter count. We cannot establish a cause, but two new findings go with it. First, the larger judges recognize their own text far better (0.69 vs 0.52 for the cohort, without the image). Second, among the nine, the five with a significant authorship effect are exactly the ones whose self-recognition the image overrides least (a drop of 0.02 to 0.10, against 0.13 to 0.24 for the others; p = 0.016). For example, GPT-4.1-mini, compared against a stand-in's donor-image answer, still recognizes its own donor-image answer when the image is shown (0.74 to 0.70) and has an authorship effect of +21.8 points; Gemini 2.5 Flash does not (0.74 to 0.50) and has none (-2.6 points). With nine judges this is descriptive: the override tracks the evidence effect (r = 0.86) and does not separate judges across all 28 (p = 0.15), although across all 28, judges with an authorship effect recognize their own text better on average (0.62 vs 0.54, p = 0.009; the ranges overlap).
Commercial results in the main text
Sec. 6 of the submission already states the key results for the nine larger judges: the predictor, applied unchanged, captures their overall level of self-favoritism (held-out R² 0.87 against the cohort mean); their average authorship effect is +4.1 points (not significant, p = 0.18); and GPT-4.1-mini favors its own answer almost equally whatever the image (+21.8 vs +22.3 points). The full table stays in App. M because these judges are measured differently from the cohort (4-bit weights or an API), and App. R adds the new recognition measures.
Thank you for proposing the recognition test; it made the paper's central claim sharper. Please let us know if you need any more evidence regarding any sections of the paper or if we can provide more details regarding the experiments and motivation. We would be happy to provide further details. We would also be very grateful if you could reconsider your score.
Request a follow up on our responses
Dear Reviewer 5GkH, Thank you so much for all your great comments and concerns! They helped us make the paper more understandable and clear in terms of our contributions. We have tried to address your questions and concerns. Is it possible for you to please take a look and let us know if there is anything else we can do? We would be happy to further explain and clarify our claims in the paper. Also, we would be extremely grateful if you could please reconsider your score if we have properly addressed your concerns. Thank you so much for all your efforts and help!
Best regards, The authors
self-recognition measured directly; with the image in view, attribution follows the scene (1/2)
Thank you for your valuable feedback and comments! They help us a lot in improving our writings and experiments. We are also encouraged by your suggestions to run further tests. You suggested measuring self-recognition directly, and we did. We showed all 28 judges (the 19 in the cohort and the nine larger ones) exactly the answer pairs of our authorship experiment and asked: "One of the two answers was written by you, the AI model reading this. The other was written by a different AI model. Which answer did you write?", once with the true image shown and once with no image. We read the verdict from token probabilities with both answer orders averaged, so 0.5 means chance (135k comparisons; App. R, Table 27; full prompt in App. B). We answer in two comments: this one covers W1; the next covers Q1, Q2 and the main-text question.
W1: self-recognition with visual context
By self-recognition we mean identifying one's own text. By self-confirmation we mean favoring whatever fits one's own reading of the scene, whoever wrote it. Two results in the submission already point to self-confirmation: when another model rewrites the judge's answer in its own style, 90% of the effect remains (Sec. 4), and the preference for stand-ins' answers also reverses when the image is swapped (App. E), which recognizing one's own text cannot explain.
Your prediction is correct. With the true image shown, judges stop claiming the answers they wrote from a mismatched ("donor") image: recognition of those falls to chance (0.51, against 0.59 for answers written from the true image; p = 0.0003). From the text alone the two are recognized about equally (0.56 vs 0.57), so most of the drop comes from the image. But notice that the result fits both accounts.* For its own answers, a judge that recognizes itself through the scene ("I would have written something that fits this image") and a judge that simply names whatever fits its reading make the same prediction. So we designed a test that separates them, and pre-specified it before collecting any of its data.
The crossed test. We pair the judge's own donor-image answer with a stand-in's answer written from the true image (C1), and the reverse (C2), and show each pair with no image, the true image and the donor image (203k comparisons; App. R, Table 26). A judge that recognized its own writing would claim its own answer. Judges do not: shown the true image, they name the stranger's answer as their own. In the two pre-specified groups, the rate of claiming their own answer is 0.44 in the cohort (p = 7e-4, below chance for 17 of 19 judges) and 0.41 over all 28 judges (26 of 28). Swapping the image reverses the attribution for the very same pairs (25 of 28 judges for C1, 28 of 28 for C2). The clearest case is the larger judges (not a pre-specified group): from the text alone they can pick out their own donor-image answer (0.61, where the cohort is at chance, 0.50), yet once the true image is shown they name the stranger's answer as their own (0.36; 9 of 9 judges, p = 0.007). Now what does this mean eventually? A judge that names a stranger's answer as its own is not recognizing itself: with the scene in view, attribution follows fit to the scene more than authorship. Identity still matters somewhat, and attribution is not the same as preference: shown the true image, GPT-4.1-mini names the stranger's answer over its own donor-image answer (0.46), yet it has the largest authorship effect of all judges. Sec. 4 now reports the crossed test.