acceptodds
Under review as a conference paper at ICLR 2027

WHEN SELF-PREFERENCE IS A VERIFICATION FAILURE: CAUSAL EVIDENCE FROM VISION-LANGUAGE JUDGES

Abstract

Vision-language models used as judges tend to prefer their own answers, and the usual explanation is narcissism: the judge recognizes its own output and rewards it. We argue instead that self-preference is mostly a verification failure: a judge checks each answer against its own interpretation of the image, so when it misinterprets the image, it confirms the wrong answer produced from that misunderstanding. We test this causally in 19 open vision-language judges from 8 model families, across five multimodal benchmarks and a text control, by changing the judge's image while keeping every candidate answer byte-identical. An outcome-matched control shows that most raw self-preference is evaluator noise, consistent with recent findings in text-only settings. The remaining bias follows the judge's interpretation of the image rather than authorship: answers are favored for the scene they were written from rather than because the judge wrote them. Degrading the judge's view steadily reduces this bias, and showing the judge the image used to generate its other answer reverses this preference. A judge's accuracy at verifying other models' answers on items it answers incorrectly predicts its favoritism, including for judges and families excluded from fitting; the signal comes from answers that repeat the judge's own mistake. Tests on nine larger judges, four commercial, show that this relationship predicts their much higher favoritism, but an authorship effect can no longer be ruled out: individual larger judges show substantial direct authorship preference, the strongest almost matching that judge's evidence effect. This mechanism has a practical cost and a diagnostic use: a model choosing among its own samples selects a correct answer less often than other judges do on average, and removing or replacing the image during evaluation changes which judges are identified as biased.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-language models, LLM-as-a-judge, self-preference bias, multimodal evaluation, verification failure, causal analysis.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5GkHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new explanation for the self-preference of VLM judge models, arguing that this issue arises from self-confirmation on the image input rather than self-recognition. To support this claim, it conducts causal tests utilizing image interventions (swapping or fading images) to separate authorship from image content. Results on VLMs of various families show that self-preference depends on the judge's visual input.

Critical strengths
  • The proposed grounded self-confirmation mechanism to account for self-preference in VLM judges, in place of the existing self-recognition hypothesis, is interesting. The predictive capability of this self-confirmation idea (i.e., using VA to predict SF) also further supports the paper's claim.
  • The experiments cover a large pool of VLMs of different families and varying parameter sizes. Both open-source and commercial models are evaluated to assess the proposed new mechanism.
Critical weaknesses
  • Experiments in this paper seem not to sufficiently separate self-confirmation from self-recognition. In particular, for the experimental design in Section 4, the statement that "recognition predicts Aalt≈Atrue>0A_{alt} \approx A_{true} > 0 regardless of image; grounded self-confirmation predicts $ A_{true} > 0$ for the true-image answers, but $A_{alt} \approx 0 $" requires further justification. Although results indeed show that $ A_{true} > 0$ for the true-image answers and $A_{alt} \approx 0 $ for many VLMs, an alternative explanation could still be self-recognition with visual context. For example, the judge may fail to recognize its response generated based on another wrong image when judging the inputs with the correct image, leading to $A_{alt} \approx 0 $. Additional clarification on the difference between self-confirmation and self-recognition, and experiments that further rule out the self-recognition possibility (e.g., directly measure the self-recognition of VLMs) will further strengthen this paper's claim on the new self-confirmation mechanism.
  • The proposed self-confirmation mechanism does not explain the results on larger commercial models, where Aalt≈0A_{alt} \approx 0 does not hold. As the authors have also acknowledged this exception, further analyses and discussions could be provided. In addition, although the abstract highlights experiments on commercial models, their detailed results only appear in the appendix. I would recommend that the authors include the key results in the main paper.
Questions for authors
  • How well do self-recognition-based measures predict SF, compared with VA in Section 6? This comparison could help distinguish self-recognition from self-confirmation.
  • Have the authors investigated why commercial models exhibit different patterns from the open-source models (e.g., authorship-related preference on GPT-4.1-mini)? Are these differences attributable to model size or any other factors?
3 comments
AuthorsOpenReview ↗

self-recognition measured directly; with the image in view, attribution follows the scene (1/2)

Thank you for your valuable feedback and comments! They help us a lot in improving our writings and experiments. We are also encouraged by your suggestions to run further tests. You suggested measuring self-recognition directly, and we did. We showed all 28 judges (the 19 in the cohort and the nine larger ones) exactly the answer pairs of our authorship experiment and asked: "One of the two answers was written by you, the AI model reading this. The other was written by a different AI model. Which answer did you write?", once with the true image shown and once with no image. We read the verdict from token probabilities with both answer orders averaged, so 0.5 means chance (135k comparisons; App. R, Table 27; full prompt in App. B). We answer in two comments: this one covers W1; the next covers Q1, Q2 and the main-text question.

W1: self-recognition with visual context

By self-recognition we mean identifying one's own text. By self-confirmation we mean favoring whatever fits one's own reading of the scene, whoever wrote it. Two results in the submission already point to self-confirmation: when another model rewrites the judge's answer in its own style, 90% of the effect remains (Sec. 4), and the preference for stand-ins' answers also reverses when the image is swapped (App. E), which recognizing one's own text cannot explain.

Your prediction is correct. With the true image shown, judges stop claiming the answers they wrote from a mismatched ("donor") image: recognition of those falls to chance (0.51, against 0.59 for answers written from the true image; p = 0.0003). From the text alone the two are recognized about equally (0.56 vs 0.57), so most of the drop comes from the image. But notice that the result fits both accounts.* For its own answers, a judge that recognizes itself through the scene ("I would have written something that fits this image") and a judge that simply names whatever fits its reading make the same prediction. So we designed a test that separates them, and pre-specified it before collecting any of its data.

The crossed test. We pair the judge's own donor-image answer with a stand-in's answer written from the true image (C1), and the reverse (C2), and show each pair with no image, the true image and the donor image (203k comparisons; App. R, Table 26). A judge that recognized its own writing would claim its own answer. Judges do not: shown the true image, they name the stranger's answer as their own. In the two pre-specified groups, the rate of claiming their own answer is 0.44 in the cohort (p = 7e-4, below chance for 17 of 19 judges) and 0.41 over all 28 judges (26 of 28). Swapping the image reverses the attribution for the very same pairs (25 of 28 judges for C1, 28 of 28 for C2). The clearest case is the larger judges (not a pre-specified group): from the text alone they can pick out their own donor-image answer (0.61, where the cohort is at chance, 0.50), yet once the true image is shown they name the stranger's answer as their own (0.36; 9 of 9 judges, p = 0.007). Now what does this mean eventually? A judge that names a stranger's answer as its own is not recognizing itself: with the scene in view, attribution follows fit to the scene more than authorship. Identity still matters somewhat, and attribution is not the same as preference: shown the true image, GPT-4.1-mini names the stranger's answer over its own donor-image answer (0.46), yet it has the largest authorship effect of all judges. Sec. 4 now reports the crossed test.

AuthorsOpenReview ↗

recognition vs. verification accuracy, larger judges, and main-text results (2/2)

(Continued from the previous comment.)

Q1: How well does self-recognition predict self-favoritism, compared with verification accuracy?

Short answer: it predicts it too, somewhat less well, and it does not replace verification accuracy.

We tested recognition exactly the way the paper tests verification accuracy: measured on one half of the items, used to predict self-favoritism on the other half, and checked on held-out judges and families. Recognition of the judge's own true-image answers predicts self-favoritism across the cohort with r = +0.79 when the image is shown and +0.76 from the text alone. Verification accuracy gives r = -0.85, and -0.93 when measured on answers from commercial models outside every cohort family (-0.90 when siblings are also removed from self-favoritism). On held-out judges and families, recognition explains less of the variation (held-out R² 0.54 and 0.50 with the image, 0.46 and 0.42 from text) than verification accuracy (0.61 and 0.67; 0.84 and 0.81 on the commercial answers); we did not test the difference. Together the two explain more than verification accuracy alone (R² from 0.72 to 0.87). So recognition is a real additional predictor, and we now report it. Part of it is style: larger judges spot their own answer even when both answers reach the same final answer (0.67).

Because this is a correlation across judges, recognition's causal role remains open, and two results limit it. First, three of the six most self-favoring judges recognize their own text only at chance (0.53), yet favor themselves by 7.7 to 13.6 points (Table 20). Second, from the text alone the cohort recognizes its donor-image answers as well as its true-image answers (equivalent within 0.02, p = 0.002), yet donor-image answers earn no extra preference on average. Recognizing an answer as one's own does not by itself earn it a preference.

Q2: Why do the larger and commercial judges differ?

The submission already tests size (App. M): within each open model family, self-favoritism rises with scale, and the mixture-of-experts judge behaves like its active, not total, parameter count. We cannot establish a cause, but two new findings go with it. First, the larger judges recognize their own text far better (0.69 vs 0.52 for the cohort, without the image). Second, among the nine, the five with a significant authorship effect are exactly the ones whose self-recognition the image overrides least (a drop of 0.02 to 0.10, against 0.13 to 0.24 for the others; p = 0.016). For example, GPT-4.1-mini, compared against a stand-in's donor-image answer, still recognizes its own donor-image answer when the image is shown (0.74 to 0.70) and has an authorship effect of +21.8 points; Gemini 2.5 Flash does not (0.74 to 0.50) and has none (-2.6 points). With nine judges this is descriptive: the override tracks the evidence effect (r = 0.86) and does not separate judges across all 28 (p = 0.15), although across all 28, judges with an authorship effect recognize their own text better on average (0.62 vs 0.54, p = 0.009; the ranges overlap).

Commercial results in the main text

Sec. 6 of the submission already states the key results for the nine larger judges: the predictor, applied unchanged, captures their overall level of self-favoritism (held-out R² 0.87 against the cohort mean); their average authorship effect is +4.1 points (not significant, p = 0.18); and GPT-4.1-mini favors its own answer almost equally whatever the image (+21.8 vs +22.3 points). The full table stays in App. M because these judges are measured differently from the cohort (4-bit weights or an API), and App. R adds the new recognition measures.

Thank you for proposing the recognition test; it made the paper's central claim sharper. Please let us know if you need any more evidence regarding any sections of the paper or if we can provide more details regarding the experiments and motivation. We would be happy to provide further details. We would also be very grateful if you could reconsider your score.

AuthorsOpenReview ↗

Request a follow up on our responses

Dear Reviewer 5GkH, Thank you so much for all your great comments and concerns! They helped us make the paper more understandable and clear in terms of our contributions. We have tried to address your questions and concerns. Is it possible for you to please take a look and let us know if there is anything else we can do? We would be happy to further explain and clarify our claims in the paper. Also, we would be extremely grateful if you could please reconsider your score if we have properly addressed your concerns. Thank you so much for all your efforts and help!

Best regards, The authors

Reviewer_26WyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies whether VLMs favor their own wrong answers when judging responses. It argues that models repeat their own mistakes about the image when checking answers and uses verification performance to predict this bias. It tests these claims across models and benchmarks using other models as controls, image changes, and answer-selection experiments.

Critical strengths
  • Models use self-verification to choose answers and score training data. Mistakes in these checks can affect both outputs and training, so studying their reliability is timely.

  • The paper links poor answer checking to stronger self-preference on questions a model gets wrong. This link also helps predict self-preference in models and families not used to build the predictor.

Critical weaknesses

The experimental design does not sufficiently support the main causal claim.

  • The donor-image condition may create a floor effect. Donor images are assigned by a fixed reassignment of image IDs, without reported matching by content or question relevance. If both self-generated and stand-in answers become clearly unsuitable, both scores may approach zero, making their difference small. This makes the claim that authorship does not explain residual self-preference too strong.

  • Image degradation may simply make judgments less decisive, moving scores toward 0.5 and shrinking self-preference. The shrinkage analysis finds that this explains most of the uncorrected contrast. Significant differences of 0.76 and 1.12 percentage points remain after correction, but these smaller effects do not uniquely identify grounded self-confirmation as the cause.

  • References and stand-ins may belong to the judge’s own family. Shared errors and family preferences may therefore affect the baseline, especially when families contribute different numbers of eligible stand-in answers. This makes model-specific self-preference difficult to separate from family-level effects. Leaving out a family when evaluating the predictor does not resolve this measurement issue.

  • The proposed causal explanation is not established. A judge evaluating answers against the displayed image could also show the reported preference reversal. Comparing two self-generated answers cannot detect an authorship preference shared by both. Image agreement and authorship preference could therefore coexist, leaving the claim that verification failure explains the residual insufficiently supported.

  • The additional evaluation still uses open models of 32B or less and older proprietary models. Newer proprietary families are excluded because their APIs do not provide token probabilities in the form required by the protocol. Their behavior therefore remains unknown. Authorship effects also remain in several tested larger judges, limiting how broadly the proposed explanation applies.

The added insight and practical impact are limited.

  • Self-preference, self-recognition, outcome-matched controls, multimodal self-preference, and the gap between generation and verification are already studied in prior work. The additions are visual interventions, the relationship between verification performance and residual self-preference, and selection and audit experiments. However, image relevance would normally affect answer scores. The uncertain causal interpretation therefore limits the added insight into the cause of self-preference.

  • The main measure only covers cases where the judge and a stand-in are wrong while the reference is correct. This captures a narrow form of harmful self-preference. It excludes cases where both answers are correct and does not measure the overall value of self-verification across the usual output distribution.

Questions for authors

Please refer to Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

floor effect, shrinkage and same-family models tested; the causal results hold(1/2)

Thank you for your valuable comments and feedback. We really appreciate your careful reading and insights. We tested each concern directly; the new analyses are in App. R (Tables 26-29). Because of length we answer in two comments: this one covers the floor effect, shrinkage and same-family models; the next covers authorship, larger judges, novelty and scope.

1. Floor effect: the donor-image answers are real attempts.

If answers written from a mismatched ("donor") image were obviously unsuitable, judges would almost never prefer them, and there would be nothing left for authorship to act on. That is not what happens. Against the correct answer, judges prefer their own donor-image answers 47.8% of the time and stand-ins' 48.1%, close to the 54.6% and 51.8% for answers written from the true image. Before we keep only the wrong ones, 38.4% of donor-image answers are actually correct (60.1% from the true image). And the design does detect authorship where it exists: Pixtral-12B (+4.8 points) and GPT-4.1-mini (+21.8 points) show it, and when the donor image is shown, the authorship premium moves onto these answers (p = 0.041; already in App. E).

2. Shrinkage: we agree, and our causal claim rests on the reversal instead.

Shrinkage explains most of the naive contrast, so our causal claim does not rest on the small self-specific residual that remains after removing it (0.76 and 1.12 points beyond stand-ins under the two corrections; App. G). It rests on the preference reversal (already in Sec. 5 and App. E). The judge wrote two answers, one from the true image and one from a donor image. Judged with the true image, it prefers the first (by 6.8 points); judged with the donor image, it prefers the second (by 5.1 points; p = 0.007 for the change). Shrinkage can only pull scores toward indifference; it cannot flip which answer is preferred. Since the judge wrote both answers, it is the image in view, not authorship, that drives the preference. That the misreading is the judge's own rests on separate evidence (point 4, in the next comment).

3. Same-family stand-ins and references: the causal results hold or get stronger without them.

We removed every stand-in and reference from the judge's own model family and recomputed everything (Table 28):

submittedwithout same family
self-favoritism+2.89 pp+3.72 (p = 0.036)
image-swap drop+2.57+3.30 (p = 0.012, 17 of 19 judges)
authorship effect (A_alt)-0.26+0.21 (p = 0.79, still null)
evidence effect (E_self)+6.80+6.59 (p = 0.003)
interaction+3.09+3.39 (p = 0.009)
reversal (change in E_self)-11.91-11.81 (p = 0.008)

For self-favoritism and the swap drop, same-family stand-ins had made the submitted estimates conservative. Grouping models by shared language backbone instead of family gives the same pattern.

The Sec. 6 predictor needed a new test. You are right that leaving a family out of the fit does not address this. Measured only on the cohort's stand-ins from other families, verification accuracy does not predict self-favoritism (r = -0.07); we report this. So we ran a pre-specified test with no siblings and no selection: each judge graded wrong answers written by four commercial models outside every cohort family (GPT-4.1, GPT-4.1-mini, GPT-4o, Gemini 2.5 Flash; Gemini was excluded for the two Gemma judges). Accuracy on those answers predicts self-favoritism (recomputed without siblings) as strongly as the submitted result: r = -0.90 against -0.85, with held-out R² of 0.76 for unseen judges and 0.65 for unseen families. As a robustness check (not pre-specified), dropping any one of the four models changes nothing (-0.895 to -0.907). The predictor does not need siblings.

One post-hoc nuance: on these commercial answers, even wrong answers that differ from the judge's own predict self-favoritism (r = -0.87). With plausible wrong answers, the predictor therefore tracks a general failure to reject them, not only the judge's own mistake. Sec. 6 now says this, and the abstract limits the own-mistake claim to the cohort's answers.

AuthorsOpenReview ↗

authorship vs. image fit, larger judges, novelty and scope (2/2)

(Continued from the previous comment.)

4. Image agreement and authorship could coexist: we agree, and we measure both.

The reversal holds authorship fixed, because the judge wrote both answers. We agree that any judge that uses the image would show it: the reversal shows that the image in view, not authorship, drives the preference. That the misreading is the judge's own comes from App. G: on items it got wrong, the judge scores stand-ins that give its exact wrong answer higher than stand-ins wrong in other ways (by 1.5 points, p < 1e-4), and this gap disappears under a wrong image. Authorship itself is measured by A_alt, the preference for the judge's own answers written from a different image: in the cohort it lies between -1.4 and +0.9 points (90% CI), next to an evidence effect of +6.8 points (already in Sec. 4, Table 12).

A new pre-specified test pits the two directly against each other (Table 26). We show the judge its own answer written from the wrong image next to a stranger's answer written from the true image, and ask which one it wrote. Shown the true image, judges name the stranger's answer as their own: they pick their own with probability 0.44 in the cohort (below chance for 17 of 19 judges) and 0.41 over all 28 (26 of 28). Swapping the image flips this for the same pairs. Attribution follows fit to the scene more than authorship.

Our claim is "mostly", not "never": 4 cohort judges and 5 of the 9 larger judges do show an authorship effect, and we report them. Finally, a judge that evaluated correctly would prefer the correct answer. Preferring a wrong answer because it fits what the judge sees requires misreading the image, which is the verification failure we mean.

5. Larger and newer commercial judges.

Our protocol needs graded verdicts from token probabilities. The GPT-5 family caps token probabilities at five alternatives, which in a 20-comparison probe left only one verdict letter, and so a 0-or-1 readout, in 15 cases; Gemini 3 returns no probabilities at all. Neither can be measured this way (already in App. M). For the nine larger judges we could measure, the authorship effect is about the same whether the stand-ins are strong or weak models (+4.7 vs +4.2 points, p = 0.80), so it is not an artifact of weak competition. The judges with an authorship effect are those whose self-recognition the image overrides least (p = 0.016); with nine judges, this describes the split rather than predicting it.

6. What is new compared with prior work.

Prior work explains self-preference by self-recognition [1] or by familiarity, measured as low perplexity [2]. Most recently, an outcome-matched control attributed about 90% of measured self-preference to evaluator uncertainty and left the remaining 10% as an open question [3]. These studies are text-only or correlational, and none changes what the judge sees. We change only the judge's image, which shows what the residual responds to. Image relevance affects both kinds of wrong answer, the judge's and the stand-ins', so it largely cancels in our self-versus-stand-in comparison. What goes beyond it: (i) under an image swap the judge's own answers lose more than stand-ins', even after removing shrinkage; (ii) the judge's preference between two of its own answers reverses with the image (point 2); (iii) authorship earns nothing on average once the answer was written from another scene; (iv) a judge's verification accuracy predicts its self-favoritism for held-out judges and families (all four in the submission); and (v), new in the revision, with the scene in view judges attribute authorship more by fit to the scene than by who wrote it, naming a stranger's answer as their own (point 4).

  • [1] Panickssery, Bowman, Feng. LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024.
  • [2] Wataoka, Takahashi, Ri. Self-Preference Bias in LLM-as-a-Judge. NeurIPS 2024 Workshop on Safe Generative AI.
  • [3] Roytburg et al. Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations. ICML 2026.

(All three are discussed in Sec. 2 of the paper.)

7. Only error items.

These are the harmful cases, and preference for a correct own answer is outside our estimand; we state this scope. Two submitted analyses do go beyond error items. On the common set that every judge grades (App. L), items the judge answers correctly behave differently: there, verification accuracy rises with self-favoritism (r = +0.79). And best-of-7 includes correct and incorrect candidates wherever samples disagree, with a 3.4-point cost relative to other judges (a practical cost, not a test of the mechanism).

We hope these answers address your soundness concerns. Please let us know if we can further explain any unclear parts of the paper. We would also be very grateful if you could reconsider your score for our submission!

AuthorsOpenReview ↗

Request follow up on review

Dear Reviewer 26Wy, Thank you again for your insights and critical question. These questions helped us further clarify our claims. Given that the rebuttal deadline is tomorrow, could you please take a look and let us know if there is anything else we can do to further enhance the paper? Also, we would be extremely grateful if you could please reconsider your score if we have properly addressed your concerns. Thank you so much for all your efforts and help!

Best regards, The authors

Thank you for the detailed response and the new analyses.

My main concern about the causal claim remains. The claim now rests on the preference reversal, but you agree that any judge that uses the image would show this reversal, and it also appears on other models' answers. It therefore shows that the image in view matters, not that the remaining self-preference comes from the judge's own misreading. The evidence against authorship comes from small judges that mostly cannot recognize their own text, while about half of the larger judges show an authorship effect. The new attribution experiment measures which answer the judge says it wrote, not which it prefers.

The predictor also no longer holds within the cohort once same-family models are removed, and on the commercial models' answers it tracks acceptance of plausible wrong answers in general rather than the judge's own mistakes.

The new analyses do resolve my concerns about the floor effect and about family effects in the main contrasts, so I will raise my soundness score, but I will keep my overall rating.

Reviewer_dzj4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether self-preference among VLM judges reflects authorship or shared errors between generation and evaluation. This paper employs autcomematched baseline method and applies interventions to the images used for generation and evaluation. Among 19 open-weight raters, the study found that visual evidence influences residual self-preference, and that each rater’s validation performance on error items can predict this preference. Additional experiments examined a larger sample of raters, “best-of-N” selection, and image ablation reviews.

Critical strengths

The image experiments show that what the judge sees affects self-preference. In Section 5, the authors change the image while keeping the candidate answers fixed. Self-preference drops by 2.57 pp across all judges (p = 0.012).

The authors also test the predictor on held-out judges and model families, using cross-fitting and reporting Q2Q^2 values of 0.61 and 0.67. The link mainly comes from other models repeating the judge’s own mistake; for different wrong answers, the correlation is almost zero (r = −0.06).

Critical weaknesses

Weakness 1: The Actual Impact of Residual Self-Preference is Unclear

The authors found that, after excluding general evaluation errors, the remaining self-preference averaged only 3.1 pp. While the Best-of-7 experiment shows that self-judgment does reduce accuracy, it's unclear how much of this decrease is due to self-preference and how much is simply due to the judge's own poor judgment.

Weakness 2: The GSC Mechanism Has Not Received Sufficient Direct Validation

The authors conducted numerous experiments to support GSC, such as the decrease or even reversal of self-preference after changing the image, and the judge's greater willingness to accept the same errors as themselves. However, these results do not fully demonstrate that the judge actually repeated previous errors during jury evaluation. It's also possible that the judge simply felt a particular answer matched the current image better, thus awarding a higher score.

Questions for authors
  1. How much of the Best-of-7 accuracy drop comes from self-preference, beyond the judge simply making mistakes?
  2. Can you show that the judge makes the same visual mistake again when evaluating, rather than just choosing the answer that seems to fit the image?
2 comments
AuthorsOpenReview ↗

best-of-7 cost separated from judge skill; evidence the judge repeats its own mistake

Thank you so much for your valuable comments and feedback! We provide answer to each one of your concerns below:

Q1. How much of the best-of-7 drop is self-preference, and how much is just a weak judge?

Short answer: about 3.3 accuracy points are specific to a model judging its own samples, beyond its general skill as a judge.

How we separate the two (already in the submission: Sec. 7; App. N, Table 22). Each of the six judges picks the best answer from every generator's samples, so all judges face identical candidates. We can therefore model accuracy over all 72 cells with a separate term for each generator, judge and dataset. The judge term captures how good a model is at choosing among other models' answers. On top of it, a model choosing among its own answers loses 3.3 points (SE 1.2, p = 0.008). A simpler comparison agrees: against the other judges on the same candidates, self-judging loses 3.4 points per generator (p = 0.012) and does worse in 11 of 12 generator-dataset cells.

New in the revision. For scale, judges' general skill differs by 4.8 points from best to worst, so the self-judging loss is about two-thirds of that gap. Your question also led us to check where the loss happens. It is not concentrated where the generator misread the item: the loss is 5.2 points (p = 0.032) when the generator's own greedy answer is right and 2.0 points (p = 0.12, not significant) when it is wrong, and self-judges do not favor samples that agree with their greedy answer more than other judges do. We therefore removed the sentences (abstract, Sec. 7, App. N) that blamed the cost on misreadings. Since every candidate in best-of-N is the generator's own, this measures a cost of self-judging, not own-versus-other preference.

How large is the preference itself? The 3.1-point headline averages all 19 cohort judges (per-judge mean 2.9), including 11 with little or negative bias. The eight judges that do prefer themselves average 8.3 points (Sec. 5); grading answers one at a time gives 7.5 points (App. C); and the nine larger judges show 8.8 to 29.9 points (App. M; partly because their stand-ins are weaker, App. R). All of these are items the judge got wrong, so none of this preference is earned.

Q2. Does the judge repeat its own visual mistake, or just pick the answer that fits the image?

Short answer: on these items the two coincide, and the evidence says the mistake being repeated is the judge's own.

On these items every answer except the correct one is wrong. A wrong answer can "fit the image" better than the correct one only for a judge that misreads the image, so the real question is whether the misreading is the judge's own. Four results say it is (the first, second and fourth were already in the submission):

  1. The judge endorses its exact wrong answer even in other models' words. On items it got wrong, it scores stand-ins that give its exact wrong answer higher than stand-ins wrong in another way (by 1.5 points, p < 1e-4; App. G). Shown a wrong image, it loses this gap.
  2. Among the cohort's stand-ins, that endorsement carries the predictive signal. Accuracy on stand-ins that repeat the judge's answer strongly predicts its self-favoritism (r = -0.89). On stand-ins wrong in a different way, every judge is near chance and there is no relation (r = -0.06) (Sec. 6, App. K).
  3. The more an answer resembles the judge's own, the more it is endorsed, mainly with the true image in view (new, post hoc). Qwen3-VL-8B gives sibling answers that repeat its mistake a preference of 0.67 over the correct answer (0.5 = indifferent), other-family answers wrong in another way 0.49, and the same sibling answers under a wrong image 0.53. Within judges, answers one standard deviation more similar to the judge's own get 2.0 points more endorsement (p = 0.02).
  4. Partial support from re-derivation. When seven judges re-answer the question at judging time, stand-ins agreeing with that fresh answer get a 4.5-point bonus (p = 0.018) that disappears with a wrong image. The own-answer effect stays larger in 5 of 7 judges, so we treat this as partial support (App. O).

In short, the preference follows the judge's own wrong reading: it extends to other models' answers that repeat that reading, and a wrong image switches it off. Authorship as such adds almost nothing on average (-0.26 points in the cohort; Sec. 4).

Thank you again; we hope this strengthens your assessment, and please let us know if anything else remains unclear, so we can elaborate on it! We would also be very grateful if you could consider raising our score if we have addressed all your concerns. Thank you so much!

AuthorsOpenReview ↗

Requesting follow up on your review

Dear Reviewer dzj4, Thanks for your assessment and your critical questions. We have tried to address your questions and concerns. Given that the deadline for reviewer discussion is today, is it possible for you to please take a look and let us know if there is anything else we can do? Also, we would be extremely grateful if you could please reconsider your score if we have properly addressed your concerns. Thank you so much for all your efforts and help!

Best regards, The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

two pre-specified experiments, new controls, and narrowed claims.

We thank all the reviewers for their careful and constructive reviews. Where a concern could be tested, we ran the test, and where the submitted paper already answered it, we say so and point to the place. All changes are marked in blue in the revised PDF; most new material is in Appendix R (Tables 26-29).

Our claim in brief. When a vision-language judge misreads an image, it favors its own wrong answer mainly because it checks every answer against its own misreading, not because it recognizes its own writing. So the judges that are worst at rejecting other models' wrong answers are the ones that favor themselves most. (Terms: a stand-in is another model's answer that is also wrong; self-favoritism is how much more the judge prefers its own wrong answer over the correct one than it prefers stand-ins' wrong answers; verification accuracy is how strongly the judge prefers the correct answer over a stand-in's wrong one.)

What we found

  1. Self-recognition (5GkH, 26Wy). We asked all 28 judges which of two answers they wrote (135k comparisons). In the decisive pre-specified test (a further 203k), judges shown the true image claim a stranger's answer that fits the image over their own answer that does not: they pick their own with probability 0.44 in the cohort (below chance for 17 of 19 judges) and 0.41 over all 28. Swapping the image flips this for the same pairs. With the image in view, attribution follows fit to the scene more than authorship. Recognizing one's own writing from the text alone does exist and also predicts self-favoritism (r = +0.76); we now report it, although its causal role is open.
  2. Same-family models (26Wy). Removing stand-ins and references from the judge's own model family leaves every causal result intact or stronger: self-favoritism rises from 2.9 to 3.7 points, the image-swap drop from 2.6 to 3.3, and the preference reversal holds (-11.8 points). Our predictor fails on the cohort's answers from other families, which we report. But in a second pre-specified test, accuracy on wrong answers written by four commercial models outside every cohort family predicts self-favoritism, with siblings removed, as well as the submitted result did (r = -0.90 vs -0.85; held-out R² 0.76 for unseen judges, 0.65 for unseen families).
  3. Floor effect (26Wy). Answers written from a mismatched image are real attempts: judges still prefer them to the correct answer almost half the time (47.8% for their own, 48.1% for stand-ins'), and 38.4% of all such answers are actually correct.
  4. Best-of-7 (dzj4). After accounting for each model's general skill as a judge, choosing among its own samples still costs it 3.3 accuracy points (p = 0.008). The cost is not concentrated where the model misread the item (5.2 points where its greedy answer is right, 2.0 and not significant where wrong), so we now present it as a practical cost of self-judging, not as evidence for the mechanism.
  5. Larger judges (5GkH, 26Wy). Their authorship effect is not an artifact of weak stand-ins: it is about the same against strong and weak ones (+4.7 vs +4.2 points, p = 0.80).

Claims we narrowed

  • Self-confirmation explains self-favoritism mostly, not always: 4 of the 19 cohort judges and 5 of the 9 larger judges also show a direct authorship effect, which we report.
  • Best-of-N shows a cost of self-judging; it is not a test of the mechanism.
  • "The predictive signal comes from stand-ins that repeat the judge's mistake" holds for the cohort's answers. On the commercial models' answers, wrong answers that differ from the judge's also predict self-favoritism (post hoc).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.