acceptodds
Under review as a conference paper at ICLR 2027

Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation

Abstract

Multimodal large language models (MLLMs) are increasingly used as automated judges for instruction-based image editing and as reward signals for model training. However, systematically auditing whether these judges are influenced by cues irrelevant to editing quality is challenging because visual interventions may themselves alter the quality being evaluated. A judgment shift can therefore be attributed to bias only when the intervention is verified to preserve the underlying editing quality. To address this challenge, we introduce EditJudgeBias, a counterfactual benchmark with verified quality preservation, comprising 1,196 real editing samples and 13 cues injected across four evaluation sites. We verify quality preservation for each cue using calibrated multimodal validators, controls, and human inspection. We then audit five MLLM judges along three complementary dimensions: invariance to quality-preserving cues, agreement with human judgments, and stability of pairwise preferences. Importantly, observed shifts are evaluated against each judge’s own zero-dose and re-query noise floors rather than against zero. Experiments show that quality-preserving cues move every judge beyond its own noise. Fabricated majority opinions increase ratings, irrelevant visual elements cause larger shifts than whole-image manipulations, and swapping candidate order reverses up to 60.9% of pairwise decisions. Edit-region cues also tend to reduce human agreement. The three measures characterize judges differently, showing that robustness cannot be captured by a single metric.

Primary area:
datasets and benchmarks
Keywords:
multimodal LLM-as-a-judge, instruction-based image editing, judge bias

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_d336OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This article proposes EditJudgeBias for the systematic study of bias issues in MLLM in image editing evaluation, including 1196 editing examples and 13 bias interference factors. The author introduces a quality preservation verification mechanism and noise baseline to distinguish between score changes caused by bias, real image quality changes, and random fluctuations in the model itself. The experiment found that MLLM is easily influenced by irrelevant visual elements and contextual information.

Critical strengths
  1. MLLM is increasingly being used for automatic evaluation of image generation and editing tasks. This article studies the bias problem of MLLM and has academic value.
  2. The experimental design is relatively rigorous, and multiple verifications have been conducted to maintain quality. In the evaluation, model bias, image quality changes, and bifurcation changes caused by random fluctuations in the model itself have been effectively distinguished. The experimental analysis is comprehensive, and multidimensional analysis was conducted on different bias interference factors.
Critical weaknesses
  1. The main research findings lack novelty. The phenomenon of being easily influenced by contextual information interference and visual factors during the MLLM evaluation process has been widely discussed in previous research, and the conclusions of this article can largely be expected.
  2. Lack of in-depth analysis of the causes of bias formation. The paper only demonstrates different types of evaluation bias through a large number of experiments, but lacks sufficient explanation for the possible reasons behind their occurrence, and lacks mitigation solutions.
Questions for authors
  1. Besides extending existing bias analysis to image editing tasks, what are the differences from existing work?
  2. What are the underlying causes and mechanisms of the different types of evaluation bias identified in the experiments? such as Why do small stickers have a greater impact than the entire image transformation? Could the authors provide some analysis to develop and evaluate corresponding strategies to alleviate these different bias?
  3. It is mentioned in the article that MLLM Judge is used for reinforcement learning reward signals. How do these biases studied by the author affect downstream reinforcement learning?
3 comments
AuthorsOpenReview ↗

Response to Reviewer d336 (1/2)

We thank the reviewer for the thoughtful assessment. We added a mechanism analysis with a causal test and representation-level mitigation, and an RL-reward experiment; they address W2/Q2 and Q3 directly.

detailed results in revision (changes in blue):

PointAnswerLocation
W1 / Q1Content-level cues and verified preservation on all three quality dimensionsSec. 3.3; App. C; Table 1
W2 / Q2Mechanism with a causal test; mitigation at the representation and output levelsApp. G (Table H); App. E
Q3Effect on an RL reward model and on GRPO advantagesApp. H (Table I)

W1 / Q1. Novelty beyond prior judge-bias work

To our knowledge, EditJudgeBias is the first audit that studies cues changing image content at the semantic level and that rigorously verifies whether the edit's quality is preserved before and after cue injection.

An audit built for editing. The judge evaluates a triplet (source image, instruction, edited image), so cues can be anchored to the edit itself (box, inset, caption, sticker). The sticker (Distraction), for example, carries the known "irrelevant-information distraction" effect from the text modality into the judge's visual input, which, to our knowledge, no prior judge-bias audit has tested.

Preservation is tested, not assumed. Prior audits of MLLM judges (FRAME, EMNLP 2025; MM-JudgeBias, ACL 2026) inject visual cues for text-to-image and general multimodal inputs, but assume without verification that the cues leave the evaluated quality unchanged. We test this with calibrated validators and blinded human ratings on all three quality dimensions (App. C).

New findings. Our pixel-level cues include the main manipulation types studied before (e.g., brightness, filters, borders and text overlays; Sec. 3.3), and the content-level cues reveal a semantic-level bias that is both subtler and larger: a sticker shifts the judges more than any pixel-level cue (Table 1).

W2 / Q2. Mechanism and mitigation; why a small sticker outweighs a global transform

App. E already evaluates two output-level remedies (two-order reconciliation and the standardized-median ensemble), and neither removes a bias that the judges share. Following UnfairJudge (Xu et al., 2026b), we added a mechanism analysis that first examines the representation geometry and then tests causality, and we evaluated mitigation at both levels.

Setting. We ran Qwen3-VL-8B-Instruct with the paper's scoring prompt on 300 breadth samples and read the residual stream at the last prompt token of layer 20. The total score S is the sum of IA, EQ and DP (3–30), and shifts are Hodges–Lehmann estimates.

The open judge reproduces the bias. Sticker, brightness and bandwagon shift its score in the same direction as for qwen3-vl-32b and qwen3.5-plus (3/3), and the sticker, the largest effect, lowers S significantly by a comparable amount (Table H, Panel A).

What drives the effect. How much a cue moves the score depends on the direction in which it pushes the internal representation, and on whether it pushes all samples the same way, not on how many pixels it changes (Table H, Panel B). The sticker changes only 1.2% of the pixels, yet pushes the samples consistently in the same direction, closely aligned with the direction that separates human-rated poor edits from good ones. Brightness changes far more pixels, but its shifts point in different directions and largely cancel. Bandwagon text is nearly orthogonal to the quality direction, so its influence bypasses the judge's perception of edit quality.

CuePixels changedShift in S: 8B / 32B / qwen3.5ConsistencyCosine with poor-edit direction
Sticker1.2%−3.12 / −2.32 / −3.530.69+0.62
Brightness≈100%−0.09 / −0.41 / −0.670.21−0.02
Bandwagon0% (text)+0.32 / +1.27 / +0.720.94−0.03
SHAM38.9%–0.15+0.03

Causal test and representation-level mitigation. Adding the captured sticker direction to uncued inputs reproduces 70% of the sticker's score drop, and subtracting it from affected inputs removes 72% (Table H, Panel C), whereas norm-matched random directions barely move the score on average. For cues such as the sticker, which act through the representation of edit quality, this is a representation-level mitigation. Bandwagon bypasses that representation, so its mitigation belongs at the input stage.

Steering along the sticker directionnShift in SShare of the real sticker shiftRandom directions
Add to uncued inputs255−2.0670%−0.09
Subtract from affected cued inputs65+12.7572%+0.39

We look forward to future work that builds on this analysis.

AuthorsOpenReview ↗

Response to Reviewer d336 (2/2)

Q3. Impact on downstream RL rewards

These cues occur naturally. The Framing and Typographic examples in Figure 1 are taken from widely used datasets, and the judges' comments explicitly cite these irrelevant elements as reasons for deducting points.

Setting. We used EditScore-7B (Luo et al., 2026), an open reward model built for online RL in image editing, and scored through its own RL reward path. Each of 611 edits is scored once before and once after cue injection; the two calls differ only in that cue, with the same instruction, source image and sampling seed.

The cues systematically change the reward. Of the five image-side cues that shift all five of the paper's judges in the same direction, four also lower the reward significantly, and the three largest effects hold under Avg@4 with larger standardized effects.

CueΔ reward [95% CI]Lower / higherdz: 1 pass / Avg@4q
Sticker−0.122 [−0.142, −0.103]59% / 18%−0.50 / −0.613×10⁻³³
Box−0.071 [−0.089, −0.053]47% / 26%−0.31 / −0.413×10⁻¹²
Text overlay−0.050 [−0.070, −0.030]44% / 32%−0.20 / −0.287×10⁻⁵
Brightness−0.020 [−0.032, −0.008]42% / 27%−0.13 / –3×10⁻⁵
Padding−0.014 [−0.031, +0.002]37% / 34%−0.07 / –0.19
SHAM (null)−0.002 [−0.012, +0.008]33% / 29%−0.02 / –p = 0.46

They bias the policy-gradient signal. In 77 GRPO-style groups, each containing the edits of eight editing models for the same instruction, replacing one candidate with its sticker version lowers its group advantage by 0.25 on average (SHAM: +0.03) and flips its sign for 24% of candidates (SHAM: 9%). A flipped advantage means that the bias reverses the direction of the policy update for that candidate.

We hope these new results, which go beyond documenting the biases to explaining and mitigating them and to measuring their effect on RL rewards, address the reviewer's concerns, and we would be grateful if the reviewer would reconsider the score.

Dear reviewer, thank you for the helpful feedback. We hope that our clarifications and the additional experiments presented in the revised manuscript have adequately addressed your concerns. If you find these new results satisfactory, we would sincerely appreciate your consideration of a score update that reflects the strengthened evidence and discussion.

Reviewer_EYRrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EditJudgeBias, comprising 1,196 image-editing samples and 13 interventions, to audit five primary MLLM judges through score invariance, human agreement, and pairwise stability. It reports sensitivity to fabricated majority opinions, visual modifications, and presentation order.

Critical strengths
  1. It addresses a consequential problem for automated evaluation and reward-based learning.
  2. It recognizes that response variability must be measured before attributing changes to interventions.
  3. It examines several complementary outcomes, such as the leaderboard changes and the coverage cost of reconciling presentation orders.
Critical weaknesses
  1. Section 3.1 defines quality across instruction adherence, editing quality, and detail preservation. However, the human verification instructions focus on whether the requested edit remains completed, as indicated by the appendix.
  2. The validator prompt describes the intervention as superficial and instructs validators to treat several transformations as cosmetic. Human annotators receive similar guidance and know the study hypothesis, which makes agreement partly a consequence of shared instructions.
  3. The paper treats a nonsignificant difference from SHAM as support for preservation, although failure to detect a difference does not establish equivalence. Would using the existing observations to report direct contrasts against SHAM and retest variability be a better idea?
Questions for authors

Please refer to the weakness section above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer EYRr

We thank the reviewer for the careful comments on verification. We addressed each point with new experiments; together they show that the cue effects are not explained by changes in edit quality.

detailed results in revision (changes in blue):

PointNew resultLocation
W1Human verification of IA, EQ and DP, blind to condition and hypothesisSec. 3.4; App. C.5 (Table C.5)
W2gpt-6-luna validator with a neutral prompt; annotators unaware of the hypothesisSec. 5.4; App. C.4 (Table A, Figure B), Table 2, Panel B; App. C.5
W3Direct contrasts with SHAM and retest variability; revised preservation wordingTables C.5, A and 1; Secs. 3.4, 5.4

W1. Human verification on IA, EQ and DP

We repeated the human verification following the three-dimensional definition of Sec. 3.1, and the original conclusion holds. We drew 25 base edits stratified over the five source datasets and created one task per base edit for each of the ten image-side cues and SHAM (275 tasks), plus 25 tasks in which the edit is reverted to the source as a positive control (300 tasks in total). Two external annotators who are not involved in this study worked independently, unaware of the hypothesis and of the condition of each task, with left/right order randomized. For IA, EQ and DP separately, they reported whether the cued version differs from the uncued edit (No / Slightly / Yes).

Injection leaves edit quality essentially unchanged. SHAM, text overlay, padding, caption and saturation drew no change reports on any dimension (0/25). Other reports are rare, e.g., sticker EQ 1/25 and DP 4/25, box EQ 1/25 and DP 2/25.

The protocol detects real changes reliably. Both annotators flagged every reverted edit on all three dimensions (25/25), and they agree on 95–97% of the labels (Cohen's κ = 0.81–0.85).

ConditionIAEQDP
SHAM000
Brightness020
Saturation000
Aesthetic filter242
Watermark002
Padding000
Text overlay000
Caption000
Box012
Inset (borderline)228
Sticker114
Reverted edit252525
Annotator agreement (raw, κ)97.3%, 0.8597.0%, 0.8595.3%, 0.81

W2. "Cosmetic" framing in the validator and annotator instructions

We switched to neutral instructions on both the MLLM and the human side, and the conclusions are unchanged. Both checks have adequate detection power, so the cue-injection audit rests on a reliable quality check.

MLLM validators. gpt-4o-mini is being retired, so we re-ran the validation with OpenAI's designated successor, gpt-6-luna, as a second validator independent of gemini. Its prompt neither calls the change cosmetic nor lists transformations (Figure B). gpt-6-luna detects controlled degradations at least as often as gemini (Table 2, Panel B).

Validator25% reversion50% reversion100% reversionBlur
gemini-3.546 (49%)69 (74%)88 (95%)91 (98%)
gpt-6-luna31 (60%)47 (90%)51 (98%)52 (100%)

Human checks. The new annotators knew neither the hypothesis nor the condition, and they still reported almost no changes (W1). The agreement between humans and validators is therefore not a product of shared instructions.

W3. Non-significance is not equivalence: direct contrasts with SHAM and retest variability

We adopted this suggestion and now report differences from paired SHAM directly, alongside retest variability:

EvidenceDirect contrast now reportedWhere
Human annotatorscue − SHAM change rate with a 95% CI, for every cue and dimensionTable C.5
gpt-6-lunaIA, cue − SHAM: −7.3 to +5.5 pp with 95% CIs, all BH q ≥ 0.70; for reference, its re-query flip rate on SHAM is 8.8%App. C.4, Table A
Judgesevery shift against the judge's own SHAM and retest floorsTable 1

The paper now states preservation as "no detectable change relative to SHAM and retest variability" (Secs. 3.4 and 5.4; App. C.4).

We believe these additions resolve the three concerns, and we would be grateful if the reviewer would reconsider the score in light of them.

Dear Reviewer, thank you for your constructive feedback. We believe the additional analyses and revisions have substantially addressed the concerns raised in your review. Should you find our responses convincing, we would be grateful if you could reconsider your initial rating in light of expanded discussion.

Reviewer_TGaLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether MLLM judges used to evaluate image editing respond to edit quality or to cues unrelated to quality, and introduces EditJudgeBias, an audit of 1,196 editing triplets drawn from five benchmarks (three independent content pools). It applies 13 counterfactual cues at four sites, namely protocol (position swap), prompt (bandwagon and model-name claims), pixel (text overlay, watermark, brightness, saturation, aesthetic filter, padding) and content (red box, inset, caption, sticker), together with a SHAM control, and checks that the requested edit is preserved using MLLM validators and human annotation (Sec. 3, Fig. 3, App. C). Effects are measured on three axes with BH correction: invariance, which compares score shifts against judge-specific sham and re-query noise floors (Eq. 1–3); agreement with human MOS or preferences, via Spearman Δρ\Delta\rho and McNemar tests (Eq. 4); and pairwise stability, via CALM-style robustness and consistency rates (Eq. 5). Across four proprietary or API judges and VIEScore, the paper reports that bandwagon claims inflate scores for every judge, stickers shift scores more than every pixel cue, order swaps reverse up to 60.9% of decisions against a 10.5% re-query rate, and 33 of 34 significant agreement changes are decreases (Sec. 5). A downstream analysis further shows that cue injection can change the top-ranked editor on EBench-18K (App. E, Table 10).

Critical strengths
  1. A careful statistical design that separates cue effects from judge noise. Each judge's score shift is compared with its own SHAM (JPEG q95) and re-query noise floors, and a cell is marked significant only if its 95% bootstrap lower bound exceeds both floors (Eq. 2–3, Table 1). Pairwise effects are calibrated against re-query flip rates (Fig. 4b: ratios of ×1.5–×38), and the analysis further uses turn-clustered bootstrap, BH correction and a separate replication arm with qwen3-vl-32b. The floors matter in practice: Gemini's sham floor alone is 15.2–17.5 (Table 1), so score differences of that size would otherwise be read as bias. The findings that pass this gate are therefore more convincing than comparisons of raw score differences.
Critical weaknesses
  1. For occluding cues, construct validity is unresolved on the Editing Quality and Detail Preservation dimensions. Preservation is verified only against the requested edit (Sec. 3; App. C; acknowledged in App. A), yet stickers, text overlays, padding, red boxes and insets change or cover regions outside the edit. Under the rubric's EQ/DP definitions, a judge that lowers EQ/DP for these cues may therefore be scoring correctly rather than showing bias, but Table 1 and the agreement analysis in Sec. 5 count such EQ/DP shifts as bias.

  2. The evidence that cue injection preserves the requested edit is weaker than presented. Only the second human package is sham-controlled and explicitly blinded to cue identity, whereas the first package (150 items, 5 pixel cues) has no SHAM control and is not described as blinded. In addition, the annotators knew the study hypothesis (App. C.2), and gpt-4o-mini, one of the two gate validators, detects reversion only 3–9% of the time (Table 2, Panel B).

Questions for authors
  1. For occluding cues (sticker, text overlay, padding, box), do the significant effects in Table 1 remain when restricted to the IA dimension only? Alternatively, could you collect human EQ/DP ratings on cue-injected images to test whether judge shifts exceed the human shifts?
  2. Can you repeat the first human preservation package (5 pixel cues) with SHAM controls, with cue identity hidden, and with annotators unaware of the hypothesis?
  3. Fig. 4d / Table 6 report 8 of 20 adjacent rank inversions across the three measures, against roughly 10 expected under independence. Why does this support the claim that the measures capture distinct properties, rather than reflecting the limited power of a comparison over only five judges?
  4. The mitigation results (two-order reconciliation, standardized-median ensemble; App. E, Tables 11–12) are preliminary. Do these remedies restore agreement with human judgments when evaluated on held-out data?
3 comments
AuthorsOpenReview ↗

Response to Reviewer TGaL (1/2)

We thank the reviewer for recognizing the noise-floor design and for the precise, constructive questions. We ran every analysis the reviewer requested; each answer below points to where the result appears in the revision.

detailed results in revision (changes in blue):

PointNew resultLocation
W1 / Q1Human IA, EQ and DP ratings of cue-injected images; IA-only effectsSec. 3.4; App. C.5 (Table C.5); Tables 1, 3
W2 / Q2Blinded, SHAM-controlled human check of all ten image-side cues; gpt-6-luna validator with a neutral promptSecs. 3.4, 5.4; App. C.4–C.5 (Tables A, C.5; Figure B); Table 2, Panel B
Q3Rankings with ten judges, with bootstrap CIsSec. 5.5; App. D.2 (Table 6, Panel C)
Q4Both remedies on held-out human preference pairsSec. 5.3; App. E.3 (Tables C–D)

W1 / Q1. Construct validity of EQ/DP shifts under occluding cues

We address this from two directions.

Human EQ/DP ratings. Following the reviewer's suggestion, we collected human ratings on all three dimensions. We drew 25 base edits stratified over the five source datasets and created one task per base edit for each of the ten image-side cues and SHAM (275 tasks), plus 25 tasks in which the edit is reverted to the source as a positive control (300 tasks in total). Two external annotators who are not involved in this study worked independently, blind to the hypothesis and to the condition of each task, with left/right order randomized. For IA, EQ and DP separately, they reported whether the cued version differs from the uncued edit (No / Slightly / Yes), judging each dimension by its definition and explicitly not ignoring visible elements such as stickers, text or borders.

The annotators almost never report EQ or DP changes for occluding cues: text overlay and padding 0/25 on every dimension, box EQ 1/25 and DP 2/25, sticker EQ 1/25 and DP 4/25, none of which differs from paired SHAM. The judges, in contrast, lower EQ significantly in 19/20 and DP in 20/20 occluding-cue cells (Tables 4–5). The judges' EQ/DP shifts therefore exceed the human shifts, which supports reading them as bias. The only nominal human signal is DP for the inset (8/25; not significant after BH correction), a cue the paper already treats as borderline and excludes from aggregate counts.

Occluding cueEQ(Humans)DP(Humans)EQ(Judges)DP(Judges)
Text overlay0/250/255/55/5
Padding0/250/254/55/5
Box1/252/255/55/5
Sticker1/254/255/55/5
SHAM0/250/25––
Reverted edit25/2525/25––

IA only (Tables 1 and 3). The effects persist when the analysis is restricted to IA. The sticker still exceeds both noise floors for 4/5 judges, and text overlay, padding and box for 3/5. The signed IA shifts are BH-significant for the sticker in 5/5 judges and for the other three cues in 4/5. The observed effects are therefore not driven by EQ or DP alone.

W2 / Q2. Strength of the preservation evidence

We strengthened both sources of evidence that the reviewer identified.

Validator25% reversion50% reversion100% reversionBlur
gemini-3.546 (49%)69 (74%)88 (95%)91 (98%)
gpt-4o-mini7 (8%)9 (10%)3 (3%)83 (89%)
gpt-6-luna31 (60%)47 (90%)51 (98%)52 (100%)

MLLM validators. We re-ran the validation with gpt-6-luna, the designated successor of gpt-4o-mini (which rarely flags reversion and is being retired), as a second validator independent of gemini; Table 2 keeps the published gate results for reference. Its prompt neither calls the change cosmetic nor lists transformations (Figure B). On held-out controls, gpt-6-luna detects every type of degradation at least as often as gemini (Table 2, Panel B).

Human checks (Q2). Yes. The study above repeats the human check for all ten image-side cues, including the five pixel cues of the first package. Each cue is paired with SHAM on the same base edit, condition identity and the hypothesis are hidden, and the annotators are external to the study. No cue shows a detectable difference from SHAM on any dimension, the positive control is detected 25/25, and the two annotators agree on 95–97% of the labels (Cohen's κ = 0.81–0.85; Table C.5). The human quality check is thus reproducible and reliable.

AuthorsOpenReview ↗

Response to Reviewer TGaL (2/2)

Q3. Distinct measures or low power with five judges?

We answer this directly by doubling the number of judges. The panel now has ten judges: the paper's six (the five primary judges and the replication judge qwen3-vl-32b) and four judges from vendors not represented in the paper (doubao-seed-2.0-lite, claude-haiku-5.5, glm-5.3-flash, deepseek-v4.1-flash). All three measures are computed with the same code on a fixed subset.

Overall agreement is close to that of unrelated measures. Adjacent orderings reverse 38 of 90 times, against 44.5 expected under independence (P(≤ 38) = 0.23), and Kendall's W across the three rankings is 0.30, against a chance level of 1/3 (permutation p = 0.58). For example, gemini-3.5 ranks 10th on invariance and 1st on stability.

JudgeInvarianceAgreementStability
gpt-5.5672
gemini-3.51061
kimi-k2.5124
qwen3.54510
VIEScore5105.5
qwen3-vl-32b217
doubao-seed-2.0-lite949
claude-haiku-5.5893
glm-5.3-flash785.5
deepseek-v4.1-flash338

Rank among the ten judges on each measure

Confidence intervals rule out interchangeable measures. Two-level bootstrap 95% CIs of Kendall's τ are [−0.69, 0.33] between stability and invariance and [−0.51, 0.29] between stability and agreement. Both upper bounds lie below 0.5, whereas interchangeable estimates of a single quantity would show strong concordance (τ ≥ 0.5).

These results support the paper's claim that the three measures capture different aspects of judge robustness.

Q4. Do the remedies restore agreement with humans on held-out data?

We evaluated both remedies on held-out data; neither restores agreement with humans, consistent with the paper's conclusion. We drew 300 human preference pairs from EditReward-Bench (Luo et al., 2026), stratified by subtask and with no detectable overlap with our content pools, and used the same five judges and prompts without fitting anything to human labels.

Two-order reconciliation raises decided-pair accuracy for all five judges, but only by abstaining: coverage drops by up to 42 percentage points, and the 253 verdicts withdrawn under the MT-Bench rule had been correct only 51% of the time, close to chance.

JudgeDecided-pair accuracyCoverageCoverage-adjusted accuracy
gpt-5.50.809 → 0.8560.997 → 0.8800.808 → 0.813
gemini-3.50.866 → 0.8890.993 → 0.9330.863 → 0.863
kimi-k2.50.853 → 0.8860.997 → 0.9030.852 → 0.848
qwen3.50.718 → 0.8750.957 → 0.5330.708 → 0.700
VIEScore0.790 → 0.8410.967 → 0.8170.780 → 0.778

Base order → MT-Bench rule on 300 held-out pairs per judge

The standardized-median ensemble still shifts significantly under a sticker or a text overlay, with all five judges moving in the same direction. When the cue is placed on the human-preferred edit, its accuracy falls to 0.505 and 0.623, comparable to the judge average (0.513 and 0.638). Without a cue, it is 0.027 above the judge average but not above the best single judge.

Accuracy against human preferenceEnsembleJudge meangemini-3.5
No cue0.8190.7920.833
Sticker on the preferred edit0.5050.5130.495
Text overlay on the preferred edit0.6230.6380.605

This confirms the conclusion of App. E: such biases cannot be removed by simple output-level remedies, which is why auditing them matters.

We hope these results resolve the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score.

Thank you for the additional analyses. The blinded, SHAM-controlled human study and the IA-only results substantially address my main construct-validity concern. Overall, the new evidence strengthens the paper, and I maintain my weak-accept recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.