- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This article proposes EditJudgeBias for the systematic study of bias issues in MLLM in image editing evaluation, including 1196 editing examples and 13 bias interference factors. The author introduces a quality preservation verification mechanism and noise baseline to distinguish between score changes caused by bias, real image quality changes, and random fluctuations in the model itself. The experiment found that MLLM is easily influenced by irrelevant visual elements and contextual information.
Critical strengths
- MLLM is increasingly being used for automatic evaluation of image generation and editing tasks. This article studies the bias problem of MLLM and has academic value.
- The experimental design is relatively rigorous, and multiple verifications have been conducted to maintain quality. In the evaluation, model bias, image quality changes, and bifurcation changes caused by random fluctuations in the model itself have been effectively distinguished. The experimental analysis is comprehensive, and multidimensional analysis was conducted on different bias interference factors.
Critical weaknesses
- The main research findings lack novelty. The phenomenon of being easily influenced by contextual information interference and visual factors during the MLLM evaluation process has been widely discussed in previous research, and the conclusions of this article can largely be expected.
- Lack of in-depth analysis of the causes of bias formation. The paper only demonstrates different types of evaluation bias through a large number of experiments, but lacks sufficient explanation for the possible reasons behind their occurrence, and lacks mitigation solutions.
Questions for authors
- Besides extending existing bias analysis to image editing tasks, what are the differences from existing work?
- What are the underlying causes and mechanisms of the different types of evaluation bias identified in the experiments? such as Why do small stickers have a greater impact than the entire image transformation? Could the authors provide some analysis to develop and evaluate corresponding strategies to alleviate these different bias?
- It is mentioned in the article that MLLM Judge is used for reinforcement learning reward signals. How do these biases studied by the author affect downstream reinforcement learning?
3 comments
Response to Reviewer d336 (2/2)
Q3. Impact on downstream RL rewards
These cues occur naturally. The Framing and Typographic examples in Figure 1 are taken from widely used datasets, and the judges' comments explicitly cite these irrelevant elements as reasons for deducting points.
Setting. We used EditScore-7B (Luo et al., 2026), an open reward model built for online RL in image editing, and scored through its own RL reward path. Each of 611 edits is scored once before and once after cue injection; the two calls differ only in that cue, with the same instruction, source image and sampling seed.
The cues systematically change the reward. Of the five image-side cues that shift all five of the paper's judges in the same direction, four also lower the reward significantly, and the three largest effects hold under Avg@4 with larger standardized effects.
| Cue | Δ reward [95% CI] | Lower / higher | dz: 1 pass / Avg@4 | q |
|---|---|---|---|---|
| Sticker | −0.122 [−0.142, −0.103] | 59% / 18% | −0.50 / −0.61 | 3×10⁻³³ |
| Box | −0.071 [−0.089, −0.053] | 47% / 26% | −0.31 / −0.41 | 3×10⁻¹² |
| Text overlay | −0.050 [−0.070, −0.030] | 44% / 32% | −0.20 / −0.28 | 7×10⁻⁵ |
| Brightness | −0.020 [−0.032, −0.008] | 42% / 27% | −0.13 / – | 3×10⁻⁵ |
| Padding | −0.014 [−0.031, +0.002] | 37% / 34% | −0.07 / – | 0.19 |
| SHAM (null) | −0.002 [−0.012, +0.008] | 33% / 29% | −0.02 / – | p = 0.46 |
They bias the policy-gradient signal. In 77 GRPO-style groups, each containing the edits of eight editing models for the same instruction, replacing one candidate with its sticker version lowers its group advantage by 0.25 on average (SHAM: +0.03) and flips its sign for 24% of candidates (SHAM: 9%). A flipped advantage means that the bias reverses the direction of the policy update for that candidate.
We hope these new results, which go beyond documenting the biases to explaining and mitigating them and to measuring their effect on RL rewards, address the reviewer's concerns, and we would be grateful if the reviewer would reconsider the score.
Dear reviewer, thank you for the helpful feedback. We hope that our clarifications and the additional experiments presented in the revised manuscript have adequately addressed your concerns. If you find these new results satisfactory, we would sincerely appreciate your consideration of a score update that reflects the strengthened evidence and discussion.
Response to Reviewer d336 (1/2)
We thank the reviewer for the thoughtful assessment. We added a mechanism analysis with a causal test and representation-level mitigation, and an RL-reward experiment; they address W2/Q2 and Q3 directly.
detailed results in revision (changes in blue):
W1 / Q1. Novelty beyond prior judge-bias work
To our knowledge, EditJudgeBias is the first audit that studies cues changing image content at the semantic level and that rigorously verifies whether the edit's quality is preserved before and after cue injection.
An audit built for editing. The judge evaluates a triplet (source image, instruction, edited image), so cues can be anchored to the edit itself (box, inset, caption, sticker). The sticker (Distraction), for example, carries the known "irrelevant-information distraction" effect from the text modality into the judge's visual input, which, to our knowledge, no prior judge-bias audit has tested.
Preservation is tested, not assumed. Prior audits of MLLM judges (FRAME, EMNLP 2025; MM-JudgeBias, ACL 2026) inject visual cues for text-to-image and general multimodal inputs, but assume without verification that the cues leave the evaluated quality unchanged. We test this with calibrated validators and blinded human ratings on all three quality dimensions (App. C).
New findings. Our pixel-level cues include the main manipulation types studied before (e.g., brightness, filters, borders and text overlays; Sec. 3.3), and the content-level cues reveal a semantic-level bias that is both subtler and larger: a sticker shifts the judges more than any pixel-level cue (Table 1).
W2 / Q2. Mechanism and mitigation; why a small sticker outweighs a global transform
App. E already evaluates two output-level remedies (two-order reconciliation and the standardized-median ensemble), and neither removes a bias that the judges share. Following UnfairJudge (Xu et al., 2026b), we added a mechanism analysis that first examines the representation geometry and then tests causality, and we evaluated mitigation at both levels.
Setting. We ran Qwen3-VL-8B-Instruct with the paper's scoring prompt on 300 breadth samples and read the residual stream at the last prompt token of layer 20. The total score S is the sum of IA, EQ and DP (3–30), and shifts are Hodges–Lehmann estimates.
The open judge reproduces the bias. Sticker, brightness and bandwagon shift its score in the same direction as for qwen3-vl-32b and qwen3.5-plus (3/3), and the sticker, the largest effect, lowers S significantly by a comparable amount (Table H, Panel A).
What drives the effect. How much a cue moves the score depends on the direction in which it pushes the internal representation, and on whether it pushes all samples the same way, not on how many pixels it changes (Table H, Panel B). The sticker changes only 1.2% of the pixels, yet pushes the samples consistently in the same direction, closely aligned with the direction that separates human-rated poor edits from good ones. Brightness changes far more pixels, but its shifts point in different directions and largely cancel. Bandwagon text is nearly orthogonal to the quality direction, so its influence bypasses the judge's perception of edit quality.
Causal test and representation-level mitigation. Adding the captured sticker direction to uncued inputs reproduces 70% of the sticker's score drop, and subtracting it from affected inputs removes 72% (Table H, Panel C), whereas norm-matched random directions barely move the score on average. For cues such as the sticker, which act through the representation of edit quality, this is a representation-level mitigation. Bandwagon bypasses that representation, so its mitigation belongs at the input stage.
We look forward to future work that builds on this analysis.