- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper trains relatively small image judges to evaluate generated images in a structured way. For each prompt–image pair, the judge selects relevant facets from 60 facets across seven dimensions and returns scores, rationales, and defect locations on a 3×3 grid where applicable. Training targets combine judgments from Kimi K2.6, Qwen3.8-Max, and Gemini 2.5 Flash with specialist checks from Qwen-VL-Max. Disputed facets undergo a challenger–defender analysis followed by separate adjudication using Gemini 3.1 Pro. The students, initialized from Qwen3-VL-8B and Qwen3.8-27B, receive SFT on about 30K production prompt–image pairs. Further refinement diagnoses recurring errors, generates targeted data, and combines single-facet DPO, full-judgment GRPO, and historical replay. Experiments evaluate facet selection, evidence grounding, localization, score consistency, generator ranking on Qwen-Image-Bench, and pairwise comparison on MJ-Bench. A downstream human study evaluates SDXL models trained using the judges to provide preferences for Diffusion-DPO.
Critical strengths
-
The structured output is useful for diagnosis. Scores, rationales, and defect locations help users understand what is wrong and where, beyond a single overall score.
-
The ablations are extensive. The paper compares single-teacher, simple-merge, and full adjudication targets under the same data budget, includes matched-token and matched-budget continued-SFT controls, and tests several data allocation strategies. These controls support gains beyond additional data or training alone. The downstream human study provides further evidence of practical value: across seven matchups with 400 comparisons each and five blinded ratings per comparison, the SDXL model trained with preferences from the 27B judge outperforms those trained with preferences from Qwen3.8-Max or Kimi K2.6.
Critical weaknesses
-
Evaluation independence and facet coverage remain concerns. Qwen3.8-Max serves as both a teacher and the auditor for two metrics. The human subset provides useful validation, but aggregate results do not establish reliability across all 60 facets. Comparisons with representative specialized reward models and the Gemini 3.1 Pro adjudicator are also missing.
-
The adjudication pipeline can retain shared teacher errors, particularly when majority-inapplicable facets bypass debate. Table 7 documents consensus errors but does not distinguish omitted facets from incorrect scores. Since specialist checks also rely on VLM outputs, it remains unclear how effectively the pipeline can detect and correct these shared mistakes.
Questions for authors
-
Qwen3.8-Max serves as a teacher, a baseline, and the auditor for selection appropriateness and unsupported claims, creating a risk of evaluator-specific bias. The 150-image human subset partly addresses this. Could you repeat these audits with an evaluator not involved in data construction and report paired confidence intervals for model differences on the human subset?
-
Could you include representative specialized reward or judge models, such as ImageReward, HPS v2, VisionReward, UnifiedReward, or LLaVA-Critic, on MJ-Bench pairwise accuracy or generator ranking? These tasks do not require the full facet-level output. Gemini 3.1 Pro, the adjudicator, is also absent as a standalone baseline. How does the student compare with it?
-
Table 7 shows that 32 of the 80 remaining errors occur when teachers agree but are wrong. Majority-inapplicable facets skip debate, so shared omissions may escape adjudication. How many of these 32 errors involve agreed-null facets versus incorrect numeric scores? Could you audit a sample of agreed-null facets and report the missed-facet rate? Since the specialist checks are also VLM outputs, would independent OCR or object-detection tools reduce errors on the relevant facets?
-
The paper defines 60 facets across seven dimensions, but their coverage by human-labeled evaluation is unclear from Table 15, and Table 1 reports only aggregate results. Could you report performance by dimension and identify which facets have human-annotated evaluation, particularly for creative generation, real-world fidelity, text rendering, and safety? For facets without independent evaluation, teacher-adjudicated supervision alone does not establish the student’s reliability.
6 comments
New Comparisons with Specialized Reward Models and the Adjudicator Gemini 3.1 Pro, and a Full Audit of Shared Teacher Errors (Part 2/2, Q2, W2, Q3)
We next answer Q2 (specialized baselines, adjudicator), then W2 and Q3 (shared teacher errors).
- Regarding Q2, we added comparisons on the two tasks you suggest, choosing the better image on MJ-Bench and ranking generators on Qwen-Image-Bench, which every judge can perform. We newly evaluated UnifiedReward-2.0 (official prompts) and Gemini 3.1 Pro; ImageReward, HPS v2, PickScore, GPT-4V, and GPT-4o are literature results from MJ-Bench's quality subset.
Table 4. Specialized reward models and other judges on shared tasks ("–": not reported by MJ-Bench).
| Judge | Human training labels | Choosing the better image (MJ-Bench) ↑ | Generator ranking ρ ↑ |
|---|---|---|---|
| ImageReward / HPS-v2.1 / PickScore (MJ-Bench) | Human preferences | 63.5% / 67.3% / 83.8% | – |
| GPT-4V / GPT-4o (MJ-Bench) | – | 90.4% / 97.6% | – |
| UnifiedReward-2.0 (new) | Preference data | 84.9% | 0.854 |
| Q-Judger | 130k+ expert pairs | 92.3% | 0.874 |
| Kimi K2.6 | – | 97.1% | 0.911 |
| Qwen3.8-Max | – | 97.8% | 0.928 |
| Gemini 3.1 Pro (new) | – | 98.5% | 0.934 |
| EviForge-8B | None | 97.0% | 0.922 |
| EviForge-27B | None | 99.0% | 0.957 |
Even on these whole-image tasks, built for reward models, EviForge-27B achieves the highest scores, and EviForge-8B clearly exceeds every specialized reward model (97.0% vs. at most 92.3%) and is comparable to the large models. The reason is the supervision: reward models learn one overall score from human preferences, whereas EviForge learns per-facet judgments checked against image evidence, which separate defects an overall score blurs. VisionReward's paper reports preference accuracy at the level of PickScore and HPS v2 (81.7% vs. 79.8% and 83.3%, HPDv2). LLaVA-Critic judges text responses of multimodal models, not generated images, so it is not included.
Comparison with the adjudicator. EviForge-27B reaches the quality of its adjudicator, Gemini 3.1 Pro, matching or exceeding it on nine of eleven measures (Table 5; paper Table 26); Gemini keeps a higher evidence hit and slightly fewer unsupported claims. The student is therefore not capped by the adjudicator: the adjudicator only settles disputed facets during construction, while refinement then trains the student on its own verified errors.
Table 5. EviForge versus the adjudicator Gemini 3.1 Pro.
| Measure | Gemini 3.1 Pro | EviForge-27B | EviForge-8B |
|---|---|---|---|
| Missed required facets ↓ | 27.1% | 25.4% | 14.0% |
| Selection appropriateness ↑ | 96.3% | 98.7% | 98.2% |
| Unsupported claims ↓ | 1.6% | 1.8% | 1.9% |
| Rationales cover annotated defects ↑ | 98.6% | 96.1% | 96.2% |
| Defects located ↑ | 72.0% | 75.4% | 85.6% |
| Score ordering ↑ (/114) | 111 | 113 | 111 |
| Low-quality detection ↑ (/128) | 126 | 127 | 127 |
| Choosing the better image ↑ | 98.5% | 99.0% | 97.0% |
| Generator ranking ρ ↑ | 0.934 | 0.957 | 0.922 |
| Changed judgments under attribute edits ↓ | 13 | 1 | 2 |
| Human evaluation, 800 images ↑ | 96.7% | 97.9% | 97.0% |
- Regarding W2 and Q3, we agree that agreed facets skip debate, so errors shared by all teachers cannot be caught by adjudication. We answer each part; all analyses are added to Appendix D.
- Omissions versus wrong scores. Of the 32 consensus errors in Table 7, 15 are agreed-null omissions and 17 are wrong scores.
- Missed-facet rate of agreed-null facets. Auditing all of them on the 800-image human set, of 33,612 facets all three teachers marked inapplicable, humans judged 357 applicable, a missed-facet rate of 1.1%.
- Does training reinforce shared errors? On the same set, all teachers agree but contradict humans on 962 of 48,000 decisions against Rater 1 (2.0%). On these, correct student answers go 397 → 384 → 572 for 8B and 472 → 463 → 682 for 27B (before training → distillation → refinement). Distillation imitates teacher labels and absorbs a few; refinement instead trains on new images built for the student's errors, kept only if their labels match what the prompt asked for, so answers are checked against the image, not teacher agreement; this corrects many shared errors. The remaining shared errors are 390/48,000 (0.81%) for 8B and 280/48,000 (0.58%) for 27B.
- Independent OCR and detection tools. Our specialist, Qwen-VL-Max, already acts as a dedicated tool via task-specific prompts for text reading, object detection, and spatial relations. We chose it for its strong text reading: on CC-OCR (ICCV 2025), general multimodal models such as Qwen-VL-Max outperform specialist OCR models, and Qwen-VL-Max ranks among the top in scene-text reading. In early development, human inspection showed it to be clearly more reliable than lightweight OCR and detection tools, which struggle with the artistic fonts, malformed characters, and stylized objects of generated images. Combining such tools with VLM evidence is future work (Limitations, Appendix M).
We sincerely hope these responses address your concerns and earn your recognition.
Thanks to the authors for their detailed responses and additional experiments! The authors’ additional analyses address most of my concerns.
I have one small follow-up question. My understanding is that the raters checked all 60 facets for each image in Appendix K, and the overall accuracy also counts correct “not applicable” judgments. Could you use the existing human annotations to report results just for the facets that humans marked as “applicable”, including those the model missed? This would help show how the model performs when a score and rationale are actually needed.
Thanks again!
A New Independent Human Evaluation on 800 Images with Paired Confidence Intervals, and Per-Dimension Results Across All 60 Facets (Part 1/2, W1, Q1, Q4)
Thank you for your careful review and for recognizing the diagnostic value of our structured output, the extensive ablations, and the downstream human study. We answer Q1 and Q4 together first because both concern W1, whether our evaluation is independent of the teachers and covers all 60 facets, and both are answered by one new human evaluation; Part 2 covers Q2, W2, and Q3.
Regarding Q1 and Q4, please allow us to first clarify which abilities and dimensions each evaluation covers, and how (Table 1). Each benchmark tests a specific ability with its own human labels; test images are held out from training, and all judges share the same rules.
Table 1. Evaluated abilities, dimensions, references, and methods.
As requested, we report paired 95% confidence intervals against the original auditor and baseline, Qwen3.8-Max, scoring both judges on the same 800 images, resampled 10,000 times (Table 2).
Table 2. Independent human assessment of facet judgments (accuracy).
EviForge-8B and EviForge-27B reach 97.0% and 97.9%, exceeding both larger judges; all three raters give this ordering, and both intervals exclude zero. DeepSeek-V4.1-Flash-1017, not involved in data construction, also re-checked the same outputs and ranks the judges nearly as humans do (ρ = 0.983). These results agree with the original audit, now confirmed without any teacher (Appendix K).
Table 3. Human evaluation by dimension (800 images, three raters combined).
Both EviForge judges reach at least 96.5% on every dimension, including the four you named, and are never below Qwen3.8-Max. Appendix I lists the human references and H800 coverage for all 60 facets, grouped by dimension, and reports these results (Tables 19–20).