acceptodds
Under review as a conference paper at ICLR 2027

EviForge: Adaptive Visual Judge Models Evidence and Weakness Guided Training

Abstract

Generated-image evaluation must adapt its criteria to each prompt-image pair while remaining fast, affordable, and private enough for routine use. Distilling multimodal large language models into a local judge is practical, but teacher errors can contaminate supervision, while untargeted follow-up training may miss the student's actual weaknesses. We therefore introduce EviForge, a training method for lightweight, adaptive judge models with three components. Sample-adaptive facet judgment selects the applicable items from 60 facets under seven dimensions and returns scores, evidence-grounded rationales, and defect locations. Evidence-adjudicated multi-teacher distillation combines generalist judgments with routed specialist evidence: agreed facets are merged, while disputed claims undergo challenge and defense before separate adjudication. Weakness-driven refinement diagnoses recurring errors by facet, scene, error type, and direction and constructs targeted prompts and images. Verified examples then train the student by correcting individual facet errors, calibrating the complete judgment, and replaying historical data to retain learned abilities. We instantiate EviForge on Qwen3-VL-8B and Qwen3.8-27B. Across eight public datasets and blinded human evaluations, we assess facet selection, score consistency, evidence-grounded rationales, defect localization, image comparison and ranking, and generator improvement when the judge is used as a reward model. Both final models improve substantially over their corresponding backbones on most measures. For example, EviForge-8B improves defect-localization coverage by 63 percentage points and evidence hit by 31 percentage points. On most reported measures, both match or surpass the much larger Qwen3.8-Max and Kimi K2.6.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-Judge, Sample-Adaptive Facet Evaluation, Evidence-Adjudicated Multi-Teacher Distillation, Weakness-Driven Refinement, Knowledge Distillation, Visual Generation Evaluation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_WBYzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper trains relatively small image judges to evaluate generated images in a structured way. For each prompt–image pair, the judge selects relevant facets from 60 facets across seven dimensions and returns scores, rationales, and defect locations on a 3×3 grid where applicable. Training targets combine judgments from Kimi K2.6, Qwen3.8-Max, and Gemini 2.5 Flash with specialist checks from Qwen-VL-Max. Disputed facets undergo a challenger–defender analysis followed by separate adjudication using Gemini 3.1 Pro. The students, initialized from Qwen3-VL-8B and Qwen3.8-27B, receive SFT on about 30K production prompt–image pairs. Further refinement diagnoses recurring errors, generates targeted data, and combines single-facet DPO, full-judgment GRPO, and historical replay. Experiments evaluate facet selection, evidence grounding, localization, score consistency, generator ranking on Qwen-Image-Bench, and pairwise comparison on MJ-Bench. A downstream human study evaluates SDXL models trained using the judges to provide preferences for Diffusion-DPO.

Critical strengths
  1. The structured output is useful for diagnosis. Scores, rationales, and defect locations help users understand what is wrong and where, beyond a single overall score.

  2. The ablations are extensive. The paper compares single-teacher, simple-merge, and full adjudication targets under the same data budget, includes matched-token and matched-budget continued-SFT controls, and tests several data allocation strategies. These controls support gains beyond additional data or training alone. The downstream human study provides further evidence of practical value: across seven matchups with 400 comparisons each and five blinded ratings per comparison, the SDXL model trained with preferences from the 27B judge outperforms those trained with preferences from Qwen3.8-Max or Kimi K2.6.

Critical weaknesses
  1. Evaluation independence and facet coverage remain concerns. Qwen3.8-Max serves as both a teacher and the auditor for two metrics. The human subset provides useful validation, but aggregate results do not establish reliability across all 60 facets. Comparisons with representative specialized reward models and the Gemini 3.1 Pro adjudicator are also missing.

  2. The adjudication pipeline can retain shared teacher errors, particularly when majority-inapplicable facets bypass debate. Table 7 documents consensus errors but does not distinguish omitted facets from incorrect scores. Since specialist checks also rely on VLM outputs, it remains unclear how effectively the pipeline can detect and correct these shared mistakes.

Questions for authors
  1. Qwen3.8-Max serves as a teacher, a baseline, and the auditor for selection appropriateness and unsupported claims, creating a risk of evaluator-specific bias. The 150-image human subset partly addresses this. Could you repeat these audits with an evaluator not involved in data construction and report paired confidence intervals for model differences on the human subset?

  2. Could you include representative specialized reward or judge models, such as ImageReward, HPS v2, VisionReward, UnifiedReward, or LLaVA-Critic, on MJ-Bench pairwise accuracy or generator ranking? These tasks do not require the full facet-level output. Gemini 3.1 Pro, the adjudicator, is also absent as a standalone baseline. How does the student compare with it?

  3. Table 7 shows that 32 of the 80 remaining errors occur when teachers agree but are wrong. Majority-inapplicable facets skip debate, so shared omissions may escape adjudication. How many of these 32 errors involve agreed-null facets versus incorrect numeric scores? Could you audit a sample of agreed-null facets and report the missed-facet rate? Since the specialist checks are also VLM outputs, would independent OCR or object-detection tools reduce errors on the relevant facets?

  4. The paper defines 60 facets across seven dimensions, but their coverage by human-labeled evaluation is unclear from Table 15, and Table 1 reports only aggregate results. Could you report performance by dimension and identify which facets have human-annotated evaluation, particularly for creative generation, real-world fidelity, text rendering, and safety? For facets without independent evaluation, teacher-adjudicated supervision alone does not establish the student’s reliability.

6 comments
AuthorsOpenReview ↗

A New Independent Human Evaluation on 800 Images with Paired Confidence Intervals, and Per-Dimension Results Across All 60 Facets (Part 1/2, W1, Q1, Q4)

Thank you for your careful review and for recognizing the diagnostic value of our structured output, the extensive ablations, and the downstream human study. We answer Q1 and Q4 together first because both concern W1, whether our evaluation is independent of the teachers and covers all 60 facets, and both are answered by one new human evaluation; Part 2 covers Q2, W2, and Q3.

Regarding Q1 and Q4, please allow us to first clarify which abilities and dimensions each evaluation covers, and how (Table 1). Each benchmark tests a specific ability with its own human labels; test images are held out from training, and all judges share the same rules.

Table 1. Evaluated abilities, dimensions, references, and methods.

Ability evaluatedDimensions coveredBenchmark and referenceEvaluation method
Facet selectionPrompt alignment, visual quality, aesthetics, safetyVisionRewardDB: human attribute annotationsCount required facets omitted
Score consistencyVisual quality, prompt alignmentVisionRewardDB: quality order; ImageRewardDB: quality labelsCheck score order and low-quality detection
Evidence-grounded rationalesPrompt alignment, visual qualityRichHF-18K, EvalMuse-40K: human defect annotationsCheck rationales against annotated defects
Defect localizationVisual quality, prompt alignmentRichHF-18K: human spatial annotationsCheck coverage of defect cells
Image comparisonVisual qualityMJ-Bench quality: human preferencesMatch human choices
Generator rankingPrompt alignment, visual quality, aesthetics, real-world fidelity, creative generationQwen-Image-Bench: expert generator orderCompare with expert ranks
Score stabilitySafetyMJ-Bench bias: matched demographic variantsMeasure score changes
Facet selection, scores, and rationales (new)All seven dimensions800 images, three independent ratersCheck every facet
  1. Regarding Q1, no benchmark labels, for all 60 facets, whether each selected facet applies and whether its rationale is correct, so the paper used Qwen3.8-Max to audit these two decisions; we acknowledge the teacher-auditor dependence and apologize for not making it clear. To address this, we added a fully independent human evaluation on 800 images, 100 from each of the eight evaluation sources. Three raters independently checked all 60 facets for each image and model (48,000 decisions per rater per model), assessing whether each facet applies and whether its score and rationale are reasonable. No teacher, adjudicator, or AI model produced or reviewed these assessments.

As requested, we report paired 95% confidence intervals against the original auditor and baseline, Qwen3.8-Max, scoring both judges on the same 800 images, resampled 10,000 times (Table 2).

Table 2. Independent human assessment of facet judgments (accuracy).

JudgeRater 1Rater 2Rater 3CombinedEviForge − Qwen3.8-Max [95% CI]
EviForge-8B97.62%96.31%97.03%97.0%+2.34 pp [+1.76, +2.92]
EviForge-27B98.08%97.71%97.86%97.9%+3.24 pp [+2.65, +3.82]
Qwen3.8-Max94.35%93.72%95.88%94.6%–
Kimi K2.690.79%89.84%92.75%91.1%–

EviForge-8B and EviForge-27B reach 97.0% and 97.9%, exceeding both larger judges; all three raters give this ordering, and both intervals exclude zero. DeepSeek-V4.1-Flash-1017, not involved in data construction, also re-checked the same outputs and ranks the judges nearly as humans do (ρ = 0.983). These results agree with the original audit, now confirmed without any teacher (Appendix K).

  1. Regarding Q4, thank you for this suggestion. In the original evaluation (Table 1, first seven rows), each ability is tested on benchmarks built for it, with their own human labels. To further evaluate all seven dimensions, including the creative generation, real-world fidelity, text rendering, and safety you mention, in one unified setting, we use the new human evaluation from Q1 (Table 1, last row): the same 800 images, all 60 facets, three raters. Table 3 reports it by dimension, with Qwen3.8-Max as reference.

Table 3. Human evaluation by dimension (800 images, three raters combined).

Dimension (facets)EviForge-27BEviForge-8BQwen3.8-Max
Prompt alignment (19)97.9%96.5%94.8%
Text rendering (5)97.7%96.9%96.5%
Creative generation (13)98.7%97.4%96.7%
Visual quality (7)97.2%96.7%95.3%
Aesthetics (5)96.9%97.5%87.9%
Real-world fidelity (8)97.6%97.6%94.3%
Safety (3)98.3%96.9%92.7%

Both EviForge judges reach at least 96.5% on every dimension, including the four you named, and are never below Qwen3.8-Max. Appendix I lists the human references and H800 coverage for all 60 facets, grouped by dimension, and reports these results (Tables 19–20).

AuthorsOpenReview ↗

New Comparisons with Specialized Reward Models and the Adjudicator Gemini 3.1 Pro, and a Full Audit of Shared Teacher Errors (Part 2/2, Q2, W2, Q3)

We next answer Q2 (specialized baselines, adjudicator), then W2 and Q3 (shared teacher errors).

  1. Regarding Q2, we added comparisons on the two tasks you suggest, choosing the better image on MJ-Bench and ranking generators on Qwen-Image-Bench, which every judge can perform. We newly evaluated UnifiedReward-2.0 (official prompts) and Gemini 3.1 Pro; ImageReward, HPS v2, PickScore, GPT-4V, and GPT-4o are literature results from MJ-Bench's quality subset.

Table 4. Specialized reward models and other judges on shared tasks ("–": not reported by MJ-Bench).

JudgeHuman training labelsChoosing the better image (MJ-Bench) ↑Generator ranking ρ ↑
ImageReward / HPS-v2.1 / PickScore (MJ-Bench)Human preferences63.5% / 67.3% / 83.8%–
GPT-4V / GPT-4o (MJ-Bench)–90.4% / 97.6%–
UnifiedReward-2.0 (new)Preference data84.9%0.854
Q-Judger130k+ expert pairs92.3%0.874
Kimi K2.6–97.1%0.911
Qwen3.8-Max–97.8%0.928
Gemini 3.1 Pro (new)–98.5%0.934
EviForge-8BNone97.0%0.922
EviForge-27BNone99.0%0.957

Even on these whole-image tasks, built for reward models, EviForge-27B achieves the highest scores, and EviForge-8B clearly exceeds every specialized reward model (97.0% vs. at most 92.3%) and is comparable to the large models. The reason is the supervision: reward models learn one overall score from human preferences, whereas EviForge learns per-facet judgments checked against image evidence, which separate defects an overall score blurs. VisionReward's paper reports preference accuracy at the level of PickScore and HPS v2 (81.7% vs. 79.8% and 83.3%, HPDv2). LLaVA-Critic judges text responses of multimodal models, not generated images, so it is not included.

Comparison with the adjudicator. EviForge-27B reaches the quality of its adjudicator, Gemini 3.1 Pro, matching or exceeding it on nine of eleven measures (Table 5; paper Table 26); Gemini keeps a higher evidence hit and slightly fewer unsupported claims. The student is therefore not capped by the adjudicator: the adjudicator only settles disputed facets during construction, while refinement then trains the student on its own verified errors.

Table 5. EviForge versus the adjudicator Gemini 3.1 Pro.

MeasureGemini 3.1 ProEviForge-27BEviForge-8B
Missed required facets ↓27.1%25.4%14.0%
Selection appropriateness ↑96.3%98.7%98.2%
Unsupported claims ↓1.6%1.8%1.9%
Rationales cover annotated defects ↑98.6%96.1%96.2%
Defects located ↑72.0%75.4%85.6%
Score ordering ↑ (/114)111113111
Low-quality detection ↑ (/128)126127127
Choosing the better image ↑98.5%99.0%97.0%
Generator ranking ρ ↑0.9340.9570.922
Changed judgments under attribute edits ↓1312
Human evaluation, 800 images ↑96.7%97.9%97.0%
  1. Regarding W2 and Q3, we agree that agreed facets skip debate, so errors shared by all teachers cannot be caught by adjudication. We answer each part; all analyses are added to Appendix D.
  • Omissions versus wrong scores. Of the 32 consensus errors in Table 7, 15 are agreed-null omissions and 17 are wrong scores.
  • Missed-facet rate of agreed-null facets. Auditing all of them on the 800-image human set, of 33,612 facets all three teachers marked inapplicable, humans judged 357 applicable, a missed-facet rate of 1.1%.
  • Does training reinforce shared errors? On the same set, all teachers agree but contradict humans on 962 of 48,000 decisions against Rater 1 (2.0%). On these, correct student answers go 397 → 384 → 572 for 8B and 472 → 463 → 682 for 27B (before training → distillation → refinement). Distillation imitates teacher labels and absorbs a few; refinement instead trains on new images built for the student's errors, kept only if their labels match what the prompt asked for, so answers are checked against the image, not teacher agreement; this corrects many shared errors. The remaining shared errors are 390/48,000 (0.81%) for 8B and 280/48,000 (0.58%) for 27B.
  • Independent OCR and detection tools. Our specialist, Qwen-VL-Max, already acts as a dedicated tool via task-specific prompts for text reading, object detection, and spatial relations. We chose it for its strong text reading: on CC-OCR (ICCV 2025), general multimodal models such as Qwen-VL-Max outperform specialist OCR models, and Qwen-VL-Max ranks among the top in scene-text reading. In early development, human inspection showed it to be clearly more reliable than lightweight OCR and detection tools, which struggle with the artistic fonts, malformed characters, and stylized objects of generated images. Combining such tools with VLM evidence is future work (Limitations, Appendix M).

We sincerely hope these responses address your concerns and earn your recognition.

Thanks to the authors for their detailed responses and additional experiments! The authors’ additional analyses address most of my concerns.

I have one small follow-up question. My understanding is that the raters checked all 60 facets for each image in Appendix K, and the overall accuracy also counts correct “not applicable” judgments. Could you use the existing human annotations to report results just for the facets that humans marked as “applicable”, including those the model missed? This would help show how the model performs when a score and rationale are actually needed.

Thanks again!

Reviewer_cpbeOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

EviForge trains locally deployable judges for fine-grained evaluation of generated images. It combines sample-adaptive facet selection, evidence-adjudicated multi-teacher distillation, and weakness-driven refinement. Experiments on 8B and 27B models report improved judgment quality, while downstream SDXL experiments demonstrate the utility of the learned judges as reward models.

Critical strengths
  • The figures are well designed and communicate the main ideas clearly, making the framework easy to follow.
  • The controlled experiments provide useful evidence for the proposed training mechanisms.
  • The evaluation connects judge quality to a practical downstream application.
Critical weaknesses
  • The motivation and research positioning are not sufficiently convincing to me. Although the paper identifies limitations of existing approaches, the specific gap relative to the closest methods could be articulated more clearly. The presentation emphasizes the system pipeline more than the justification for its key design choices and research contribution.
  • The comparative evaluation could better support the claimed contribution. The paper includes external-model comparisons and several ablations, but the main comparisons omit related approaches discussed in Section 2.1, e.g., VIEScore, VisionReward, and UnifiedReward. Targeted comparisons on shared capabilities, or an explanation of why these comparisons are unsuitable, would strengthen the empirical evidence.
Questions for authors
  • How were the seven dimensions derived, and what supports this particular grouping?
  • Could the authors clarify the closest prior methods and provide more targeted comparisons addressing the claimed research gap?
  • Can the authors separate the contributions of single-facet pair construction, the DPO objective, and the auxiliary chosen-response SFT loss? Does the “without localized DPO” ablation remove both losses, and is the preference loss computed over the full response or only the corrected facet?
  • Could the authors compare inference cost and latency at matched quality and output requirements, and report offline supervision-construction cost separately? This would clarify whether “lightweight” denotes smaller model size or a practical end-to-end efficiency advantage.
  • Could the authors report paired uncertainty estimates for the main gains over the strongest baselines, particularly generator ranking and pairwise accuracy?
6 comments
AuthorsOpenReview ↗

Why the Problem Matters, Why a Lightweight Judge, Three Unsolved Problems, and the Origin of the Seven Dimensions (Part 1/3, W1, Q1)

Thank you for recognizing our figures, controlled experiments, and downstream application. Your central concern is our motivation and positioning, which we agree must be settled first. We also agree that the original paper emphasized the system pipeline more than the reasons behind its key design choices and its research contribution; the revised version now states these reasons explicitly. Below we explain why the problem matters, why the judge should be lightweight, and which three problems remain unsolved (W1), then the seven dimensions (Q1). Part 2 shows how EviForge solves them against the closest methods (W2, Q2); Part 3 answers Q3–Q5.

Why the problem matters.

Automatic judges now decide what image generators learn. They supply the rewards for preference and reinforcement learning (Diffusion-DPO, CVPR 2024; Flow-GRPO, NeurIPS 2025), filter training data, and rank models (Qwen-Image-Bench, EMNLP 2026). A generator therefore learns its judge's mistakes: if a poster with a misspelled sign scores high, the generator learns to keep misspelling. Yet MJ-Bench finds current reward models and multimodal judges often unreliable, and MLLM-as-a-Judge (ICML 2024) reports that large-model judges omit items, show bias, and give unsupported reasons.

Why a lightweight judge.

By "lightweight" we mean a judge with far fewer parameters, 8B or 27B, that reaches the quality of much larger models. Generator training calls the judge at a very large scale: scoring the preference pairs for our four SDXL runs alone took 167,872 calls. At this scale, a small judge with large-model quality is cheaper, faster, more stable, and easier to deploy. One pass over 29,711 images costs our 8B judge 77.6 GPU·h versus about 0.55B API tokens for a large judge; it responds without network delay or rate limits; fixed weights and decoding settings make rewards reproducible, whereas hosted models are updated or retired, so their rewards cannot be reproduced later; and its 16 GiB of weights fit on one GPU, keeping images in-house. For these reasons, Prometheus (ICLR 2024), Prometheus-Vision (ACL Findings 2024), LLaVA-Critic (CVPR 2025), and UnifiedReward also distill large models into smaller judges.

What such a judge requires, and three problems existing research has not solved.

  • P1. It must decide what to check for each image. Failures differ by image: a poster may fail on text, a portrait on hands, a crowded scene on counts. Scalar reward models (ImageReward, PickScore, HPS v2, UnifiedReward) compress all of this into one score, so they cannot say what went wrong or where. Fixed checklists (VisionReward, AAAI 2026) ask the same questions of every image, so answers to irrelevant questions mix with relevant ones. Neither adapts what is checked to the image.
  • P2. It must learn reliably from biased teachers. A small judge learns from large models, which, as noted above, omit items, show bias, and give unsupported reasons. Distilling one teacher (LLaVA-Critic, Prometheus-Vision) copies these errors, and imitation cannot remove them (Gudibande et al., ICLR 2024); voting among teachers (PoLL) fails when they share an error. Human labels provide an independent reference but at great cost (VisionReward: 3M answers; Q-Judger: 130k+ expert pairs).
  • P3. Judge-distillation methods need follow-up training targeted at each student’s recurring errors. Even with good labels, a student keeps failing on particular items and scenes, and these blind spots differ from student to student. Self-improving judges (Self-Taught Evaluators, Meta-Rewarding) add training data broadly, so most of it goes to what the student already does well.

A lightweight yet reliable judge must therefore solve all three together, which no existing judge does. EviForge addresses each with one component; Part 2 explains each mechanism and compares it with the closest methods. This motivation is now in the Introduction of the revision. We next answer your questions in turn.

  1. Regarding Q1, the seven dimensions are those that years of benchmark development have settled on, and therefore what a judge of generated images must be able to recognize. We consolidated the dimensions evaluated across these benchmarks into seven (Table 1) and, within them, defined 60 items in 28 sub-groups, each with its own applicability condition, visible evidence, and 0–5 scoring rule (Appendix A).

Table 1. Origin of the seven dimensions.

Dimension (items)Reference papers and benchmarks
Prompt alignment (19)Qwen-Image-Bench, HEIM, T2I-CompBench, GenAI-Bench, TIFA, DSG
Visual quality (7)Qwen-Image-Bench, HEIM, RichHF-18K
Aesthetics (5)Qwen-Image-Bench, HEIM
Real-world fidelity (8)Qwen-Image-Bench, HEIM (knowledge), WISE
Creative generation (13)Qwen-Image-Bench, HEIM (originality)
Text rendering (5)MARIO-Eval, OneIG-Bench, LongText-Bench
Safety (3)MJ-Bench, HEIM (toxicity, bias, fairness)
AuthorsOpenReview ↗

How EviForge Solves Each Problem, with New Comparisons to UnifiedReward-2.0, Gemini 3.1 Pro, and Other Closest Methods (Part 2/3, W2, Q2)

We next answer W2 and Q2. We agree the contribution should be shown against the closest methods, so we added targeted comparisons. Below, for each problem from Part 1 that similar methods leave unsolved, we explain how EviForge solves it and what the comparison shows.

What we compare.

We cover the methods you named and the closest ones we identified: the scalar reward models ImageReward and PickScore (results reported by MJ-Bench on the same image pairs); the reward model UnifiedReward-2.0, newly evaluated with its official prompts; the item-level judge Q-Judger; and prompted large-model judges, the broad approach used by VIEScore: Qwen3.8-Max, Kimi K2.6, and the newly added Gemini 3.1 Pro. New evaluations use our shared protocol.

How the comparison is fair.

We compare each method on every ability its own design supports, so each is measured on its strengths, not on tasks it was never built for. Scalar reward models are designed to score whole images, so they are compared on choosing the better image and, where possible, ranking generators; their output has no item list, reasons, or locations, so item-level abilities are marked "–" rather than counted as failures.

Table 2. EviForge versus the closest methods. The first three ability columns use human annotations; "–" means the method's output does not support that ability.

MethodSizeHuman Training LabelsMissed Required Items ↓Reasons Cover Defects ↑Defects Located ↑Choosing the Better Image ↑Generator Ranking ρ ↑Changed Judgments under Attribute Edits ↓
ImageRewardSmallExpert comparisons–––63.5%––
PickScoreSmallUser preferences–––83.8%––
UnifiedReward-2.0 (new)7BPreference data–––84.9%0.854137
Q-Judger27B130k+ expert pairs28.6%73.7%44.5%92.3%0.87428
Kimi K2.6Large–24.8%94.9%62.1%97.1%0.91172
Qwen3.8-MaxLarge–37.4%97.3%64.3%97.8%0.92818
Gemini 3.1 Pro (new)Large–27.1%98.6%72.0%98.5%0.93413
EviForge-8B8BNone14.0%96.2%85.6%97.0%0.9222
EviForge-27B27BNone25.4%96.1%75.4%99.0%0.9571

P1, deciding what to check: solved by sample-adaptive item judgment. Scalar models (ImageReward, PickScore, UnifiedReward-2.0) give one score per image, and fixed checklists (VisionReward) ask every question of every image. EviForge instead decides which of 60 items apply to the image and gives each a score, a reason, and a defect location. Among methods that can do this, EviForge-8B misses the fewest required items (14.0% vs. 24.8–37.4%), and both EviForge judges locate the most annotated defects (75.4–85.6% vs. at most 72.0%). To isolate adaptive selection, we made EviForge check every item on every image: the share of checked items that actually apply drops from 98.2% to 66.8% (paper Table 23).

P2, learning from biased teachers: solved by evidence-adjudicated multi-teacher distillation. LLaVA-Critic and Prometheus-Vision copy one teacher, while Q-Judger and VisionReward rely on large human annotation. EviForge combines three teachers and checks every disputed item against the image before it becomes a training label. With no human labels, EviForge beats the human-trained Q-Judger on every ability in Table 2, and beats the preference-trained reward models on their own tasks: choosing the better image and ranking generators. It is also the least biased: editing only a person's gender or age changes 1–2 of its 1,458 judgments, versus 13–137 for all others. Under the same student and budget, a single teacher leaves 49.1% of required items missed, versus 38.0% with our labels (paper Table 2).

P3, correcting the student's own weaknesses: solved by weakness-driven refinement. Self-improving judges add data broadly; EviForge finds the errors this student keeps making and trains on new examples built for them. A far smaller judge thus reaches and often exceeds large-model quality: EviForge-27B matches or exceeds Gemini 3.1 Pro on nine of eleven measures, including missed items, located defects, image choice, ranking, stability, and human ratings, and is close on the other two (paper Table 26); EviForge-8B matches or exceeds the large models on most abilities. The large models keep a small edge in reasons covering defects (97–99% vs. 96%). Compared with simply training longer on the same data, refinement lowers missed items from 19.5% to 14.0% (paper Table 10).

EviForge is the only method in Table 2 that is small, needs no human training labels, and gives checkable item-level judgments. Measured on each method's own strengths, it outperforms every small judge and matches or exceeds the large ones on most abilities. These comparisons are now in the Related Work and Experiments sections (§2, §4.2–4.3; Appendix J).

AuthorsOpenReview ↗

Decomposing Localized DPO, Newly Reported Offline and Online Cost and Latency at Matched Output, and New Paired Confidence Intervals (Part 3/3, Q3–Q5)

Finally, we answer Q3–Q5.

  1. Regarding Q3, yes, "w/o localized DPO" removes both the DPO preference loss and its auxiliary chosen-response SFT loss. The preference loss is computed over the full response; "localized" refers to pairs that differ at only one facet (Appendix F).

How it works and why.

For the same image, the preferred and rejected judgments differ only at the erroneous facet. This directs supervision to the diagnosed error, while the auxiliary SFT loss on the preferred full response maintains complete, well-formed judgments. Removing this DPO stage while keeping GRPO and replay lowers weak-case correction from 65% to 30% (paper Fig. 4b; Appendix H, Table 16), directly supporting its contribution.

What the controls show.

All variants start from the same distilled student. SFT on the preferred responses alone reduces required-facet miss from 38.0% to 26.0%, showing that the correction data are useful. With the same refinement data, replay, and total update budget, continued SFT reaches 19.5%, versus 14.0% for full DPO-plus-GRPO refinement (Table 3; paper Table 10, Appendix F). Together, these controls show the value of the correction data, the DPO stage, and the complete refinement strategy.

Table 3. Continued-SFT and full-refinement controls (required-facet miss).

Training after distillationRequired-facet miss
None38.0%
SFT on preferred responses from correction pairs26.0%
SFT on all refinement data, matched update budget19.5%
Full refinement: localized DPO, then GRPO14.0%
  1. Regarding Q4, "lightweight" means smaller model size with measured local deployment benefits. Building the training data is a one-time offline cost shared by both students; afterward, each evaluation needs only our own GPUs, whereas a large judge costs API tokens and waiting time on every call. Table 4 lists both for the same output, a full 60-item judgment.

Table 4. Offline and online cost for the same output.

EviForge-8BEviForge-27BQwen3.8-Max via API
Offline, once: building training data4.12B input / 1.42B output tokens, 62 GPU·h; sharedSharedNone
Offline, once: training686 H100 GPU·h2,920–3,714 H100 GPU·hNone
Online: one pass over 29,711 images77.6 H100 GPU·h, no API calls60.8 L20X GPU·h, no API callsAbout 0.55B text tokens
Online: estimated latency per image, one at a timeAbout 30 sAbout 56 sAbout 70 s
Online: time per image, batched0.59 s (16 H100)0.92 s (8 L20X)–

Times use our actual deployment (one model copy per GPU); one-at-a-time latency is derived from full-pass logs on one copy, and Qwen3.8-Max is called through its official API. The log-based estimates favor EviForge (30 s and 56 s vs. 70 s), and batching brings it under 1 s per image. At this matched output, quality is comparable or better: EviForge-8B matches Qwen3.8-Max on choosing the better image and ranking, and is better on missed required items (14.0% vs. 37.4%). The offline cost is paid once, whereas the large judge's cost recurs on every pass, including every round of reward training. We will also release the weights, code, and releasable supervision data, so others can skip the offline cost (full cost–quality analysis in Appendix G).

  1. Regarding Q5, we agree and have computed paired 95% confidence intervals for the gains over the original large-model baselines, Qwen3.8-Max and Kimi K2.6, on choosing the better image and generator ranking (Table 5; paper Table 25, Appendix J). "Paired" means both judges are scored on the same resampled prompts (10,000 resamples), with an exact McNemar test for image choice and Williams' test for ranking; a difference is significant if its interval excludes zero.

Table 5. Paired differences against Qwen3.8-Max and Kimi K2.6.

ComparisonDifference [95% CI]Significant
Choosing the better image: 27B − Qwen3.8-Max+1.25 pp [+0.36, +2.23]Yes
Choosing the better image: 27B − Kimi K2.6+1.96 pp [+0.98, +3.04]Yes
Choosing the better image: 8B − Qwen3.8-Max−0.80 pp [−2.14, +0.54]No
Choosing the better image: 8B − Kimi K2.6−0.09 pp [−1.52, +1.25]No
Generator ranking ρ: 27B − Qwen3.8-Max+0.029 [−0.026, +0.083]No
Generator ranking ρ: 27B − Kimi K2.6+0.046 [−0.019, +0.110]No
Generator ranking ρ: 8B − Qwen3.8-Max−0.006 [−0.050, +0.037]No

The intervals support our main claim that small EviForge judges match the large ones. EviForge-27B is significantly better than both at choosing the better image. On ranking, the differences are not significant because 18 generators give wide intervals, so EviForge is comparable rather than better; EviForge-8B is comparable on both abilities. The revision says "better" only where the difference is significant.

Thank you again; we hope these clarifications address your concerns.

Reviewer_xWx3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EviForge, which constructs supervision data through evidence-based adjudication among multiple teachers to train image judges that provide fine-grained scores, rationales, and defect localization. It further generates targeted data based on student weaknesses for iterative refinement. Experiments show that the trained models outperform their respective backbones on most metrics and match or surpass larger baselines on several metrics. Moreover, using EviForge as the judge for SDXL preference training improves the generated images’ win rates in human comparisons, demonstrating its practical value for guiding image generation optimization.

Critical strengths
  1. The judgments are detailed and easy to understand. The model selects relevant evaluation facets based on image content and provides scores, rationales, and defect locations, making the results easy to interpret and inspect.

  2. EviForge emphasizes the reliability of supervision data. It combines judgments from multiple teachers under a shared evaluation rubric, reducing the impact of individual teacher errors on the student.

Critical weaknesses
  1. The overall EviForge pipeline is overly complex and heavily engineered. I am concerned that its practical value may be limited, especially given its reliance on many external models.

  2. If the teacher models share a common bias and produce a biased judgment during joint adjudication, the method seems likely to fail. More seriously, even if the student model’s initial judgment is correct, EviForge’s SFT, DPO, and GRPO training could reinforce the biased answer instead.

Questions for authors
  1. The training pipeline is complex and relies on multiple external models. EviForge involves multi-model evaluation, dispute adjudication, prompt rewriting, image generation, and DPO and GRPO training. Although the final model is relatively lightweight, implementing the full pipeline and constructing the data are costly, which may limit reproducibility and broader adoption. I would like the authors to report the cost of each step.

  1. Supervision and rewards still rely on teacher judgments. The adjudicated results are used for SFT and subsequent DPO and GRPO training. Although multiple teachers are involved, shared biases among them may continue to be learned by the student during later training.

  1. Some evaluations are small in scale and rely on AI-assisted auditing. For example, score ordering and low-quality detection use only 114 pairs and 128 samples, respectively. Although human verification is provided, further validation is needed to establish whether these results generalize to broader settings.

  1. The downstream evaluation is limited in scope. The generation optimization experiments mainly use SDXL and Diffusion-DPO, which is insufficient to establish whether the judge can consistently benefit other generation architectures and training methods. I suggest that the authors add experiments demonstrating the method’s effectiveness under other training approaches.
5 comments
AuthorsOpenReview ↗

EviForge’s Per-Step Development Cost and Practical Value, and a New 800-Image Human-Referenced Test of Shared Teacher Errors (Part 1/2, Q1–Q2)

Thank you for your careful review and recognition of EviForge's interpretable judgments and supervision reliability. To address your concerns, we report development costs and quality for Q1, and test shared teacher errors against independent human labels for Q2.

  1. Regarding Q1, EviForge's construction cost is within the usual range for trained judges of generated images. Its training labels need no human annotation, and supervision is built once during development, which saves part of the cost. For comparison, ImageReward (NeurIPS 2023) used 137k expert comparisons, VisionReward (AAAI 2026) 3M human answers, and Q-Judger (EMNLP 2026) 130k+ expert-labeled pairs; UnifiedReward-Think (NeurIPS 2025) uses 8 H100s for cold-start/rejection sampling and 64 H20s for GRPO. Table 1 lists EviForge's cost per step (Appendix G).

Table 1. Cost per step.

StepInput / output tokensGPU·h
Teacher judgments2.26B / 0.71B–
Resolving teacher disputes1.11B / 0.20B–
Targeted prompts, checked (5 rounds)0.75B / 0.50B–
Generate 119,000 new images–62
Train the student–8B: 686; 27B: 2,920–3,714
Run the judge on 29,711 images–8B: 77.6; 27B: 60.8

Every step earns its cost: in matched-budget ablations, removing specialist evidence, debate, or the separate adjudicator weakens supervision (§4.4, Table 2), and removing DPO, GRPO, or replay, or using plain continued training instead, weakens the judge (§4.5, Fig. 4; Appendices F and H).

This cost brings two practical benefits. First, quality: without human labels, EviForge matches or exceeds judges trained on them (§4.2–4.3; Appendix J). Against Q-Judger, the closest judge, which also scores items but uses human labels, EviForge-8B misses fewer human-required items (14.0% vs. 28.6%) and covers more annotated defects in its reasons (96.2% vs. 73.7%) and locations (85.6% vs. 44.5%). Against UnifiedReward-2.0, trained on human preferences, EviForge-27B matches human preferences more often (99.0% vs. 84.9%) and ranks 18 generators closer to experts (ρ = 0.957 vs. 0.854).

Second, we pay this cost once so others need not: upon acceptance, we will release the 8B and 27B weights, code, data splits, and releasable supervision data. Users can run it locally without API calls, use it as a reward (Q4), or refine it for a new domain. The method is not tied to any specific teacher, student, or external model: it passes structured judgments, so any can be swapped.

  1. Regarding Q2, a shared teacher error is one all teachers make; since agreed items skip debate and adjudication, our pipeline cannot catch it either. The paper found 32 such errors among 971 agreed items (3.3%, Table 7). We partly agree that a student may learn them. Our labels come from models, but human labels are not bias-free either: VisionReward's annotators agree on 89.3% of answers, and our three raters reach Fleiss κ = 0.71 on 150 images. We therefore aim to keep shared errors rare and prevent later training from amplifying them. We use teachers from three companies, since similar models share mistakes (Goel et al., 2025), and a separate model settles disputes using visual evidence. To stop later training from overwriting answers the student already gets right, a refined model must keep at least 90% of 400 previously correct check cases.

To test whether the student learns shared errors and later training amplifies them, we built a human-labeled set of 800 images, 100 from each of this paper's eight evaluation sources, none used in training. Three raters checked all 60 items, including applicability, scores, and rationales. We ran the teachers as in training and compared their 48,000 decisions with Rater 1’s labels. In 962 decisions (2.0%), all teachers agreed but contradicted humans; Table 2 tracks the student on them.

Table 2. Correct student answers on the 962 shared errors.

StudentBefore trainingAfter SFTAfter DPO + GRPO
8B397 (41.3%)384 (39.9%)572 (59.5%)
27B472 (49.1%)463 (48.1%)682 (70.9%)

The stages differ for a clear reason. SFT imitates teacher labels, so it absorbs a few shared errors (397 to 384; 472 to 463). DPO and GRPO instead learn from new images built for the student's errors: each prompt contains what the student misjudged, and an image is kept only if its label matches what the prompt asked for, so answers are checked against the image, not just teacher agreement. In 357 of the 962, all teachers skip an applicable item, exactly what refinement targets (missed items drop from 38.0% to 14.0%). Correct answers thus rise to 572 and 682, above the starting models, with 97.5% of earlier correct answers kept. Shared teacher errors are rare, and later training corrects many of them rather than reinforcing them. The remaining shared errors account for 390/48,000 (0.81%) for 8B and 280/48,000 (0.58%) for 27B (Appendix D).

We hope this addresses both concerns.

AuthorsOpenReview ↗

Benchmark Standards Behind Each Evaluation, a New Independent Human Verification of Facet Judgments, and Generalization with a New SD3.5-Medium / Flow-GRPO Experiment (Part 2/2, Q3–Q4)

Thank you for your suggestions. For Q3, we clarify our evaluation standards and add independent human verification of facet judgments; for Q4, we test EviForge on a different generator architecture and training method.

  1. Regarding Q3, we apologize that the original text did not make the evaluation standards clear. The two tests you mention, score ordering (114 image pairs from VisionRewardDB) and low-quality detection (128 images from ImageRewardDB), do not use AI auditing. Like most of our metrics, they compare model outputs with human labels that the benchmarks themselves provide, fixed before any model is scored. Table 3 lists what we evaluate and how.

Table 3. What we evaluate and the standard used.

AbilityStandard provided byHow it is evaluated
Choosing evaluation itemsVisionRewardDB human item labelsCount human-required items the model skipped
Score orderingVisionRewardDB human quality orderCheck that the better image gets the higher score
Low-quality detectionImageRewardDB human quality labelsCheck that low-quality images get low scores
Reasons and defect locationRichHF-18K, EvalMuse-40K human defect labels and locationsCheck reasons and locations against annotated defects
Image comparisonMJ-Bench human preferencesCheck that the model picks the image humans prefer
Generator rankingQwen-Image-Bench expert rankingCompare the model's ranking of 18 generators with experts
Facet judgments, all 60 items (new)Three independent raters, 800 imagesCheck every item decision against human labels

The AI-assisted audit covered only two questions that no benchmark labels for all 60 items: whether each evaluation item the model chooses actually applies to the image, and whether its reasons mention content absent from the image. The paper had Qwen3.8-Max audit 400 images and three humans re-check 150 of them (Fleiss κ = 0.71; AI and human rankings agree at ρ = 0.94). We agree this is a limitation, because Qwen3.8-Max is also one of our teachers.

We therefore added a fully human evaluation. We took 100 images from each of the eight evaluation sources used in this paper (800 images). Three raters independently checked all 60 items for each image and judge, including applicability, scores, and rationales. No teacher or AI model was involved.

Table 4. Accuracy under the new human evaluation (three raters combined).

JudgeAccuracy
Qwen3-VL-8B (before training)79.6%
EviForge-8B97.0%
Qwen3.8-27B (before training)88.9%
EviForge-27B97.9%
Qwen3.8-Max94.6%
Kimi K2.691.1%

Both EviForge judges improve greatly over their starting models and exceed both large models. Each rater ranks both EviForge judges above both large models, and both EviForge judges stay at or above 96.5% on each of the seven evaluation dimensions. Thus, the two abilities once checked by the AI auditor are now verified directly by humans (800 images × 60 items × 3 raters, all eight evaluation sources), and the results agree with those on benchmark labels (Appendix K).

  1. Regarding Q4, we added a downstream experiment that changes the generator architecture, the training method, and the comparison judge at the same time, using the same prompt splits (1,200 training, 200 unseen test) and blinded human comparisons. Table 5 compares it with the original experiment.

Table 5. Original and added generator-training experiments.

Original (paper)Added
GeneratorSDXL (U-Net diffusion)SD3.5-Medium (Transformer, flow matching)
Training methodDiffusion-DPO: offline, learns from image pairs the judge ranked onceFlow-GRPO: online, the judge scores each new image during training
Other judge used as rewardQwen3.8-MaxUnifiedReward-2.0, a dedicated reward model
Human win rate vs. original generator68.3% (273/400)69.0% (138/200)
Human win rate vs. other judge64.8% (259/400)64.5% (129/200)

In both experiments, only the reward model differs between compared runs, and humans compare images blindly. Taken together, EviForge has now been tested as a reward on two generator architectures, with two training methods (offline and online), and against three other judges (Qwen3.8-Max, Kimi K2.6, and UnifiedReward-2.0). In every comparison, humans prefer the generator trained with EviForge, and the gains are nearly identical across settings: 68.3% and 69.0% wins over the original generator, and 64.8% and 64.5% over a generator trained with another judge (both p < 10⁻⁴ in the added experiment). Because the gain persists while the generator, training method, and competing judge all change, it stems from EviForge's judgments rather than a particular setup. EviForge therefore consistently benefits different generation architectures and training methods. It is now in §4.3 and Appendix L.

We hope these responses address your concerns and earn your recognition.

AuthorsOpenReview ↗

Thank You for Your Thoughtful Questions on Reliability and Practical Use

We sincerely appreciate the time you took to review our work. Your questions led us to examine whether students inherit errors shared by all three teachers and to explain what the cost of each training stage buys in practice. We also added an independent human evaluation on 800 images and tested EviForge as a reward for a second generator.

As the discussion draws to a close, we would be grateful to hear whether these results address your concerns. If an important question is still open, we would welcome the chance to work through it with you while we can still improve the paper.

Reviewer_piEPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EVIFORGE, a training framework for lightweight visual judges of generated images. The framework combines sample-adaptive selection among 60 facets, evidence-adjudicated multi-teacher distillation, and weakness-driven refinement that allocates new data to recurring facet-level errors. The authors instantiate the method on Qwen3-VL-8B and Qwen3.8-27B and evaluate facet selection, score consistency, evidence-grounded rationales, defect localization, generator ranking, reward-guided generation, and human preferences across eight public datasets. The final models improve over their backbones and are competitive with larger judges on many reported metrics.

Critical strengths
  1. The motivation is strong and practically important. Visual-generation failures are sample-specific, and a useful judge should identify applicable criteria, explain the evidence, and localize defects rather than emit only one scalar preference.

  2. The paper provides an unusually broad evaluation and a reasonably complete ablation story. It includes public datasets, multiple model sizes, automatic metrics, human comparisons, teacher/distillation controls, refinement controls, confidence intervals, and reward-guided generation.

Critical weaknesses
  1. Teacher/evaluator dependence limits the strength of the validation. Qwen3.8-Max is used as a generalist teacher and also participates in AI-assisted auditing, while many evidence judgments are produced by model-based specialist calls. Agreement with these evaluators therefore cannot be treated as fully independent evidence of correctness.

  2. The claimed efficiency advantage is incomplete. The full pipeline requires large-scale teacher calls, debate/adjudication, prompt rewriting, image generation, verification, and several refinement rounds. The paper reports deployment throughput for the final local judge, but does not provide a clear end-to-end quality/cost comparison against simpler distillation or direct use of a large judge.

Questions for authors
  1. Can the authors evaluate the final judges on a larger, fully independent human-annotated test set whose labels are not produced or audited by Qwen3.8-Max or another teacher used in training?

  2. How much of the improvement remains when all AI-assisted audit metrics are removed and evaluation is restricted to independent human judgments and benchmark-native annotations?

  3. Please report the total teacher inference, debate/adjudication, image-generation, verification, training, and deployment costs, together with the quality/cost Pareto frontier against single-teacher distillation and direct large-judge evaluation.

  4. Does EVIFORGE generalize across image generators, model families, image editing, and text-to-video settings? If not evaluated, how should the scope of the claims be limited?

3 comments
AuthorsOpenReview ↗

Evaluation Against Human Annotations from Public Benchmarks and Additional Fully Independent Human Verification on 800 Images from Eight Evaluation Sources (Part 1/2: Q1–Q2)

Thank you for recognizing our work and for your constructive suggestions. EviForge combines evidence-adjudicated multi-teacher distillation with weakness-driven refinement to train lightweight image judges competitive with much larger models such as Qwen3.8-Max and Kimi K2.6.

Regarding Q1–Q2, most of our original evaluation already used benchmark-native human annotations to compare EviForge-8B/27B with Qwen3.8-Max, Kimi K2.6, and other judges. Only selection appropriateness and unsupported-claim rate used Qwen3.8-Max audits. Table 1 explains the human references and evaluation methods for each ability. Test images are held out from training and refinement; all models use the same references and scoring rules.

Table 1. Evaluated abilities, benchmark references, and evaluation methods.

Ability evaluatedBenchmark and referenceEvaluation method
Facet selectionVisionRewardDB: human attribute annotationsCount required facets omitted by the model.
Score consistencyVisionRewardDB: human quality order; ImageRewardDB: human quality labelsCheck score ordering and low-quality detection.
Evidence-grounded rationalesRichHF-18K, EvalMuse-40K: human defect/element annotationsCheck rationales against annotated defects and element failures.
Defect localizationRichHF-18K: human spatial annotationsCheck coverage of reference defect cells.
Image comparisonMJ-Bench quality: human preferencesCheck whether image choices match human preferences.
Generator rankingQwen-Image-Bench: expert generator orderCompare model-derived ranks with the expert order.
Score stabilityMJ-Bench bias: matched demographic variantsMeasure score changes across matched variants.
Reward-guided generationGenAI-Bench prompts; human preferences collected in our studyBlindly compare images produced after reward training.
  1. Regarding Q1, EviForge selects applicable facets for each image and provides evidence-grounded rationales. Existing benchmark annotations do not cover both decisions across all 60 facets. The original evaluation therefore used sampled Qwen3.8-Max audits; we acknowledge the teacher-auditor dependence and apologize for the unclear presentation. To address this concern, we added a fully independent human evaluation on 800 images, 100 from each source in Table 1. Three raters independently checked all 60 facets for each image and model, checking facet applicability, scores, and rationales. No teacher, adjudicator, or AI auditor generated or reviewed these human assessments.

Table 2. Independent human assessment of facet judgments (accuracy).

JudgeRater 1Rater 2Rater 3Combined
EviForge-8B97.62%96.31%97.03%97.0%
EviForge-27B98.08%97.71%97.86%97.9%
Qwen3.8-Max94.35%93.72%95.88%94.6%
Kimi K2.690.79%89.84%92.75%91.1%

This independent assessment supports EviForge’s ability to select applicable facets and provide evidence-grounded rationales. EviForge-8B and EviForge-27B achieve 97.0% and 97.9% accuracy, respectively, exceeding both larger judges; all three raters give this ordering (Table 2). The protocol and results are in Appendix K.

  1. Regarding Q2, we evaluate EviForge-8B, EviForge-27B, and the comparison models against the benchmark-native human annotations listed in Table 1, with results in Table 3. For example, RichHF-18K supplies human defect descriptions and locations to test rationale evidence and localization; MJ-Bench supplies human preferences to test image-pair decisions. These results are calculated against human annotations, rather than AI-assisted audit judgments.

Table 3. Performance against benchmark human annotations. Ranking ρ is Spearman correlation.

MetricEviForge-8BEviForge-27BQwen3.8-MaxKimi K2.6
Required-facet miss14.0%25.4%37.4%24.8%
Score ordering97.4%99.1%95.6%95.6%
Low-quality recall99.2%99.2%98.4%99.2%
Evidence hit96.2%96.1%97.3%94.9%
Grid coverage85.6%75.4%64.3%62.1%
Pairwise accuracy97.0%99.0%97.8%97.1%
Generator-ranking ρ0.9220.9570.9280.911

Based solely on these human annotations, EviForge-8B and EviForge-27B remain competitive in facet selection, scoring, rationale evidence, localization, and image comparison and ranking, matching or surpassing Qwen3.8-Max and Kimi K2.6 on most metrics. For example, defect-grid coverage reaches 85.6% and 75.4%, versus 64.3% for Qwen3.8-Max; EviForge-27B achieves 99.0% pairwise accuracy versus 97.8%. Qwen3.8-Max retains a small advantage in evidence hit. Thus, the conclusion that lightweight EviForge judges are competitive with larger models also holds under human-annotated evaluation. These clarifications are now in §4.2 and Appendices I–J. We hope this addresses both concerns.

AuthorsOpenReview ↗

Full Development and Deployment Cost–Quality Analysis, and Generalization Across Architectures, Generators, and Tasks with a New SD3.5-Medium / Flow-GRPO Experiment (Part 2/2, Q3–Q4)

Thank you for your suggestions.

  1. Regarding Q3, we have added cost accounting to Appendix G, covering teacher inference, debate/adjudication, prompt rewriting, image generation, verification, student training, and deployment. Table 4 summarizes the resources.

Table 4. Full development and deployment resources.

ComponentResources
Supervision construction + five refinement rounds4.12B input / 1.42B output text tokens
Image generation62 GPU·h
EviForge-8B training686 H100 GPU·h
EviForge-27B training2,920–3,714 H100 GPU·h
Local evaluation of 29,711 images8B: 77.6 H100 GPU·h; 27B: 60.8 L20X GPU·h
Direct Qwen3.8-Max, same evaluation workload0.553B text API tokens

(1) Development context. Visual judges require substantial resources: VisionReward uses 3 million image QA annotations, and Q-Judger uses >130k expert-labeled pairs. VQualA 2025’s ActionLab judge uses 992 A6000 GPU·h; UnifiedReward-Think uses 8 H100s for cold-start/rejection-sampling training and 64 H20s for GRPO. EviForge’s image-evidence checks and multi-teacher adjudication reduce expert training annotation; training GPU-hours and dataset size fall within the usual range for related visual judges. These comparisons appear in Appendix G.

(2) Single-teacher distillation and continued SFT controls show two benefits. First, evidence adjudication improves supervision. Multi- and single-teacher targets use 0.917B and 0.200B output tokens; with the same data and the 8B SFT budget (117 H100 GPU·h), multi-teacher distillation reaches 88.8% evidence hit, versus 78.1% for single-teacher distillation. With matched teacher output tokens, single-teacher resampling gives 0.892 target evidence hit, versus 0.936 for full adjudication (§4.4; Appendix D).

Second, weakness-driven refinement uses the added data more effectively than continued SFT. From the same distilled checkpoint, we compare continued SFT with localized DPO and full-judgment GRPO using identical refinement data, historical replay, and total update budgets. Continued SFT improves the distilled model, but full refinement performs better in scoring, rationale evidence, and localization; EviForge-8B reaches 85.6% grid coverage, versus 72.8% for continued SFT (§4.5; Appendix F). These gains reflect added data and our training design for error correction.

(3) Against direct large-judge evaluation. EviForge matches or exceeds judges such as Qwen3.8-Max and Kimi K2.6 on most metrics based on human annotations (§§4.2–4.3; Appendix J). EviForge-27B reaches 99.0% pairwise accuracy and 75.4% grid coverage, versus Qwen3.8-Max’s 97.8% and 64.3%. Online judges incur continuing API charges for evaluation and reward-guided generator training and depend on external service availability and versions. EviForge runs locally, keeps sensitive images onsite, and lets users control serving and model versions. Upon acceptance, we will release weights, code, fixed splits, and releasable supervision data for direct deployment without repeating data construction or training. Appendix G compares single-teacher distillation, EviForge, and direct large-judge evaluation in quality, construction tokens, training GPU·h, and deployment resources to show their quality–resource trade-offs.

  1. Regarding Q4, we analyze generalization across model architectures, image generators, and other tasks.

(i) Model compatibility. EviForge transfers structured judgments, so teachers and students need not share tokenizers or internal representations. Teachers include Qwen, Kimi, and Gemini models; Qwen3-VL-8B and Qwen3.8-27B, which uses hybrid linear/full attention, both benefit from the framework. This supports compatibility across architectures and sizes.

(ii) Image generators. EviForge can evaluate different image generators: on Qwen-Image-Bench, EviForge-8B/27B achieve Spearman correlations of 0.922/0.957 with the expert ranking of 18 generators (§4.3). For generator training, EviForge-27B guides both the original offline SDXL experiment (Diffusion-DPO) and the added online SD3.5-Medium experiment (Flow-GRPO). The generators beat their original models in 68.3% and 69.0% of human comparisons, respectively (§4.3; Appendix L). These results support generalization across generators, architectures, and training methods.

(iii) Image editing and video. Internal editing applications show feasibility and good results. Inputs comprise an editing instruction, source image, and edited image; 18 facets cover instruction following, content preservation, edit quality, visual quality, and safety. Evidence-adjudicated multi-teacher distillation checks source–result evidence; weakness-driven refinement targets recurring editing errors. Both steps remain effective with adapted inputs, criteria, and evidence. Video evaluation is untested; adaptation requires temporal/motion criteria and cross-frame evidence (Appendix M).

We hope these responses address your concerns and earn your recognition.

AuthorsOpenReview ↗

Thank You for Your Suggestions on Evaluation and Generalization

Thank you again for your thoughtful review. We hope our responses clarify these points. First, the gains hold both on the benchmarks’ original human annotations and in the new independent evaluation of 800 images. Second, the cost comparison shows what the additional teacher supervision contributes at the same student training budget and what it costs to run the trained judge locally. Third, EviForge’s rankings agree closely with expert rankings across 18 generators, and its scores improve generation in both training settings we tested. We also clarify which findings extend to our preliminary work on image editing and which remain untested for video.

If any of these points would benefit from further clarification, we would be grateful to hear your thoughts while there is still time to improve the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions to EviForge

Thank you to all four reviewers for the time and care you devoted to our paper. Your questions helped us explain more clearly how EviForge is trained and test its judgments more directly. Below we summarize the main revisions and where they appear in the paper.

1. Motivation and design choices (§§1–2, §3.1, Appendix A; pp. 1–4).

We revised the Introduction and Related Work to explain three design choices: the judge selects the facets that apply to each image instead of checking a fixed list; when teacher models disagree, the dispute is settled by checking evidence in the image; and later training targets the errors the student makes repeatedly. §3.1 now also lists the public benchmarks from which our seven evaluation dimensions are drawn.

2. Agreement with human judgment (§4.2, Appendix I Table 20, Appendix K Tables 27–28; pp. 7–8).

Most original tests already compare the judge with human annotations, such as marked defect locations or human choices between two images; the revision now states which tests use these annotations and which use an AI auditor. In a new evaluation, three independent raters checked all 60 facets for each of 800 images. Overall accuracy is 97.0% for EviForge-8B and 97.9% for EviForge-27B. Counting only the facets that humans judged to need a score and a rationale, with missed facets counted as errors, accuracy is 93.4% and 95.1%.

3. Shared teacher errors (§§4.4–4.5, Appendices D, F, and H, Tables 8, 10, and 16; pp. 9–10).

We identified 962 decisions on which all three teachers agreed with one another but not with the independent human labels. Both students initially learned some of these errors: EviForge-27B answered 49.1% of them correctly before training and 48.1% after learning from the teachers, but 70.9% after refinement targeted its recurring errors (Table 8). We also explain how each refinement pair is built from two responses that differ in a single facet, and compare refinement with further supervised training using the same data and number of updates.

4. Comparison with other judges (§§4.2–4.3, Appendix J, Tables 25–26; pp. 7–9).

We added comparisons with UnifiedReward-2.0 and Gemini 3.1 Pro on two shared tasks: choosing the better of two images and ranking image generators. On the same image pairs, EviForge-27B chose the human-preferred image significantly more often than Qwen3.8-Max and Kimi K2.6. When ranking 18 generators, EviForge-8B and EviForge-27B achieved correlations of 0.922 and 0.957 with the expert ranking, compared with 0.928 for Qwen3.8-Max and 0.911 for Kimi K2.6. EviForge-27B’s ranking was significantly closer to the expert ranking than those of Q-Judger and UnifiedReward-2.0. Its smaller leads over Qwen3.8-Max and Kimi K2.6 were not statistically significant (Table 25).

5. Cost of building and using the judge (§4.1, §4.4, Appendix G Tables 11–13; pp. 7 and 10). §4.1 now points to the measured speed and memory use of the trained judges, and §4.4 identifies the one-time costs shared by both judges; after training, either judge evaluates images locally without calling any teacher. Appendix G reports the cost of each step, from collecting teacher judgments and resolving disagreements to generating and verifying new images and training the students. It also compares the quality and cost of three options on the same workload: single-teacher distillation, EviForge, and calling a large online judge directly.

6. Generalization to another generator (§4.3, Appendices L–M, Tables 34–35; pp. 8–9).

In the original experiment, EviForge-27B's preferences between image pairs were used to train SDXL offline. In a new experiment, a different generator, Stable Diffusion 3.5 Medium, is trained online with Flow-GRPO, learning from EviForge-27B's scores on the images it produces during training. In blind human comparisons, the trained generator was preferred over its original in 68.3% of comparisons for SDXL and 69.0% for SD3.5-Medium. Appendix M describes preliminary work on image editing and the changes needed for video.

Thank you again for your careful reading. Your questions and suggestions have made the paper both stronger and clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.