acceptodds
Under review as a conference paper at ICLR 2027

How Can We Achieve Human-Aligned Evaluation for Vision Generation Tasks?

Abstract

The rapid advancement of visual generation models has outpaced traditional evaluation approaches, necessitating the adoption of Vision-Language Models as surrogate judges. In this work, we systematically investigate the reliability of the prevailing absolute pointwise scoring standard, across a wide spectrum of visual generation tasks. Our analysis reveals that this paradigm is limited due to stochastic inconsistency and poor alignment with human perception. To resolve these limitations, we introduce GenArena, a unified evaluation framework that leverages a pairwise comparison paradigm to ensure stable and human-aligned evaluation. Crucially, our experiments uncover a transformative finding that simply adopting this pairwise protocol enables off-the-shelf open-source models to outperform top-tier proprietary models. Notably, our method boosts evaluation accuracy by over 20% and achieves 78.7% agreement with human annotators. It also achieves a Spearman correlation of 0.86 with the LMArena leaderboard, surpassing the 0.36 correlation of pointwise methods. Based on GenArena, we benchmark state-of-the-art visual generation models across diverse tasks, providing the community with a rigorous and automated evaluation standard for visual generation.

Primary area:
generative models
Keywords:
Visual Generation, Human-Aligned Evaluation, Vision-Language Models, Pairwise Comparison, GenArena, Automatic Evaluation, Generative Models, Multimodal Evaluation, Human Preference, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8DByOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisit the idea of VLM as a judge under absolute score and pairwise manner, then exposed the problem of absolute scoring. Authors introduced "GenArena", an elo system benchmark based on pairwise scoring. The benchmark is validated, coming up a conclusion that elo systems are the way to achieve human-aligned evaluation for visual generation tasks.

Critical strengths

S1: Human aligned evaluation for visual generation tasks is an important problem. While the drawback of pointwise scoring is intuitively known, the paper does a good job of formally quantifying it by consistency metrics and correlation metrics.

Critical weaknesses

W1: The conceptual novelty feel overstated. The psychological framing is intuitive, but it is not a direct port to the problem. Elo is already a standard practice in existing benchmarks (such as GenAI Arena and Chatbot Arena), and the paradigm of replacing human voters with automated pairwise AI judges has already been extensively established by prior VLM-as-a-Judge literatures. This work functions as a confirmatory empirical study validating known techniques on visual context.

W2: The paper wasted a lot of space over-explaining standard background concepts rather than spotlighting the core empirical results. The introduction also reads bulky and can be trimmed. Also poor data visualization on Table 1 as it is heavily fragmented with "--" dashes, there might be alternative choices for visualizing the results (e.g. charts) other than tables. In the abstract "Crucially, our experiments uncover a transformative finding that simply adopting this pairwise protocol enables off-the-shelf open-source models to outperform top-tier proprietary models." is also confusing whether the subject is the judge or the generation. Overall the paper is still generally understandable, but it would need a major revision to improve readability.

W3: I feel skeptical about the observation of inconsistent pointwise scoring. If I understood correctly, there are ways e.g. temperature = 0, fixed seed, etc. to reproduce the consistent scoring even at pointwise manner. Or under same context, pairwise scoring might also be inconsistent due to temperature and other settings. It may be misleading to claim that VLM cannot do consistent pointwise scoring, or it should have clarified in the main text.

Questions for authors

See W1, W2, W3

From Table 3, does it make sense to assume model rankings should naturally align across two entirely different data distributions? LMArena relies on open-ended user prompts that lean heavily toward general text-to-image tasks. But GEdit-EN is another specific benchmark.

Submission title "Vision" is different from paper title "Visual". Typo?

GenArena feels a weird name as it only include visuals, but not other GenAI elements e.g. text / audio. Since the paper only discussed image and video, would something like "VisualArena" be a better name. Justifications?

Reviewer_ooroOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies pointwise versus pairwise VLM judging and introduces GenArena, which combines bidirectional comparisons with BT/Elo aggregation for visual-generation evaluation. It also curates a 6,086-prompt suite covering basic editing, reasoning-intensive editing, and multi-reference composition.

Critical strengths
  1. The protocol comparisons show useful accuracy gains across multiple tasks, and the benchmark construction represents substantial work.
  2. The additional human annotations, inter-annotator agreement, same-judge controls, tie analyses, and sampling experiments substantially strengthen the empirical evidence.
Critical weaknesses
  1. Limited incremental contribution. Pairwise judging, order reversal, and BT/Elo aggregation are established mechanisms. The main contribution remains empirical validation and benchmark integration, rather than a substantially new evaluation methodology. The paper should clarify its distinctive scientific insight and moderate the broad “protocol over scale” narrative: Appendix N shows that the relative advantage of open-source and proprietary judges depends on the task, even under the same pairwise protocol.
  2. Human alignment does not yet establish precise leaderboard reliability. The extended study adds labels, but still covers only 300 unique battles, with repeated annotation on 100. The human-derived ranking audit is valuable; however, Basic has a Spearman correlation of 0.464 with a confidence interval including zero, and the other tracks also have wide intervals (Table 11). The authors acknowledge this uncertainty, but the headline claims remain stronger than the evidence for model ordering. More human comparisons among closely matched models would better validate the leaderboard’s discriminability.
  3. Robustness is narrower than the presentation suggests. Bidirectional consistency mitigates order effects but does not rule out stable, content-dependent judge errors or aesthetic preferences. Removing ties without changing ranks is an internal sensitivity check, not evidence that retained judgments are correct. Moreover, low-temperature pointwise scoring is more self-consistent than the pairwise reference on EditScore-Bench (Table 8), although less accurate. The paper should distinguish repeatability from validity and strengthen cross-judge or style-stratified validation before claiming a generally reliable evaluation standard.
  4. Practical efficiency remains insufficiently quantified. The 5% sampling experiment convincingly recovers the full judge-derived rankings, but measures agreement with the same automated reference rather than human accuracy per unit cost. It also does not establish how the required budget changes with a larger or more closely matched model pool. Reporting judge-call costs, latency or GPU time, and budget-versus-ranking-uncertainty curves would better support the scalability claims.
Questions for authors

Please refer to weaknesses.

Reviewer_WggdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies VLM-based evaluation for visual generation, with emphasis on image editing and multi-reference composition. It claims that pointwise judging, where each output is scored independently in a VIEScore-style setup and preferences come from comparing scalars, agrees poorly with humans and is unstable across repeated runs. It advocates pairwise judging under the same prompt, plus bidirectional order checks, forced binary choices with ties only when swapped-order votes conflict, and Elo or Bradley–Terry aggregation.

The authors claim three contributions: evidence that pairwise judging improves accuracy and self-consistency without fine-tuning the judge; GENARENA, a benchmark with 6,086 prompts in Basic, Reasoning, and MultiRef tracks, VLM-judged battles, and per-track Elo leaderboards; and rankings of editing and composition models validated against human labels and LMArena.

Validation compares pointwise and pairwise judges on human preference datasets, measures self-consistency, compares rankings to LMArena, reports GENARENA leaderboards with a 32B-FP8 judge, ablates ties and judge scale, and includes a human agreement study on roughly 300 battles sampled from GENARENA logs.

Critical strengths

The paper gives a systematic, same-model comparison of pointwise versus pairwise VLM judging on established human-preference benchmarks (image generation, editing, and video), with large gains in preference accuracy and clearer gains in self-consistency. For a field that still leans on VIEScore-style pointwise pipelines for editing and related tasks, this is consequential because it targets a widely used practice rather than a strawman, and it supports the claim that the evaluation protocol is a first-order lever before investing in larger judges or heavy reward-model fine-tuning.

Critical weaknesses

The core scientific contribution is largely confirmatory and overlaps with established practice. The paper’s main takeaway that pairwise judging is more reliable than pointwise scoring for VLMs is already the default intuition in LLM evaluation and is closely related to recent visual pairwise judging work cited in the paper. GENARENA mainly combines known mechanisms with a new prompt pool and battle pipeline. Here the value is primarily empirical packaging and benchmarking, while the writing (e.g., “transformative,” outperforming top proprietary models) suggests a stronger novelty claim than the evidence and prior art support.

Questions for authors

Is the pairwise setup used in the early judge comparisons exactly the same as the one used to build the final leaderboard, including forced choice and order swapping? If not, what differs, and do the main accuracy gains still hold with the full procedure?

For pointwise judging, is each image scored in a separate call? If so, have you tested whether the advantage of pairwise is mostly due to seeing both images in one context rather than to relative versus absolute scoring?

When you compare model rankings to LMArena, are pointwise and pairwise results produced by the same judge on the same generated outputs? If not, can you provide that matched comparison for the main claims?

How are prompt–model pairs chosen for battles, and will you release enough data or code for others to reproduce the published rankings?

How much does the benchmark prompt set overlap with datasets used to tune or select the default judge, and could that affect reported alignment or leaderboard results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.