- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper revisit the idea of VLM as a judge under absolute score and pairwise manner, then exposed the problem of absolute scoring. Authors introduced "GenArena", an elo system benchmark based on pairwise scoring. The benchmark is validated, coming up a conclusion that elo systems are the way to achieve human-aligned evaluation for visual generation tasks.
Critical strengths
S1: Human aligned evaluation for visual generation tasks is an important problem. While the drawback of pointwise scoring is intuitively known, the paper does a good job of formally quantifying it by consistency metrics and correlation metrics.
Critical weaknesses
W1: The conceptual novelty feel overstated. The psychological framing is intuitive, but it is not a direct port to the problem. Elo is already a standard practice in existing benchmarks (such as GenAI Arena and Chatbot Arena), and the paradigm of replacing human voters with automated pairwise AI judges has already been extensively established by prior VLM-as-a-Judge literatures. This work functions as a confirmatory empirical study validating known techniques on visual context.
W2: The paper wasted a lot of space over-explaining standard background concepts rather than spotlighting the core empirical results. The introduction also reads bulky and can be trimmed. Also poor data visualization on Table 1 as it is heavily fragmented with "--" dashes, there might be alternative choices for visualizing the results (e.g. charts) other than tables. In the abstract "Crucially, our experiments uncover a transformative finding that simply adopting this pairwise protocol enables off-the-shelf open-source models to outperform top-tier proprietary models." is also confusing whether the subject is the judge or the generation. Overall the paper is still generally understandable, but it would need a major revision to improve readability.
W3: I feel skeptical about the observation of inconsistent pointwise scoring. If I understood correctly, there are ways e.g. temperature = 0, fixed seed, etc. to reproduce the consistent scoring even at pointwise manner. Or under same context, pairwise scoring might also be inconsistent due to temperature and other settings. It may be misleading to claim that VLM cannot do consistent pointwise scoring, or it should have clarified in the main text.
Questions for authors
See W1, W2, W3
From Table 3, does it make sense to assume model rankings should naturally align across two entirely different data distributions? LMArena relies on open-ended user prompts that lean heavily toward general text-to-image tasks. But GEdit-EN is another specific benchmark.
Submission title "Vision" is different from paper title "Visual". Typo?
GenArena feels a weird name as it only include visuals, but not other GenAI elements e.g. text / audio. Since the paper only discussed image and video, would something like "VisualArena" be a better name. Justifications?