- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how task formulation affects MLLM evaluation. It finds that model rankings align more closely with benchmark identity than shared capability labels. Changing between binary, multiple-choice, and open-ended formulations can substantially alter rankings even when images and semantic targets are preserved. It further proposes formulation-aware IRT subset selection to reduce the cost of multi-formulation evaluation while preserving much of the full ranking structure.
Critical strengths
Changes in the relative rankings of models were measured across different tasks, with baselines derived from intra-task sampling helping to isolate task-specific effects. The formulation-aware subset selection provides a practical way to reduce multi-formulation evaluation costs. Comparing joint and union subsets also clarifies the trade-off between overall ranking fidelity and fidelity within each formulation.
Critical weaknesses
The generality and additional value of the proposed evaluation framework remain insufficiently established. The experiments focus on older benchmarks, leaving their relevance to current evaluation challenges unclear. Since the 3 formulations impose different task demands, ranking changes do not establish that testing multiple formulations provides a more valid assessment. The methodological gain over existing efficient evaluation approaches also needs clearer evidence.
Questions for authors
- The experiments rely on MME, MMBench, and OCRBench, which were introduced in 2023~2024. Although recent models are included, how representative are these benchmarks of the evaluation challenges facing current MLLMs? Or do they also occur on newer, more challenging tasks that better reflect the capabilities of current MLLMs?
- Binary, multiple-choice, and open-ended QA introduce different task demands and provide different answer cues, so changes in model rankings seem understandable. Why should these changes be considered an evaluation problem? What evidence shows that evaluate all three formats gives a better assessment than choosing the format that matches the intended use?
- Previous work has already shown that different prompt formats can change MLLM rankings. For example, tinyBenchmarks uses small representative subsets to estimate full benchmark scores, PromptEval estimates performance distributions across prompt variants under limited evaluation budgets. What new insight does this paper provide beyond these studies? Could the authors compare their formulation-aware subset selection with relevant existing approaches under the same evaluation settings (budgets and so on)?
8 comments
Q3.1 New insight beyond tinyBenchmarks and PromptEval. First, the paper shows that the rankings commonly reported for a capability are conditional on formulation, so they should not be read as rankings of the capability itself.
On the method side, tinyBenchmarks estimates average performance under a fixed format and PromptEval estimates performance across prompt templates of the same task, both on text-only benchmarks.
We vary the input and output structure on the same images, which changes rankings (Q2.2), and we select items to preserve the ranking in the worst-case formulation rather than to estimate an average score. Section 5 now makes this distinction.
Q3.2 Comparison under the same budget. We added this comparison (subsection 4.4, App. J.1). At the same number of model calls, our method preserves rankings better than all four baselines, and the difference is largest in the worst-case formulation:
| Method (calls per model) | MME | MMBench | OCRBench |
|---|---|---|---|
| Random (100) | 0.34 / 0.04 | 0.40 / 0.15 | 0.60 / 0.07 |
| Stratified random (100) | 0.34 / 0.05 | 0.39 / 0.14 | 0.61 / 0.07 |
| Anchor points (100) | 0.45 / 0.05 | 0.49 / 0.21 | 0.55 / 0.04 |
| Standard IRT, tinyBenchmarks-style (100) | 0.65 / 0.21 | 0.17 / −0.36 | 0.57 / 0.31 |
| Ours, joint (100) | 0.77 / 0.31 | 0.80 / 0.26 | 0.80 / 0.59 |
| Ours, union (75) | 0.70 / 0.59 | 0.76 / 0.56 | 0.73 / 0.59 |
Each cell shows overall tau / worst-case tau_min, with the best value per column in bold. Standard IRT, which ignores formulation, falls to tau_min = −0.36 on MMBench, while union subsets give the best worst-case ranking with fewer calls.
We are happy to give more detail on any of these points.
Dear Reviewer zL6c,
Thank you again for your review. Please respond to our rebuttal and let us know whether any of your concerns remain. We have replied to all of your points above and updated the paper (changes are in blue). We added a saturation analysis and a test of whether the formats only change difficulty, along with the same-budget comparison with other baselines.
If anything is still unclear, we would be glad to answer before the discussion period ends.
Thank you for taking the time to review the revision, Authors
Thank you for recognizing that the "baselines derived from intra-task sampling" help "to isolate task-specific effects", and that "comparing joint and union subsets also clarifies the trade-off between overall ranking fidelity and fidelity within each formulation."
Q1.1 How representative are these benchmarks? We chose MME, MMBench and OCRBench because they are still widely used to compare models: they are part of the OpenVLM leaderboard and VLMEvalKit and appear in recent model releases.
Our question is whether these comparisons depend on formulation, and the effect appears with recent models on all three. We would be glad to add a newer benchmark that the reviewer considers most representative in the final version.
Q1.2 Saturation of older benchmarks. One concern with older benchmarks is that models saturate them. We added a saturation analysis, which shows that the effect does not depend on saturated categories: after removing every category in which more than half of the models score above 90%, all nine cross-formulation values remain significantly below the within-format baseline. On MMBench, the categories that average 93.6% under MCQ average only 58.9% in the open-ended version of the same images, so the high scores reflect the formulation more than easy content (App. F.4).
Q2.1 Why is this an evaluation problem? Benchmarks and leaderboards report one score per capability, and practitioners choose models by capability, not by answer format. We show that the ranking behind such a score can change when only the format changes, so a single-format ranking describes the capability under that format. The abstract and Sections 1 and 6 now say this explicitly.
Q2.2 Different task demands. You note that the three formats impose different demands, so changes in rankings seem understandable. We agree that the demands differ, and we added an analysis that tests whether this alone can explain the ranking changes.
Different demands can act in two ways. A format can make items harder or easier for all models, which shifts accuracy and keeps the order of models. Or a format can favor some models over others, which is the only way rankings can change.
To tell the two apart, we fit two models to the responses and compared how well they predict held-out responses. In the first, each MLLM has one ability and the formulation only changes item difficulty. In the second, each MLLM has a separate ability per formulation. The second predicts better on every fold of all three benchmarks (2.7–4.6% lower log-loss), while the same model with shuffled formulation labels gives no gain (subsection 3.2, App. I.6).
So the formulations do not only change difficulty: they favor different models, and a single-format ranking mixes the capability with a format-specific skill.
Q2.3 Practical consequences. To make the consequence concrete, we added a count of pairwise reversals: switching formulation reverses the order of 29–62% of model pairs, and no two formulations agree on the top model on any benchmark. For example, the top model on MME Binary ranks last on MME Open-ended (App. F.5). So the changes are large enough to change which model is chosen.
Q2.4 Using the format that matches the intended use. We agree, and Section 6 now says so: when the deployment formulation is known, its ranking is the relevant one. However, each benchmark we study offers a single formulation, each a different one, while real use is usually open-ended (Röttger et al., ACL 2024), so the matching formulation is often not available. When it is not known, results across formulations show whether a comparison is robust: the model chosen by the average over formulations never ranks below 3rd / 4th / 6th in any formulation, while the top model of one formulation can rank last in another (App. F.5).