acceptodds
Under review as a conference paper at ICLR 2027

Same Image, Different Ranking: Task Formulation Matters in MLLM Evaluation

Abstract

Multimodal Large Language Models (MLLMs) are typically evaluated using benchmark scores that are often interpreted as stable measures of model capability. In this work we demonstrate inconsistencies in the VLM/MLLM performance measures in popular benchmarks. Instead, rankings tend to align with benchmark identity. That is, tasks within a single benchmark, even those targeting different capabilities, yield similar model rankings. % Crucially, we identify task formulation, the textual definition of the task input and output format, as a source of such discrepancies. To study this, we construct controlled variants of benchmarks, holding the visual content and semantic goal fixed for each item, while varying the task formulation (binary, multiple-choice, or open-ended). We demonstrate that this change can substantially alter both model success on the same example and the resulting overall model rankings. A natural response is to evaluate tasks under multiple formulations, but this substantially increases evaluation costs. To mitigate this, we propose a \em formulation-aware subset selection based on Item Response Theory (IRT). This approach efficiently approximates multi-formulation evaluation using compact subsets, which we confirm recover the performance trends as the full matrix. We conclude that reliable MLLM evaluation must account for variation across task formulations, and offer an effective and efficient protocol to do so.

Primary area:
datasets and benchmarks
Keywords:
evaluation, reliability, methodology, vlm

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zL6cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how task formulation affects MLLM evaluation. It finds that model rankings align more closely with benchmark identity than shared capability labels. Changing between binary, multiple-choice, and open-ended formulations can substantially alter rankings even when images and semantic targets are preserved. It further proposes formulation-aware IRT subset selection to reduce the cost of multi-formulation evaluation while preserving much of the full ranking structure.

Critical strengths

Changes in the relative rankings of models were measured across different tasks, with baselines derived from intra-task sampling helping to isolate task-specific effects. The formulation-aware subset selection provides a practical way to reduce multi-formulation evaluation costs. Comparing joint and union subsets also clarifies the trade-off between overall ranking fidelity and fidelity within each formulation.

Critical weaknesses

The generality and additional value of the proposed evaluation framework remain insufficiently established. The experiments focus on older benchmarks, leaving their relevance to current evaluation challenges unclear. Since the 3 formulations impose different task demands, ranking changes do not establish that testing multiple formulations provides a more valid assessment. The methodological gain over existing efficient evaluation approaches also needs clearer evidence.

Questions for authors
  1. The experiments rely on MME, MMBench, and OCRBench, which were introduced in 2023~2024. Although recent models are included, how representative are these benchmarks of the evaluation challenges facing current MLLMs? Or do they also occur on newer, more challenging tasks that better reflect the capabilities of current MLLMs?
  2. Binary, multiple-choice, and open-ended QA introduce different task demands and provide different answer cues, so changes in model rankings seem understandable. Why should these changes be considered an evaluation problem? What evidence shows that evaluate all three formats gives a better assessment than choosing the format that matches the intended use?
  3. Previous work has already shown that different prompt formats can change MLLM rankings. For example, tinyBenchmarks uses small representative subsets to estimate full benchmark scores, PromptEval estimates performance distributions across prompt variants under limited evaluation budgets. What new insight does this paper provide beyond these studies? Could the authors compare their formulation-aware subset selection with relevant existing approaches under the same evaluation settings (budgets and so on)?
8 comments

Thank you for recognizing that the "baselines derived from intra-task sampling" help "to isolate task-specific effects", and that "comparing joint and union subsets also clarifies the trade-off between overall ranking fidelity and fidelity within each formulation."

Q1.1 How representative are these benchmarks? We chose MME, MMBench and OCRBench because they are still widely used to compare models: they are part of the OpenVLM leaderboard and VLMEvalKit and appear in recent model releases.

Our question is whether these comparisons depend on formulation, and the effect appears with recent models on all three. We would be glad to add a newer benchmark that the reviewer considers most representative in the final version.

Q1.2 Saturation of older benchmarks. One concern with older benchmarks is that models saturate them. We added a saturation analysis, which shows that the effect does not depend on saturated categories: after removing every category in which more than half of the models score above 90%, all nine cross-formulation values remain significantly below the within-format baseline. On MMBench, the categories that average 93.6% under MCQ average only 58.9% in the open-ended version of the same images, so the high scores reflect the formulation more than easy content (App. F.4).

Q2.1 Why is this an evaluation problem? Benchmarks and leaderboards report one score per capability, and practitioners choose models by capability, not by answer format. We show that the ranking behind such a score can change when only the format changes, so a single-format ranking describes the capability under that format. The abstract and Sections 1 and 6 now say this explicitly.

Q2.2 Different task demands. You note that the three formats impose different demands, so changes in rankings seem understandable. We agree that the demands differ, and we added an analysis that tests whether this alone can explain the ranking changes.

Different demands can act in two ways. A format can make items harder or easier for all models, which shifts accuracy and keeps the order of models. Or a format can favor some models over others, which is the only way rankings can change.

To tell the two apart, we fit two models to the responses and compared how well they predict held-out responses. In the first, each MLLM has one ability and the formulation only changes item difficulty. In the second, each MLLM has a separate ability per formulation. The second predicts better on every fold of all three benchmarks (2.7–4.6% lower log-loss), while the same model with shuffled formulation labels gives no gain (subsection 3.2, App. I.6).

So the formulations do not only change difficulty: they favor different models, and a single-format ranking mixes the capability with a format-specific skill.

Q2.3 Practical consequences. To make the consequence concrete, we added a count of pairwise reversals: switching formulation reverses the order of 29–62% of model pairs, and no two formulations agree on the top model on any benchmark. For example, the top model on MME Binary ranks last on MME Open-ended (App. F.5). So the changes are large enough to change which model is chosen.

Q2.4 Using the format that matches the intended use. We agree, and Section 6 now says so: when the deployment formulation is known, its ranking is the relevant one. However, each benchmark we study offers a single formulation, each a different one, while real use is usually open-ended (Röttger et al., ACL 2024), so the matching formulation is often not available. When it is not known, results across formulations show whether a comparison is robust: the model chosen by the average over formulations never ranks below 3rd / 4th / 6th in any formulation, while the top model of one formulation can rank last in another (App. F.5).

Q3.1 New insight beyond tinyBenchmarks and PromptEval. First, the paper shows that the rankings commonly reported for a capability are conditional on formulation, so they should not be read as rankings of the capability itself.

On the method side, tinyBenchmarks estimates average performance under a fixed format and PromptEval estimates performance across prompt templates of the same task, both on text-only benchmarks.

We vary the input and output structure on the same images, which changes rankings (Q2.2), and we select items to preserve the ranking in the worst-case formulation rather than to estimate an average score. Section 5 now makes this distinction.

Q3.2 Comparison under the same budget. We added this comparison (subsection 4.4, App. J.1). At the same number of model calls, our method preserves rankings better than all four baselines, and the difference is largest in the worst-case formulation:

Method (calls per model)MMEMMBenchOCRBench
Random (100)0.34 / 0.040.40 / 0.150.60 / 0.07
Stratified random (100)0.34 / 0.050.39 / 0.140.61 / 0.07
Anchor points (100)0.45 / 0.050.49 / 0.210.55 / 0.04
Standard IRT, tinyBenchmarks-style (100)0.65 / 0.210.17 / −0.360.57 / 0.31
Ours, joint (100)0.77 / 0.310.80 / 0.260.80 / 0.59
Ours, union (75)0.70 / 0.590.76 / 0.560.73 / 0.59

Each cell shows overall tau / worst-case tau_min, with the best value per column in bold. Standard IRT, which ignores formulation, falls to tau_min = −0.36 on MMBench, while union subsets give the best worst-case ranking with fewer calls.

We are happy to give more detail on any of these points.

Dear Reviewer zL6c,

Thank you again for your review. Please respond to our rebuttal and let us know whether any of your concerns remain. We have replied to all of your points above and updated the paper (changes are in blue). We added a saturation analysis and a test of whether the formats only change difficulty, along with the same-budget comparison with other baselines.

If anything is still unclear, we would be glad to answer before the discussion period ends.

Thank you for taking the time to review the revision, Authors

Reviewer_7dQHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how different task formulations affect the reliability of multimodal large language model (MLLM) evaluations. The authors begin by analyzing the performance of more than 250 MLLMs across a range of benchmarks. They find that model rankings are often influenced more by the specific benchmark used than by the underlying capabilities being tested. To better understand this effect, the authors create binary, multiple-choice, and open-ended versions of questions from MME, MMBench, and OCRBench, while keeping the images and intended answers unchanged. Their results show that changing the question format can lead to noticeable differences in both individual model responses and overall model rankings. To make evaluations across multiple formats more efficient, the authors also introduce a subset-selection method based on Item Response Theory (IRT). Their experiments suggest that carefully selected smaller subsets can produce model rankings that closely match those obtained from much larger evaluation sets.

Critical strengths
  1. Strong empirical evidence that task formulation affects evaluation reliability. The paper raises an important concern about current MLLM benchmarks: whether model rankings remain consistent when the same capability is evaluated using different question formats. The controlled experiments provide convincing evidence that this is not always the case. For example, Table 2 reports Kendall's tau values of 0.10 for binary versus open-ended questions on MME, -0.16 on MMBench, and 0.01 on OCRBench. These values are much lower than the ranking agreement observed within the same question format, suggesting that task formulation can have a substantial impact on model rankings.
  2. A practical approach to making multi-formulation evaluation more efficient. Another strength is that the paper goes beyond identifying the problem and proposes a practical way to address it. Evaluating models across multiple question formats can be expensive, so the proposed IRT-based subset-selection method is a useful contribution. The results are encouraging, showing that a relatively small subset of examples can achieve a Kendall correlation of around 0.8 with rankings obtained from the full evaluation set.
Critical weaknesses
  1. It is unclear whether the different task formulations are always semantically equivalent. My main concern is whether the converted questions consistently test the same underlying capability. Although the authors aim to preserve the original meaning when converting questions between formats, some transformations seem to change what the model is actually being asked to do. For example, Appendix E describes converting a binary question about whether a cat is present into an open-ended question asking which animals appear in the image. The open-ended question may require the model to identify multiple animals rather than simply determine whether a cat is present. This introduces additional difficulty that is not part of the original question. As a result, some of the observed differences in model rankings may come from changes in question difficulty or semantic requirements, rather than the question format itself. A more systematic check of semantic equivalence would help clarify how much of the reported instability can be attributed specifically to task formulation.
  2. Some practical limitations of the subset-selection method need further discussion. The proposed IRT model uses a shared latent ability parameter across formulations and mainly accounts for formulation differences through item difficulty. However, this setup does not directly capture cases where two models reverse their relative rankings across question formats. The authors acknowledge this limitation and partly address it by combining selected examples from different formulations, but it remains unclear how well the approach handles more substantial ranking changes. I also think the computational cost deserves more attention. The subset-selection method requires an existing model-response matrix to fit the IRT model before informative examples can be selected. While the paper shows that smaller subsets can reduce evaluation costs, it does not fully account for the initial calibration effort. This could matter in practice, especially when new models are introduced frequently or benchmarks are updated. A clearer discussion of the calibration cost and how often the selection process needs to be repeated would make the practical contribution stronger.
Questions for authors
  1. Semantic equivalence: How did the authors verify that the binary, multiple-choice, and open-ended versions of each question test the same underlying information? In particular, would the large changes in model rankings still hold if the analysis were limited to manually verified examples where the different formats have equivalent semantic requirements?
  2. Robustness of the proposed method: Since the IRT model assumes a shared ability parameter across formulations, it cannot directly capture cases where model rankings reverse across question formats. How much does this assumption affect the subset-selection results? Also, does the proposed union strategy consistently help address this limitation across the different benchmarks?
  3. Evaluation cost: Could the authors provide more details on the initial cost of fitting the IRT model and how often the selected subsets would need to be updated? For example, would recalibration be necessary when evaluating new models with substantially different capabilities? Clarifying these points would help readers better understand the actual efficiency gains of the proposed method.
4 comments

Thank you for recognizing the "strong empirical evidence that task formulation affects evaluation reliability", and that "the paper goes beyond identifying the problem and proposes a practical way to address it."

Q1. Semantic equivalence and results on verified items. The large ranking changes remain when we keep only the items verified to be equivalent across formulations.

We agree that the "cat" example was not equivalent. It was a poorly chosen illustration in the appendix text (all MME existence items in the data use a singular question), and we replaced it with real items.

Equivalence check: In the submission we relied on the shared image and reference answer of each item, and we now check every item explicitly. Two VLMs from different model families checked that the three versions target the same fact, have a single answer that can be determined from the image, add no subtask and have consistent references. 80–89% of the items pass, and a blind manual check of 1,500 items agrees with the automatic decision on 88.2% of them.

Results on verified items: Table 2 recomputed on the verified items:

Kendall tau (all → verified items)MMEMMBenchOCRBench
Binary vs. MCQ0.39 → 0.40*0.18 → 0.19*−0.05 → −0.08*
Binary vs. Open−0.12 → −0.13*−0.17 → −0.20*−0.03 → −0.02*
MCQ vs. Open0.30 → 0.29*0.22 → 0.18*0.39 → 0.40*
Within-format baseline0.72 → 0.730.73 → 0.730.73 → 0.72

No value changes by more than 0.04, and all nine remain significantly below the within-format baseline, so the ranking changes are not explained by differences in what the formulations ask (subsection 3.1, Table 2 bottom, App. E.4).

Q2.1 Effect of the shared ability. The shared θ_m affects which items are selected, but not how models are ranked. As you note, the submission acknowledges that the IRT model does not capture reversals (footnote in subsection 4.1, now in the main text), and the paper does not rely on it to do so. θ_m only scores how informative each item is. Rankings are computed from the models' actual responses on the subset, and reversals are measured directly by tau_min. The cost of this assumption shows in the worst-case formulation of joint subsets, and union subsets reduce it (Q2.2). We added the assumption to the limitations (Section 7).

Q2.2 Does the union strategy consistently help? Yes, on all three benchmarks. Selecting items separately for each formulation protects the worst-case formulation, which is where joint subsets are weakest:

Worst-case tau_minMMEMMBenchOCRBench
Joint, 100 calls per model0.310.260.59
Union, 75 calls per model0.590.560.59
Union, k = 100 per formulation0.660.550.64

The gain is largest on MME and MMBench, where formulations reverse rankings most often. On OCRBench, union subsets match joint subsets with fewer calls and exceed them with more (Fig. 4 right, App. J.1).

Q3.1 Initial cost. The cost of calibration is the one-time evaluation of the reference models, not the IRT fit. We added a cost analysis: fitting the model takes 1.0–4.6 s on a laptop CPU, and selecting all subsets takes under 5 ms. The reference evaluation takes 42K–182K responses per benchmark, and after it new models are evaluated only on the subset (subsection 4.2, App. J.2).

Q3.2 When to recalibrate. The subsets transfer well to new models, and recalibration is recommended mainly when a new model family is added.

We added a 5-fold cross-validation over models: held-out models are a median of 1.0 rank from their full-data rank (5.3 for a random ranking), and the error decreases as more calibration models are added (learning curve in App. J.2). As you expected, the largest error (10 ranks) is for a model with an unusual profile: Claude 4 Sonnet on MME, the only model of its family in our pool. We now recommend recalibrating when a new model family is added or the benchmark changes (Sections 4.2 and 7).

Thank you again for the constructive suggestions. We are happy to give more detail on any of these points.

Dear Reviewer 7dQH,

Thank you again for your review. Please respond to our rebuttal and let us know whether any of your concerns remain. We have replied to all of your points above and updated the paper (changes are in blue). The biggest changes are the ranking changes are held on the verified items, and we added the cost and recalibration analyses you asked about.

If anything is still unclear, we would be glad to answer before the discussion period ends.

Thank you for taking the time to review the revision, Authors

I thank the authors for their response. I have decided to maintain my original rating.

Dear Reviewer 7dQH,

As the discussion period ends, here is a short summary of how we addressed your main concerns.

The ranking changes remain on items verified to be equivalent across formulations (no τ changes by more than 0.04), and a blind manual check of 1,500 items agrees with the automatic check on 88.2% of them.

We show that the shared ability only affects which items are selected and union subsets improve the worst-case formulation on all three benchmarks.

We also added the cost analysis and a cross-validation over models, which shows that subsets transfer well to new models and recommend when to do recalibration (mainly needed when a new model family is added).

Thank you again for your constructive review, and if our responses resolve your concerns, we hope you will consider this in your final evaluation.

Thanks, authors

Reviewer_QmFnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether MLLM leaderboard rankings actually tell us that a model is better at a given "capability." The authors do three things: They collect scores for 250+ models on about 90 tasks from nine benchmarks and run a factor analysis. Tasks mostly group by which benchmark they come from, and cross-benchmark task pairs that are supposed to measure the same capability agree poorly in their rankings (Kendall's τ\tau of only 0.12–0.50). They rewrite MME, MMBench and OCRBench items into Binary, multiple-choice and open-ended versions and evaluate 16 models. Rank agreement across formats is clearly lower than a within-format bootstrap baseline. They propose a format-aware 2PL IRT subset-selection method and claim that 3–5% of the items are enough to recover the full ranking under multi-format evaluation.

Critical strengths
  1. The question matters, and the core control is solid. Table 2 does not just report cross-format τ\tau. It compares that τ\tau with agreement between disjoint item samples of the same format, and uses a paired bootstrap to test the difference. This rules out the explanation that rankings differ only because a different set of items was used. It is the main reason I think the claim that formulation changes rankings mostly holds up.
  2. The instance-level analysis shows where the effect comes from. Most of the difference comes from items that models solve as Binary or multiple-choice questions but fail when the same item is asked open-ended. So the finding rests on more than aggregate numbers, and it gives useful guidance for designing future benchmarks.
Critical weaknesses
  1. The paper's main storyline is unclear. Sections 2–3 argue that rankings follow the benchmark and the task format. Section 4 then switches to IRT subset selection to make evaluation cheaper. The only link between the two is a cost argument on pp. 2 and 7, and the paper reads like two papers stitched together. The evidence for the second part is also thin. This matters because the method contribution rests mainly on this part. If it does not hold up, the paper is left with one observational finding.
  2. How "capabilities" are defined is unclear, and figures map loosely to claims.
    • Page 2 gives only a one-sentence verbal definition. In practice, the capability labels are just each benchmark's own category names (e.g., MME_OCR).
    • Table 1 calls its cross-benchmark task pairs "representative" but gives no selection rule. This matters because "rankings follow the benchmark, not the capability" is the paper's starting point. Whether that claim holds depends directly on how capability is defined.
  3. The rewritten items may no longer ask the same question.
    • The rewrite example on p. 20 turns "Is there a cat?" into "What animals are present?", which is a different question.
    • The paper does not describe at all how the MMBench and OCRBench items were rewritten. The paper's central control is "change only the format and keep the meaning fixed." If that does not hold, the ranking changes cannot be attributed to format.
  4. The reliability of the scoring itself is under-reported.
    • Open-ended answers are scored by an LLM judge, but the judge was validated against a single human annotator, and κ\kappa is given as 0.83 in the main text and as "83" in the appendix.
    • The paper says results are robust to exact-match scoring but gives no numbers.
Questions for authors
  1. In one sentence, what is the paper's main claim? Beyond saving cost, how does the subset selection in Section 4 support it? Please also add a random-selection baseline and run-to-run variance to Figure 4, and state the item-pool size behind "3–5%". (Weakness 1)
  2. How were capability labels assigned to tasks, and what rule was used to choose the task pairs in Table 1? Could you list every pair you considered with its τ\tau, and explain why OCRBench text recognition is paired with RealWorldQA? (Weakness 2)
  3. How were the MMBench and OCRBench items rewritten, and how did you check that their meaning was preserved? How many items does each benchmark have in each format? Do the Table 2 conclusions still hold if you keep only the items whose open-ended answer is unique? (Weakness 3)
  4. Could you report judge–human agreement separately for each format and give the exact κ\kappa? Do the Table 2 conclusions change with a different judge, or if Binary and multiple-choice items are scored by exact match? (Weakness 4)
4 comments

Thank you for recognizing that "the question matters, and the core control is solid", and that the instance-level analysis shows that "the finding rests on more than aggregate numbers".

Q1.1 Storyline and main claim. We revised the paper so that it reads as one argument. The claim is that MLLM rankings are conditional on task formulation, so a ranking obtained under one formulation should not be read as a ranking of the capability itself. Sections 2 and 3 establish this claim, and Section 4 is its continuation: if rankings depend on formulation, a fair comparison has to cover several formulations, and Section 4 makes this practical, with fewer model calls than running the full benchmark in a single formulation. Beyond cost, Section 4 supports the claim in two ways. Subsets selected for one formulation transfer poorly to the others (Fig. 5), and selection that ignores formulation fails in the worst-case formulation (standard IRT, subsection 4.4). We revised the abstract, section 1 and the opening of section 4 to make this link explicit.

Q1.2 Random baseline and variance. We added a comparison with random subsets and run-to-run variance to Fig. 4. Our subsets preserve the ranking much better than random subsets of the same size.

At k = 100, they reach tau = 0.77 ± 0.05 / 0.80 ± 0.07 / 0.80 ± 0.07 on MME / MMBench / OCRBench, compared with 0.34 ± 0.09 / 0.40 ± 0.13 / 0.60 ± 0.10 for random subsets (std over 5 folds for our method and over 100 draws for random subsets). The baseline is described in subsection 4.2, and App. J.1 has the full comparison, which also includes stratified random selection, anchor points and standard IRT.

Q1.3 Pool size behind "3–5%". This share was computed over the three benchmarks together, which was not clear. Per benchmark, k = 100 corresponds to 8.4% / 2.3% / 10.0% of the 1,187 / 4,329 / 1,000 items, and sections 1 and 4.2 now report it this way.

Q2.1 Definition of capability. You are right that the submission did not define the term precisely and used each benchmark's category name in its place. We did so because several benchmarks present their categories as the abilities they measure (for example, MMBench organizes its tasks into "ability dimensions", and MME into perception and cognition abilities). However, a capability and a category are not the same. A category, which we call a task, is a group of items as defined and reported by a benchmark, such as MME OCR or OCRBench Text Recognition. A capability is the skill a task is designed to measure, and tasks from different benchmarks can target the same capability. Subsection 2.1 now gives this definition.

To link the two, we added an explicit mapping from tasks to capabilities. One author and an LLM independently assigned each of the 87 tasks to one of 14 capability labels, using the task's name, its benchmark and the benchmark's description of the task (κ = 0.92; disagreements were resolved by the author). For instance, MME OCR and OCRBench Text Recognition both receive the label OCR / text recognition. Two tasks share a capability if they have the same label (subsection 2.1, App. D.1).

Q2.2 Selection rule for Table 1, and all pairs. In the submission, Table 1 and its appendix version showed only four representative pairs, without a selection rule. We replaced them with a full analysis in which no pair is hand-picked: the pairs follow from the mapping above. App. D.1 now lists all 158 cross-benchmark pairs of tasks that share a capability label (Table 6), and compares them with all other groups of task pairs. The result supports the starting point of the paper: two tasks with the same capability label from different benchmarks agree no more than two unrelated tasks from the same benchmark.

Task pairsPairsMedian tau
Same benchmark, same capability1690.64
Same benchmark, different capability4050.44
Different benchmarks, same capability1580.43
Different benchmarks, different capability3,0090.35

Table 1 now shows one pair per capability, chosen as the most direct one-to-one match between task definitions, and its caption states this rule.

Q2.3 OCRBench vs. RealWorldQA. You are right that RealWorldQA is not an OCR task, and we removed this pair.

Q3.1 Rewriting MMBench and OCRBench, and item counts. We added a description of both conversions to App. E.2 and E.3, with item counts per benchmark and formulation. In short, the MMBench open-ended version removes the options and uses the correct option as the reference, and the OCRBench MCQ version adds four generated distractors to the reference.

Q3.2 Preserving the meaning. The conclusions do not change when we keep only the items verified to be equivalent across formulations (below). We agree that the "cat" example was not equivalent. It was a poorly chosen illustration in the appendix text (all MME existence items in the data use a singular question), and we replaced it with real items.

Equivalence check: In the submission we relied on the shared image and reference answer of each item, and we now check every item explicitly. Two VLMs from different model families checked that the three versions target the same fact, have a single answer that can be determined from the image, add no subtask and have consistent references. 80–89% of the items pass, and a blind manual check of 1,500 items agrees with the automatic decision on 88.2% of them (This is described in subsection 3.1 and in a new subsection of Appendix E).

Results on verified items: Table 2 recomputed on the verified items:

Kendall tau (all → verified items)MMEMMBenchOCRBench
Binary vs. MCQ0.39 → 0.40*0.18 → 0.19*−0.05 → −0.08*
Binary vs. Open−0.12 → −0.13*−0.17 → −0.20*−0.03 → −0.02*
MCQ vs. Open0.30 → 0.29*0.22 → 0.18*0.39 → 0.40*
Within-format baseline0.72 → 0.730.73 → 0.730.73 → 0.72

No value changes by more than 0.04, and all nine remain significantly below the within-format baseline, so the ranking changes are not explained by differences in what the formulations ask (subsection 3.1, Table 2 bottom, App. E.4).

Q3.3 Items with a unique open-ended answer. The conclusions also hold on this subset. Keeping only items whose open-ended reference is a single short answer gives almost the same items and the same results (App. E.4).

Q4.1 Per-format agreement and κ. Thank you for catching the typo. The correct value is κ = 0.83, and we fixed it. Judge–human agreement is 97% on Binary (n = 100), 97% on MCQ (n = 100) and 89% on open-ended responses (n = 300), now reported in subsection 3.1 and App. E.

Q4.2 A different judge and EM. The conclusions hold under both, which means that the ranking changes are not an artifact of the judge or of how the answers are scored.

We re-scored open-ended responses with a second judge from a different model family (Claude Haiku 4.5), which agrees with our judge on 93.8% of responses (κ = 0.87), and cross-formulation tau changes by at most 0.10.

EM agrees with the judge on 99.2% of Binary and 98.9% of MCQ responses, and under EM scoring 8 of the 9 values remain significant. The exception is OCRBench MCQ vs. Open (tau = 0.53), which was already the closest to the within-format baseline (subsection 3.1, App. F.3).

We hope these answers address your concerns, and we are happy to give more detail on any of them.

Dear Reviewer QmFn,

Thank you again for your review. Please respond to our rebuttal and let us know whether any of your concerns remain. We have replied to all of your points above and updated the paper (changes are in blue). The biggest changes are the clearer storyline, the definition of capability with the task mapping, and the results on verified items.

If anything is still unclear, we would be glad to answer before the discussion period ends.

Thank you for taking the time to review the revision, Authors

Dear Reviewer QmFn,

As the discussion period ends, here is a short summary of how we addressed your main concerns.

We revised the paper so that Section 4 follows from the main claim, added the random baseline with variance (our subsets reach τ = 0.77–0.80 at k = 100, against 0.34–0.60 for random).

We defined capability explicitly and replaced the hand-picked pairs with all 158 same-capability pairs.

we also showed that the results hold on items verified to be equivalent across formulations (with a blind manual check of 1,500 items agreeing with the automatic check on 88.2%) and under a second judge and EM scoring.

We also addressed your other points, including the per-benchmark pool sizes, the conversion details and item counts, and the κ value.

If these changes address your concerns, we would appreciate it if you would take them into account in your final assessment.

Thanks, authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful and constructive reviews. We uploaded a revised PDF.

The main claim of the paper is that MLLM rankings are conditional on task formulation, so a ranking obtained under one formulation should not be read as a ranking of the capability itself. From this Section 4 follows that if rankings depend on formulation, a fair comparison has to cover several formulations, which multiplies the cost of evaluation, which we are able to more than mitigate.

In response to the reviews we added several analyses (you can find them in the attached new revision of the paper). All of them support the conclusions of the submission, and the main result now holds under each additional check: on items verified to be equivalent, under alternative scoring, and after removing saturated categories.

The main changes are:

  1. Main claim and storyline (QmFn Q1, zL6c Q2): We now state the claim explicitly in the abstract, sections 1 and 6, and section 4 opens by presenting subset selection as a consequence and a test of it.
  2. Semantic equivalence (QmFn Q3, 7dQH Q1): We verified all items with two VLMs and manually checked all of them. On the verified items, every cross-formulation tau changes by at most 0.04, and all nine remain significantly below the within-format baseline (subsection 3.1, Table 2, App. E.4).
  3. Scoring (QmFn Q4): We report human agreement per format, and results under EM scoring and with a second judge from a different model family. The conclusions hold (subsection 3.1, App. E, App. F.3).
  4. Capability labels (QmFn Q2): We mapped all tasks to shared capability labels and analyzed all 158 same-capability task pairs across benchmarks, where the submission showed four examples (subsection 2.1, App. D.1).
  5. Why rankings change, and what it means in practice (zL6c Q1, Q2): A held-out test shows that formulations favor different models rather than only making items harder or easier (subsection 3.2, App. I.6).
  6. Saturation (zL6c Q1): The effect is not caused by saturated benchmarks: after removing the categories in which most models score above 90%, all nine cross-formulation values remain significantly below the within-format baseline (App. F.4).
  7. Practical impact (zL6c Q2): Switching formulation reverses the order of 29–62% of model pairs, and no two formulations agree on the top-ranked model on any benchmark (App. F.5).
  8. Subset selection (QmFn Q1, 7dQH Q2–Q3, zL6c Q3): We compared our method with random, stratified, anchor-point and standard IRT selection at the same number of model calls, and we now report pool sizes, run time and results on held-out models (section 4, Fig. 4, App. J).
  9. Other additions: We now report the pool size per benchmark behind the compression rates (QmFn Q1), the calibration cost and results on held-out models (7dQH Q3; subsection 4.2, App. J.2), results on items with a unique open-ended answer (QmFn Q3; App. E.4), and how the MMBench and OCRBench formulations were built (QmFn Q3; App. E.2–E.3). We also expanded the related work (section 5; zL6c Q3) and the limitations (Section 7; 7dQH Q2).

To keep all the analysis experiments consistent, we recomputed the Section 3 results with all 16 models on a single bootstrap procedure. All nine cross-formulation values the agreement between ranking from two different formulations is significantly lower than the ranking from two disjoint item samples of the same formulation, and the conclusions remained unchanged. Detailed answers to each reviewer follow below.

Dear Reviewers and Area Chair,

Thank you for the helpful reviews.

We have replied to each reviewer and uploaded a revised paper, with the changes in blue. The main additions are a check that the results hold on items verified to be equivalent across formulations, an explicit definition of capability with a mapping of all 87 tasks, a saturation analysis, and a comparison with four baselines at the same number of model calls.

We would be happy to answer any further questions before the discussion period ends. Thanks again, Authors

Dear AC and Reviewers,

Thank you for the helpful reviews. We replied to each reviewer and uploaded a revised paper (changes in blue). The main additions are results on items verified to be equivalent across formulations, a definition of capability with a mapping of all 87 tasks, a saturation analysis, and a comparison with four baselines at the same budget.

During the discussion we also added results on MathVista, where rankings again change across formulations, and showed that the results for it are inline w/ the trend we present in this paper where the model chosen by averaging over formulations stays within 3.44 points of the best model, while a single-format winner can fall 30.75 points behind (replies to Reviewer zL6c).

We will add both to the final version.

Thanks, authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.