acceptodds
Under review as a conference paper at ICLR 2027

Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling

Abstract

Multiple-choice benchmarks are cheap to grade and are running out of room, and the standard remedy, writing harder items, is slow and repeated for every benchmark. A saturated benchmark still holds a harder task. Each question's wrong options are written for that question alone, so a model can score by eliminating a few options. We propose AnswerPool: take questions that share a context, pool all their options into one list, and ask the model to assign every question its answer. No item is written and no label changes. The chance of guessing a group right falls from to for five four-option questions, and a model that recognizes its answers keeps its multiple-choice score, so the accuracy lost to pooling measures the credit the format gave for elimination. Deleting answers from the pool makes questions unanswerable with exact ground truth, so abstention is scored in the same pass. Across eight text, image, and video benchmarks and eighteen models, pooling is harder for every model, the elimination credit is largest for the weakest models, and seven of eight open-weight models answer 87 to 100% of unanswerable questions.

Primary area:
datasets and benchmarks
Keywords:
large language models, evaluation, benchmarks, multiple choice question, abstention, unanswerable questions, robustness, multimodal

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CeEiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Answer Pooling, a new evaluation protocol for multiple-choice benchmarks designed to address the potential saturation of current LLM evaluation. Instead of evaluating each multiple-choice question independently, Answer Pooling combines answer options from multiple related questions into a shared candidate pool and requires models to assign answers under a uniqueness constraint. The authors argue that conventional multiple-choice evaluation may overestimate model ability because models can exploit elimination strategies among answer choices. The paper evaluates Answer Pooling across multiple text, image, and video benchmarks with a broad range of language and multimodal models. The authors further investigate two related properties: whether models can recognize unanswerable questions when correct answers are removed, and how much contextual information contributes to performance by comparing original and context-removed settings. The experimental results show that Answer Pooling generally reduces model accuracy and changes model rankings, suggesting that conventional multiple-choice benchmarks may underestimate the difficulty of evaluation tasks.

Critical strengths

1.The paper introduces a simple transformation that can be applied to existing multiple-choice benchmarks without requiring expensive human annotation. This is consequential because benchmark saturation is an important challenge for evaluating increasingly capable LLMs. 2.The evaluation across multiple text and multimodal benchmarks demonstrates that the proposed phenomenon is not limited to a single dataset or modality.

Critical weaknesses

1.The interpretation of Answer Pooling results is not sufficiently isolated. The paper interprets the accuracy gap between standard multiple-choice and Answer Pooling as evidence of "elimination credit". However, the proposed setting introduces additional reasoning requirements beyond elimination, including global matching and constraint satisfaction. Without stronger controls, it remains unclear whether the method measures hidden reasoning ability or simply creates a harder task format. 2.A more difficult benchmark does not automatically imply a better benchmark. The paper needs stronger evidence that Answer Pooling scores correlate better with meaningful model capabilities. 3.Cross-modal conclusions are too strong. The paper suggests that Answer Pooling reveals a general limitation of multiple-choice evaluation. However, the meaning of performance drops differs substantially between text, image, and video tasks.The authors should avoid assuming that the same mechanism explains all modalities.

Questions for authors

1.Can the authors provide experiments separating semantic answer recognition from global assignment ability? For example, what happens if models first predict answers independently and a separate algorithm performs the matching? 2.How does Answer Pooling performance correlate with external reasoning benchmarks or human performance? Does the proposed metric provide improved construct validity? 3.How sensitive are results to different grouping strategies? Would different question grouping methods substantially change model rankings?

4 comments

We thank the reviewer for the constructive comments. We answer each point below.

Isolating the drop (W1). Global matching and constraint satisfaction do not produce the drop: it is complete before the questions are answered together (step 5 below). To find the cause, we went from multiple choice to the pooled format one change at a time, on the same questions. The table gives what each change costs in mean accuracy over the 11 text models.

StepChangeWhat it isolatesQuALITYRACE
Multiple choice (starting accuracy)0.8070.910
1The question's four options are shuffled among themselvesOrder of the question's own options0.000.00
2Options from another passage are listed after the itemA longer list-0.02-0.02
3All options of that list are shuffledThe question's options no longer next to each other-0.06-0.04
4The other options are those of the other questions on the passagePlausible alternatives-0.07-0.08
5All questions on the passage are answered jointlyJoint assignment+0.01+0.01
Pooled (final accuracy)0.6680.782
  • Order and a longer list (steps 1 and 2) cost almost nothing. A model ignores options that do not belong to its question: with them on the page it still picks one of its own four options in 98% of cases, and its choice among the four is unchanged.
  • Separating the four options (step 3) is the first half of the drop. A model still knows which options are its own (97% of its picks), but it chooses among them less well: wrong picks among the four rise from 19% to 24% on QuALITY. This is consistent with the paper's criticism of multiple choice, that it lets a model answer by comparing its options side by side.
  • Plausible alternatives (step 4) are the second half. When the implausible options are replaced with plausible ones, the options of the other questions on the same passage, a model picks an option that is not its own in 17% of cases instead of 3%, and accuracy falls by 7 points on QuALITY and 8 points on RACE. This shows the central point of the paper: the options a benchmark wrote for its other questions act as hard wrong answers for this one. A benchmark with several questions per passage therefore already contains a harder test, and multiple choice leaves it unused.
  • Joint assignment (step 5) costs nothing. The drop is complete before the questions are answered together.

These steps show that a multiple-choice item helps a model in two ways that are separate from knowing the answer: it places the options side by side for comparison, and it hides every other plausible answer. Answer pooling removes both, using only the options the benchmark already has. We will add this table to Section 4.3 and define the elimination credit by these two parts.

Recognition against global assignment (W1, Q1). We ran the experiment the reviewer proposes, and global assignment is not what makes the pooled format harder. Each question is first answered independently against the pool. A separate algorithm then computes the best one-to-one assignment from the scores the model gave to the options. The table gives the mean accuracy on QuALITY of the five models that return these scores.

Who does the matchingAccuracy
Multiple choice, for reference0.758
Nobody: each question is answered independently against the pool0.584
A separate algorithm, from the model's independent answers0.614
The model itself: all questions assigned jointly (pooled format)0.596
  • The drop is there before any matching. Answered independently, with no uniqueness constraint, the questions already lose 0.17 against multiple choice.
  • Matching helps a little, whoever does it. The algorithm adds 0.03, and a model that does the assignment itself is within 0.02 of the algorithm.

The pooled format therefore measures whether a model recognises the answer among plausible options, and not its ability to solve an assignment. We will add this table to Section 4.3.

A more difficult benchmark and a better benchmark (W2, Q2). We agree, and we do not claim a better benchmark. We leave the benchmark as it is and change only the format in which it is asked. A score depends on four things:

Part of an evaluationWho sets itWhat usually happens to it
The benchmark: its questions, options and labelsIts designerReplaced by a new benchmark when scores saturate
The format in which it is askedIts designer, as the defaultTaken as given
How the model is run: prompt, examples, reasoning, toolsWhoever tests the modelVaried all the time
The modelIts developerImproved all the time

What a benchmark tests, and how well this reflects a meaningful capability or human performance, is decided in the first row, and pooling does not touch it. Pooling changes the second row only: how much of what the benchmark holds is read out. As an outside reference we use MMLU-Pro, which changed the content of MMLU: six more options were written for every question. We changed only the format, and pooled the four original options of MMLU. The table gives both on the 5,956 questions the two benchmarks share, for 11 models.

What is changedMean accuracyGap between the best and the worst model
Nothing: MMLU, multiple choice0.7560.335
The content: six options written for every question (MMLU-Pro)0.6380.420
The format: the four original options, pooled0.6280.561
  • The pooled format keeps what the benchmark measures. It ranks the 11 models exactly as MMLU-Pro does, and as multiple choice does.
  • It reads out more of it. With nothing written, the gap between the models is wider than after the six hand-written options.

So a pooled score inherits its validity from the benchmark it is applied to, and the gain is in how much of that benchmark the format uses. We will state this distinction in the introduction.

Cross-modal conclusions (W3). We agree with the reviewer, and we will rewrite this conclusion. The pooled format is the same in every modality, but what it reads out depends on the benchmark it is applied to, so the drop need not have the same explanation everywhere. The revision will state the cross-modal conclusion in these terms.

Grouping strategies (Q3). The ranking of the models did not change in any grouping we tried. What changes with the grouping is how much the pooled format reads out of the benchmark: the more plausible the pooled options are for a question, the more it reads out.

  • Groups on one passage give the most plausible options. There, the options of the other questions cost 0.07 on QuALITY and 0.08 on RACE (step 4 above).
  • Groups on one subject give less plausible options. On MMLU, where a group shares a subject and no passage, the same step costs 0.02.
  • The ranking is the same in every case. With a second random seed for forming the groups, every score changes by less than 0.03 (Table 7). With three to six questions per group, the five local models keep the same order, apart from one pair that ties (Figure 2).

So the grouping decides how plausible the pool is, and questions that share a context give the most.

Thanks, the responses pretty much address my worries about IRT robustness and downstream sensitivity. Two things still bug me: the resampling stays within the same model pool instead of removing whole families and refitting the full pool, and the uncertainty analysis keeps the discovered hypotheses fixed instead of re-running discovery under perturbation. The Jaccard overlap is helpful, but it doesn’t fully rule out different wordings of the same factor, so I’d like that clarified in revision. i think the paper can be borderline accepted

We sincerely thank the reviewer for the careful reading and for the updated assessment. We are very glad that the responses addressed the reviewer’s concerns, and the discussion has helped us make the paper clearer. To address the remaining points precisely, could the reviewer kindly point us to the analysis or section they refer to for the hypothesis-discovery and Jaccard-overlap points? Thank you again for the constructive discussion and for the time given to our work.

Reviewer_Fe5WOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new form of LLM/MLLM evaluation that repurposes existing MCQA datasets. The authors propose to pool together a set of multiple-choice questions and their answers, and then task a model with matching each question with its respective answer. The method relies on the fact that there is still only one correct/ground-truth answer for each question after pooling (i.e., no other correct or distractor answers from other questions are valid for the question in question), which the authors validate at least to some extent (sec 3.4). The benefits are that the method reduces the random chance baseline substantially by adding many more incorrect options. The authors also test ablations where they remove context (choices-only evaluation) and remove the correct answer (models' ability to abstain from answering). The authors demonstrate that models' performance drops substantially on this new format compared to standard MCQA, with some models having far larger drops than others.

Critical strengths
  1. It's a simple and creative idea that may be practically useful for those looking to repurpose existing datasets for harder evaluations, or who want to get a more discriminative signal between models on MCQA tasks.
  2. The evaluation is multi-modal and rather extensive (eight text, image, and video benchmarks and eighteen models).
  3. The paper is complete/self-contained. Very few if any missing details that I noticed in the manuscript.
Critical weaknesses
  1. Most notably, the scope of the paper's contribution is very narrow (changing the task evaluated for an existing set of MCQA questions, and showing models' performance drops, but not explaining in detail why this is). The contribution is, in my opinion, unfortunately not large enough for a full conference paper.
  2. The paper does not explain why some models perform worse on this format than others. This could be due to a number of reasons, e.g., better zero-shot generalization to a new task format models have not been trained to perform, or benchmark leakage of the original MCQA datasets. Given that fixed-answer-selection-style evaluation gives increasingly limited insights into the strengths and weaknesses of modern models, the paper would be much stronger if it demonstrated clear takeaways that this new evaluation format provides in terms of how it changes either our training or inference-time use of the models in question.
  3. The paper has a terse and unnatural writing style (i.e., either LLM-generated or heavily LLM-assisted) that is quite difficult to parse at times and is ungenerous to the reader. For example, the conclusion starts with the sentence "A saturated multiple-choice benchmark still holds a harder task in the labels it ships." I kind of understand what this means, but it took me a second.
Questions for authors
  1. Could you please clarify your second contribution ("Two read-outs from the same labels, exact abstention by removing answers and context dependence by removing the context, each compared with multiple choice on identical items")? How is this unique to the pooled answer format? I believe that both ablations can also be done on the standard MCQA format, as they have been done in prior work.
  2. Can you please provide more details on Section 3.4 "validation", e.g., the sample size you manually validated, were there some datasets where this property held more clearly than others, etc.? The validity of the proposed restructuring depends on the extent to which this was done comprehensively and questions do not share multiple valid answers. I don't feel enough details have been provided about this.
8 comments

We thank the reviewer for the constructive comments. We answer each point below.

Scope of the contribution (W1). The paper changes one thing, the format in which an existing benchmark is asked, and shows that this alone separates the models again on a benchmark that looks saturated. A score depends on four things:

Part of an evaluationWho sets itWhat usually happens to it
The benchmark: its questions, options and labelsIts designerReplaced by a new benchmark when scores saturate
The format in which it is askedIts designer, as the defaultTaken as given
How the model is run: prompt, examples, reasoning, toolsWhoever tests the modelVaried all the time
The modelIts developerImproved all the time

The paper changes the second row only. To show what the format is worth, we compared a change of format with a change of content. MMLU-Pro changed the content of MMLU: six more options were written for every question. We changed only the format, and pooled the four original options of MMLU. The table gives both on the 5,956 questions the two benchmarks share, for 11 models.

What is changedMean accuracyGap between the best and the worst model
Nothing: MMLU, multiple choice0.7560.335
The content: six options written for every question (MMLU-Pro)0.6380.420
The format: the four original options, pooled0.6280.561
  • A change of format does what new content does, with nothing written. Pooling lowers the mean as much as the six hand-written options, and it separates the models more.
  • It applies wherever questions share a context. This includes text, image and video benchmarks (Tables 1 and 2). The designers of new benchmarks can plan for it: with several questions per context, a benchmark can be released in both formats from the same labels.
  • It opens a direction. The format is a layer where more read-outs can be explored to unlock more value of existing benchmarks.

The contribution is therefore the finding that the format is a part of an evaluation that can be changed, and that a benchmark which looks saturated still holds more than multiple choice reads out of it.

Why accuracy drops in the pooled format (W1, W2). Accuracy drops because a multiple-choice item gives a model two things beyond the answer itself, and the pooled format removes both. To find the cause, we went from multiple choice to the pooled format one change at a time, on the same questions. The table gives what each change costs in mean accuracy over the 11 text models.

StepChangeWhat it isolatesQuALITYRACE
Multiple choice (starting accuracy)0.8070.910
1The question's four options are shuffled among themselvesOrder of the question's own options0.000.00
2Options from another passage are listed after the itemA longer list-0.02-0.02
3All options of that list are shuffledThe question's options no longer next to each other-0.06-0.04
4The other options are those of the other questions on the passagePlausible alternatives-0.07-0.08
5All questions on the passage are answered jointlyJoint assignment+0.01+0.01
Pooled (final accuracy)0.6680.782
  • Order and a longer list (steps 1 and 2) cost almost nothing. A model ignores options that do not belong to its question: with them on the page it still picks one of its own four options in 98% of cases, and its choice among the four is unchanged.
  • Separating the four options (step 3) is the first half of the drop. A model still knows which options are its own (97% of its picks), but it chooses among them less well: wrong picks among the four rise from 19% to 24% on QuALITY. This is consistent with the paper's criticism of multiple choice, that it lets a model answer by comparing its options side by side.
  • Plausible alternatives (step 4) are the second half. When the implausible options are replaced with plausible ones, the options of the other questions on the same passage, a model picks an option that is not its own in 17% of cases instead of 3%, and accuracy falls by 7 points on QuALITY and 8 points on RACE. This shows the central point of the paper: the options a benchmark wrote for its other questions act as hard wrong answers for this one. A benchmark with several questions per passage therefore already contains a harder test, and multiple choice leaves it unused.
  • Joint assignment (step 5) costs nothing. The drop is complete before the questions are answered together.

These steps show that a multiple-choice item helps a model in two ways that are separate from knowing the answer: it places the options side by side for comparison, and it hides every other plausible answer. Answer pooling removes both, using only the options the benchmark already has. We will add this table to Section 4.3.

Why some models drop more than others (W2). Weaker models lose on both halves of the drop, and stronger models lose almost only on the second. To see this, we split the drop of each model into the cost of the format (steps 2 and 3 above) and the cost of plausible alternatives (step 4). The table groups the 11 models by their multiple-choice accuracy on QuALITY.

Models, by multiple-choice accuracyCost of the formatCost of plausible alternatives
The three strongest0.040.05
The four in the middle0.090.07
The four weakest0.100.08
  • The cost of the format shrinks with the strength of the model. It is 0.112 for the weakest model and 0.015 for the strongest. So the reviewer's first candidate, generalization to a new task format, explains part of the difference between models, and it matters less the stronger the model is. It is not a lack of practice: with two solved examples of the pooled format in the prompt, the drop on RACE is unchanged (0.128 and 0.129).
  • The cost of plausible alternatives stays for every model. It is 0.05 even for the strongest model. This part is the difficulty the benchmark itself contains.
  • Leakage does not explain the drop. A leaked answer helps in both formats, and a leaked answer position is removed by step 1, which changes nothing.

In other words, the stronger the model, the less of its drop comes from the format and the more from the plausible alternatives of the benchmark. As models improve, the pooled score therefore becomes a cleaner measure of what the benchmark is meant to test. We will add this split per model to Section 4.3.

Takeaways (W2). The three points above change one part of an evaluation at a time: the format, then the model, and here how the model is run. The table collects what follows for the use of models and for benchmarks, with the measurement behind each line.

TakeawayWhat we measured
Running a model. Reasoning helps a model handle a long list of options. It does not help a model reject a plausible wrong answer.With reasoning, the cost of the format falls from 0.070 to 0.036, and the cost of plausible alternatives stays at 0.07 (Qwen3-8B, Qwen3-32B, Gemini 3.6 Flash and Gemini 3.1 Flash-Lite, with reasoning off and on, QuALITY).
Reading a score. The cost of the format can be absorbed, by a stronger model or by reasoning. The cost of plausible alternatives cannot. So the stronger the model, the more its pooled score reflects the benchmark.The format costs the weakest model 0.112 and the strongest 0.015.
Designing a benchmark. Write several questions per context. This opens the benchmark to formats that read more out of it.Pooling the four original options of MMLU separates the models more than six hand-written options per question.
Designing a benchmark. Group the questions so that the pooled options are plausible for each other.Plausible options cost 0.07 and 0.08 for groups on one passage (QuALITY, RACE), and 0.02 for groups on one subject (MMLU).
Designing a benchmark. Let the questions share a context and stay independent: no option of one question should also answer another, and no question should give away the answer to another.Where several questions on one short passage overlap, as in RACE, 16.8% of the questions have a second option that answers them, against 1.0% on GPQA.
Designing a benchmark. Make the groups larger when models approach the ceiling again.From three to six questions per group, the best local model gets a whole group right 35% and then 9% of the time (Figure 2).

So the pooled format says what reasoning does and does not buy at inference time, and it tells a benchmark designer how to get more out of the questions already written.

Writing style (W3). We apologize for the difficulty. All authors read and revised the manuscript several times before submission, and we will revise it again for readability.

The second contribution (Q1). The reviewer is right: both read-outs can be done in multiple choice, and they are not unique to the pooled format. They carry over to it, because multiple choice is the smallest case of the pooled format: a pool that holds one question with its own options. What the paper shows is how each read-out behaves when the format changes. The two behave differently, because one belongs to the benchmark and the other to the format:

  • Context dependence belongs to the benchmark, and the format does not change it. Removing the context costs a similar share of accuracy under both formats (Table 6: on RACE, 43% under multiple choice and 37% pooled).
  • Abstention depends on the format. On the same questions with the same answers removed, models still answer 17% to 100% of the unanswerable questions in the pooled format, and 10% to 56% in multiple choice with a printed option to decline (Table 5).

The second contribution was therefore meant to say that the pooled format keeps the read-outs of multiple choice, from the same labels, and shows how each of them behaves, and has to be read, when the format changes. We will rewrite it in these terms.

Validation of the pooled items (Q2). The check asks one thing of every pooled question: does any other option in the pool also answer it? We answer it in two ways. First, three screening models (Gemini 3.6 Flash, Gemini 3.1 Flash-Lite and Gemma 4 26B) each see the context, one question, the pool and the correct answer, and a question is flagged when two of the three name another option. Second, two of the authors verified a sample of 50 questions per benchmark by hand. They agreed with each other on 93% of the questions and found the screening mostly correct: on average 6% false positives and 4% false negatives, mostly from implicit meanings or ambiguous wording. The screening flags from 1% to 16% of the questions, depending on the benchmark. We found:

  • The property holds most clearly where every question stands alone and was written and validated by experts (GPQA).
  • Questions on one short passage overlap the most (RACE). Many of its questions ask for the main idea or for what can be learned from the passage, and its options are full sentences about the same 300 words. On RACE, 5.3% of the questions are flagged because the correct answer of another question also fits, against 2.4% on QuALITY and less than 0.5% on the rest.
  • Where options were generated by a model, the flags come from wrong options. On MMLU-Pro and MMMU-Pro nearly every flag names a wrong option and not the answer of another question (142 of 145, and 41 of 44).
Reviewer_S6KDOpenReview ↗
Rating
3(+1)
Confidence
4(+2)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes answer pooling, a way to make existing multiple choice benchmarks harder without new labels. For N questions that share context, all options are merged into one pool, and the model assigns each question an answer, using each option at most once. The authors argue that this lowers the guessing floor and withdraws the credit that multiple choice gives for eliminating a questions own wrong options. They call the accuracy lost to pooling the elimination credit. Deleting correct answers from the pool gives exact-ground-truth abstention scoring, and removing the context measures context dependence. They convert eight text, image and video benchmarks and evaluate 18 models, reporting that pooling is harder for every model, that the credit is largest for weaker models, and that most open-weight models answer 87 to 100% of unanswerable questions.

Critical strengths

The proposed method is simple and does not require new questions or labels. It is tested on text, image, and video benchmarks and allows the authors to evaluate accuracy, abstention, and context dependence using the same existing labels. This is a useful contribution because many benchmarks already contain multiple questions sharing the same context, making the method relatively easy to apply.

Critical weaknesses

1- The paper treats the drop from multiple-choice to pooled accuracy as elimination credit, but this interpretation is not fully proveed. Pooling also adds many more candidate answers, changes the option order, and requires joint assignment across questions. The unrelated-pool control is useful but does not fully isolate elimination. The paper shows that pooling makes the task harder, but not that the drop measures what the authors claim, and much of the analysis relies on this interpretation.

2- The motivation around benchmark saturation is not fully supported by the experiments where several evaluated benchmarks are far from saturated. For example, accuracy on MedXpertQA-MM ranges from 22.7% to 42.9% (Table 2). On benchmarks with high accuracy, the strongest models sometimes show only small drops under pooling, such as around 0.03 on QuALITY and 0.01 on MMLU-Pro. While pooling makes benchmarks harder, it remains unclear how much it helps where saturation is actually a problem. This weakens the connection between the paper’s motivation and its results.

Questions for authors

1- How can we distinguish the effect of eliminating incorrect options from the additional difficulty introduced by the larger answer pool and joint assignment? For example, would multiple choice with shuffled option order or a single question item expanded with neighbouring questions options but without joint assignment, show the same drop?

2- On benchmarks such as QuALITY and MMLU-Pro, some of the strongest models show relatively small accuracy drops under pooling. How should these results be interpreted in relation to the papers goal of addressing benchmark saturation?

4 comments

We thank the reviewer for the constructive comments. We answer each point below.

Isolating the drop (W1, Q1). We ran the two controls the reviewer proposes (steps 1 and 4 below) and three more. The order of the options, the size of the pool and joint assignment do not produce the drop. To find the cause, we went from multiple choice to the pooled format one change at a time, on the same questions. The table gives what each change costs in mean accuracy over the 11 text models.

StepChangeWhat it isolatesQuALITYRACE
Multiple choice (starting accuracy)0.8070.910
1The question's four options are shuffled among themselvesOrder of the question's own options0.000.00
2Options from another passage are listed after the itemA longer list-0.02-0.02
3All options of that list are shuffledThe question's options no longer next to each other-0.06-0.04
4The other options are those of the other questions on the passagePlausible alternatives-0.07-0.08
5All questions on the passage are answered jointlyJoint assignment+0.01+0.01
Pooled (final accuracy)0.6680.782
  • Order and a longer list (steps 1 and 2) cost almost nothing. A model ignores options that do not belong to its question: with them on the page it still picks one of its own four options in 98% of cases, and its choice among the four is unchanged.
  • Separating the four options (step 3) is the first half of the drop. A model still knows which options are its own (97% of its picks), but it chooses among them less well: wrong picks among the four rise from 19% to 24% on QuALITY. This is consistent with the paper's criticism of multiple choice, that it lets a model answer by comparing its options side by side.
  • Plausible alternatives (step 4) are the second half. When the implausible options are replaced with plausible ones, the options of the other questions on the same passage, a model picks an option that is not its own in 17% of cases instead of 3%, and accuracy falls by 7 points on QuALITY and 8 points on RACE. This shows the central point of the paper: the options a benchmark wrote for its other questions act as hard wrong answers for this one. A benchmark with several questions per passage therefore already contains a harder test, and multiple choice leaves it unused.
  • Joint assignment (step 5) costs nothing. The drop is complete before the questions are answered together. The reviewer's second control, a single question expanded with its neighbours' options and no joint assignment, is step 4, and it already shows the whole drop.

These steps show that a multiple-choice item helps a model in two ways that are separate from knowing the answer: it places the options side by side for comparison, and it hides every other plausible answer. Answer pooling removes both, using only the options the benchmark already has. We will add this table to Section 4.3 and define the elimination credit by these two parts.

Saturation and the strongest models (W2, Q2). A benchmark is saturated when it no longer tells models apart, and this is what pooling restores, most where the benchmark is most saturated. To show it, we measured saturation directly. The table orders four text benchmarks from the most to the least saturated, and gives two measures over the same 11 models.

BenchmarkQuestions that every model solves: multiple choiceQuestions that every model solves: pooledGap between the best and the worst model: multiple choiceGap between the best and the worst model: pooled
RACE73%45%0.0970.292
QuALITY44%21%0.2510.455
MMLU-Pro19%10%0.4740.636
GPQA5%3%0.5760.578
  • The more saturated the benchmark, the more pooling restores. On RACE, where three questions in four are solved by every model, the gap between the models triples. On GPQA, which is not saturated, the gap is unchanged.
  • The strongest models are still told apart. On RACE the two strongest models are 0.008 apart under multiple choice and 0.052 apart in the pooled format.
  • The ranking is kept. The order of the models is nearly the same in both formats (rank correlation 0.87 to 0.97), so pooling spreads the models without reordering them.
  • Benchmarks that are not saturated. The reviewer is right that MedXpertQA-MM is far from saturated. Such benchmarks are in the paper to show that the conversion applies to every modality, and we will state the saturation claim only for the benchmarks near the ceiling.

So pooling helps where saturation is the problem: it turns questions that every model solves into questions that tell models apart, and it does so with the options the benchmark already has. We will add this table to Section 4.2.

The small drop of the strongest models (Q2). The strongest models drop little because the format costs them almost nothing, and what they still lose is the difficulty the benchmark itself contains. To see this, we split the drop of each model into the cost of the format (steps 2 and 3 above) and the cost of plausible alternatives (step 4). The table groups the 11 models by their multiple-choice accuracy on QuALITY.

Models, by multiple-choice accuracyCost of the formatCost of plausible alternatives
The three strongest0.040.05
The four in the middle0.090.07
The four weakest0.100.08
  • The cost of the format shrinks with the strength of the model. It is 0.112 for the weakest model and 0.015 for the strongest.
  • The cost of plausible alternatives stays for every model. It is 0.05 even for the strongest model. This part is the difficulty the benchmark itself contains.

In other words, the stronger the model, the less of its drop comes from the format and the more from the plausible alternatives of the benchmark. A small drop for a strong model is therefore what the measure should give, and as models improve, the pooled score becomes a cleaner measure of what the benchmark is meant to test. We will add this split per model to Section 4.3.

Thanks for the detailed response. The additional experiments address most of my concerns, and I am happy to increase my score to 3

Dear Reviewer S6KD,

We sincerely thank you for the careful reading and for the updated assessment. We are very glad that the responses addressed the concerns. Thank you for the time given to our work.

Comments on the paper

1 comment

Dear Reviewers and Area Chair, we warmly thank you for your time and constructive feedback, which has made the paper clearer and stronger. We have uploaded a revised PDF in which every added or changed text is marked in blue. The changes are listed below by section, with the reviewers who asked for them.

  • Abstract and introduction. The paper now separates the benchmark from the format in which it is asked, and states that answer pooling changes only the format (CeEi, Fe5W). The contributions are reworded, including the second one on the read-outs (Fe5W).
  • Section 3.2. The elimination credit is defined by its two parts, the cost of the format and the cost of plausible alternatives (S6KD, CeEi).
  • Sections 3.4 and 4.6, Table 11, Appendix F. The ambiguity screen, the hand check, the flagged share of the benchmarks, and an iterative repair of the pool with its result on RACE (Fe5W).
  • Section 4.2, Tables 3 and 4. Saturation measured directly, and pooled MMLU compared with MMLU-Pro (S6KD, Fe5W, CeEi).
  • Section 4.3, Tables 5 to 7. The format changed one step at a time, the drop split by model strength, reasoning, and a separate matching algorithm (S6KD, CeEi, Fe5W). The per-model tables are in Appendix D.
  • Section 5. The cross-modal conclusion is reworded (CeEi), and the limitations are updated.

In the revised PDF, abstention is Table 8, context dependence Table 9, the protocol changes Table 10, and group size Figure 3.

Thank you again for your time and thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.