- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Answer Pooling, a new evaluation protocol for multiple-choice benchmarks designed to address the potential saturation of current LLM evaluation. Instead of evaluating each multiple-choice question independently, Answer Pooling combines answer options from multiple related questions into a shared candidate pool and requires models to assign answers under a uniqueness constraint. The authors argue that conventional multiple-choice evaluation may overestimate model ability because models can exploit elimination strategies among answer choices. The paper evaluates Answer Pooling across multiple text, image, and video benchmarks with a broad range of language and multimodal models. The authors further investigate two related properties: whether models can recognize unanswerable questions when correct answers are removed, and how much contextual information contributes to performance by comparing original and context-removed settings. The experimental results show that Answer Pooling generally reduces model accuracy and changes model rankings, suggesting that conventional multiple-choice benchmarks may underestimate the difficulty of evaluation tasks.
Critical strengths
1.The paper introduces a simple transformation that can be applied to existing multiple-choice benchmarks without requiring expensive human annotation. This is consequential because benchmark saturation is an important challenge for evaluating increasingly capable LLMs. 2.The evaluation across multiple text and multimodal benchmarks demonstrates that the proposed phenomenon is not limited to a single dataset or modality.
Critical weaknesses
1.The interpretation of Answer Pooling results is not sufficiently isolated. The paper interprets the accuracy gap between standard multiple-choice and Answer Pooling as evidence of "elimination credit". However, the proposed setting introduces additional reasoning requirements beyond elimination, including global matching and constraint satisfaction. Without stronger controls, it remains unclear whether the method measures hidden reasoning ability or simply creates a harder task format. 2.A more difficult benchmark does not automatically imply a better benchmark. The paper needs stronger evidence that Answer Pooling scores correlate better with meaningful model capabilities. 3.Cross-modal conclusions are too strong. The paper suggests that Answer Pooling reveals a general limitation of multiple-choice evaluation. However, the meaning of performance drops differs substantially between text, image, and video tasks.The authors should avoid assuming that the same mechanism explains all modalities.
Questions for authors
1.Can the authors provide experiments separating semantic answer recognition from global assignment ability? For example, what happens if models first predict answers independently and a separate algorithm performs the matching? 2.How does Answer Pooling performance correlate with external reasoning benchmarks or human performance? Does the proposed metric provide improved construct validity? 3.How sensitive are results to different grouping strategies? Would different question grouping methods substantially change model rankings?
4 comments
Rebuttal
A more difficult benchmark and a better benchmark (W2, Q2). We agree, and we do not claim a better benchmark. We leave the benchmark as it is and change only the format in which it is asked. A score depends on four things:
| Part of an evaluation | Who sets it | What usually happens to it |
|---|---|---|
| The benchmark: its questions, options and labels | Its designer | Replaced by a new benchmark when scores saturate |
| The format in which it is asked | Its designer, as the default | Taken as given |
| How the model is run: prompt, examples, reasoning, tools | Whoever tests the model | Varied all the time |
| The model | Its developer | Improved all the time |
What a benchmark tests, and how well this reflects a meaningful capability or human performance, is decided in the first row, and pooling does not touch it. Pooling changes the second row only: how much of what the benchmark holds is read out. As an outside reference we use MMLU-Pro, which changed the content of MMLU: six more options were written for every question. We changed only the format, and pooled the four original options of MMLU. The table gives both on the 5,956 questions the two benchmarks share, for 11 models.
| What is changed | Mean accuracy | Gap between the best and the worst model |
|---|---|---|
| Nothing: MMLU, multiple choice | 0.756 | 0.335 |
| The content: six options written for every question (MMLU-Pro) | 0.638 | 0.420 |
| The format: the four original options, pooled | 0.628 | 0.561 |
- The pooled format keeps what the benchmark measures. It ranks the 11 models exactly as MMLU-Pro does, and as multiple choice does.
- It reads out more of it. With nothing written, the gap between the models is wider than after the six hand-written options.
So a pooled score inherits its validity from the benchmark it is applied to, and the gain is in how much of that benchmark the format uses. We will state this distinction in the introduction.
Cross-modal conclusions (W3). We agree with the reviewer, and we will rewrite this conclusion. The pooled format is the same in every modality, but what it reads out depends on the benchmark it is applied to, so the drop need not have the same explanation everywhere. The revision will state the cross-modal conclusion in these terms.
Grouping strategies (Q3). The ranking of the models did not change in any grouping we tried. What changes with the grouping is how much the pooled format reads out of the benchmark: the more plausible the pooled options are for a question, the more it reads out.
- Groups on one passage give the most plausible options. There, the options of the other questions cost 0.07 on QuALITY and 0.08 on RACE (step 4 above).
- Groups on one subject give less plausible options. On MMLU, where a group shares a subject and no passage, the same step costs 0.02.
- The ranking is the same in every case. With a second random seed for forming the groups, every score changes by less than 0.03 (Table 7). With three to six questions per group, the five local models keep the same order, apart from one pair that ties (Figure 2).
So the grouping decides how plausible the pool is, and questions that share a context give the most.
Thanks, the responses pretty much address my worries about IRT robustness and downstream sensitivity. Two things still bug me: the resampling stays within the same model pool instead of removing whole families and refitting the full pool, and the uncertainty analysis keeps the discovered hypotheses fixed instead of re-running discovery under perturbation. The Jaccard overlap is helpful, but it doesn’t fully rule out different wordings of the same factor, so I’d like that clarified in revision. i think the paper can be borderline accepted
Thank you
We sincerely thank the reviewer for the careful reading and for the updated assessment. We are very glad that the responses addressed the reviewer’s concerns, and the discussion has helped us make the paper clearer. To address the remaining points precisely, could the reviewer kindly point us to the analysis or section they refer to for the hypothesis-discovery and Jaccard-overlap points? Thank you again for the constructive discussion and for the time given to our work.
Rebuttal
We thank the reviewer for the constructive comments. We answer each point below.
Isolating the drop (W1). Global matching and constraint satisfaction do not produce the drop: it is complete before the questions are answered together (step 5 below). To find the cause, we went from multiple choice to the pooled format one change at a time, on the same questions. The table gives what each change costs in mean accuracy over the 11 text models.
These steps show that a multiple-choice item helps a model in two ways that are separate from knowing the answer: it places the options side by side for comparison, and it hides every other plausible answer. Answer pooling removes both, using only the options the benchmark already has. We will add this table to Section 4.3 and define the elimination credit by these two parts.
Recognition against global assignment (W1, Q1). We ran the experiment the reviewer proposes, and global assignment is not what makes the pooled format harder. Each question is first answered independently against the pool. A separate algorithm then computes the best one-to-one assignment from the scores the model gave to the options. The table gives the mean accuracy on QuALITY of the five models that return these scores.
The pooled format therefore measures whether a model recognises the answer among plausible options, and not its ability to solve an assignment. We will add this table to Section 4.3.