- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper study the forced-choice evaluation, where model is scored only on the listed answers. The authors point our that model can give higher score to some answer that is not in the menu. The authors propose two tools. First is off-menu ratio. where w star is the best candidate from an excluded set and a-hat is the top listed answer. It need no gold label. Second is swap, it replaces the weakest distractor with w-star, while keeping the gold. The main findings are three. Firstly, menu effect is bigger that model gap. second is within-menu margin is better than tau for selecting correct answers. Finally tau helps calibration on the altered menu.
Critical strengths
-
The question is interesting and the framing is clear. Separating within-menu confidence, menu sensitivity, and correctness is useful view for evaluation method.
-
The displacement identify explains effect size nicely. It shows why the effect is large on CSQA and small on ARC.
Critical weaknesses
-
The headline result is mostly by construction. If we insert a candidate that the model itself already scores higher than the menu leader, the leader will be replaced on τ>1 items, and since w* is never gold, accuracy must drop. So P(d) is almost same as flag rate, and "drop = displacement × accuracy" is an algebraic identity, not empirical law. The 0.998 correlation is not surprising. In the end it say "if you insert model-specific adversarial distractor, accuracy goes down". I am not convinced this is measuring menu sensitivity.
-
The comparison with model gap is not fair. Model gap is measured on same menu, but swap effect use a different adversarial menu for each model. Without a common altered menu, the claim "menu matters more than model" is hard to support. Authors also admit this in limitations.
-
w is mostly not a real answer.* It come from filtered next tokens, so many are shallow continuations like "kitchen", "home", "library". In authors' own audit only 37% are in gold vocabulary, and the 8B judge says w* is as good or better than gold only 13–33% of time. So what is measured looks more like cloze-style likelihood scoring is attracted by frequent words, not menu incompleteness.
-
Important related work is missing. This is very close to surface form competition (Holtzman et al., 2021) and PMI/DC-PMI calibrated scoring, but there is no citation or comparison.
-
The setting is far from current benchmark practice. Paper only study cloze-style scoring, where menu is not in prompt and each option is scored as continuation. In modern MMLU-style eval, options are in prompt and model answers with letter A/B/C/D, so "off-menu" means something quite different. This setting is not tested, so it is hard to generalize to "forced-choice evaluation" in general as the title says.
-
Practical contribution is weak. τ lose to margin for selection, the calibration gain is about .042 only on host, and on 2 of 4 models it is null or worse. Also, re-menu draw risk is risk on a menu that include model's own w*, so the target is somewhat circular. It is not clear why practitioner should care about this metric.
-
Scale and data are limited. Models are only 0.6B–4B (mostly Qwen, only one non-Qwen). In the 941 CSQA slice, 838 items are from train split. The selection experiment (half of 1197) and intervention experiment (941) use different slices, which make it hard to connect the results.
-
Multi-token τ may be biased. Comparing a model-generated continuation with human-written options by mean token log-likelihood can inflate τ, because generated text naturally has high per-token likelihood. The over-flagging in the ARC span experiment seems to show this problem.
-
Writing is very hard to follow. There are too many self-made terms. The count-poll vs argmax part is weakly related to main claim, and it is quite obvious that plurality from sampling a fixed softmax is worse than argmax.
Questions for authors
Firstly, please refer Critical Weakness Section.
-
Do the results hold in MCF format, where options are in the prompt and scored by letter (A/B/C/D)?
-
How do you see relation to surface form competition? If you use calibrated scoring like PMI/DC-PMI, how much do flag rate and swap drop decrease?
-
Can you compare the drop with model gap on a common altered menu for all models (for example, w* from a reference model, or union of w* from multiple models)?
-
How different are P(d) and P(τ≥1)? If displacement rate is basically the flag rate, what new information does the identity give beyond flag rate?
-
What exactly is the filtering rule for top-4096 candidates (stopwords, subwords, punctuation, etc.)? How much do results change with different filter?
-
How are multi-token excluded candidates generated (greedy? any length limit?)? How do you control the bias when comparing generated continuations with human-written options by mean log-likelihood?
-
Do flag rate and drop hold on bigger models (7B–70B)? Does menu sensitivity get smaller as model gets bigger?
-
What concrete recommendation do you give to benchmark builders or users? Is there an actionable protocol beyond "report the menu and excluded search"?
-
If the CSQA single-token slice uses only validation rows, do the selection and calibration results still hold?