acceptodds
Under review as a conference paper at ICLR 2027

Off-Menu Preferences Diagnose Menu Sensitivity in Forced-Choice Evaluation

Abstract

Forced-choice benchmarks score a language model only on the answers they list (the menu), yet a model can score an unlisted answer above every listed one. We ask whether such off-menu preferences indicate sensitivity to the menu or an incorrect answer. A gold-free off-menu ratio compares the best-scoring candidate in a specified excluded set with the top listed answer; a controlled swap replaces the weakest distractor with that candidate while retaining the gold answer. Across four open-weight checkpoints, controlled cloze tasks, and four established benchmarks, the menu can matter more than the model: on 941 single-token CommonsenseQA items, the swap lowers Qwen2.5-1.5B's accuracy from 66.0% to 21.6%, a 44.4-point drop, more than twice the largest accuracy gap between checkpoints (20.3 points). Because the inserted candidate is never the gold answer, the drop equals the proportion of items on which it displaces a correct answer; displacement rate times overall accuracy tracks drops of 3 to 44 points across three checkpoints and four benchmarks (correlation 0.998). At equal coverage, the margin between the top two listed answers is the stronger selector of correct answers (81.1% vs. 68.4% on CommonsenseQA; likewise on ARC and OpenBookQA). The ratio instead flags risk under the altered menu: used in labelled calibration at 50% coverage, it lowers the proportion of Qwen2.5-1.5B's samples from that menu that miss the gold answer from .652 to .610. Evaluations should report the menu, the excluded-candidate search, and the calibration target alongside accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
forced-choice evaluation, language model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_z5EkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper study the forced-choice evaluation, where model is scored only on the listed answers. The authors point our that model can give higher score to some answer that is not in the menu. The authors propose two tools. First is off-menu ratio. where w star is the best candidate from an excluded set and a-hat is the top listed answer. It need no gold label. Second is swap, it replaces the weakest distractor with w-star, while keeping the gold. The main findings are three. Firstly, menu effect is bigger that model gap. second is within-menu margin is better than tau for selecting correct answers. Finally tau helps calibration on the altered menu.

Critical strengths
  • The question is interesting and the framing is clear. Separating within-menu confidence, menu sensitivity, and correctness is useful view for evaluation method.

  • The displacement identify explains effect size nicely. It shows why the effect is large on CSQA and small on ARC.

Critical weaknesses
  • The headline result is mostly by construction. If we insert a candidate that the model itself already scores higher than the menu leader, the leader will be replaced on τ>1 items, and since w* is never gold, accuracy must drop. So P(d) is almost same as flag rate, and "drop = displacement × accuracy" is an algebraic identity, not empirical law. The 0.998 correlation is not surprising. In the end it say "if you insert model-specific adversarial distractor, accuracy goes down". I am not convinced this is measuring menu sensitivity.

  • The comparison with model gap is not fair. Model gap is measured on same menu, but swap effect use a different adversarial menu for each model. Without a common altered menu, the claim "menu matters more than model" is hard to support. Authors also admit this in limitations.

  • w is mostly not a real answer.* It come from filtered next tokens, so many are shallow continuations like "kitchen", "home", "library". In authors' own audit only 37% are in gold vocabulary, and the 8B judge says w* is as good or better than gold only 13–33% of time. So what is measured looks more like cloze-style likelihood scoring is attracted by frequent words, not menu incompleteness.

  • Important related work is missing. This is very close to surface form competition (Holtzman et al., 2021) and PMI/DC-PMI calibrated scoring, but there is no citation or comparison.

  • The setting is far from current benchmark practice. Paper only study cloze-style scoring, where menu is not in prompt and each option is scored as continuation. In modern MMLU-style eval, options are in prompt and model answers with letter A/B/C/D, so "off-menu" means something quite different. This setting is not tested, so it is hard to generalize to "forced-choice evaluation" in general as the title says.

  • Practical contribution is weak. τ lose to margin for selection, the calibration gain is about .042 only on host, and on 2 of 4 models it is null or worse. Also, re-menu draw risk is risk on a menu that include model's own w*, so the target is somewhat circular. It is not clear why practitioner should care about this metric.

  • Scale and data are limited. Models are only 0.6B–4B (mostly Qwen, only one non-Qwen). In the 941 CSQA slice, 838 items are from train split. The selection experiment (half of 1197) and intervention experiment (941) use different slices, which make it hard to connect the results.

  • Multi-token τ may be biased. Comparing a model-generated continuation with human-written options by mean token log-likelihood can inflate τ, because generated text naturally has high per-token likelihood. The over-flagging in the ARC span experiment seems to show this problem.

  • Writing is very hard to follow. There are too many self-made terms. The count-poll vs argmax part is weakly related to main claim, and it is quite obvious that plurality from sampling a fixed softmax is worse than argmax.

Questions for authors

Firstly, please refer Critical Weakness Section.

  • Do the results hold in MCF format, where options are in the prompt and scored by letter (A/B/C/D)?

  • How do you see relation to surface form competition? If you use calibrated scoring like PMI/DC-PMI, how much do flag rate and swap drop decrease?

  • Can you compare the drop with model gap on a common altered menu for all models (for example, w* from a reference model, or union of w* from multiple models)?

  • How different are P(d) and P(τ≥1)? If displacement rate is basically the flag rate, what new information does the identity give beyond flag rate?

  • What exactly is the filtering rule for top-4096 candidates (stopwords, subwords, punctuation, etc.)? How much do results change with different filter?

  • How are multi-token excluded candidates generated (greedy? any length limit?)? How do you control the bias when comparing generated continuations with human-written options by mean log-likelihood?

  • Do flag rate and drop hold on bigger models (7B–70B)? Does menu sensitivity get smaller as model gets bigger?

  • What concrete recommendation do you give to benchmark builders or users? Is there an actionable protocol beyond "report the menu and excluded search"?

  • If the CSQA single-token slice uses only validation rows, do the selection and calibration results still hold?

Reviewer_bffYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies forced-choice evaluation, where a model is scored only on the listed answers (the "menu"). It defines an off-menu ratio that compares the best-scoring unlisted candidate with the top listed answer, and a swap that replaces the weakest distractor with that candidate while keeping the gold answer. On four small open-weight models and CommonsenseQA, ARC, OpenBookQA and RiddleSense, the swap lowers accuracy by up to 44 points on CommonsenseQA, more than the gap between models. The paper also finds that the within-menu margin selects correct answers better than the ratio, and that the ratio helps a little in calibrating risk on the altered menu.

Critical strengths
  1. The question is reasonable: forced-choice scores depend on what is on the menu, and keeping the gold answer while swapping one distractor is a clean, item-level way to measure that.

  2. The paper reports its negative results honestly, for example that the ratio loses to the within-menu margin for selecting correct answers, and it releases code and per-item results.

Critical weaknesses
  1. The headline result is true by construction.

The swap inserts the model's own highest-scoring unlisted candidate, which is never the gold, so accuracy has to drop whenever that candidate beats the listed leader. The displacement identity and its 0.998 correlation restate this construction rather than test anything, and the limitations section itself says the direction of the effect is built into the design. I mean if you curate an adversarial, model-specific distractor, you would expect the accuracy to drop significantly. I don't see the novelty of this.

  1. The setup doesn't match how multiple-choice evaluation is done today.

The options are not shown in the prompt, and each answer is scored as a free continuation (Sec. 2), so the model has no way to know which answers are allowed and preferring an unlisted word is expected. All four models are also small (0.6B–4B), so it's unclear whether any of this holds for the models people actually compare.

  1. Many of the "wrong" answers after the swap may be valid.

The paper's own judge rates the inserted candidate as at least as good as the gold on about one in five flagged items on CommonsenseQA and one in three on OpenBookQA. Counting those as errors inflates the drop, because the swap can make an item ambiguous rather than expose a model weakness. There is no human check of these candidates.

  1. The proposed ratio isn't useful for the tasks it is tested on.

For selecting correct answers, the standard within-menu margin beats the ratio on three of four benchmarks, by about 13 points on CommonsenseQA. The calibration gain is small and clear for only one checkpoint, and for two of the four it is slightly worse.

  1. The paper is hard to read. Need to work on this substantially.

It introduces many internal terms ("host", "hostage fraction", "threat", "under-inclusion", "re-menu draw risk") and spreads the evidence over a 55-page appendix that points to result files more than it explains them. The abstract alone carries more than ten numbers, and the main message gets lost. Menu is also a weird term, like why can't we just say options or multiple choice.

Questions for authors
  1. How do the results change when the options are shown in the prompt and the model picks a letter, as in most current evaluations?
  2. What happens if the inserted distractor is random or human-written instead of the model's own top candidate?
  3. How often do human annotators judge the inserted candidate to be a valid answer?
  4. Do the effects hold for larger models, e.g. 7B to 70B?
Reviewer_hBs5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents off-menu preferences in forced-choice LLM evaluation cases where a model assigns a higher score to some answer not among the benchmark's listed options than to any listed option, including the correct one. It separates three quantities that standard multiple-choice accuracy conflates which are within-menu confidence (the margin between the top two listed options), menu sensitivity, and answer correctness (agreement with gold). The authors employ wwo tools operationalize this, a gold-free off-menu ratio \tau, which compares the best-scoring candidate from a specified excluded set against the top listed answer with no gold label involved, and a gold-retaining "swap" intervention, which replaces only the weakest listed distractor with that excluded candidate while keeping the gold answer on the menu, so every resulting accuracy change is traceable to a specific displaced item.

Critical strengths

[+] The design of the excluded candidate condition (τ and the gold-retaining swap) is a novel contribution that the community can benefit by adopting it to, and extend to multiple evaluation frameworks - that follow the "menu" format.

[+] The fact that the swap can only ever turn a correct answer wrong, and does not lead to the reverse - where a wrong answer is tagged correct, by construction, is valuable guardrails.

Critical weaknesses

[-] While I understood the motivation upuntil fig-1, fig-1 actually confused me more than help. The question "Where would you keep a clean plate?" lists {cupboard (gold), sink, table, oven, floor} with kitchen as the off-menu winner but table is an entirely ordinary place to keep a clean plate (set at a place setting, left to dry, etc.), so the example confused me. Why is "kitchen" the golden answer here? This is an underspecified problem, and hence the cocnept of a "right" answer is incorrect.

[-] The authros attribute cross-benchmark heterogeneity in to menu and model sensitivity. Isn't a discussion on gold-label noise not more appropriate here? Especially looking at differences in ARC-Challenge (vs) OpenBookQA?

Questions for authors

The forced-pairwise judge rates the excluded candidate at least as good as gold on 32.7% of OpenBookQA's flagged items. What is to be concluded from this observation?

Similarly, I see in the supplementary material that the judge used is llama-3.1-8b. I wonder how much of these observations hold, if the authors update a) the models under consideration, and b) the judge to more recent state-of-the-art ones? Given it is just inference, I would encourage the authors to also run this experiment on the latest models!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.