acceptodds
Under review as a conference paper at ICLR 2027

Scoring the Wrong Question: Readout Failures in Constrained-Option Evaluation

Abstract

Constrained-option scoring reads a model's probabilities for a fixed set of permitted answers, so it returns a score even when the model is about to write something else. We study a prompt that quotes a multiple-choice item, ending in the item's own answer instruction, and asks for a forecast of whether a reader model given only a short note will answer it correctly. The model can begin answering the quoted item: on three Qwen3 checkpoints an answer letter is the most probable token on nearly every passage, and the forecast ranks the reader's correctness indistinguishably from chance. Beyond prior work's argmax check and prefill repair, we contribute a label-free diagnostic of the declared options' support, their first-token mass in context; a deletion test tracing the Qwen3 answer-letter start to the quoted instruction; and rendering and tokenisation checks for dropped separators and coinciding first tokens. Prefilling an answer stem lifts the median option mass from below to above one half on all nine checkpoints scored with this prompt and, with options and renormalisation unchanged, raises the Qwen3-averaged AUC from 0.4929 to 0.6152. lm-polygraph's default P(True) estimator shows a related failure: it reads True where Qwen3 without thinking favours an answer letter (MMLU) or the prompt's (A)/(B) label (TriviaQA), and its Qwen3-averaged AUROC is indistinguishable from chance on MMLU and below chance on TriviaQA. Scoring and renormalising the (A)/(B) labels after an answer stem raises it to 0.632 and 0.868, renormalising True against False in place reaches 0.660 and 0.860, and on TriviaQA both exceed all fourteen default single-answer library estimators. The original forecast is renormalised too, yet stays indistinguishable from chance: option mass flags low support in both settings without labels, and only checking the intended target shows which score still ranks.

Primary area:
datasets and benchmarks
Keywords:
constrained-option scoring, elicitation, evaluation methodology, measurement validity, tokenisation, chat templates, negative results, language model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pyVYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the "constrained-option scoring" scenario in LLM evaluation, where a prompt presents the model with a multiple-choice question and the task is to forecast whether the model will answer it correctly. The authors find that the forecast ranks the model's correctness at chance level, analyze the answer distributions, and find that prefilling an answer stem increases the success rate.

Critical strengths

None

Critical weaknesses

The paper is overall incomprehensible, and I can't make much sense of what it is trying to do after reading it several times. The scope of this paper is very small (and I would even say trivial or irrelevant). It focuses on the niche (and clearly outdated) scenario where the model must choose from a fixed set of answers.

This so-called "constrained-option scoring" is essentially a classification problem. In real-world scenarios, we would almost certainly use an embedding or reranking model for them. If it's a reasoning-intensive classification problem (e.g., a problem in GPQA, or a medical diagnosis / financial decision in the real world), then it's the reasoning process that actually matters, not the answer.

The only scenario that I can think of where "constrained-option scoring" could be useful is language model evaluation on simple benchmarks (like the example in Figure 1). However, instruction following is also one aspect of language models' capabilities to be evaluated. If we explicitly asked the model to answer with one of A, B, C, D and the model generated something else, then it means the model is not strong enough and doesn't have much real-world use. Trying to extract one answer from the model's logits doesn't achieve anything in this case.

An additional note on the example in Figure 1 (and the complete one in Appendix B): this example doesn't make any sense. Based on my understanding, the prompt says explicitly that the answer to a problem is A, and asks the model to predict how many future readers will answer it correctly, without providing any information on who these future readers are. So this question is unanswerable.

Questions for authors
  • I shall ask this one single question: is there any real-world implication of "constrained-option scoring"?
Reviewer_6yZDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates prompt which include after a MCQ a different but related question whose solution is not one of the answer labels of the MCQ. The paper finds that for Qwen3, an answer letter for the MCQ is the most probable token on every prompt. It explores how the problem can be solved.

Critical strengths

None

Critical weaknesses
  • The paper seems to be written by an LLM. Further, it has an almost unreadable information-dense writing style.
  • The finding is little surprising and insignificant, effectively there are 2 questions after each other, each with instruction to be answered. The finding of the paper shows that the first one is answered first. The paper does not have any practical implications.
  • The experimental evaluation is very limited.
Questions for authors

None

1 comment
Reviewer_6yZDOpenReview ↗

Answer to Author's Rebuttal

I would like to thank the reviewers for their rebuttal and I would like to apologize for the late reply. While I very much appreciate that the authors took the effort of rewriting large parts of the paper, it is still extremely hard to read and follow. Further, I am not convinced by the practicality argument of the authors. I believe that many benchmarks can be improved by prompting the model differently or reading out answers in a different way which is inherently connected to how the model was posttrained. Still, I do not think that the contribution is significant enough to meet the standards of ICLR. I am keeping my score. Thank you again for spending time on rebottling and communicating to the reviewers.

Reviewer_JH7aOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a failure mode in constrained-option scoring: a model's probability mass over a declared set of legal answers can be misleading when the prompt contains another instruction that the model begins to execute at the scored token position. The central example quotes a multiple-choice question and then asks the model to forecast how many readers would answer it correctly. Although the intended output is a numeric forecast, the model often begins by answering the quoted question, so the scorer reads an answer letter rather than a forecast. The authors propose an option-support diagnostic based on the total first-token probability mass of the declared options, use deletion controls to localize the failure to the quoted instruction, and analyze tokenization and rendering issues. They also test the default P(True) estimator in lm-polygraph on MMLU and TriviaQA. Prefilling an answer stem improves the forecast AUC from 0.4929 to 0.6152 averaged over three Qwen3 checkpoints. For P(True), changing the scored labels and renormalization substantially improves AUROC, particularly on TriviaQA.

Critical strengths
  1. The paper identifies a concrete and consequential measurement failure. Constrained-option scoring can return a numerical score even when the model's next-token distribution is dominated by a different instruction in the same prompt. This is a useful warning for confidence estimation and LLM-as-a-judge pipelines.
  2. The paper goes beyond reporting an argmax mismatch. The declared-option first-token mass diagnostic quantifies whether intended options have meaningful support before renormalization. Deletion controls and tokenization checks make the explanation more persuasive than a purely anecdotal example.
  3. The authors test consequences in maintained evaluation tools. The analysis of lm-polygraph's P(True) estimator, plus checks of configurations in AlpacaEval and OpenCompass, connects the problem to practical evaluation pipelines rather than a single bespoke prompt.
  4. The repair is simple and experimentally testable. Prefilling an answer stem and scoring the intended label set are straightforward interventions. The reported AUROC changes show that prompt position and label design can materially affect evaluation outcomes.
Critical weaknesses
  1. The main forecasting setup is narrow and potentially idiosyncratic. The core experiment asks a model to forecast reader correctness after quoting a multiple-choice item with its own answer instruction. This is a well-designed diagnostic, but it is not sufficient by itself to establish how often constrained-option scoring fails in common evaluation settings. Broader task formats and independently constructed prompts would strengthen the generality claim.
  2. The option-mass diagnostic flags weak support but does not establish that the corrected score is valid. Low mass over declared options is a useful warning, yet the original forecast remains near chance even after option support is considered. A diagnostic for invalid readout should be distinguished from a method that reliably repairs calibration or ranking.
  3. The improvement after prefilling remains moderate in the original task. AUC increases from 0.4929 to 0.6152, which is meaningful but still far from strong discrimination. The paper should avoid implying that an answer stem fully resolves the underlying forecasting problem. More calibration metrics and uncertainty estimates would help assess the practical benefit.
  4. The analysis of existing evaluation software may overstate prevalence without a systematic audit. Counts of configurations that use constrained-option scoring or quoted lettered items establish that the pattern exists, but do not by themselves show how often it causes incorrect rankings or materially changes benchmark conclusions. The scope and selection criteria should be explicit.
  5. The mechanisms may differ across model families and prompt formats. Deletion and tokenization effects vary by checkpoint; some models require additional changes to remove the answer-letter start. This heterogeneity suggests that one universal repair may not work across architectures and templates.
  6. Prompt-following failure and confidence-estimation failure are not the same. The model may be executing the wrong instruction rather than expressing a miscalibrated belief about the intended target. These are related but distinct failure modes, and their consequences for benchmark scoring should be discussed separately.
Questions for authors
  1. How often does the failure occur across a broader, independently sampled set of constrained-option evaluation prompts beyond the quoted-item forecasting setup?
  2. Can you quantify the relationship between option mass and downstream ranking/calibration quality across models and tasks, including cases where option mass is low but ranking remains useful?
  3. After answer-stem or label-renormalization repairs, how do AUC, Brier score, and calibration compare with simple baselines such as a constant forecast?
  4. What audit protocol was used to select configurations from lm-polygraph, AlpacaEval, and OpenCompass, and how many actual evaluation outcomes were materially affected?
  5. Can you distinguish prompt-instruction competition from tokenization/whitespace artifacts with controlled prompts that vary one factor at a time?
  6. Do the proposed checks generalize to multi-token options or structured outputs beyond the numeric and True/False examples?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.