acceptodds
Under review as a conference paper at ICLR 2027

The Monty Gap: Selection Evidence and Its Limits

Abstract

A verifier's choice of which answer to reject can carry evidence beyond the rejection itself. We audit whether this selection evidence improves model decisions: no tested prompted critic establishes a positive subject-cluster-robust accuracy gain over re-prompting, and a separate confidence study finds no reliable decision-cost benefit over calibrated controls. To interpret these results, we define the Monty gap, the expected posterior information discarded by masking rejected answers. We prove that every conclusive verifier that always spares the model's commitment at a positive fixed budget is non-ignorable. This guarantees a posterior discrepancy, not a decision gain. Under uniform selection among wrong alternatives, switching has a simple threshold: the surviving rival's prior probability relative to the commitment's must exceed . We characterize when masking is exact and show how intermediate discrepancies can disappear over a fixed budget. Scripted oracles establish the mechanism on model output distributions; prompted critics violate its assumptions, and the only positive four-option pairing is concentrated in one subject block. The resulting contribution is an audit of verifier protocols and their limits: selection evidence can exist without being reliably usable by the decoder.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian inference, selective feedback, uncertainty quantification, coarsening at random, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2srWOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper poses a question whether information from critic/verifier can be used to guide answering of a question by a student model. They formalize the problem mathematically and derive a bound based on the ratio between the student's confidence on commitment and the best surviving alternative which specifies when the answer should be switched to the best surviving alternative. The derived approach is evaluated across a variety of policies/budgets/models/datasets and is shown to benefit the correctness of the final answer particularly under conflict aware critic elimination policy with temperature adjusted threshold.

Critical strengths
  • The paper looks at the problem both theoretically and empirically. Theoretically it establishes mathematical foundation and proves/derives a number of mathematical properties whose correctness I couldn't verify. Empirically it contains a large suite of experiments, settings, models and datasets.
  • Presentation is relatively good adjusted to the density of the material. I especially like Figure 1 visualizing the boundary when the answer should be switched which helps better grasp the mathematical takeaways.
Critical weaknesses
  • The density of the content of the paper is at the same time a pro and a con in my eyes. It is extremely dense, with a lot of references on supplementary material which is for the lack of better word quite long. I have a feeling that a lot of work went into this paper and that the authors have to compress it into the format of the conference which resulted in text that tries to present as much as possible at the cost of quality of presentation. This raises a concern of the suitability of the venue considering that the manuscript format is clearly not suitable for the quantity of the work.
  • Section 4.1 shows that the method does not really benefit the inference, moreover it arguably harms it on average. While I acknowledge that this is for case without temperature, it just means the solution is highly dependent to a new hyper parameter which I would assume would be highly susceptible to temporal shifts during deployment.

Some issues I came across:

  • Line 55 mentions a study that is neither cited nor references
  • In Eq (1) it is not defined what is p(S(r))p(S(r)) as pp is supposed to have Y\mathcal{Y} for domain from my understanding and S(r)∈P(Y)S(r) \in \mathcal{P}(\mathcal{Y})
  • I feel Introduction jumps into the solution too fast and while the answers reveal itself further into the paper I kept wondering as I was reading it: "Where does this rule come from? How are options eliminated? If we eliminate options and renormalize eliminating 98 options shouldn't affect the decision." and so on. Meanwhile I feel it lacks examples of applications of these methods.
Questions for authors
  • This method can be considered in a very loose way a form of ensembles. Have you at any point considered relationship with ensembles and incorporating other well-known ensemble methods as baselines to put your solution into perspective?
  • Line 310 you mentioned that each cell has 2000 items, however you never specify how 2000 items are sampled? Are they the entire dataset? Are items sampled consistently across pairs/methods/seeds in Table 2? Can you be more precise on this?
  • Do you have any insights how having critic and student from the same family affects the results in for example Table 2? It would also be useful to see accuracy of no-critic (just plain student without elimination) in Table 2, so it can be seen whether the proposed solution harms the performance.
  • Line 355 states "Temperature fitting and evaluation use separate halves". Does this refer to temperature tuning (as a hyper parameter) and how are halves obtained
  • I finished reading the paper and while I saw a number of experiments and text I didn't leave with any conclusive feeling. In my opinion the paper reads more like "this is what we have done" rather than "this is what we have found out". Considering "Conclusion" itself is missing and the paper does not make any conclusive remarks, recommendations or guidelines, could the authors summarize main takeaways/remarks in their response/revision?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2srW

Dear Reviewer 2srW,

Many thanks for the careful reading, and for telling us candidly that you finished the paper without a clear sense of its conclusion. That is a fair criticism: we did not communicate the paper's main message well.

For us, the main message is this. A verifier's choice of which answer to reject carries evidence beyond the rejection itself, and that evidence changes the best answer only past a simple threshold. When the verifier follows the rule that spares the model's answer -- as our scripted verifiers do -- using that evidence gains 1.7 to 19.2 points at four options, and when it selects uniformly or eagerly, it gains nothing, exactly as the theory predicts. With the LLM critics we tested, however, it does not reliably help, and we observe that they neither follow the rule they are given nor avoid rejecting the correct answer. And when we enforce the rule in code (the new Section 4.6), the evidence gives a real gain in the pair with the strongest critic, but even there it does not beat simply asking the critic for its answer. The revision now says all of this in a short Conclusion (Section 6), and we have made a light pass through the prose to make it less dense.

Regarding Section 4.1, you are right that it shows no benefit, and we intended that as one of the paper's main findings. A new post-hoc analysis, now in Supplement O, is our attempt to explain why. The temperature correction does not rescue the result: under real four-option critics it changes accuracy by −1.3 to +0.9 points (Section 4.4). On your question about line 355, the temperature is a single number per model and dataset, fitted on a random half of the items (seed 7) and evaluated on the other half, so it is not tuned on the items we report -- although this addresses leakage only, and we did not test whether it transfers across distributions.

Regarding Table 2, the 2,000 items are a fixed, seeded draw: ten-option MMLU-Pro questions from twelve categories chosen with the same seed, shuffled once with seed 7, with the first 2,000 used in every pair, so every method within a pair is scored on the same questions and the same rejections. The revised Table 2 adds the baseline you asked for. Accuracy after eight rounds:

Student × criticStudent aloneMaskingDecoderBest re-prompting
Llama-8B × Qwen-14B33.960.455.860.7
Mistral-7B × gemma-9B33.061.564.162.6
Qwen-7B × gemma-9B48.471.272.071.4
Mistral-7B × Llama-8B-Inst33.059.050.955.8
Qwen-7B × Llama-8B-Inst48.467.562.163.3

Every method is far above the student alone, mostly because the critics often reject the student's initial wrong answer, and the decoder falls below plain masking in three of the five pairs. The last column is from the corrected re-run (see our general comment).

On same-family pairs, Table 1's same-family pairing (Qwen-14B critic with Qwen-7B student) gives −0.6 [−1.4, +0.2], against −0.4 [−1.0, +0.2] for a cross-family critic with the same student. This neither establishes nor rules out a family effect, since the two critics differ in more than family. Supplement L also contains one exploratory, unregistered same-family pair for Table 2 (Llama-8B student, Llama-8B-Instruct critic), which gives −3.8 against masking, in line with the −8.0 and −5.5 of the two cross-family pairs that use the same critic.

On ensembles, the decoder reweights a single model's distribution and needs no further passes of the student, so it is not itself an ensemble, and we did not run a compute-matched comparison against ensemble methods. The closest experiment is in Supplement M, which builds the distribution from sixteen sampled reasoning chains and compares it with re-prompting over eight fresh chains (+0.6 points, [−1.5, +2.8]). It uses a scripted oracle, missed its parse-validity floor, and is not compute-matched, so we regard it as exploratory only.

Finally, on your specific points. The "confidence study" mentioned early in the Introduction is our own experiment in Section 4.5, and the text now says so. In Equation (1), p(S(r)) is the total probability of the surviving options -- the sum of p(y) over y in S(r) -- and it is now defined; if you meant a different symbol, please let us know. The Introduction now explains how options are eliminated and why renormalizing alone cannot change the answer while the model's answer survives, and gives an example application: a fact-checker with access to records who flags one wrong claim in a draft. And we accept that the paper leans heavily on its supplement; we have used the tenth page for the new experiment and the Conclusion, rather than for expanding what was already there.

Thank you again for the careful reading.

All the best,

the Authors

I thank the authors for their detailed response. While I look at the paper more positively after the changes, from my very limited perspective it does yet warrant being above the acceptance threshold, however it is very close to it.

Reviewer_1qjkOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

As I understood this paper, the authors ask whether a verifiers choice of which answer to reject supplies information beyond simply removing the answer. They define a "Monty gap" as the expected difference between a policy aware posterior and a simple masking approach.

Critical strengths

The distinction between information and decision improvement is genuinely useful and the core mathematics appears mostly coherent under the assumptions given. I actually performed an independent screen of 90 separate kernel configurations and ~3000 report posteriors without any clear discrepancy between tested identity and decision rules.

Critical weaknesses

The negative results don't try to isolate the source of the failure, the ten-option critics strike the commitment by round eight on 65–91% of initially wrong-commitment items. I find protocol violations, decoder misspecification and a lack of usable selection evidence remain difficult to separate.

Questions for authors
  1. Do you have data reporting all four fixed re-prompting-arm comparisons? How does the main conclusion vary among them?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 1qjk

Dear Reviewer 1qjk,

Many thanks for the review, and in particular for independently checking the identities and decision rules, which was a generous use of your time.

You asked for all four re-prompting arms. These cross whether the student is told that an expert removed options (told or blind) with whether the surviving options are re-lettered (relabel or keep); Table 2 reports only the best arm in each pair. The full set after eight rounds is below, as decoder minus re-prompting, in points, with 95% intervals from resampling the twelve subject categories. They come from the corrected re-run described in our general comment (Table O.1).

Student × criticblind, relabelblind, keeptold, relabeltold, keep
Llama-8B × Qwen-14B−4.85 [−7.23, −2.34]−2.80 [−4.71, −0.68]−3.70 [−5.25, −2.23]−4.40 [−7.32, −1.24]
Mistral-7B × gemma-9B+3.10 [−0.88, +6.36]+4.45 [+2.06, +6.59]+3.15 [−0.44, +6.12]+1.50 [−1.52, +4.15]
Qwen-7B × gemma-9B+0.55 [−1.81, +2.70]+3.90 [+1.53, +6.61]+1.15 [−2.00, +4.36]+1.35 [−0.47, +3.39]
Mistral-7B × Llama-8B-Inst−4.85 [−7.22, −2.37]+0.45 [−1.44, +2.17]−4.20 [−6.89, −1.68]−0.85 [−3.10, +0.87]
Qwen-7B × Llama-8B-Inst−1.30 [−3.14, +0.61]+3.30 [+1.94, +4.63]−0.85 [−2.94, +1.23]+0.55 [−0.92, +1.95]

The conclusion depends on the arm in one respect: against "blind, keep", the weakest arm in every pair, the decoder is reliably ahead in three pairs; against each of the other three arms it is reliably ahead in none. It is reliably behind every arm with the Llama-8B student and Qwen-14B critic, and behind both relabelling arms with the Mistral-7B student and Llama-8B-Instruct critic. With matched prompts, plain masking is never reliably worse than the best re-prompting arm, and is reliably better in two pairs, whereas the submitted numbers suggested that re-prompting beat masking in one.

You also asked us to separate the sources of failure at ten options. We have tried to do so in two ways.

The first is the new experiment in Section 4.6. When code enforces the selection rule, the decoder recovers a gain of 8.2 points with the strongest critic after eight rejections, where it is in effect choosing between the student's answer and the critic's, though not reliably across the three pairs, and the enforced protocol does better than the free one in all three. This is the cleanest evidence we have that the critics' failure to follow the rule accounts for much of the negative result, though not all of it -- at least for critics without the key, since we did not re-run the keyed critics of Table 2 under enforcement.

The second is a post-hoc analysis of the saved ten-option logs, now in Supplement O (Table O.2). We define a clean subset of items with eight valid rejections, none of them on the model's answer or on the correct answer; membership uses the answer key after the fact. We read it as showing three things. First, once the model's answer is rejected, the decoder and masking pick the same answer by construction, so the entire difference between them arises where that answer survives. Second, the critics rejected the model's answer on 65 to 91% of items where it was wrong, and on only 1 to 15% where it was right -- so survival is itself evidence that the answer is right, raising its odds by a factor of 3 to 10, which the ideal rule ignores and which masking already exploits. This is why masking gains 19 to 28 points over the student alone, and why the ideal rule loses 12 to 62 points on the clean subset, which consists mostly of items the model already had right. Third, on the clean subset the decoder reduces to a single threshold, so the best threshold chosen with the answers is an upper limit for that family of decoders. For the two critics that reject the model's answer most often, that limit is +0.0 to +0.2 points, while the primary decoder lost 14 to 31; with the gemma critic the limit is +6.6 and +3.3, and the decoder reaches +5.1 and +1.1, neither interval excluding zero. On all 2,000 items, by contrast, a scripted oracle following the rule gains 25 to 44 points over masking, so the rule does work on these students' distributions when the protocol holds.

We should also say what this analysis does not show. The threshold describes one restricted family of decoders, on a subset chosen after the fact, and it does not bound decoders that use the rank of the rejected options, or the selection evidence in general. Nor, where the fitted threshold loses, can it separate table estimation from the student's miscalibration. In our view, the critics' selective rejection of wrong answers is an important part of the explanation, but not the whole of it.

Many thanks again for a review that has made the paper better.

All the best,

the Authors

Reviewer_ut8yOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper acknowledges that a verifier’s model answer rejection may provide additional information beyond the rejection itself. Therefore, the paper also investigates whether using such additional information can be beneficial in artificial intelligence pipelines. This is an interesting problem.

The paper offers a variety of theoretical oriented results including conditions under which consideration of a verifier’s choices can be used to improve model predictions . The paper also offers experimental results confirming that with ideal verifiers one can successful exploit this additional information. However, and importantly, the paper also notes that real critics do not follow the assumptions underlying the mathematical framework — this raises some issues about the utility of the study (see further comments below)

Critical strengths

The paper raises an interesting question relating to whether a verifier’s choice / verdifict can carry additional information beyond the rejection; the paper then draws an analogy between this problem and the Monty Hall problem. The paper offers a variety of theoretical results including when masking is exact along with conditions under which consideration of a verifier’s choices can be used to improve model predictions. The paper is also complemented with various empirical results.

Critical weaknesses

I think the paper exhibits a substantial weakness: the paper assumes that the critic is infallible in the sense that it can choose to reveal incorrect answers, however, in practice a critic (e.g. an LLM judge) is certainly fallible implying that it could incorrectly reject the correct answer. Therefore, i am struggling to understand the practical implications and ramifications of this study given the strong assumption.

I think it would be far much more interesting to consider a setting where the critic typically reveals the incorrect answers with perhaps high probability p but ocasionally also rejects the correct answer with low probability 1—p. It would then be useful to provide some theoretical insights (e.g. whether such a critic is beneficial), and importantly how to deploy such a critic/verifier in practice (including quantifying possible advantages in relation to simple masking).

Questions for authors

Please see critical weaknesses above. I would be particularly interested in the authors’ view on how to use these ideas / results in practical settings where a verifier can incorrectly reject a true answer; note that it is also possible that the verifier rejection may be correlated with a model generation. I would also be interested in understanding whether the method outperforms standard masking in situations where the verifier can make errors.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ut8y

Dear Reviewer ut8y,

Many thanks for your careful attention to this submission. You observe, correctly, that the paper itself reports real critics violating the assumptions of its theory, and you ask what the theory is then good for, and why the case of a verifier that makes errors is not treated. These are fair questions, and the submission did not answer them adequately. We would like to do so here.

To your direct question -- whether the approach beats standard masking when the verifier can make errors -- the answer is no, not reliably, and we think that is the most useful thing the paper reports. None of the LLM critics in the submission gives a gain over masking or re-prompting that survives resampling over subjects. Their error rates vary with the setting: with the answer key, they reject the correct answer on 2.3% of four-option verdicts and on 2 to 13% of ten-option items over eight rounds; without it, on 12.5% of four-option verdicts and on around 40% of ten-option items.

The new experiment in Section 4.6 bears directly on your question. There the selection rule is enforced in code, but the critic is fallible -- it is wrong on about half the items. Modelling the critic's errors helped in all three pairs we tested, and with the strongest critic the decoder gained 8.2 points over the student after eight rejections. Across the three pairs, however, the gain over masking was not reliable, and where the critic was the stronger model, the decoder stayed about 11.6 points below simply using the critic's own answer.

On why the theory starts from a critic that never errs: the paper does not assume that critics in general are infallible, but treats this as the cleanest case of the principle, and not a hypothetical one. When the task is narrowed to verifying known facts -- past meteorological observations, historical market prices, names and dates in a registry, a chain of custody -- checkers that are, for practical purposes, never wrong when they reject a claim are common. The right picture is therefore not an LLM judge, but a checker holding a record that the model does not have, who may decline to answer but does not reject a true claim. Our scripted oracles are this case, and it is where the gains appear (Section 4.2). That said, a reliable record is not by itself enough. The additional evidence exists only when the checker, and not the model, chooses what to reject by a known rule, and where the checker is free simply to state the answer, the paper gives no reason to prefer elimination -- a limit that Section 4.6 now supports directly.

As for a theory of the fallible case, we agree that it belongs in the paper, and Section 3.2 of the revision adds a first, limited version. Suppose the critic follows the rule that spares the model's answer with probability 1−η, and otherwise rejects an option at random. For a single rejection among N options that does not land on the model's answer, each surviving rival's odds against that answer are multiplied by

B_η = [ (1−η)/(N−2) + η/N ] / [ (1−η)/(N−1) + η/N ],

which falls smoothly from (N−1)/(N−2) for a perfect critic to 1 for pure noise. The switching rule keeps its form, with the threshold raised to 1/B_η, and if the model's own answer is rejected, the rejection can only be noise, and the posterior is unchanged. As Appendix B of the submission already noted, η here is a mixture weight rather than an error rate -- a random rejection lands on the correct answer one time in N, so the false-rejection rate is η/N. With four options, our keyed critic's 2.3% rate gives η = 0.09 and moves the switching threshold from 0.667 to 0.683, and the keyless critic's 12.5% gives η = 0.50 and a threshold of 0.778. This quantifies how much the evidence weakens; it does not by itself promise a gain in accuracy, and it covers only a single rejection under uniform errors.

On correlation between the critic's rejections and the model's output, a new post-hoc analysis of the ten-option logs found the strongest form of it. Although the critics were told to spare the model's answer, they rejected it within eight rounds on 65 to 91% of items where it was wrong, and on only 1 to 15% where it was right -- so survival raises the odds that the answer is right by a factor of 3 to 10, which the ideal rule treats as carrying no information. We believe this is an important part of why the selection-aware update failed at ten options, and the details are in Supplement O and in our reply to Reviewer 1qjk.

The new Conclusion (Section 6) turns this into practical guidance: before treating a critic's choices as evidence, measure how often it spares the model's answer and how often it rejects the truth, and where a record exists, enforce the selection rule in code.

We take these criticisms with sincere appreciation, and would be grateful to hear whether this answers your concern.

All the best,

the Authors

Reviewer_ut8yOpenReview ↗

The Monty Gap: Selection Evidence and Its Limits

I appreciate the clarifications. I've decided to raise my score from 1 to 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision, a corrected baseline, and the experiment proposed in Section 5

Dear Reviewers and Area Chair,

Many thanks to all three reviewers for their careful attention to this submission. We have uploaded a revised PDF and supplementary archive, and would like to summarize what changed: a new experiment, a correction to one baseline, and smaller edits.

The new experiment (Section 4.6) is the one the submitted version proposed in Section 5. It asks whether the negative result reflects a limit of selection evidence itself, or only critics that do not follow the rule they are given. A critic without the answer key gives only its own answer, and code rejects options at random from those that are neither the student's answer nor the critic's. The rule therefore holds by construction, while the critic is still wrong on roughly half the items. We ran three pairs on the 2,000 ten-option questions of Table 2, with the analysis fixed before any results were seen.

The primary prediction, that the decoder would reliably beat masking (here, the student alone) in at least two of three pairs, was not met. With the strongest critic (Qwen-14B, 53.6% accurate against the student's 33.9%) the decoder gained 8.2 points [+3.8, +12.4] after eight rejections; with gemma-9B it gained 0.7 [−4.3, +6.4] for the Mistral-7B student and lost 3.5 [−6.5, +0.9] for the Qwen-7B student, the one pair where the critic is weaker. It beat the best re-prompting arm in none. At eight rejections of ten options the survivors are the student's answer and, when they differ, the critic's, so the decoder is then in effect choosing between the two; with the strongest critic the gain at two and four rejections was −0.2 and +2.0.

Two other registered predictions were met in all three pairs: modelling the critic's errors beat assuming an infallible critic (by 1.7, 2.9 and 11.1 points), and the enforced protocol beat the free critic, largely because with the free critic the decoder fell 1.6 to 6.7 points below masking. Two more held in only one pair: a gain that grows with the number of rejections, and a free critic that rejects a wrong student answer at least half the time. Where the critic is the stronger model, the decoder stayed about 11.6 points below simply adopting its answer. In our view this sharpens the paper's conclusion rather than changing it: enforcing the rule recovered a real gain only with the strongest critic, and even there not one that beats asking the checker for its answer.

The correction concerns the ten-option re-prompting baseline (Table 2, Supplement L). Following up a failed check in our own run notes, we found that the re-prompting arms were scored with a one-line subject header that the stored distributions used by masking and the decoders lacked; comparisons against masking were not affected. We re-scored all four arms in all six pairs (five registered, one exploratory) with the matching prompt, one item at a time, changing nothing else. The corrected prompt reproduces the stored top answer on 99.95% of items for two students and 98.85% for the third, where every remaining difference is a near-tie, against 69 to 85% before. The conclusions do not change: the decoder's difference from the best arm moved by −0.3 to +1.6 points per pair, it is reliably ahead of that arm in no pair and reliably behind it in two (three as submitted), and masking is now never reliably worse than the best arm, and reliably better in two pairs. Table 2 now uses the corrected numbers. The two four-option re-prompting comparisons (Section 4.1, Supplement C) carry the opposite mismatch and have not been re-run; the revision says so.

The smaller changes are a short Conclusion (Section 6); an Introduction that explains why masking alone cannot change the model's answer and gives the record-checker picture of the theory's assumption, with its limit; a first treatment of fallible verdicts (Section 3.2); a definition of p(S(r)) in Equation (1); absolute accuracies in Table 2; and plainer prose in Sections 2 and 3. Supplement O holds all four re-prompting arms, a post-hoc decomposition of the ten-option losses, the false-rejection rates, and every registered reading and deviation for the new experiment. Code, logs and operator notes are in the archive.

All the best,

the Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.