acceptodds
Under review as a conference paper at ICLR 2027

Confidence Is Not Enough: Decision-Value Finding Selection Beats Confidence-Gated Test-Time Compute

Abstract

Test-time compute methods decide how much extra computation to spend by reading the model's own confidence. We show that signal is invalid exactly when the model lacks the information it needs to answer: on interactive MedQA-USMLE the confidence–correctness correlation is near zero under minimal information and becomes predictive only as the record is read. 16 self-consistency at low information buys nothing (, ) while revealing two findings gains (), even beating a gold-aware compute allocator (, ). We make finding selection first-class and control it by the classical decision value of information: the finding most likely to flip the model's greedy answer. The criterion is training-free, gold-free, and survives miscalibration because a flip is a behavioral event. The prevailing selection criterion, greedy entropy reduction (the literature's “EIG”), is empirically indistinguishable from confidence gain and inherits the same pathology. Decision-value and entropy-reduction coincide under calibration and diverge under insufficiency, at two layers—selection and stopping—with which layer pays off set by how the belief is miscalibrated: selection wins under rank miscalibration (–), stopping under level miscalibration, where the continuous signal saturates (– of states ) but the binary flip transfers (– across Qwen2.5 scales, / on reasoning models, at matched reading budget). One decision-value currency arbitrating read/compute/stop Pareto-dominates the deployed controller (, ) and clears the entire pure-reading frontier, beating its best point by at less budget. Scope: four-way multiple-choice reading over an available record, against a self-consistency compute baseline; we do not evaluate acquisition of genuinely unobserved outcomes, nor compute controllers built on verifiers, process rewards, or search.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, test-time compute, value of information, confidence calibration, active information acquisition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nwgnOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper investigates test-time resource allocation under information insufficiency, arguing that confidence-based compute allocation can be unreliable when models lack necessary information. The authors propose VOI-Δ\Delta, a training-free finding-selection heuristic that prioritizes information capable of changing the model's predicted answer. They further explore decision-value-based stopping and a unified controller for reading, computing, and stopping. Experiments on MedQA-USMLE, RACE, and DDxPlus evaluate the proposed methods against confidence-based and entropy-reduction baselines, reporting improvements in accuracy and resource efficiency.

Critical strengths

None.

Critical weaknesses
  1. Serious presentation and organization issues. This paper is difficult to follow even after careful reading. The research motivation, problem formulation, theoretical analysis, and proposed algorithm are not presented in a clear and coherent manner. The paper frequently introduces notation and technical concepts without sufficient explanation, while several theoretical claims lack clearly stated assumptions or adequate justification. These issues go beyond minor writing problems and substantially hinder the reader's ability to understand and evaluate the paper's contributions.

  2. Limited novelty and significant concerns about conceptual and theoretical soundness. The paper does not establish a convincing logical connection between its motivation, theoretical claims, and proposed method. In particular, confidence is used for different purposes--compute allocation, finding selection, and stopping--without adequately distinguishing their respective decision objectives. The observed limitations of confidence in predicting correctness do not directly establish its inadequacy for these different decisions. Moreover, several theoretical results are elementary or insufficiently justified. For example, the necessary condition for answer correction in Observation 2 does not establish the effectiveness of flip-based selection; the application of Proposition 3 to multiclass temperature scaling is questionable; and the "iff" condition in the corollary lacks a rigorous definition and proof. Consequently, the proposed flip-based heuristic is not sufficiently connected to actual decision-value improvement, weakening both the claimed novelty and the theoretical foundation of the work.

Questions for authors
  1. What exactly is confidence insufficient for? Confidence can, in principle, guide both when additional information is needed and when more computation should be allocated. However, VOI-Δ\Delta mainly addresses which finding to select. How does this method resolve the limitation of confidence identified in the Introduction?

  2. Why does the proposed method deviate from the original research motivation? The Introduction motivates the need to distinguish between acquiring more information and allocating additional computation. However, the main algorithm focuses on selecting which information to reveal, rather than when additional information or computation is needed. How is this shift in the research problem justified?

  3. Does the method contradict the motivation of efficient test-time resource allocation? The fundamental motivation of this research area is to improve inference through efficient use of computational resources. However, the proposed finding-selection method itself requires substantial additional model inference. How can such a computationally expensive procedure be justified within this resource-efficiency setting, and what is its practical value?

  4. Why should answer flips indicate beneficial information? Observation 2 establishes only a necessary condition for correcting errors. How does VOI-Δ\Delta account for harmful flips from initially correct answers, or for reinforcing incorrect beliefs through its confidence-based fallback?

  5. Can the theoretical claims be rigorously justified? Claim 1's binary-entropy ordering does not generally hold when the maximum class probability is below 0.5. Proposition 3(ii) applies only to scalar monotone transformations, while the Corollary states an insufficiently justified "if and only if" condition. Can the authors clarify the assumptions and provide rigorous proofs?

  6. How are interactions among findings handled? Multiple findings may jointly change an answer even when none causes an individual flip. How can one-step selection and flip-based stopping handle such cases? Does the limited two-step experiment in Appendix D adequately address this issue?

  7. What is the practical applicability of the proposed selection procedure? The method evaluates the actual contents of candidate findings before selecting them. How can it operate when information is genuinely unavailable? Conversely, when all information is already available, why not process the complete record directly?

2 comments
AuthorsOpenReview ↗

Response to Reviewer nwgn(1/2)

Part 1 — W1 and W2

Thank you for the detailed critique. Your technical points are well taken, and we have fixed them in the revision. For context, we first list the results that these fixes do not affect, each re-checked.

  • Reading two findings beats 16× self-consistency at minimal information (+0.179, p=0.0004).
  • At matched reading budget, flip-gated stopping beats confidence-gated stopping on instruction-tuned models (+0.05 to +0.13) and on reasoning models (+0.24/+0.29).
  • On one temperature-scaled DDxPlus trajectory, decision-value stopping beats calibrated confidence stopping at all three budgets (+0.0154/+0.0103/+0.0035).
  • In the new experiments on 1255 cases and two model families, confidence-gated compute at minimal information stays below 0.43 on the test half over its whole threshold × sample sweep (m ≤ 16).

W1: presentation. We agree, and we have made targeted structural edits within the 9-page limit. §3 now opens with a problem setup that defines the case, the revealed set, the belief, the read, compute and stop actions with their costs, and the objective. §4 opens with the three decisions (compute, select, stop), each with its objective, its confidence rule and its decision-value rule (Table 9, Appendix G). Every formal statement now states its assumptions, and Appendix G gives the proofs and a notation table. The Scope paragraph of §1 ends with a roadmap.

W2: soundness.

  1. Confidence serves three different decisions. We agree that a low correlation with correctness does not by itself show that confidence fails for each decision, so we have measured each decision against its own objective (Table 9). For compute, confidence does not predict the gain from 8 samples at zero findings (Spearman -0.05/-0.03, p≥0.07, on 1255 cases) and starts to do so only after two findings (-0.11/-0.25). For selection, post-read confidence barely ranks the candidates within a state by post-read correctness (Spearman 0.05-0.21). For stopping, confidence loses to the flip at matched budget (Table 1).

  2. Observation 2 is only a necessary condition. It is, and it also has a second side, which the revision now states: at a correct state every flip is an error, so flip-seeking selection is a bet that the current answer is wrong. The new transition table (Table 13) measures both sides. VOI-Δ corrects 33-51% of wrong states (ER 19-34%) but overturns 16-24% of correct ones (ER 4-8%), and its net gain per step is at most ER's in nine of ten settings. This is why, measured in tokens, VOI-Δ reaches a correct answer first yet ties ER over the whole budget (§5.1).

  3. Prop 3 and multiclass temperature scaling. A counterexample confirms this concern. The beliefs (0.51, 0.49, 0, 0) and (0.50, 0.25, 0.25, 0) have confidence 0.51 > 0.50, but at T=0.75 they become 0.513 < 0.558. Temperature scaling is therefore not a monotone function of confidence. It falls under 3(iii) instead of 3(ii), and we have deleted the sentence "temperature scaling ... is inert".

    The same example shows that 3(i) and Prop 4 were stated too broadly. Under any strictly increasing transformation shared by all classes, what stays invariant is the flip indicator, that is, whether the argmax changes, and with it every flip-based stop along a given reading order. Confidence-ranked selection (VOI-conf, ER and VOI-Δ's tie-break) can change, and we have narrowed both statements accordingly.

    The "-0.03 to -0.04" frontier shift compared two trajectories with different states, so we have removed it. We report instead that temperature scaling reverses 5.5% of state pairs at step 0, where the states are identical, together with the same-trajectory comparison above.

  4. The corollary's "iff". We have removed it and replaced it with three statements.

    • (a) Necessity. Beating confidence stopping requires a signal outside the monotone-transform family, which follows from 3(ii).
    • (b) A standard sufficient condition, stated for our setting and proved in Appendix G. For a fixed read order, a finite horizon and a constant per-step cost, if the expected one-step accuracy gain is non-increasing along every trajectory, then stopping when that gain falls below the cost is optimal. This is the monotone case of optimal stopping (Chow & Robbins, 1961; Chow, Robbins & Siegmund, 1971), and we do not assume that the condition holds on our data.
    • (c) ρ, which the "iff" used as a criterion, is now reported as an empirical diagnostic.
  5. Novelty. VOI-Δ shares ER's one-step lookahead and probe bill. What differs is the key, an ordinal flip in place of a cardinal entropy reduction (Prop 4), and that difference shows in behavior. ER selects the same finding as confidence at every observed state in the six Qwen2.5 runs of the submission (88% of states on Llama-3.1-8B), whereas the flip corrects more wrong states and reaches a correct answer 9-20 tokens sooner. It does so at the price of more harmful flips, which we now report.

AuthorsOpenReview ↗

Response to Reviewer nwgn(2/2)

Part 2 — Questions

Q1. What exactly is confidence insufficient for? Confidence falls short for each of the three decisions when it is measured against that decision's own objective (Table 9, and item 1 of Part 1). We replace it with one test, "would this action change the answer?", which we use to choose what to read, when to stop and whether to compute.

Q2. Does the method depart from the motivation? We see why it reads this way, and the revision makes the link explicit. The paper answers the question of when more information or more computation is needed at two levels. §3 shows that at minimal information compute buys nothing, while reading two findings gains +0.179 over 16× SC, and the stopping and read/compute rules (§5.2-5.4) then answer the question at each step of a case. Which finding to read is the step that must be decided before reading, and the revised Scope paragraph of §1 now ends with this roadmap.

Q3. Is the selection procedure computationally justified? It is not when the whole record is free to read, and the revision now says so. E2 counts the cost. The controller makes about 11 single-token candidate probes per case (12.8/13.1 calls in all), and nearly all of its 1.8-2.1k input tokens are spent on probe prompts. Matched on tokens, reading the full record and voting beats the controller (-0.064/-0.070, Holm p=0.0008), and matching on calls gives the same result. In a post-hoc check that counts only generated tokens, the controller (135-137 per case) spends about as much as reading the full record and drawing one sample (156-157), and the two tie in accuracy.

The procedure pays where reading is the costly step. This is the case in acquisition (DDxPlus) and with an expensive reader. When a 1.5B/3B model computes the probes, the 14B/32B reader makes one call per case and keeps 65-80% of its own gain over fixed order (E3b, Holm p≤0.031).

Q4. Harmful flips and the confidence fallback. VOI-Δ does flip correct answers, in 16-24% of correct states against 4-8% for ER (Table 13), while it corrects 33-51% of wrong states against 19-34% for ER. The fallback does not drive the results. When no candidate flips, falling back to fixed order instead changes token-AUC by -0.017 to +0.001, which is significant in none of the ten settings (Appendix H.2).

Q5. Rigor. We agree that H_b(m) orders candidates like confidence only for m ≥ 1/2. The revised Claim 1 states the condition under which the entropy and confidence orderings must agree, namely that both candidates have m ≥ 1/2 and H_b(m2) - H_b(m1) > (1 - m1) log(K-1). Outside this condition, Claim 1 remains empirical. Beliefs with top probability below 1/2 make up 0.4-1.0% of candidate beliefs on Qwen2.5 and 21-24% on Llama-3.1-8B, where agreement is 88%. Prop 3(ii) is now stated for transforms of the confidence score only, and 3(i), Prop 4 and the corollary are narrowed and rewritten as described in Part 1. The proofs are in Appendix G, and the Claim 1 condition is derived in Appendix A.

Q6. Interactions among findings. We have probed every pair of unread findings at the flip-stop state, where no single finding changes the answer (Table 16). A pair that changes the answer exists in 12-25% of cases, so one-step stopping does miss joint effects. However, when the answer at the stop is wrong and no pair changes it, reading the whole record corrects it in only 10 of 895 cases. Two-step lookahead (Appendix H.2, and among the negative controls in Appendix D) reaches the first correct answer no sooner, after 1.61 findings against 1.56 for one step on Qwen2.5-7B and after 1.60 against 1.54 on Llama-3.1-8B, while using 4-5× the probes.

Q7. Practical applicability. When content is unavailable before acquisition, an observation model is needed. DDxPlus has one, a Bayesian answer model, and computes decision value as an expectation over the possible answers. This is acquisition of unseen information, and decision-value stopping wins there at mid budget (+0.014). On LLMs the analogue is the expected flip, which we have not tested. When everything is available, the complete record should be processed, since this is better at matched compute (E2), and the paper now says so. Selection pays when reading itself is costly, as with long records, many retrieved documents, a human reviewer or an expensive model, and in those cases a cheap model can screen the record (E3b).

If any passage of the revised §3 and §4 is still hard to follow, or if any statement still lacks an assumption, please point us to it, and we will revise it within the discussion period. We hope that the revision and these responses merit a reconsideration of your evaluation.

Reviewer_1e4TOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the limitations of confidence-based test-time compute allocation under insufficient information. It proposes VOI-Δ, which selects findings based on whether they change the model's predicted answer, and extends this idea to stopping and read/compute allocation. Experiments on MedQA, RACE, and DDxPlus show improvements over confidence-based and entropy-reduction approaches.

Critical strengths

The paper highlights an important limitation of confidence-based compute allocation when the model lacks necessary information, and decision changes provide a simple alternative signal.

Critical weaknesses
  1. VOI-Δ evaluates each candidate finding by revealing its content during lookahead, this makes the task closer to reordering available information than acquiring genuinely unknown information, also the metho requires additional probing calls, limiting its practical advantage.

  2. The improvement over the strongest oracle-tuned reading baseline is not statistically significant +0.071, p=0.056

Questions for authors

How would VOI-Δ work when candidate findings are genuinely unavailable before acquisition, such as in retrieval or tool-use settings?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1e4T

Thank you for the focused review. The revision addresses both weaknesses.

W1: lookahead reveals content, so the task is closer to reordering than to acquisition. We agree. On the LLM testbeds, every criterion scores a candidate by reading its content, and all criteria use the same lookahead probes, so these experiments compare selection criteria over an available record and do not measure the acquisition of unseen information. In the paper, the acquisition setting is DDxPlus, where a Bayesian answer model supplies the possible answers to each question. The revision now states this in §1 (Scope) as well as in Limitations.

W1: probing cost. E2 now quantifies this cost on the full MedQA test set with Qwen2.5-7B and Llama-3.1-8B. It counts findings+samples, LLM calls and tokens, under an analysis plan fixed before the runs. The controller makes about 11 single-token candidate probes per case (12.8/13.1 calls in all), and nearly all of its 1.8-2.1k input tokens are probe prompts. When the whole record is free to read, this cost does not pay off. On the 628-case test half and matched on tokens, reading the full record and voting over 8 samples beats the controller by 0.064/0.070 (Holm p=0.0008), and matching on calls gives the same result. We therefore withdraw the claim that the controller beats simple read-then-compute pipelines. The break-even analysis in Fig. 2B assumed that probes see candidate content for free, and its caption now states this premise.

The probing cost can also be moved to a cheaper model, which E3b tests. Qwen2.5-1.5B or 3B computes the flip-based order, and Qwen2.5-14B or 32B reads only the two selected findings. At two findings, this beats the main model's fixed order by +0.096 to +0.110 (Holm p≤0.031) and keeps 65-80% of the main model's own VOI-Δ gain, with one main-model call per case instead of 10.6 (Appendix H.3, Table 17).

W2: the improvement over the strongest oracle-tuned reading baseline is not significant (+0.071, p=0.056). We agree, and we have withdrawn the number. It rested on a cost that under-counted the controller's samples, as explained in the general response and Appendix C.4. With corrected costs, the contrasts on the 628-case test half are as follows.

  • When LLM tokens or calls are the cost, the controller is behind, as described above.
  • When reading a finding is the cost (findings+samples), the controller beats every fixed-, ER- and confidence-order pipeline (+0.048, p=0.010, on Qwen2.5-7B and +0.092, p<2e-4, on Llama-3.1-8B). It ties the best pipeline of any order, VOI-Δ flip-stopping answered greedily (-0.006, p=0.72, and +0.032, p=0.060).
  • Relative to the same controller with its compute branch removed and its low-confidence fallback kept, the compute branch adds -0.006 (p=0.72) on Qwen2.5-7B and +0.037 (p=0.033) on Llama-3.1-8B.

Two effects do not depend on how the controller's cost is counted. The controller's flip-based stop rule beats a fixed-k reader at equal reads (+0.074/+0.058, p<2e-4, on all 1255 cases), and the flip beats confidence stopping at matched budget (Table 1). Information also matters more than compute, since confidence-gated SC at minimal information stays below 0.43 on the test half over its whole threshold × sample sweep (m ≤ 16).

Q: how would VOI-Δ work when candidates are genuinely unavailable, as in retrieval or tool use? The realized flip is then replaced by its expectation under an observation model, so that an action a is scored by E over o ~ q(o | S, a) of 1{argmax changes}.

  • DDxPlus is the setting in the paper that has such a model. A Bayesian answer model supplies each question's possible answers, and decision value is computed as an expectation over them, using the continuous gain in top-class probability in place of the flip. There, decision-value stopping beats confidence and EIG stopping in the mid-budget regime (+0.014, p<2e-4), and it still beats calibrated confidence stopping on the same temperature-scaled trajectory (+0.0154/+0.0103/+0.0035). Selection loses to EIG there (0.710 vs 0.870), as Observation 2 predicts for a calibrated belief.
  • In retrieval after the fetch, the passages exist, but reading all of them with the main model is costly. E3b tests this pattern on MedQA findings, with a small model screening the candidates and the main model reading the ones it selects.
  • For tool calls, or for choosing which query to issue before retrieval, the outcome is unknown before the call. Computing the expected flip then needs an observation model, for example one that samples outcomes, and we have not tested this on LLMs.

Does this cover the setting that you had in mind? If it falls outside these three cases, we would be glad to discuss how the expected flip would be computed there.

Reviewer_hkSZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of confidence-guided test-time computation when language models operate under information insufficiency. The authors show that model confidence becomes less predictive of correctness when relevant information is missing, and that additional self-consistency sampling may be less effective than acquiring relevant information.

To address this problem, the paper introduces VOI-Δ, a training-free finding-selection criterion that prioritizes information capable of changing the model's current greedy answer. The authors further extend this decision-based perspective to stopping decisions and propose a unified controller that allocates resources among reading, computing, and stopping. Experiments across MedQA, RACE, DDxPlus, and multiple language models demonstrate promising improvements over confidence- and entropy-based strategies.

Critical strengths

S1. Important and well-motivated problem.

The paper addresses an important distinction between insufficient information and insufficient computation. The confidence–correctness analysis and compute-versus-information experiments provide useful empirical evidence that additional inference-time sampling cannot always compensate for missing information (§3, Table 3, Table 8).

S2. Simple and interpretable method.

VOI-Δ is conceptually straightforward, requires no additional training, and does not rely on ground-truth labels at inference time. Using decision changes rather than confidence magnitude provides an interesting alternative perspective on information selection (§4).

S3. Extensive mechanism-oriented analysis.

The paper goes beyond reporting accuracy improvements by analyzing flip probabilities, correction rates conditional on flips, calibration effects, and stopping behavior. In particular, Table 7 provides useful evidence that the gains of VOI-Δ are primarily associated with a higher probability of changing an initially incorrect decision.

S4. Careful empirical reporting.

The experiments cover multiple model scales and families, and the authors report confidence intervals, paired statistical tests, cost-matched stopping comparisons, and several negative results. The explicit discussion of limitations, including unsuccessful stopping signals and the restricted compute baseline, is appreciated.

Critical weaknesses

W1. The unified controller experiments do not sufficiently isolate the contribution of the proposed joint decision strategy.

The unified controller achieves a substantial improvement over the deployed confidence-gated compute controller (0.714 vs. 0.459 accuracy; Table 2). However, the principal comparison involves controllers with different action capabilities: the proposed approach can selectively acquire additional information, while the confidence-gated compute baseline primarily allocates additional inference computation.

Although the paper also includes reading-based controls, these comparisons do not fully establish whether the improvements arise from the proposed decision-value arbitration or simply from enabling information acquisition before computation.

This distinction is important because Appendix C.3 already shows that reading additional findings followed by self-consistency can be effective.

A more convincing evaluation would compare against simple joint strategies, such as fixed-order or confidence-guided reading followed by self-consistency, under the same test cases, action space, and total computational budget. An explicit ablation of the compute branch would further clarify its contribution.

Moreover, the reported improvement over the best fixed-budget reading baseline is relatively modest (+0.041) and does not reach conventional statistical significance (p=0.056p=0.056; Appendix C.4). This result should therefore be interpreted more cautiously than the improvement over the deployed confidence-gated baseline.

W2. Unequal finding lengths may confound the information-selection results and efficiency claims.

Appendix E describes a sentence-based segmentation procedure in which passages are limited to six findings. When a passage contains more than six sentences, the first five sentences are retained individually, while all remaining sentences are merged into the sixth finding.

Consequently, different findings can contain substantially different amounts of information, yet the main selection experiments measure reading budgets in terms of the number of findings.

This introduces a potential confounding factor: a selection method that preferentially chooses longer findings may obtain more relevant information per reading action, even without necessarily identifying more informative findings at a comparable token budget.

This concern is particularly relevant to Fig. 2 and the reported reductions in findings required to reach the first correct answer.

I would appreciate an analysis of the length distribution of findings, the frequency with which each selection strategy chooses the merged final finding, and whether the main conclusions remain valid when reading cost is measured in tokens rather than finding counts. A simple length-aware baseline or a token-matched comparison would help rule out this alternative explanation.

Questions for authors

Q1: Can the authors compare the unified controller against a simple read-then-compute baseline that uses the same action space and total computational budget as Table 2? How much improvement remains after controlling for the availability of information acquisition?

Q2: How different are the lengths of the findings selected by VOI-Δ, entropy reduction, and fixed/random order? Do the advantages in Fig. 2 remain when reading budgets are measured by cumulative tokens rather than finding counts?

2 comments
AuthorsOpenReview ↗

Response to Reviewer hkSZ(1/2)

Part 1 — W1 and Q1: isolating the controller

Thank you for the careful review. We agree that the submitted comparison did not separate the controller's decision rule from the benefit of being allowed to read, and we have run the comparison you suggest (E2, in the revised §5.4, Table 2 and Appendix H.1).

Setup and accounting. E2 runs Qwen2.5-7B and Llama-3.1-8B on the full MedQA test set (1255 cases), with findings capped at six. Every case is probed along four reading orders at every prefix, with 16 self-consistency samples at every visited state, so that all policies are simulated on the same states. Every operating point, including the controller's m and each baseline pipeline, is chosen on a validation half, and all contrasts are paired bootstraps (5000 resamples) on the test half (628 cases). We count cost in three currencies: findings read plus samples drawn (findings+samples), LLM calls, and tokens (prompt plus generated). Every lookahead probe is charged in full, with no credit for prefix caching. The primary contrasts and their Holm family were fixed before the runs.

Same action space, same budget (Q1). Each pipeline reads findings, either a fixed number k in fixed, ER or confidence order, or until the answer stops flipping, or until confidence passes a threshold, and then answers greedily or by SC-m. Among the pipelines that spend no more tokens than the controller, we choose the best on validation.

The result is negative for the controller. The best pipeline reads the whole record in fixed order and votes over 8 samples, and it is ahead by 0.064 [0.029, 0.100] on Qwen2.5-7B and by 0.070 [0.035, 0.105] on Llama-3.1-8B (Holm p=0.0008). Matching on calls, a secondary contrast, gives the same result, and so do both sensitivity analyses, one voting on the explicit answer letter and, for Qwen2.5-7B, one dropping the 200 items that the submitted grid used. The gap comes from the controller's lookahead probes. Of its 12.8/13.1 calls per case, about 11 are single-token candidate probes, and nearly all of its 1.8-2.1k input tokens are probe prompts. We therefore withdraw the claim that decision-value arbitration beats simple read-then-compute pipelines.

How much improvement remains once information acquisition is controlled for. None remains at matched LLM compute, and the controller's advantage lies in findings read. Matched on findings+samples, the paper's original currency, the controller beats every fixed-, ER- and confidence-order pipeline, by +0.048 [0.013, 0.084] (p=0.010) on Qwen2.5-7B and +0.092 [0.056, 0.131] (p<2e-4) on Llama-3.1-8B. It ties the best pipeline of any order, VOI-Δ flip-stopping answered greedily with no compute and no low-confidence fallback (-0.006, p=0.72, and +0.032, p=0.060). Against the deployed confidence-gated controller it gains +0.164/+0.239 (p<2e-4), and the best point over confgate's whole sweep is 0.417/0.428 (Appendix H.1). As you suspected from Appendix C.3, this gain comes from being able to read.

Compute-branch ablation. We also removed the compute branch from the controller while keeping its low-confidence fallback. Relative to this ablation, the compute branch adds -0.006 [-0.038, 0.025] (p=0.72) on Qwen2.5-7B and +0.037 [0.005, 0.070] (p=0.033, secondary) on Llama-3.1-8B, at 0.8-0.9 extra samples per case. Once findings have been read, extra samples add little. The reading efficiency comes from the stopping rule, since this ablated controller beats a fixed-k reader at the same mean number of reads by +0.074 [0.057, 0.090] and +0.058 [0.043, 0.073] (p<2e-4). This contrast selects no operating point, so it uses all 1255 cases (§5.4).

The +0.041. Your caution about this number was warranted. It rested on an under-counted cost, as explained in the general response, and we have withdrawn it together with "45% less budget".

In the revision, we have rewritten §5.4 and Table 2 with these results, and the abstract and contribution 3 now say that the controller's savings are in reading. Appendix C.4 documents the cost correction, and Appendix H.1 (Tables 10-11) gives the primary, secondary and sensitivity contrasts and the post-hoc full-record baselines.

AuthorsOpenReview ↗

Response to Reviewer hkSZ(2/2)

Part 2 — W2 and Q2: finding length

We agree that the capped segmentation makes findings unequal. The merged sixth finding averages 92 tokens on Qwen2.5 in the 75 of 146 cases that have one (99 tokens over the 664 merged cases of the full test set) and 87 tokens on Llama-3.1-8B (60 of 117 cases), against about 21 tokens for the other findings.

We have rerun selection with every step and every candidate probe logged (E3, in the revised §5.1 and Appendix H.2). The reruns cover the paper's four models and cases, plus all 1255 cases for Qwen2.5-7B, each under the capped segmentation and under one finding per sentence with no merge.

Length distribution (Table 15). On the full test set under the cap, a finding has a mean of 29.5 tokens, a median of 21, a 10th percentile of 10 and a 90th percentile of 49. Without merging, these are 21.6, 18, 9 and 37.

How often each method picks the merged finding (Table 14).

VOI-ΔERrandomlongest-firstfixed
at step 118-29%15-31%15-17%94-96%0
within two steps47-56%25-39%33-35%97%0

On the full capped test set (Qwen2.5-7B), the findings that VOI-Δ selects at steps 1 and 2 average 31 and 34 tokens, against 30 and 23 for ER, 29 and 30 for random order, and 26 and 20 for fixed order (Table 15).

Do the advantages in Fig. 2 survive a token budget? Partly, and the revision now states which parts do.

  • Per finding, VOI-Δ is below ER at k=1 and above it at k=2 and k=3 in all ten settings, as in Fig. 2.
  • For tokens to the first correct answer, the efficiency claim you singled out, VOI-Δ needs 9-20 fewer tokens than ER in all ten settings (p<2e-4 each). A case that is never answered correctly counts its whole record.
  • For whole-curve token-AUC, our pre-registered primary contrast with Holm correction over the ten settings, VOI-Δ minus ER is significant in no setting. All five capped estimates are negative (-0.004 to -0.029), and the unmerged ones range from -0.004 to +0.015. VOI-Δ stays above fixed order in all ten settings, eight of them nominally significant, and VOI-hybrid has the highest token-AUC of all realizable criteria in all ten (+0.003 to +0.031 over ER, two nominally significant).
  • Among length-aware baselines, longest-first is below VOI-Δ in token-AUC in eight of ten settings (Table 12). Breaking VOI-Δ's ties toward shorter findings changes its token-AUC by -0.021 to +0.031, with one setting nominally significant.

VOI-Δ reaches a correct answer first and still ties ER over the whole curve because flip-seeking also overturns correct answers, in 16-24% of correct states against 4-8% for ER (Table 13). Over a full budget, the two effects balance.

In the revision, the §5.1 heading now reads "VOI-Δ leads per finding and reaches a correct answer first", and the abstract, contribution 2 and Related Work no longer claim that VOI-Δ beats ER in accuracy. The Fig. 2 caption states that budgets count findings, and Appendix H.2 adds Tables 12-16.

Taken together. Your two controls changed what the paper claims. It no longer claims that the controller beats read-then-compute pipelines at matched compute, or that VOI-Δ is more accurate than ER over a token budget. Three results hold with these controls in place.

  • The flip-based stop rule is more accurate than a fixed-k reader at the same mean number of reads (+0.074/+0.058).
  • Matched on findings+samples, the controller beats every fixed-, ER- and confidence-order pipeline (+0.048/+0.092).
  • VOI-Δ reaches a correct answer in fewer tokens than ER in all ten settings.

The paper's starting point is also unchanged. At minimal information, confidence-gated self-consistency tops out at 0.417/0.428 over its whole sweep, so extra compute does not make up for missing findings. If a comparison that you had in mind is still missing from Table 2 or Tables 10-16, we would be glad to add it during the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading and constructive comments. We have run the three experiments that the reviews asked for and uploaded a revision with targeted edits, and the main text still ends on page 9. Changes are marked in blue in the revised PDF. The analysis plan for the new experiments, covering the primary contrasts, Holm families, selection rules and cost accounting, was fixed before any result was available, and we report the outcomes as they came out.

A correction we found ourselves. The submitted code under-charged the unified controller's samples. The controller draws m self-consistency samples at every step where it consults its compute branch, but the code charged them only when the vote changed the answer. Once every drawn sample is charged, the submitted m=8 configuration costs 6.99 findings+samples on the submitted grid, against the 2.50 we reported, and it makes about 10 single-token candidate probes per case on top of this. We therefore withdraw "+0.041 over the best reading point at 45% less budget" and the marginal "+0.071 (p=0.056)" comparison. With corrected costs, the paper's knob rule selects m=1 (Appendix C.4).

New experiments.

  • E2 (hkSZ W1/Q1, 1e4T W2, nwgn Q3) compares the controller with read-then-compute pipelines on the full MedQA test set (1255 cases) for Qwen2.5-7B and Llama-3.1-8B, counting cost as findings+samples, as LLM calls and as tokens.
  • E3 (hkSZ W2/Q2, nwgn Q4/Q6) reruns selection on the paper's cases and four models, plus all 1255 cases for Qwen2.5-7B, under the capped segmentation and an unmerged one. It logs every step and every candidate probe and measures budgets in tokens.
  • E3b (1e4T, nwgn Q3/Q7) lets a small model (Qwen2.5-1.5B/3B) select the findings that the main model (14B/32B) then reads.

What the results say.

  1. Unchanged, now on 1255 cases and two model families. Under insufficiency, what is missing is information, and extra compute does not make up for it. Confidence-gated self-consistency at minimal information tops out at 0.417/0.428 over its whole threshold × sample sweep. Confidence also does not predict which cases gain from eight samples until findings are read (Spearman -0.05/-0.03 at zero findings, -0.11/-0.25 after two).
  2. Unchanged. The flip is the right stopping signal. The controller's flip-based stop rule beats a fixed-k reader at equal mean reads (+0.074/+0.058, p<2e-4), and the matched-budget wins over confidence stopping in Table 1 stand.
  3. Narrowed, for the controller. On the test half (628 cases per model) and matched on tokens, reading the whole record and voting over 8 samples beats the controller (-0.064/-0.070, Holm p=0.0008). Matching on calls gives the same result. We withdraw the claim that the controller beats simple read-then-compute pipelines, and the revision now states that its savings are in findings read.
  4. Narrowed, for selection against entropy reduction (ER). Measured in tokens, VOI-Δ reaches a correct answer 9-20 tokens sooner than ER in all ten settings. However, it ties ER over the whole token budget, because flip-seeking also overturns more correct answers (16-24% of correct states, against 4-8% for ER). We no longer claim that VOI-Δ is more accurate than ER over a token budget, although per finding read it still leads from two findings on.
  5. New. The probes can be delegated to a small model. A 1.5B/3B selector beats the main model's fixed order at two findings (Holm p≤0.031) and keeps 65-80% of the main model's own gain, with one main-model call per case.

Theory. We have narrowed Prop 3(i) and Prop 4 to what holds. The flip indicator is invariant under any strictly increasing transformation shared by all classes, temperature scaling included, whereas confidence comparisons across candidates are not. The corollary's "iff" is replaced by a necessity statement, a standard sufficient condition with proof, and ρ as an empirical diagnostic. Claim 1 now states where its geometric argument holds, namely when the top probability is at least 1/2. The proofs are in the new Appendix G.

What changed in the revision.

  • The abstract and contributions 2-3 are narrowed as above.
  • The Scope paragraph of §1 says what the LLM testbeds compare and ends with a roadmap.
  • §3 opens with a problem setup, and §4 with the three decisions (Table 9 in Appendix G).
  • Claim 1, Observation 2 (now with both sides of a flip), Prop 3, Prop 4 and the corollary are revised.
  • §5.1 adds the token results and harmful flips, and §5.2 compares temperature scaling on one trajectory.
  • §5.4 and Table 2 are replaced by E2, and Fig. 3 has moved to Appendix B.1 for space.
  • Appendix C.4 documents the cost correction, and Appendix E gives the answer-parsing rule as implemented.
  • Appendix G adds notation and proofs, and Appendix H reports the three experiments (Tables 10-17).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.