- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
This paper investigates test-time resource allocation under information insufficiency, arguing that confidence-based compute allocation can be unreliable when models lack necessary information. The authors propose VOI-, a training-free finding-selection heuristic that prioritizes information capable of changing the model's predicted answer. They further explore decision-value-based stopping and a unified controller for reading, computing, and stopping. Experiments on MedQA-USMLE, RACE, and DDxPlus evaluate the proposed methods against confidence-based and entropy-reduction baselines, reporting improvements in accuracy and resource efficiency.
Critical strengths
None.
Critical weaknesses
-
Serious presentation and organization issues. This paper is difficult to follow even after careful reading. The research motivation, problem formulation, theoretical analysis, and proposed algorithm are not presented in a clear and coherent manner. The paper frequently introduces notation and technical concepts without sufficient explanation, while several theoretical claims lack clearly stated assumptions or adequate justification. These issues go beyond minor writing problems and substantially hinder the reader's ability to understand and evaluate the paper's contributions.
-
Limited novelty and significant concerns about conceptual and theoretical soundness. The paper does not establish a convincing logical connection between its motivation, theoretical claims, and proposed method. In particular, confidence is used for different purposes--compute allocation, finding selection, and stopping--without adequately distinguishing their respective decision objectives. The observed limitations of confidence in predicting correctness do not directly establish its inadequacy for these different decisions. Moreover, several theoretical results are elementary or insufficiently justified. For example, the necessary condition for answer correction in Observation 2 does not establish the effectiveness of flip-based selection; the application of Proposition 3 to multiclass temperature scaling is questionable; and the "iff" condition in the corollary lacks a rigorous definition and proof. Consequently, the proposed flip-based heuristic is not sufficiently connected to actual decision-value improvement, weakening both the claimed novelty and the theoretical foundation of the work.
Questions for authors
-
What exactly is confidence insufficient for? Confidence can, in principle, guide both when additional information is needed and when more computation should be allocated. However, VOI- mainly addresses which finding to select. How does this method resolve the limitation of confidence identified in the Introduction?
-
Why does the proposed method deviate from the original research motivation? The Introduction motivates the need to distinguish between acquiring more information and allocating additional computation. However, the main algorithm focuses on selecting which information to reveal, rather than when additional information or computation is needed. How is this shift in the research problem justified?
-
Does the method contradict the motivation of efficient test-time resource allocation? The fundamental motivation of this research area is to improve inference through efficient use of computational resources. However, the proposed finding-selection method itself requires substantial additional model inference. How can such a computationally expensive procedure be justified within this resource-efficiency setting, and what is its practical value?
-
Why should answer flips indicate beneficial information? Observation 2 establishes only a necessary condition for correcting errors. How does VOI- account for harmful flips from initially correct answers, or for reinforcing incorrect beliefs through its confidence-based fallback?
-
Can the theoretical claims be rigorously justified? Claim 1's binary-entropy ordering does not generally hold when the maximum class probability is below 0.5. Proposition 3(ii) applies only to scalar monotone transformations, while the Corollary states an insufficiently justified "if and only if" condition. Can the authors clarify the assumptions and provide rigorous proofs?
-
How are interactions among findings handled? Multiple findings may jointly change an answer even when none causes an individual flip. How can one-step selection and flip-based stopping handle such cases? Does the limited two-step experiment in Appendix D adequately address this issue?
-
What is the practical applicability of the proposed selection procedure? The method evaluates the actual contents of candidate findings before selecting them. How can it operate when information is genuinely unavailable? Conversely, when all information is already available, why not process the complete record directly?
2 comments
Response to Reviewer nwgn(2/2)
Part 2 — Questions
Q1. What exactly is confidence insufficient for? Confidence falls short for each of the three decisions when it is measured against that decision's own objective (Table 9, and item 1 of Part 1). We replace it with one test, "would this action change the answer?", which we use to choose what to read, when to stop and whether to compute.
Q2. Does the method depart from the motivation? We see why it reads this way, and the revision makes the link explicit. The paper answers the question of when more information or more computation is needed at two levels. §3 shows that at minimal information compute buys nothing, while reading two findings gains +0.179 over 16× SC, and the stopping and read/compute rules (§5.2-5.4) then answer the question at each step of a case. Which finding to read is the step that must be decided before reading, and the revised Scope paragraph of §1 now ends with this roadmap.
Q3. Is the selection procedure computationally justified? It is not when the whole record is free to read, and the revision now says so. E2 counts the cost. The controller makes about 11 single-token candidate probes per case (12.8/13.1 calls in all), and nearly all of its 1.8-2.1k input tokens are spent on probe prompts. Matched on tokens, reading the full record and voting beats the controller (-0.064/-0.070, Holm p=0.0008), and matching on calls gives the same result. In a post-hoc check that counts only generated tokens, the controller (135-137 per case) spends about as much as reading the full record and drawing one sample (156-157), and the two tie in accuracy.
The procedure pays where reading is the costly step. This is the case in acquisition (DDxPlus) and with an expensive reader. When a 1.5B/3B model computes the probes, the 14B/32B reader makes one call per case and keeps 65-80% of its own gain over fixed order (E3b, Holm p≤0.031).
Q4. Harmful flips and the confidence fallback. VOI-Δ does flip correct answers, in 16-24% of correct states against 4-8% for ER (Table 13), while it corrects 33-51% of wrong states against 19-34% for ER. The fallback does not drive the results. When no candidate flips, falling back to fixed order instead changes token-AUC by -0.017 to +0.001, which is significant in none of the ten settings (Appendix H.2).
Q5. Rigor. We agree that H_b(m) orders candidates like confidence only for m ≥ 1/2. The revised Claim 1 states the condition under which the entropy and confidence orderings must agree, namely that both candidates have m ≥ 1/2 and H_b(m2) - H_b(m1) > (1 - m1) log(K-1). Outside this condition, Claim 1 remains empirical. Beliefs with top probability below 1/2 make up 0.4-1.0% of candidate beliefs on Qwen2.5 and 21-24% on Llama-3.1-8B, where agreement is 88%. Prop 3(ii) is now stated for transforms of the confidence score only, and 3(i), Prop 4 and the corollary are narrowed and rewritten as described in Part 1. The proofs are in Appendix G, and the Claim 1 condition is derived in Appendix A.
Q6. Interactions among findings. We have probed every pair of unread findings at the flip-stop state, where no single finding changes the answer (Table 16). A pair that changes the answer exists in 12-25% of cases, so one-step stopping does miss joint effects. However, when the answer at the stop is wrong and no pair changes it, reading the whole record corrects it in only 10 of 895 cases. Two-step lookahead (Appendix H.2, and among the negative controls in Appendix D) reaches the first correct answer no sooner, after 1.61 findings against 1.56 for one step on Qwen2.5-7B and after 1.60 against 1.54 on Llama-3.1-8B, while using 4-5× the probes.
Q7. Practical applicability. When content is unavailable before acquisition, an observation model is needed. DDxPlus has one, a Bayesian answer model, and computes decision value as an expectation over the possible answers. This is acquisition of unseen information, and decision-value stopping wins there at mid budget (+0.014). On LLMs the analogue is the expected flip, which we have not tested. When everything is available, the complete record should be processed, since this is better at matched compute (E2), and the paper now says so. Selection pays when reading itself is costly, as with long records, many retrieved documents, a human reviewer or an expensive model, and in those cases a cheap model can screen the record (E3b).
If any passage of the revised §3 and §4 is still hard to follow, or if any statement still lacks an assumption, please point us to it, and we will revise it within the discussion period. We hope that the revision and these responses merit a reconsideration of your evaluation.
Response to Reviewer nwgn(1/2)
Part 1 — W1 and W2
Thank you for the detailed critique. Your technical points are well taken, and we have fixed them in the revision. For context, we first list the results that these fixes do not affect, each re-checked.
W1: presentation. We agree, and we have made targeted structural edits within the 9-page limit. §3 now opens with a problem setup that defines the case, the revealed set, the belief, the read, compute and stop actions with their costs, and the objective. §4 opens with the three decisions (compute, select, stop), each with its objective, its confidence rule and its decision-value rule (Table 9, Appendix G). Every formal statement now states its assumptions, and Appendix G gives the proofs and a notation table. The Scope paragraph of §1 ends with a roadmap.
W2: soundness.
Confidence serves three different decisions. We agree that a low correlation with correctness does not by itself show that confidence fails for each decision, so we have measured each decision against its own objective (Table 9). For compute, confidence does not predict the gain from 8 samples at zero findings (Spearman -0.05/-0.03, p≥0.07, on 1255 cases) and starts to do so only after two findings (-0.11/-0.25). For selection, post-read confidence barely ranks the candidates within a state by post-read correctness (Spearman 0.05-0.21). For stopping, confidence loses to the flip at matched budget (Table 1).
Observation 2 is only a necessary condition. It is, and it also has a second side, which the revision now states: at a correct state every flip is an error, so flip-seeking selection is a bet that the current answer is wrong. The new transition table (Table 13) measures both sides. VOI-Δ corrects 33-51% of wrong states (ER 19-34%) but overturns 16-24% of correct ones (ER 4-8%), and its net gain per step is at most ER's in nine of ten settings. This is why, measured in tokens, VOI-Δ reaches a correct answer first yet ties ER over the whole budget (§5.1).
Prop 3 and multiclass temperature scaling. A counterexample confirms this concern. The beliefs (0.51, 0.49, 0, 0) and (0.50, 0.25, 0.25, 0) have confidence 0.51 > 0.50, but at T=0.75 they become 0.513 < 0.558. Temperature scaling is therefore not a monotone function of confidence. It falls under 3(iii) instead of 3(ii), and we have deleted the sentence "temperature scaling ... is inert".
The same example shows that 3(i) and Prop 4 were stated too broadly. Under any strictly increasing transformation shared by all classes, what stays invariant is the flip indicator, that is, whether the argmax changes, and with it every flip-based stop along a given reading order. Confidence-ranked selection (VOI-conf, ER and VOI-Δ's tie-break) can change, and we have narrowed both statements accordingly.
The "-0.03 to -0.04" frontier shift compared two trajectories with different states, so we have removed it. We report instead that temperature scaling reverses 5.5% of state pairs at step 0, where the states are identical, together with the same-trajectory comparison above.
The corollary's "iff". We have removed it and replaced it with three statements.
Novelty. VOI-Δ shares ER's one-step lookahead and probe bill. What differs is the key, an ordinal flip in place of a cardinal entropy reduction (Prop 4), and that difference shows in behavior. ER selects the same finding as confidence at every observed state in the six Qwen2.5 runs of the submission (88% of states on Llama-3.1-8B), whereas the flip corrects more wrong states and reaches a correct answer 9-20 tokens sooner. It does so at the price of more harmful flips, which we now report.