acceptodds
Withdrawn from ICLR 2027

Same Uncertainty, Different Remedies: Failure-Aware Routing of Test-Time Compute

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adaptive test-time scaling decides where and how much inference compute to spend, but after an initial sampling batch the system must decide what the next computation should do. Scalar uncertainty can signal risk, yet it does not specify whether to retain the majority, verify competing candidates, or abstain. We study whether answer-distribution structure provides a routing state using answer counts alone. Across 1,577 GPQA question–model pairs, the distributions admit operational regimes with different candidate concentration and intervention profiles. This motivates a failure-aware router that keeps majority voting outside a contestation gate, verifies concentrated top-two contests with a sampler-specific screened verifier, and abstains on scattered cases. In the primary Qwen3-32B/GPQA setting, verification gains percentage points on 17 two-mode flags. With 51 verifier calls, full-coverage accuracy rises from 54.5% to 58.7%; abstaining on 11 scattered flags yields 61.4% accuracy at 92.3% coverage. The router has the largest observed equal-budget point estimate, although its advantage over runner-up-share allocation is not statistically established. Verification utility also varies across sampler–verifier pairs. These results suggest that uncertainty identifies risk, structure identifies candidate availability, and realizing that opportunity additionally requires a competent resolver. Routing should therefore follow the observed failure configuration and screen the resolver it depends on.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time scaling, inference-time compute, self-consistency, majority voting, answer distribution, verification, adaptive routing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FQVeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper frames the step after an initial self-consistency batch as a next-action routing problem and uses only answer counts (majority share p1p_1, runner-up share η\eta, top-two mass t2t_2, entropy HH) as the routing state. A descriptive Dirichlet–multinomial mixture over 1,577 GPQA question–model pairs yields three regimes (stable consensus, structured contest, diffuse uncertainty), and the authors separate candidate availability (top-two headroom) from resolver competence (a verifier's net rescue minus damage). The resulting count-only policy keeps majority vote (MV) when η<0.3\eta < 0.3, verifies the top two with an offline-screened verifier when η≥0.3\eta \ge 0.3 and t2≥0.9t_2 \ge 0.9, and abstains otherwise. On Qwen3-32B/GPQA, the policy raises accuracy from 54.5% to 58.7% (61.4% at 92.3% coverage with abstention), though its edge over scalar allocation is not significant and gains do not transfer reliably across samplers or datasets.

Critical strengths
  1. Useful framing. The paper separates two questions: is the correct answer already among the top candidates, and can a verifier pick it without breaking correct answers? This framing and evaluation style is useful beyond this specific router.
  2. Transparent and careful reporting. The paper reports both the fixes and the breakages of each action, and openly states where its results are not significant or where the method fails (e.g., MMLU-Pro, ScienceQA and AIME).
Critical weaknesses
  1. Unclear main claim and contribution. The title promises a failure-aware router, but the paper plays down most of it. The answer patterns are only descriptive, the thresholds are "simple count-based proxies," and the router's advantage is "not statistically established." The overall gain is only 6 net questions out of 143, with no significant edge over simply verifying the questions with the highest runner-up share η\eta (p=0.375p = 0.375). The paper even states that "the resolver, not the structure, is the binding constraint." It is unclear whether the contribution is a method or an analysis.
  2. No new takeaways as an analysis paper. If read as an analysis, the main findings restate known effects. The correct answer is often among the top candidates even when majority vote is wrong, which the paper itself says Cai et al. (2026) also report as "the same kind of recoverable top-two headroom". Reflection or self-verification often hurts, as already shown by Huang et al. (2024a).
  3. Limited generality. The full policy shows a clear gain only for Qwen3-32B on GPQA. Verification adds only 1.1 pp on InternLM3-8B fails screening on Qwen3-8B. Other datasets were tested only without screening, and on AIME the gate triggers on no question (Sec. 7). The cost of per-pair screening is not reported.
  4. Missing baselines and cost accounting. All baselines are count-based rankings, random allocation or resampling. There is no comparison with cited methods such as Lin et al. (2026) or Fu et al. (2026) (Sec. 2). Cost is counted in verifier calls, which the authors admit do "not equalize tokens, latency, or FLOPs" (Sec. 8).
Questions for authors
  1. What is the single main claim the paper is meant to establish? A deployable routing method or an empirical analysis? If it is the method, what result shows that the proposed routing method is valuable? If it is an analysis paper, what takeaways or findings do you consider new relative to prior work?
  2. How does the method compare with existing approaches (e.g., Lin et al., 2026; confidence-based self-consistency; reward-model selection) under an equal total-token budget, and what is its wall-clock latency?
Reviewer_hmGXOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies post-sampling adaptive test-time scaling for large language models, focusing on how to select the next remedial action after an initial batch of responses is generated. The key insight is that scalar uncertainty signals risk but fails to distinguish whether a correct candidate is already present in the sample pool or whether a verifier can reliably select it without overturning correct majority answers. Using only answer-count statistics, specifically the runner-up share and top-two concentration, the authors propose a lightweight routing policy that retains majority voting under strong consensus, invokes an offline-screened external verifier on concentrated two-way contests, and abstains on scattered distributions. Experiments on GPQA-Diamond across multiple model families show that this failure-aware router improves full-pool accuracy and accuracy–coverage trade-offs over naive interventions, while the paper also transparently examines its statistical boundaries and failure modes on open-ended math tasks.

Critical strengths
  1. The paper offers a novel perspective by observing that questions with similar uncertainty can exhibit distinct answer-count structures that require different remedies, and proposes a failure-aware router to improve post-sampling accuracy. Instead of applying a uniform intervention to all uncertain cases, the authors show that naive reflection or verification often damages correct majority answers, and therefore route concentrated top-two contests to a screened verifier while abstaining on scattered distributions.

  2. The proposed routing state relies solely on answer counts and introduces zero additional inference overhead. Because the policy only computes the runner-up share and top-two concentration from the initial sampling batch, it requires no token log-probabilities, hidden states, or extra model calls, making it directly applicable to black-box LLM APIs.

Critical weaknesses
  1. The experimental evaluation fails to statistically establish that the proposed structural routing rule outperforms a simple scalar baseline. In the primary equal-budget comparison on GPQA-Diamond, the accuracy gain of the failure-aware router over majority voting is only marginally significant with a p-value of 0.070, while its direct comparison against a scalar runner-up-share allocator yields a p-value of 0.375 and results in an exact tie on the frozen test split.

  2. The effective sample sizes in the main experiments are too small to support robust empirical conclusions. Although the descriptive analysis pools 1,577 pairs, the core routing evaluation on Qwen3-32B relies on only 143 questions, within which verification is triggered on just 17 questions, abstention is tested on 11 questions, and verifier screening on the discovery split is decided by only 3 recoverable errors.

  3. The empirical benefits are heavily restricted to a single four-option benchmark and fail to generalize to open-ended tasks or uncalibrated datasets. On open-ended AIME problems, dispersed numerical errors prevent any question from triggering the contestation gate and yield zero runner-up accuracy, while transferring the policy to MMLU-Pro and ScienceQA without dataset-specific screening produces zero or negative net gains.

Questions for authors

Given that the proposed router ties with the scalar runner-up-share allocator on the frozen test split and shows no statistically significant advantage on the full pool, what concrete empirical evidence supports the necessity of the two-dimensional structural gate over a single scalar metric?

Reviewer_YzdfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies adaptive test-time scaling, where the goal is to decide the next action after the model provides a batch of initial answers. The authors study the performance of an adaptive decision rule solely based on answer counts. They identify various operating regimes based on the answer count, and propose a simple threshold-based action rule which relies on majority voting (MV) in certain case, verifies only the top-two answers in others, and abstains in the rest. The results based on this scheme seems mixed.

Critical strengths

The paper addresses an important and timely question on adaptive test-time scaling.

Formulating adaptive test-time scaling as a next-action decision problem makes sense.

It proposes a simple strategy that requires limited compute (no more model calls after the initial sampling stage).

Critical weaknesses

The paper appears heavily AI-written. The authors already acknowledge AI usage, and this it not a problem in itself, but the current paper is very difficult to read as there are many vague concepts, undefined variables, and convoluted uninformative sentences, which make the paper very difficult to read and follow.

Also, from a technical point of view, the idea is quite basic, and I'm not sure the results confirm its validity. It seems the reported results are mixed and dataset-dependent.

Therefore, I'm not sure the paper provides significant new findings to the literature. Then, there are sentences such as "Indeed, the mixture fit does not uniquely select three components." What is the mixture fit here? It's later explained in the paper that a Dirichlet-multinomial mixture is fit to the answer count distribution; however, at this point this is a completely vague statement. There are many such sentences throughout the paper. The next sentence is "We then evaluate candidate availability through top-two selection headroom." What is a candidate? What do you mean by top-two selection headroom (which is again defined later in the paper) How is this evaluated? during test-time or offline?

The contributions are also difficult to understand and evaluate. In the first point, it states "AIME exposes a boundary where top-two routing misses dispersed failures." So far, top-two routing has not been explained. The reader does not know when it is used, and what it exactly means. Moreover, what are dispersed failures? The second contribution states "In structured contest, top-two coverage is 76.9% compared with 48.6% MV accuracy, exposing selection headroom. The verification matrix further shows that realized gains depend on the sampler–verifier pair." But, this is simply an observation, so I don't see how this is a contribution. It is quite natural to expect that the gains will depend on the sampler–verifier pair. The last continuation states the proposed strategy, with modest gains. It doesn't seem to provide statistically significant gains compared to other alternatives. Therefore, it is hard to say there is a significant contribution to the literature.

When the Dirichlet-multinominal distribution fit is introduced, variables z, \pi or \alpha_z are not defined. In the paragraph following (4), it says "BIC decreases from 14,348 at two components to 13,881 at six; it does not identify three components as optimal." Please define BIC and explain why it is relevant/ important here. What are the components exactly? Please do not leave these to the reader to guess.

The first sentence in the following paragraph is "In stable consensus, MV accuracy is 75.3% and the runner-up adds only 3.1 pp of selection headroom." But, stable consensus has just been mentioned as one of the three components, but it has not been properly defined. At this point, the reader has no idea what it means so these numbers also don't help much. Same applies for the following sentences in the same paragraph. The last sentence is completely cryptic: "Appendix F.5 shows at the flag level that the availability contrast survives conditioning on η."

A new parameter called contestation score is defined in the next paragraph as P(contest| n), but I don't know what the variable `contest' exactly refers to. Is it referring to one of the four regimes (show in Table 2)? What do you mean by "Its 0.5 crossing near η=0.30 motivates a simple count gate."? Why does this motivate such a gate?

The concept of "screened verification" is used in paragraphs before it is defined. Also, please explain what you mean by "unflagged questions" Does that refer to one of the four contests ? flagged/unflagged is used throughout the paper, but I don't think it's been defined anywhere explicitly.

There are many similar problematic sentences and statements throughout the paper. This applies also to the results. Lots of numbers are provided, but they don't seem to be systematic, and the conclusions seem quite mixed. Sentences like "On Qwen3-8B, forced verification loses 0.5 pp; its selected policy therefore abstains on scattered flags without verification,... " What is "forced verification" and what do you mean by "selected policy"? What does "scattered flags" mean?

Overall, I think the paper is very difficult to read. While it is possible to finally understand what the authors are proposing, it could be presented in a much clear and straightforward manner. While treating test time compute as a decision problem between possible actions makes sense, I think the proposed strategy in the paper is too simplistic and the results are not strong enough to signal efficacy.

Questions for authors
  • In the Introduction, the authors explain that they identify three regimes based solely on the answer counts; however, they do not explain what these regimes correspond to (at least approximately), and how they're exactly decided, which makes it difficult to appreciate and understand the later discussion of the contributions.

  • In (1), why define \eta, given that p_2 is defined afterwards as exactly the same quantity?

  • In (3), please explicitly define what S is. What is a_1(q)? Also, variables N_{rescue} and N_{damage} have not been defined.

  • Please explain what "reflection" exactly refers to. Which specific strategy did you use? From the results in Table 2, it never seems to help. Why is it considered at all?

  • At the end of Section 4, it is stated that "This gap motivates an action that changes selection rather than only increasing the sample count." While only increasing the sample count may not be the best action, it seems that the proposed solution does not sample at all. Can you please clarify? How do we decide how many initial samples will be enough? Shouldn't this depend on the question, and hence, be adaptive as well?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.