acceptodds
Under review as a conference paper at ICLR 2027

Reliability-Guided Objective Allocation for Test-Time Reasoning

Abstract

Test-time adaptation can improve large language model reasoning without additional supervision, yet existing approaches typically apply the same optimization objective to every input. We argue that this uniform treatment ignores a fundamental asymmetry in test-time learning: reliable solutions should be consolidated, whereas uncertain ones should be explored. We introduce DiSCTT, which uses agreement among independently sampled reasoning trajectories as a label-free estimate of instance-level reliability and dynamically allocates the adaptation objective accordingly. High-consensus inputs are routed to supervised fine-tuning using modal-answer-supported pseudo-targets, while low-consensus inputs are routed to reinforcement learning with a modal-consistency-gated, population-aware reward. The routing partition is periodically recomputed as the model adapts, yielding a self-evolving curriculum between consolidation and exploration. Across diverse reasoning benchmarks and models ranging from 0.5B to 32B parameters, DiSCTT consistently improves over uniform reinforcement-learning adaptation, including gains of 9.6 and 8.7 percentage points on the three-task average for 20B and 32B models, respectively. Controlled comparisons against random and static routing, sequential SFT+RL, a label-free comparator based on SRFT (LF-SRFT), and an external-verifier variant support consensus-conditioned objective allocation under the evaluated configurations. DiSCTT also substantially reduces adaptation compute relative to full-dataset reinforcement learning and extends to open-ended generation through semantic consensus. These results suggest that test-time adaptation should treat the choice of optimization objective itself as an instance-level decision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Training, Uncertainty Estimation, Dynamic Objective Allocation, Label-Free Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SAGpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

DiSCTT is a label-free test-time adaptation framework that allocates different optimization objectives according to answer consensus. High-consensus questions are assigned to SFT, while low-consensus questions are assigned to GRPO,. The authors report substantial accuracy improvements over existing test-time RL approaches across multiple models and reasoning benchmarks, together with reduced adaptation compute.

Critical strengths
  1. The idea of allocating different optimization objectives based on instance-level reliability is interesting.

  2. The paper also includes extensive experimental comparisons and routing ablations.

Critical weaknesses
  1. I have concerns about the reported evaluation variance. Table 1 reports exceptionally small standard deviations across 10 runs described as independent in both training and sampling. For example, Qwen3-4B reports 51.4 ± 0.06 on MATH-500 and 27.3 ± 0.16 on GPQA-Diamond, with temperature 0.9 and top-p 0.9. Let Yi∼Bernoulli(pi)Y_i \sim \mathrm{Bernoulli}(p_i) indicate whether question ii is answered correctly in one independently sampled generation. Assuming independent sampling across questions, accuracy in percentage points is A=100n∑i=1nYi,A=\frac{100}{n}\sum_{i=1}^{n}Y_i, with variance Var(A)=104n2∑i=1npi(1−pi).\mathrm{Var}(A)=\frac{10^4}{n^2}\sum_{i=1}^{n}p_i(1-p_i). For MATH-500 (n=500n=500), if the reported SD of 0.06 percentage points is representative of the true sampling SD, it implies
\approx \left(\frac{500\times0.06}{100}\right)^2 =0.09.$$ This implies extremely limited variability in correctness outcomes across independent generations. Even if only 10 of the 500 questions had correctness probability $p_i=0.5$, while all remaining questions were perfectly deterministic, the true accuracy SD would already be $$\sigma_A =\frac{100}{500}\sqrt{10\times0.5\times0.5} \approx0.316\text{ pp},$$ more than five times the reported value. We can further illustrate the statistical sensitivity under a hypothetical larger true variance. Assuming that the true accuracy SD is $\sigma=1$ percentage point and that the ten independent evaluation accuracies are approximately normally distributed, $$\frac{(m-1)s^2}{\sigma^2}\sim\chi^2_{m-1},\qquad m=10.$$ Therefore, $$P(s\leq0.06\mid\sigma=1\text{ pp}) =P\left(\chi_9^2\leq\frac{9(0.06)^2}{1^2}\right) =P(\chi_9^2\leq0.0324) \approx1.65\times10^{-10}.$$ This is just an illustration as the true SD is unknown. Nevertheless, the reported variances appear unusually small under the stated independent stochastic evaluation method. These results raise concerns about the evaluation procedure and statistical reporting. The authors should address this concern first. 2. Table 2 reports 36.7% for the unadapted Qwen3-32B (thinking is enabled) on AIME-2024, compared with 81.4% in the official Qwen3 Technical Report. Even after DiSCTT adaptation, the accuracy reaches only 54.7%, below the officially reported baseline. Although the evaluation protocols differ, this large discrepancy requires explanation. One possible cause is the restrictive 2,048-token generation limit, compared with the larger budget used in the official evaluation. If truncation is a major factor, the reported improvements may partly reflect changes in response length or answer completion rather than improved reasoning capability. Also, The method relies on a fixed consensus threshold of 0.45. The theoretical analysis does not justify this assumption. The authors also write "These local results do not determine ρ = 0.45 or establish optimality of multi-round DiSCTT." Although the paper provides empirical sensitivity analyses, the evidence for the precise routing boundary remains limited. Please also consider changing Figure 2 into vector image.
Questions for authors
  1. How can the exceptionally small SDs in Table 1 be explained?

  2. Are the wall-clock times in Table 6 measured per run or aggregated across ten seeds?

  3. Why does Qwen3-32B achieve only 36.7% on AIME-2024, compared with 81.4% in the official Qwen3 Technical Report? Could the 2,048-token limit explain this discrepancy? Please report truncation rates and results with larger output budgets to clarify the source of the improvements.

  4. What justifies the fixed consensus threshold of 0.45? The theoretical analysis assumes monotonic conditional risk differences but does not derive this relationship or the chosen threshold. Can the authors provide stronger theoretical or empirical justification?

Reviewer_gWmXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiSCTT uses answer agreement to route unlabeled test questions between SFT on confident pseudo-solutions and RL on uncertain examples, refreshing the partition during adaptation. It reports sizable accuracy gains and lower compute than uniform RL. The idea is practical, though the evidence is more convincing for the overall recipe than for the proposed consolidation-versus-exploration explanation.

Critical strengths
  • The method is fairly simple and addresses a sensible problem: spending expensive RL updates on every input may be unnecessary.
  • Evaluation covers several model families, scales, and tasks, with ten independent runs and shared hyperparameters. Reporting measured runtime alongside estimated FLOPs is useful.
  • The random/static routing controls and coverage–precision analyses go beyond headline accuracy. The paper also clearly acknowledges confidently wrong consensus and the transductive setting.
Critical weaknesses
  • The sequential SFT → RL baseline in Figure 1 seems almost similar level of performance as DiSCTT. Given the typical level of variance observed in such methods, they might heavily or fully overlap in their variance bands. Without such a comparison, the claim that DiSCTT has subsantial benefits over SFT + RL does not hold weight.
  • Table 10 includes a useful uniform-RL control with DiSCTT’s reward, but only for one backbone and two tasks. Meanwhile, adding novelty contributes 9.2 points in Table 17. The main suite needs this same-reward RL control and iterative SFT with refreshed pseudo-targets; the pure-SFT baseline instead freezes targets once. Since SRFT already adapts the SFT/RL balance using uncertainty, these comparisons matter for establishing what exclusive consensus routing adds.
  • Every non-modal answer receives zero reward, so a correct minority answer gets a negative group-relative advantage whenever modal completions receive positive rewards. Novelty rewards variation within the selected answer cluster, which need not help discover a better answer. The paper acknowledges this failure mode but provides little direct evidence about answer diversity or minority-answer recovery. SCRL, which explicitly handles unreliable consensus, is a particularly relevant missing baseline.
  • The theory is mathematically sound but largely states that choosing the lower-risk branch is better; threshold structure follows from an assumed ordering of branch risks. The empirical crossover uses one checkpoint and dataset, separately trained bin-specific models, and unequal branch budgets. The high-consensus SFT advantage is only six additional correct answers. This is suggestive evidence, rather than a strong explanation of the full shared-model curriculum.
  • Appendix I.4 calls majority voting generation-budget matched without stating the actual number of samples. Its Qwen3-1.7B results exactly match the eight-sample routing diagnostic, whereas DiSCTT generates repeatedly throughout adaptation. Without explicit sample counts and accuracy-versus-total-compute curves, this comparison does not convincingly rule out spending the adaptation budget on additional inference.

Minor points:

  • Equation (46) defines precision using incorrect pseudo-labels. The inequality should be equality; the subsequent decomposition otherwise contradicts the definition.
  • Token-distribution JSD compares identical token positions across different reasoning prefixes. It is unclear how well this measures meaningful reasoning novelty rather than differences in wording or structure.
Questions for authors
  • What are the exact prompts, answer extractors, and reasoning-step boundaries? How is enable_thinking=True applied to the Qwen3 Base checkpoints?
  • How are trajectories longer than the 2,048-token optimization limit handled, particularly when their rewarded final answer lies beyond that limit?
  • Do the held-out transfer gains exceed those from the same-reward uniform-RL and refreshed-SFT controls?
Reviewer_qM2dOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose DiSCTT, a method for test-time adaptation without ground-truth labels. The method assumes that agreement among the model's own samples on a test input is a reliable signal of correctness. For each test question, the model generates 8 answers, and a consensus score (the fraction of samples that match the most common answer) decides how the question is trained. High-agreement questions (at least 4 of 8 agreeing) go to supervised fine-tuning, using a sampled response whose answer matches the majority answer as the target. Low-agreement questions go to reinforcement learning (GRPO). There, the reward is based on matching the majority answer of fresh rollouts, scaled by a novelty term (how different the reasoning is from other majority-answer rollouts) and a relevance term (how close the reasoning stays to the question). The routing is recomputed at the start of each of five rounds, so questions can move between the two branches as the model changes. The authors also provide a theoretical analysis framing the choice between SFT and RL as a per-question routing decision.

Critical strengths

Strengths

  1. Simple idea. The idea is simple and makes sense, especially applying SFT and RL to different subsets of the data. However, the unnecessarily complicated and unclear presentation obscures this simplicity.
  2. Compute efficiency. Routing high-agreement questions to SFT reduces estimated adaptation FLOPs by 52.8–61.6% (Table 5) and measured wall-clock time by 40–49% (Table 6) relative to TTRL. This part highlights a straightforward advantage of the method and avoids vanishing gradient issues that can further destabilize GRPO. I recommend putting this part in the main paper (currently Table 5 and 6 are in the Appendix) and doing an additional stability estimation over the Reinforcement Learning part, which can potentially show improvement over vanilla GRPO.
Critical weaknesses

The presentation is scattered, incomplete, and hard for human readers to follow.

The writing feels unpolished and was hard for me to follow. I read the paper without AI tools and found major gaps in how the method (§3) is presented, which made it very hard to read. Many readers now use AI tools, but a submission should still be readable by humans on its own, so I consider this a critical weakness. The specific problems are:

  1. L164–166: r_{j,i} and a_{j,i} are never named (presumably the reasoning and the final answer). The split between π_θ (the "raw model distribution") and q_θ (the "decoding distribution") adds notation readers don't need. I suggest naming r and a, and dropping q_θ or explaining in one sentence why it matters.
  2. L171 (Eq. 1): V_j depends on "valid" answers, but the main paper never defines "valid." I suggest defining it here.
  3. L180, L187 (Eq. 2): TieBreak(·) is never defined in the main paper. The name a_route suggests a routing decision, but it is just the majority-vote answer used as the SFT pseudo-label. I suggest a clearer name and a one-line definition of the tie-break.
  4. L189: a*(x) (the ground truth) appears among the method's notation without any statement that it is never used during adaptation. The "working assumption" is heavily formalized and hedged, but in plain words it only says "more agreement means the majority answer is more likely correct." I suggest stating it in one plain sentence and noting that a* is used only for analysis.
  5. L192–195 (Eq. 3): D_conf and D_unc are introduced without explanation. Their roles (SFT vs. RL) are only implied by later section titles, and ρ has no value until §3.5 (L286). "Policy-dependent… instances may migrate" just means the split is recomputed each round. I suggest explaining both sets in words, giving ρ here, and saying plainly that the split is recomputed each round.
  6. L197 ("Hard routing."): This paragraph never describes the routing; it only points elsewhere. I suggest describing the routing procedure here.
  7. L199–211 (§3.3): "Supervised consolidation" is jargon for SFT on the majority answer. "Active generated-response positions" (L202) just means response tokens. "At least ⌈ρM⌉ sampled support" (L210) means at least 4 of 8 samples agree. "Both SFT epochs" (L211) refers to a schedule that is only introduced in §3.5. I suggest plain wording and introducing the schedule before §3.3.
  8. L216: The RL branch draws a fresh set of 32 samples and computes its own majority answer, a_RL. Why it doesn't reuse the 8 routing samples is never explained. I suggest stating the reason.
  9. L226 (Eq. 6): The terms "modal-consistency gate," "population-relative novelty," and "relevance shaping" are hard to parse. In plain words, the reward is 1 if the rollout matches the majority answer, scaled up for novel reasoning and down for off-topic reasoning. The paper never motivates why the novelty and relevance terms are needed. I suggest describing the reward plainly and motivating each term.
  10. L236–247 (Eqs. 7–8): π_θroll ("rollout-snapshot probabilities," L242) is undefined. Sentences such as "singleton references contribute zero" (L246) and "retained distributions are detached and reused" (L247) are implementation details that hide the main idea, which is roughly: how different this trajectory's token distributions are from the other majority-answer trajectories. I suggest a one-sentence intuition in place of these details.
  11. L250–255 (Eq. 9): How reasoning steps s_{i,j} are segmented is never stated. The "fixed encoder e(·)" is never identified in the main paper, even though it adds an external model to the method. "Semantic drift" (L255) is undefined. I suggest stating the segmentation rule and naming the encoder.
  12. L258–279 (GRPO update): This is mostly standard GRPO with extra notation (G_i vs. T_i, L268). The paper then says ratio clipping is inactive (L276), so the clipped objective in Eq. 11 is unnecessary. K̂ (L278) is never defined in the main paper. I suggest citing GRPO, stating only what differs from it, and defining K̂.
  13. L284–287 (§3.5): "Self-evolving curriculum" is jargon for recomputing the SFT/RL split at the start of each of 5 rounds. "A final routing refresh is diagnostic only" (L287) is unclear. Algorithm 1 is not in the main paper. I suggest a plain description and an algorithm box in §3.
  14. L290–298 (§3.6): "Local analysis of objective allocation" and "conditional allocation regret" dress up a near-trivial identity: picking the worse branch costs the gap between the two branches. The paper itself admits this does not determine ρ or justify the method (L298–299). I suggest shortening it to one sentence.

Method

  1. The method needs more than the stated assumption. The working assumption (L189) only says that more agreement means the majority answer is more likely correct. The method needs something stronger: that a 4/8 majority is reliable enough to use as an SFT label. That is not established. For example, pseudo-label precision on AMC starts at only 66.7% (L1349), so about a third of the early SFT targets are wrong. I suggest reporting majority-answer accuracy at the chosen ρ for each benchmark.
  2. Confirmation bias under SFT is not analyzed. SFT on a wrong majority answer reinforces that error directly, and 11 of 62 confidently wrong answers remain wrong after adaptation (Table 3). I suggest reporting how many initially wrong pseudo-labels are corrected vs. locked in, and whether pass@k or output diversity drops.
  3. The need for SFT is not justified. On high-agreement questions, most GRPO rollouts match the majority answer, so advantages are already near zero. This suggests SFT mainly saves compute rather than improving accuracy. The accuracy evidence for SFT over RL on these questions is small (67.86 vs. 65.48 on a single checkpoint, L358). Meanwhile, RL is applied exactly where the majority-vote reward is least reliable (low-agreement questions). Please justify this design, ideally against the baseline in Experiments point 1.

Experiments

  1. Simple baselines are missing. (a) RL only on D_unc, skipping high-agreement questions entirely. (b) TTRL with dynamic prompt filtering that drops zero-advantage groups (as in DAPO's dynamic sampling), the standard fix for this problem; it is neither compared against nor cited. (c) TTRL at matched compute. Without (a) and (b), the benefit of SFT routing cannot be separated from simply not training on easy questions.
  2. The reported variance is implausible. Table 1 (L324) reports SDs as small as ±0.01 across 10 end-to-end runs, which is below sampling noise for one generation per question. Table 2 reports no SDs at all. Please explain how the SDs are computed and report per-run results.
  3. Gains may partly come from answer formatting. Several base models score below chance on 4-choice GPQA (e.g., 14.1 and 7.1 in Table 1), which suggests answer-extraction failures. SFT on well-formatted majority answers would fix formatting quickly. Please report format-controlled accuracy (accuracy on parseable answers only).
  4. Evaluation is transductive. The model adapts on the same test questions it is scored on, and held-out transfer is only in the appendix (L314). Please report the main results on a held-out split.
  5. Large-scale coverage is limited. The 20B/32B results cover only three tasks without SDs, and no already RL-tuned reasoning models are tested. Those models may have little left to gain from majority-vote self-training.
Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.