- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses a failure mode of self-consistency (SC) under a hard per-call token cap: many reasoning chains do not finish, so SC may have no answer to vote on. The authors propose an eight-call portfolio that replaces three of the eight SC samples with one greedy chain and two repairs of it. If any SC sample answers, the portfolio returns the SC plurality; otherwise it returns the plurality of the repair answers. With Qwen3.5 models on MATH Level 4–5, the portfolio improves accuracy over SC@8 by 3.7 to 11.7 points across an exploratory study, a pre-registered archival evaluation, and a pre-registered prospective replication. The analysis attributes the gain mainly to answer supply: the repairs finish more often when no SC chain does.
Critical strengths
The motivation is clear and practically relevant. The paper identifies a concrete failure of SC under per-call caps and shows that simply adding more samples closes this gap only slowly.
The evaluation is careful. It includes pre-registered evaluations, a prospective replication on unused questions, a full manual audit of scored answers, and paired bootstrap tests.
The mechanism analysis is convincing. Controls on temperature, budget instructions, answer recovery, and the aggregation rule all support the claim that the gain comes from answer supply rather than from changes in voting.
Critical weaknesses
-
A natural baseline seems to be missing: continuing truncated SC chains. The gains come mainly from records where every SC chain hits the token cap. A simple alternative would be to spend the three extra calls on continuing these stopped chains instead of generating repairs. Prior work combines sampling with self-correction [1], and concurrent work finds that continuing a model’s own reasoning prefix outperforms restarting under matched token budgets [2]. The authors themselves note that this allocation is outside their evaluation (§7). Without this comparison, it may be unclear whether the repairs are better than simply reusing the unfinished chains.
-
The efficiency evidence may be incomplete. Although the portfolio uses the same number of calls as SC@8, its total token count is 14.8% higher in the original study, because the repair prompts include the greedy chain (§6.1). The comparison with SC@16 partly addresses this (§6.4, Table 9). However, at the prospective 4,096-token cap, the eight-call portfolio is not significantly better than SC@16 (+3.4 [−0.4, 7.6]) or SC@24 (+0.7). Latency is also not measured (§6.1, Appendix A), even though the repairs must wait for the greedy chain and the lazy variant adds two sequential rounds when SC abstains (§6.4). The practical cost therefore may be underestimated.
-
When the portfolio should be preferred over SC is not clearly established. The gains seem limited to settings where many SC chains fail to finish. When SC chains mostly finish, the portfolio is significantly worse on Llama-3.1-8B at 4,096 tokens (−4.2 [−7.8, −0.7]), and shows no significant gain for Qwen3.5-9B with thinking disabled (+1.2 [−0.5, 3.8]) or on GSM8K (+0.11 [−0.26, 0.54]) (Tables 7 and 12). The authors suggest that the gain can be estimated from the SC no-answer rate (§1, §7), but this prediction does not appear to be tested on held-out models or budgets.
-
Section 3 focuses on plurality inversion, yet the gains come from answer supply, and almost no SC-vote inversions are observed (Table 8). The theory also does not explain the 5+3 split, which the authors chose after seeing the per-arm results of the original study (§4). The design may therefore rest mainly on empirical tuning.
[1] SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling. Transactions on Machine Learning Research (TMLR), 2025. arXiv:2501.19306.
[2] It’s the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories. arXiv:2609.03436, 2026.
Questions for authors
See above.
2 comments
Update on W2: comparison at equal mean total tokens
An update on W2, efficiency. You noted that at equal calls the eight-call portfolio uses 14.8% more total tokens than SC@8 in the original study, and you pointed to the prospective cohort at the 4,096-token cap. The updated PDF now compares at equal mean total tokens in both of these sets and at the 16,384-token cap (Appendix G, Table 17). When SC receives the extra tokens as extra samples, the portfolio stays ahead by +8.3, +7.2 and +10.9 points.
The three sets are the original study with Qwen3.5-27B at 4,096 tokens and the prospective cohort with Qwen3.5-9B at 4,096 and 16,384 tokens, each with 100 questions × 3 seeds. This is a descriptive comparison made after the results of the paper were known, with no new generation. SC at equal tokens is the mixture of SC@k and SC@(k+1) with the same mean prompt-plus-completion tokens per question-seed pair, drawn from the 24 pooled SC samples of a question at the same per-call cap. Intervals are question-bootstrap intervals of the per-question differences.
- Original study, 4,096 tokens: the portfolio uses 14.8% more tokens than SC@8 and reaches 0.770 against 0.687 for SC at that token count: +8.3 points [4.6, 12.4].
- Prospective cohort, 4,096 tokens: it uses 14.8% more tokens than SC@8, 36.1k against 31.4k per pair, and reaches 0.467 against 0.395: +7.2 points [3.5, 11.4]. For comparison, SC@16 and SC@24 of Table 9 use 60.8k and 91.2k completion tokens per question at this cap.
- Prospective cohort, 16,384 tokens: it uses 5.7% more tokens than SC@8, 87.5k against 82.7k per pair, and reaches 0.980 against 0.871: +10.9 points [5.6, 16.7].
The portfolio can also be run with fewer tokens than SC@8. The deferred form, ACDR-Lazy of Section 4, generates the greedy chain and its two repairs only if the five SC samples return no answer, at least four distinct answers, or a plurality below half of their returned votes. On the same generated candidates it returns the same decisions as the eight-call portfolio, so its accuracy is the same. It uses 6.1, 7.0 and 5.5 calls per pair and 14.7%, 0.3% and 27.4% fewer tokens than SC@8 in the three sets. Against the mixture matched to its own lower token count it is at +10.4 [6.3, 14.8], +8.2 [4.4, 12.5] and +12.3 [6.9, 18.2] points.
On the two sequential rounds: the deferred form generates the repairs in 35.9%, 65.7% and 15.3% of the cases in the three sets, counting every pair with every choice of five SC samples (2,563 of 16,800 at 16,384 tokens). In each of these cases none of the five SC samples returned an answer. There deferral adds two rounds after the five SC calls, the greedy chain and then its two repairs, three further calls; in every other case it issues only the five SC calls. Table 16 gives composed latency in the held-out test, where the deferred concise close has a median parallel latency of 110 s against 122 s for SC@8. Those figures are for Qwen3.8-27B; for the three Qwen3.5 sets above we report token costs and trigger rates.
The held-out test of our first reply uses Qwen3.8-27B at the 4,096-token cap. There the label-free rule selected concise close on 30 calibration questions, and Table 15 reports +4.8 [1.1, 9.2] for it at equal total tokens. The three sets above are the original exploratory study and the prospective evaluations of the Qwen3.5 portfolio, in which the repairs finish more often than SC samples (Sections 6.1, 6.3 and 6.4).
Response to Reviewer Aqyu
Thank you for the careful reading. We answer the four points in turn. The new measurements are in Section 6.6 and Table 13 of the revised PDF.
They come from a held-out comparison: 100 MATH Level 4-5 questions used in no earlier analysis of this project, 3 seeds, Qwen3.8-27B (released after the submission's models), a 4,096-token per-call cap, and the submission's prompts, return policy and automatic evaluator. All policies share five SC samples and differ in the three remaining calls: three more SC samples (SC@8); a greedy chain and two repairs (the portfolio); a continuation of up to three stopped chains (continuation); or a new call that shows a stopped chain and asks for a concise final solution (concise close). The close wording and the choice among these uses were settled on 30 separate calibration questions.
W1, continuing truncated SC chains. Continuation works, and it is the same idea as repair: both are instances of Proposition 1 (Section 3), which says to use the replaced calls for candidates that are more likely to finish. Accuracy side by side is 0.948 for concise close, 0.925 for continuation, 0.897 for SC@8 and 0.893 for the portfolio (Table 13). On the 22 of 300 question-seed pairs where none of the five shared SC chains finishes, concise close returns an answer on 19, continuation on 12 and the portfolio on 3. At equal calls the portfolio is at -5.5 [-9.7, -2.0] points against concise close and -3.2 [-5.7, -1.2] against continuation. The repairs are the instance chosen on Qwen3.5, where they finish more often than fresh chains (68% and 57% against 47% in the original study, Section 6.4) and gain +3.7, +9.4 and +11.7 points over SC@8 (Tables 3 and 4). The continuation result agrees with arXiv:2609.03436, which finds that continuing a model's own prefix beats restarting at matched budgets; SETS (TMLR 2025) verifies and corrects candidates that finish, the complementary case. Section 2 now cites both.
W2, efficiency. Prompt-plus-output tokens per pair are 13.1k for SC@8, 16.2k for the portfolio, 14.8k for continuation and 14.0k for concise close. Against SC with the same mean total tokens (a mixture of SC@k and SC@k+1 from the pooled samples at the same cap, Table 15), concise close is at +4.8 [1.1, 9.2] points, continuation at +2.3 [0.2, 5.0] and the portfolio at -1.0 [-2.7, 0.6]. Generating the three concise-close calls only when the five SC samples return no answer uses 5.2 calls and 9.6k tokens per pair at accuracy 0.948, which is +5.6 [1.7, 10.1] against SC at equal tokens. Latency is now measured by composing the service time of each call along its dependencies: repairs wait for the greedy chain, continuations and closes for the five SC samples. With every ready call on its own replica, the deferred form of concise close has a median / 90th-percentile latency of 110 / 415 s against 122 / 411 s for SC@8, at accuracy 0.948 against 0.897 (Table 16).
W3, when to prefer it over SC. The relation of Section 7 combines three quantities: the SC no-answer rate, which needs no labels; the accuracy of the extra calls where SC abstains; and the effect of voting with five SC samples instead of eight elsewhere. The accuracy where SC abstains depends on model and cap: 32%, 35%, 16% and 85% in the original study, the archive and the two prospective caps (Sections 6.3-6.4). So it has to be estimated on a small sample at the deployment's own model and cap, and we tested exactly that. From the held-out no-answer rate (19 of 300 pairs, read without references) and the 30 calibration questions, the relation, computed before any held-out answer was graded, predicted +3.8 points (band +1.3 to +5.7) for concise close. The observed gain is +5.2 [1.5, 9.7]; all three uses are in Appendix F. The runs where SC chains mostly finish follow the same relation (new Table 14): with 0, 0 and 7 of 300 pairs without an SC answer there is no missing answer to supply.
W4, Section 3 and the 5+3 split. The method rests on Proposition 1: when chains finish with probability below one, every aggregator that returns only supplied answers abstains with probability at least tau^k, so raising the probability that a candidate finishes is an alternative to adding samples. The revised Section 3 leads with it. Theorem 1 and Proposition 2 (Appendix C) follow as background on why re-weighting the same answers cannot help, and inversions are rare here (0 of 112 and 1 of 259 cases, Table 8). The proposition says which kind of call to put in the replaced slots. The 5+3 split was chosen after the per-arm results of the original study, as Section 4 states; every later evaluation uses that split unchanged.