acceptodds
Under review as a conference paper at ICLR 2027

Changing the Candidates, Not the Vote: A Repair Portfolio for Self-Consistency under a Per-Call Token Cap

Abstract

Self-consistency (SC) votes over sampled reasoning chains, but under a hard per-call token cap many chains never finish, and a system that discards unfinished generations has nothing to vote for. We study an alternative at an equal number of calls: replace three of eight SC samples by a greedy chain that does not vote and two repairs of it, and return the SC plurality when any SC sample answers and the repair plurality otherwise. With Qwen3.5 models on MATH Level 4–5, this frozen configuration raises accuracy over SC@8 by 10.0 percentage points with 95% confidence interval in an exploratory study at a 4,096-token cap, by 3.7 in a pre-registered evaluation of a 475-question archive, and by 9.4 and 11.7 in a pre-registered replication on 100 unused questions at caps of 4,096 and 16,384 tokens. The gain comes from answer supply: it arises where no SC chain finishes, because the repairs finish more often. It persists when answers are recovered from unfinished chains and when SC is sampled at the repairs' temperature or with a budget instruction in every prompt, and the 15-call portfolio exceeds SC@16 at the same cap in the original study and at both prospective caps.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-consistency, plurality voting, test-time compute, reasoning, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AqyuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a failure mode of self-consistency (SC) under a hard per-call token cap: many reasoning chains do not finish, so SC may have no answer to vote on. The authors propose an eight-call portfolio that replaces three of the eight SC samples with one greedy chain and two repairs of it. If any SC sample answers, the portfolio returns the SC plurality; otherwise it returns the plurality of the repair answers. With Qwen3.5 models on MATH Level 4–5, the portfolio improves accuracy over SC@8 by 3.7 to 11.7 points across an exploratory study, a pre-registered archival evaluation, and a pre-registered prospective replication. The analysis attributes the gain mainly to answer supply: the repairs finish more often when no SC chain does.

Critical strengths

The motivation is clear and practically relevant. The paper identifies a concrete failure of SC under per-call caps and shows that simply adding more samples closes this gap only slowly.

The evaluation is careful. It includes pre-registered evaluations, a prospective replication on unused questions, a full manual audit of scored answers, and paired bootstrap tests.

The mechanism analysis is convincing. Controls on temperature, budget instructions, answer recovery, and the aggregation rule all support the claim that the gain comes from answer supply rather than from changes in voting.

Critical weaknesses
  • A natural baseline seems to be missing: continuing truncated SC chains. The gains come mainly from records where every SC chain hits the token cap. A simple alternative would be to spend the three extra calls on continuing these stopped chains instead of generating repairs. Prior work combines sampling with self-correction [1], and concurrent work finds that continuing a model’s own reasoning prefix outperforms restarting under matched token budgets [2]. The authors themselves note that this allocation is outside their evaluation (§7). Without this comparison, it may be unclear whether the repairs are better than simply reusing the unfinished chains.

  • The efficiency evidence may be incomplete. Although the portfolio uses the same number of calls as SC@8, its total token count is 14.8% higher in the original study, because the repair prompts include the greedy chain (§6.1). The comparison with SC@16 partly addresses this (§6.4, Table 9). However, at the prospective 4,096-token cap, the eight-call portfolio is not significantly better than SC@16 (+3.4 [−0.4, 7.6]) or SC@24 (+0.7). Latency is also not measured (§6.1, Appendix A), even though the repairs must wait for the greedy chain and the lazy variant adds two sequential rounds when SC abstains (§6.4). The practical cost therefore may be underestimated.

  • When the portfolio should be preferred over SC is not clearly established. The gains seem limited to settings where many SC chains fail to finish. When SC chains mostly finish, the portfolio is significantly worse on Llama-3.1-8B at 4,096 tokens (−4.2 [−7.8, −0.7]), and shows no significant gain for Qwen3.5-9B with thinking disabled (+1.2 [−0.5, 3.8]) or on GSM8K (+0.11 [−0.26, 0.54]) (Tables 7 and 12). The authors suggest that the gain can be estimated from the SC no-answer rate (§1, §7), but this prediction does not appear to be tested on held-out models or budgets.

  • Section 3 focuses on plurality inversion, yet the gains come from answer supply, and almost no SC-vote inversions are observed (Table 8). The theory also does not explain the 5+3 split, which the authors chose after seeing the per-arm results of the original study (§4). The design may therefore rest mainly on empirical tuning.

[1] SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling. Transactions on Machine Learning Research (TMLR), 2025. arXiv:2501.19306.

[2] It’s the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories. arXiv:2609.03436, 2026.

Questions for authors

See above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Aqyu

Thank you for the careful reading. We answer the four points in turn. The new measurements are in Section 6.6 and Table 13 of the revised PDF.

They come from a held-out comparison: 100 MATH Level 4-5 questions used in no earlier analysis of this project, 3 seeds, Qwen3.8-27B (released after the submission's models), a 4,096-token per-call cap, and the submission's prompts, return policy and automatic evaluator. All policies share five SC samples and differ in the three remaining calls: three more SC samples (SC@8); a greedy chain and two repairs (the portfolio); a continuation of up to three stopped chains (continuation); or a new call that shows a stopped chain and asks for a concise final solution (concise close). The close wording and the choice among these uses were settled on 30 separate calibration questions.

W1, continuing truncated SC chains. Continuation works, and it is the same idea as repair: both are instances of Proposition 1 (Section 3), which says to use the replaced calls for candidates that are more likely to finish. Accuracy side by side is 0.948 for concise close, 0.925 for continuation, 0.897 for SC@8 and 0.893 for the portfolio (Table 13). On the 22 of 300 question-seed pairs where none of the five shared SC chains finishes, concise close returns an answer on 19, continuation on 12 and the portfolio on 3. At equal calls the portfolio is at -5.5 [-9.7, -2.0] points against concise close and -3.2 [-5.7, -1.2] against continuation. The repairs are the instance chosen on Qwen3.5, where they finish more often than fresh chains (68% and 57% against 47% in the original study, Section 6.4) and gain +3.7, +9.4 and +11.7 points over SC@8 (Tables 3 and 4). The continuation result agrees with arXiv:2609.03436, which finds that continuing a model's own prefix beats restarting at matched budgets; SETS (TMLR 2025) verifies and corrects candidates that finish, the complementary case. Section 2 now cites both.

W2, efficiency. Prompt-plus-output tokens per pair are 13.1k for SC@8, 16.2k for the portfolio, 14.8k for continuation and 14.0k for concise close. Against SC with the same mean total tokens (a mixture of SC@k and SC@k+1 from the pooled samples at the same cap, Table 15), concise close is at +4.8 [1.1, 9.2] points, continuation at +2.3 [0.2, 5.0] and the portfolio at -1.0 [-2.7, 0.6]. Generating the three concise-close calls only when the five SC samples return no answer uses 5.2 calls and 9.6k tokens per pair at accuracy 0.948, which is +5.6 [1.7, 10.1] against SC at equal tokens. Latency is now measured by composing the service time of each call along its dependencies: repairs wait for the greedy chain, continuations and closes for the five SC samples. With every ready call on its own replica, the deferred form of concise close has a median / 90th-percentile latency of 110 / 415 s against 122 / 411 s for SC@8, at accuracy 0.948 against 0.897 (Table 16).

W3, when to prefer it over SC. The relation of Section 7 combines three quantities: the SC no-answer rate, which needs no labels; the accuracy of the extra calls where SC abstains; and the effect of voting with five SC samples instead of eight elsewhere. The accuracy where SC abstains depends on model and cap: 32%, 35%, 16% and 85% in the original study, the archive and the two prospective caps (Sections 6.3-6.4). So it has to be estimated on a small sample at the deployment's own model and cap, and we tested exactly that. From the held-out no-answer rate (19 of 300 pairs, read without references) and the 30 calibration questions, the relation, computed before any held-out answer was graded, predicted +3.8 points (band +1.3 to +5.7) for concise close. The observed gain is +5.2 [1.5, 9.7]; all three uses are in Appendix F. The runs where SC chains mostly finish follow the same relation (new Table 14): with 0, 0 and 7 of 300 pairs without an SC answer there is no missing answer to supply.

W4, Section 3 and the 5+3 split. The method rests on Proposition 1: when chains finish with probability below one, every aggregator that returns only supplied answers abstains with probability at least tau^k, so raising the probability that a candidate finishes is an alternative to adding samples. The revised Section 3 leads with it. Theorem 1 and Proposition 2 (Appendix C) follow as background on why re-weighting the same answers cannot help, and inversions are rare here (0 of 112 and 1 of 259 cases, Table 8). The proposition says which kind of call to put in the replaced slots. The 5+3 split was chosen after the per-arm results of the original study, as Section 4 states; every later evaluation uses that split unchanged.

AuthorsOpenReview ↗

Update on W2: comparison at equal mean total tokens

An update on W2, efficiency. You noted that at equal calls the eight-call portfolio uses 14.8% more total tokens than SC@8 in the original study, and you pointed to the prospective cohort at the 4,096-token cap. The updated PDF now compares at equal mean total tokens in both of these sets and at the 16,384-token cap (Appendix G, Table 17). When SC receives the extra tokens as extra samples, the portfolio stays ahead by +8.3, +7.2 and +10.9 points.

The three sets are the original study with Qwen3.5-27B at 4,096 tokens and the prospective cohort with Qwen3.5-9B at 4,096 and 16,384 tokens, each with 100 questions × 3 seeds. This is a descriptive comparison made after the results of the paper were known, with no new generation. SC at equal tokens is the mixture of SC@k and SC@(k+1) with the same mean prompt-plus-completion tokens per question-seed pair, drawn from the 24 pooled SC samples of a question at the same per-call cap. Intervals are question-bootstrap intervals of the per-question differences.

  • Original study, 4,096 tokens: the portfolio uses 14.8% more tokens than SC@8 and reaches 0.770 against 0.687 for SC at that token count: +8.3 points [4.6, 12.4].
  • Prospective cohort, 4,096 tokens: it uses 14.8% more tokens than SC@8, 36.1k against 31.4k per pair, and reaches 0.467 against 0.395: +7.2 points [3.5, 11.4]. For comparison, SC@16 and SC@24 of Table 9 use 60.8k and 91.2k completion tokens per question at this cap.
  • Prospective cohort, 16,384 tokens: it uses 5.7% more tokens than SC@8, 87.5k against 82.7k per pair, and reaches 0.980 against 0.871: +10.9 points [5.6, 16.7].

The portfolio can also be run with fewer tokens than SC@8. The deferred form, ACDR-Lazy of Section 4, generates the greedy chain and its two repairs only if the five SC samples return no answer, at least four distinct answers, or a plurality below half of their returned votes. On the same generated candidates it returns the same decisions as the eight-call portfolio, so its accuracy is the same. It uses 6.1, 7.0 and 5.5 calls per pair and 14.7%, 0.3% and 27.4% fewer tokens than SC@8 in the three sets. Against the mixture matched to its own lower token count it is at +10.4 [6.3, 14.8], +8.2 [4.4, 12.5] and +12.3 [6.9, 18.2] points.

On the two sequential rounds: the deferred form generates the repairs in 35.9%, 65.7% and 15.3% of the cases in the three sets, counting every pair with every choice of five SC samples (2,563 of 16,800 at 16,384 tokens). In each of these cases none of the five SC samples returned an answer. There deferral adds two rounds after the five SC calls, the greedy chain and then its two repairs, three further calls; in every other case it issues only the five SC calls. Table 16 gives composed latency in the held-out test, where the deferred concise close has a median parallel latency of 110 s against 122 s for SC@8. Those figures are for Qwen3.8-27B; for the three Qwen3.5 sets above we report token costs and trigger rates.

The held-out test of our first reply uses Qwen3.8-27B at the 4,096-token cap. There the label-free rule selected concise close on 30 calibration questions, and Table 15 reports +4.8 [1.1, 9.2] for it at equal total tokens. The three sets above are the original exploratory study and the prospective evaluations of the Qwen3.5 portfolio, in which the repairs finish more often than SC samples (Sections 6.1, 6.3 and 6.4).

Reviewer_2WMAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies self-consistency under a head token budget cap, where reasoning chains often exhaust the budget without finishing a answer. It proposes a new strategy to call. It includes some experiments on MATH level 4-5 with Qwen 3.5, showing accuracy improvements over self consistency at the samll call count.

Critical strengths
  1. The paper provides substantial evidence for a conrete failure mode and an effective intervention within its setting. The prospective evaluation is particularly valuable because the initial configuration was selected after observing exploratory results.
  2. The mechanism analysis is informative and interesting.
Critical weaknesses

1.The practival importance of the assumed constraints is not very clear. 2.In general the evaluation doesn't show a significant improvement over baselines.

Questions for authors

1.Under the same per call cap and call budget, how does the portfolio compare with allocating the additional calls to continuing truncated sc chains or prompting them to product a concise final solution? 2.How much of the accuracy advantage remains when comparing methods at matched totoal token cost or measured inference latency?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2WMA

Thank you. We ran both comparisons you asked for on held-out questions with a newer model; the revised PDF reports them in Section 6.6 and Table 13.

The setup: 100 MATH Level 4-5 questions used in no earlier analysis of this project, 3 seeds, Qwen3.8-27B (released after the submission's models), a 4,096-token per-call cap, and the submission's prompts, return policy and automatic evaluator. All policies share five SC samples and differ in the three remaining calls: three more SC samples (SC@8); a greedy chain and two repairs (the portfolio); a continuation of up to three stopped chains (continuation); or a new call that shows a stopped chain and asks for a concise final solution (concise close). The close wording and the choice among these uses were settled on 30 separate calibration questions before the held-out questions were scored.

Q1: continuing truncated chains, or asking them for a concise final solution. Both work, and both are instances of the principle the method rests on (Proposition 1, Section 3): use the replaced calls for candidates that are more likely to finish. Accuracy side by side is 0.948 for concise close, 0.925 for continuation, 0.897 for SC@8 and 0.893 for the portfolio (Table 13). On the 22 of 300 question-seed pairs where none of the five shared SC chains finishes, concise close returns an answer on 19, continuation on 12 and the portfolio on 3. At equal calls the portfolio is at -5.5 [-9.7, -2.0] points against concise close and -3.2 [-5.7, -1.2] against continuation. The repairs are the instance chosen on Qwen3.5, where they finish more often than fresh chains (68% and 57% against 47% in the original study, Section 6.4); for this model the label-free calibration rule selected concise close (Section 6.6). The gain relation of Section 7 predicted +3.8, +2.5 and +0.0 points for concise close, continuation and the portfolio, against observed +5.2, +2.8 and -0.3 (Appendix F). With Qwen3.5, the repairs keep their gain when answers stated in stopped chains are recovered for every candidate (+10.4 and +4.1 points, Table 5).

Q2, total token cost. Prompt-plus-output tokens per pair are 13.1k for SC@8, 16.2k for the portfolio, 14.8k for continuation and 14.0k for concise close. Against SC with the same mean total tokens (a mixture of SC@k and SC@k+1 from the pooled samples at the same cap, Table 15), concise close is at +4.8 [1.1, 9.2] points, continuation at +2.3 [0.2, 5.0] and the portfolio at -1.0 [-2.7, 0.6]. Generating the three concise-close calls only when the five SC samples return no answer uses 5.2 calls and 9.6k tokens per pair at accuracy 0.948, which is +5.6 [1.7, 10.1] against SC at equal tokens.

Q2, latency. We compose the measured service time of each call along its dependencies: repairs wait for the greedy chain, continuations and closes wait for the five SC samples. With every ready call on its own replica, the deferred form of concise close has a median / 90th-percentile latency of 110 / 415 s against 122 / 411 s for SC@8, at accuracy 0.948 against 0.897. Generating all eight calls takes 219 / 710 s for concise close and 214 / 761 s for the portfolio (Table 16). At equal latency, against SC@k for the largest k among 8, 16 and 24 whose median parallel latency does not exceed that of concise close (k = 24), concise close is at +3.8 [0.7, 8.0] points (Appendix F).

W1, practical importance of the constraint. A per-call output cap is how a serving system bounds the latency and cost of each call, and raising it does not remove the problem. At 16,384 tokens a third of the SC chains of Qwen3.5-9B still stop at the cap and SC@8 returns no answer on 41 of 300 pairs (Table 4). In the new comparison, with a newer 27B model, 14% of SC chains stop at 4,096 tokens and SC@8 returns no answer on 19 of 300 pairs.

W2, significance against baselines. Against SC@8 the portfolio gains +3.7 [2.1, 5.6] points on the 475-question archive and +9.4 [5.5, 13.8] and +11.7 [6.3, 17.7] in the prospective evaluation (Tables 3 and 4). Against more samples at the same cap, it exceeds SC@16 by 5.0 [1.6, 8.7] in the original study and by 9.3 [4.0, 15.0] at the prospective 16,384 cap (Table 9). On the held-out questions, concise close, the use of the three calls selected on the calibration questions, improves on SC@8 by +5.2 [1.5, 9.7] points.

AuthorsOpenReview ↗

Update on Q2: comparison at equal mean total tokens

An addition to our answer on Q2. The updated PDF now gives the equal-token comparison for the eight-call portfolio itself, with Qwen3.5-27B and Qwen3.5-9B at the 4,096- and 16,384-token caps (Appendix G, Table 17). When SC receives the same total tokens as extra samples, the portfolio stays ahead by +8.3, +7.2 and +10.9 points in the three sets.

Our first reply gave this comparison for the held-out test, which uses Qwen3.8-27B at the 4,096-token cap; there the label-free rule selected concise close on 30 calibration questions, and Table 15 reports +4.8 [1.1, 9.2] for it at equal total tokens. The three sets here are the original exploratory study with Qwen3.5-27B at 4,096 tokens and the prospective cohort with Qwen3.5-9B at 4,096 and 16,384 tokens, each with 100 questions × 3 seeds (Sections 6.1, 6.3 and 6.4). In these sets the repairs finish more often than SC samples. This is a descriptive comparison made after the results of the paper were known, with no new generation.

SC at equal tokens is defined as in Appendix F: the mixture of SC@k and SC@(k+1) with the same mean prompt-plus-completion tokens per question-seed pair, drawn from the 24 pooled SC samples of a question at the same per-call cap. Intervals are question-bootstrap intervals of the per-question differences.

  • Original study, 4,096 tokens: the portfolio reaches 0.770 against 0.687 for SC at that token count: +8.3 points [4.6, 12.4].
  • Prospective cohort, 4,096 tokens: 0.467 against 0.395: +7.2 points [3.5, 11.4].
  • Prospective cohort, 16,384 tokens, with 87.5k tokens per pair: 0.980 against 0.871: +10.9 points [5.6, 16.7].

The portfolio can also be run with fewer tokens than SC@8. The deferred form, ACDR-Lazy of Section 4, generates the greedy chain and its two repairs only if the five SC samples return no answer, at least four distinct answers, or a plurality below half of their returned votes. On the same generated candidates it returns the same decisions as the eight-call portfolio, so its accuracy is the same. It uses 6.1, 7.0 and 5.5 calls per pair and 14.7%, 0.3% and 27.4% fewer tokens than SC@8 in the three sets. Against the mixture matched to its own lower token count it is at +10.4 [6.3, 14.8], +8.2 [4.4, 12.5] and +12.3 [6.9, 18.2] points.

On the extra calls: the deferred form generates the repairs in 35.9%, 65.7% and 15.3% of the cases in the three sets, counting every pair with every choice of five SC samples. In each of these cases none of the five SC samples returned an answer. These are the cases in which it adds two rounds after the five SC calls, three further calls. For these Qwen3.5 sets we report token costs and trigger rates; Table 16 reports latency composed from the measured service time of each call in the separate Qwen3.8-27B held-out test, as in our first reply.

Reviewer_7Jh2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a method to address the issue of discarding samples that are incomplete during self-consistency. By replacing incomplete samples, self consistency can converge to a specific answer, where it would otherwise abstain due to incompleteness. The method proposed is an an alternative to self-consistency over 8 (SC@8) samples that instead generates 5 self-consistency samples, 1 greedy sample (with temperature 0). The greedy sample is then used to feed into 2 repair samples and the plurality is taken over the 7 voting samples (greedy doesn't vote). The decision rule ADCR essentially ensures that the method never overrides plurality SC@8 when that produces a specific answer. The lazy version of ADCR ensures that repairs are only done when necessary i.e in the cases where self-consistency from the 5 samples returned no answer, or at least four distinct answers, or provided a plurality below half of its returned votes. The results show that this method on a specific dataset for a specific model outperforms SC@8 because the repairs generally tend to be more complete compared to the SC samples.

Critical strengths

Overall, the paper introduces an interesting method for overcoming limitations with self-consistency in cases where the samples are subject to token cap. The method introduced is well motivated and maintains the same budget with SC. The repair approach especially the targeted repair which groups tokens and masks out high entropy token blocks during regeneration seems to be well motivated.

Critical weaknesses

The method really seems to be primarily applicable to cases where there is an output token constraint on the self-consistency samples generated by the models. While this is perhaps common in practice, it seems like an artificial constraint at the sample level since users can always change this. In fact, the caps are varied in the experiments. An interesting angle would have been looking at allocating overall token budget across a varied number of samples, instead of relying on a token cap for each sample, to make the comparisons fair to SC@8.

Second the set of experiments on a single foundation model family (Qwen 3.5) and single dataset do not demonstrate the method generalizes as a way to improve SC at fixed budgets in general. GSM8K is called out but not included in the main paper but in the appendix. Similarly llama results are only reported in the appendix and do not seem to support the conclusions drawn in the main paper.

Questions for authors

Why were experimental results for other datasets and task types not included in the main experiments? Why were comparisons not made for a fixed overall token budget instead of per call caps?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7Jh2

Thank you for the review. In short: the per-call cap is the constraint a serving system sets, the gain remains when SC is given the same total tokens, and the method helps where SC chains run out of room.

W1 and Q2: why a per-call cap, and what a fixed overall budget gives. A per-call output cap is how a serving system bounds the latency and cost of each call, so the paper asks what to do with the calls once that cap is given. We have four comparisons on the overall budget.

Raising the cap. This is the first lever when the deployment allows it (Table 8, Section 7), but it does not remove the problem. At 16,384 tokens a third of the SC chains still stop at the cap and 41 of the 300 question-seed pairs have no SC@8 answer; there the eight-call portfolio gains +11.7 [6.3, 17.7] points over SC@8 (Table 4).

More samples at the same cap. The eight-call portfolio uses 57% of the tokens of SC@16 and exceeds it by 5.0 points [1.6, 8.7] in the original study. It uses 35% of the tokens of SC@24 and exceeds it by 9.0 [3.7, 15.0] at the prospective 16,384 cap (Section 6.4, Table 9).

Fewer tokens than SC@8. If the three extra calls are generated only when the five SC samples return no answer, the decisions are the same as with all eight calls, and the prompt-plus-completion total is 15% and 28% below that of SC@8 in the original study and the archive (Section 6.4).

Equal total tokens. We measured this in a new held-out comparison (Section 6.6 and Tables 13 and 15 of the revised PDF): 100 MATH Level 4-5 questions used in no earlier analysis of this project, 3 seeds, Qwen3.8-27B (released after the submission's models), a 4,096-token per-call cap, and the submission's prompts, return policy and automatic evaluator. All policies share five SC samples and differ in the three remaining calls: three more SC samples (SC@8); a greedy chain and two repairs (the portfolio); a continuation of up to three stopped chains (continuation); or a new call that shows a stopped chain and asks for a concise final solution (concise close). The close wording and the choice among these uses were settled on 30 separate calibration questions before the held-out questions were scored.

Prompt-plus-output tokens per question-seed pair are 13.1k for SC@8, 14.8k for continuation and 14.0k for concise close. Against SC with the same mean total tokens (a mixture of SC@k and SC@k+1 at the same cap, Table 15), concise close is at +4.8 [1.1, 9.2] points and continuation at +2.3 [0.2, 5.0]. Generating the three concise-close calls only when the five SC samples return no answer uses 5.2 calls and 9.6k tokens at accuracy 0.948, which is +5.6 [1.7, 10.1] against SC at equal tokens.

At the same cap and call budget, concise close reaches 0.948 and continuation 0.925 against 0.897 for SC@8: +5.2 [1.5, 9.7] and +2.8 [0.7, 5.7] points. On the 22 of 300 pairs where none of the five shared SC chains finishes, concise close returns an answer on 19 and continuation on 12.

W2 and Q1: other models and tasks. The runs with thinking disabled, with Llama-3.1-8B at 4,096 and 16,384 tokens, and on GSM8K are settings where SC chains mostly finish. SC calls end within the cap on 90%, 92% and 96% of calls, 7, 0 and 0 of 300 pairs have no SC answer, and GSM8K SC@8 is at 0.968. There is no missing answer to supply, which is where the mechanism predicts no gain. The revised PDF moves these runs into the main results with the no-answer rate of each (Section 6.4, new Table 14), because together with the positive results they show when the method applies. The new comparison adds a model generation that the submission did not contain: 14% of its SC chains stop at 4,096 tokens, SC@8 returns no answer on 19 of 300 pairs, and concise close improves on SC@8 by +5.2 [1.5, 9.7] points.

AuthorsOpenReview ↗

Follow-up on Q2: comparison at equal mean total tokens

You asked why the comparison was not made at a fixed overall token budget. Our first reply covered the choice of the cap itself (Table 8). The updated PDF now also compares at equal mean total tokens with the per-call cap held fixed (Appendix G, Table 17): when SC receives the same total tokens as extra samples, the eight-call portfolio stays ahead by +8.3, +7.2 and +10.9 points in the three sets.

The three sets are the original study with Qwen3.5-27B at 4,096 tokens and the prospective cohort with Qwen3.5-9B at 4,096 and 16,384 tokens, each with 100 questions × 3 seeds. This is a descriptive comparison made after the results of the paper were known; it uses the samples behind Tables 2 and 4 and no new generation.

For each policy we take its mean prompt-plus-completion tokens per question-seed pair and compare it with SC given the same mean: a mixture of SC@k and SC@(k+1) drawn from the 24 pooled SC samples of a question at the same cap, so that SC spends the same average total tokens on more samples. Intervals are question-bootstrap intervals of the per-question differences.

  • Original study, 4,096 tokens: the portfolio uses 14.8% more tokens than SC@8 and reaches 0.770 against 0.687 for SC at that token count: +8.3 points [4.6, 12.4].
  • Prospective cohort, 4,096 tokens: it uses 14.8% more tokens than SC@8 and reaches 0.467 against 0.395: +7.2 points [3.5, 11.4].
  • Prospective cohort, 16,384 tokens, the largest cap in the paper: it uses 87.5k tokens per pair, 5.7% more than SC@8 at 82.7k, and reaches 0.980 against 0.871: +10.9 points [5.6, 16.7]. SC with all 24 pooled samples of a question reaches 0.890 at 248k tokens.

The mixtures build on SC@8 among the pooled samples, which is at 0.676, 0.386 and 0.870 in the three sets. SC@8 within each seed, the comparator of Tables 2 and 4, is at 0.670, 0.373 and 0.863, and against it at equal calls the portfolio is at +10.0 [5.8, 14.5] (Table 2) and +9.4 [5.5, 13.8] and +11.7 [6.3, 17.7] (Table 4). In the equal-token comparison SC both receives the extra tokens and is estimated from the pooled samples.

The portfolio can also be run with fewer tokens than SC@8. The deferred form, ACDR-Lazy of Section 4, generates the greedy chain and its two repairs only if the five SC samples return no answer, at least four distinct answers, or a plurality below half of their returned votes. On the same generated candidates it returns the same decision as the eight-call portfolio for every pair and every choice of the five SC samples, so its accuracy is that of the eight-call portfolio. It uses 6.1, 7.0 and 5.5 calls per pair and 14.7%, 0.3% and 27.4% fewer tokens than SC@8 in the three sets. Against the mixture matched to its own lower token count it is at +10.4 [6.3, 14.8], +8.2 [4.4, 12.5] and +12.3 [6.9, 18.2] points; with SC@k and SC@(k+1) enumerated within each seed, at +11.0 [6.7, 15.6], +9.4 [5.5, 13.8] and +12.7 [7.3, 18.7].

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.