acceptodds
Under review as a conference paper at ICLR 2027

Parallel Test-Time Scaling with Multi-Sequence Verifiers

Abstract

Parallel test-time scaling—generating multiple candidate solutions for a single problem—is a powerful technique for improving large language model performance, but it faces two related challenges: selecting the correct solution from a large candidate pool is difficult, and generating every candidate to completion can be slow. Predicting candidate correctness can address both problems by guiding answer selection and informing when to stop generation. However, verifiers that score candidates independently cannot exploit the relationships between candidates' underlying representations. To address this, we introduce the Multi-Sequence Verifier (MSV), a lightweight verifier that predicts each candidate's correctness by jointly attending over the hidden representations of all sampled candidates, rather than processing each one in isolation. We apply MSV's predictions to best-of-N selection, confidence estimation, and parallel early stopping, evaluating each use case separately. Across challenging mathematical reasoning benchmarks, MSV improves best-of-64 accuracy by up to 6% relative to the strongest baselines, and a streaming variant of MSV improves the estimated accuracy-latency tradeoff over the evaluated baseline verifiers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, test-time compute, verifiers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_idSYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

To reduce the difficulty of selecting the correct answer from a large number of candidates and to alleviate the issue of slow candidate generation, this paper proposes the Multi-Sequence Verifier (MSV), a lightweight verifier that predicts each candidate’s correctness by jointly attending over the hidden representations of all sampled candidates, rather than processing each one in isolation. MSV has shown effectiveness in mathematical reasoning tasks, in some cases outperforming the best baseline's "best-of-64" performance by approximately 6 percent, and a streaming variant of MSV improves the estimated accuracy-latency tradeoff.

Critical strengths
  1. The method proposed in the paper serves multiple functions: it guides the selection of answers while simultaneously facilitating early stopping.
  2. The method designed in the paper is lightweight.
  3. The paper is well written.
Critical weaknesses
  1. It may be beneficial to discuss whether model improvements stem from comparing the content of different responses or from leveraging information about the ordinal positions of the candidates. In other words, given that the model employs per-sequence embeddings, it might be beneficial to verify whether the model's final predictions remain stable with respect to the numbering of the candidates.
  2. Can a verifier trained on a specific NN generalize better to different NN values? (This is distinct from simply reusing grouped segments in integer multiples, the two concepts seem to differ slightly). One possible example is to train on N=64, and then evaluate and test on N<64, rather than retraining or using grouping.
  3. The majority of experiments are based on models within the 1.x billion parameter range, and the baselines used in the paper are generally not the most up-to-date (the newest is GenSelect, from 2025, in the appendix). The authors are encouraged to test generalizability using SOTA baselines and models (for baselines, some relevant baesline such as TrajSelector (arXiv:2510.16449), some recent baselines such as CAPS (arXiv:2605.15513) and One-Token Verification (arXiv:2603.01025). NOTE: These are merely examples of baselines. They are not intended as mandatory points of comparison. However, the baselines currently included in the paper do not seem sufficiently comprehensive.), which helps to show the potential and novelty of the proposed method in the manuscript. Moreover, the tested datasets are mainly mathematical tasks. The performance of the proposed method on other datasets remains unclear.
  4. Is MSV still useful for tasks where reliable tools like SymPy are not available?
Questions for authors

Please see the Critical Weaknesses.

5 comments

We thank the reviewer for the feedback and questions.

W1. Sensitivity to candidate ordering

We tested 100 permutations on each of the problems on AIME, and found that the mean change in best-of-64 relative to the original ordering had 95% confidence interval [−0.18%, +1.22%]. This suggests that the ordering was not a key contributing factor to MSV’s performance. A remaining question is how much variance there is in MSV’s predictions, subject to permutations. We found that the standard deviation of best-of-64 accuracy across the 100 permutations was 0.24%, indicating a relatively small variance. For context, the standard deviation of best-of-64 accuracy across five MSV checkpoints was 0.73%. Moreover, across pairs of permutations, MSV selected different answer classes on only 2% of problems on average.

W2. Generalization across candidate-set sizes

We agree this is an important question, and it is addressed in Appendix B.7. We copy the weights of an MSV trained with N=64N=64 into an N=16N=16 model, retain the first 16 sequence embeddings, and apply it to groups of 16 candidates without retraining or partitioning the candidates into fixed-size subgroups. Across the five benchmarks, the difference between direct transfer and separate N=16N=16 training ranges from −0.46-0.46 to +0.36+0.36 percentage points. These results show that the candidate-set size used during training does not fix the inference candidate-set size, and that MSV transfers directly from N=64N=64 to N=16N=16 without a meaningful loss in observed accuracy.

W3. Model scale, baseline coverage, and task diversity

Regarding the lack of experiments with models larger than 1.x billion parameter models, Section 4.5 and Appendix D.4 report experiments with three additional base models, including DeepSeek-R1-Distill-Llama-8B. MSV consistently achieves the highest mean best-of-64 accuracy among the compared methods.

Regarding the lack of baselines, the paper compares MSV against independent outcome verification with Probe and MSV$_1$, the Qwen2.5-Math-PRM-7B process reward model, self-consistency, weighted-voting variants, the generative joint selector GenSelect, and the parallel early-stopping methods ST-BoN and Parallel-Probe. These baselines cover independent verification, process verification, voting, generative joint selection, trajectory pruning, and parallel early stopping.

We provide two additional baselines, CAPS (arXiv:2605.15513) and TrajSelector (arXiv:2510.16449) on the five math reasoning benchmarks.

Table R.5 Best-of-16 accuracy compared with recent test-time scaling baselines

DatasetPass@1CAPS N=16N=16TrajSelector N=16N=16Probe best-of-16MSV16 best-of-16
AIME27.48%33.48%37.89%44.70%46.15%
MATH67.55%61.38%70.93%73.45%74.72%
OlympiadBench41.05%44.98%45.54%50.39%51.54%
Omni-MATH24.32%26.90%28.29%31.76%31.98%
AMC1239.65%44.22%47.57%51.98%53.99%

MSV16 achieves the highest accuracy on all five benchmarks, outperforming both CAPS and TrajSelector. These results provide a broader comparison against more recent verification and test-time scaling methods. We thank the reviewer for this suggestion, and will make sure to incorporate these results into our revised manuscript.

Regarding task diversity, Appendix B.13 trains the verifiers on 224 ARC-Challenge questions and evaluates them on 448 MMLU test questions. MSV16 achieves the highest best-of-16 accuracy of 54.0%, compared with 49.0% for MSV$_1$ and 48.5% for Probe.

As detailed in our response to Reviewer DMgB, we have also added experiments on MBPP+ code generation and TL;DR free-form summarization. MSV16 achieves the highest best-of-16 performance in both settings, reaching 66.15% accuracy on MBPP+ and a mean reward of 3.6391 on TL;DR. Together with the MMLU results, these experiments extend our evaluation beyond mathematical reasoning to multiple-choice knowledge, complete program generation, and free-form text generation.

W4. Reliance on symbolic equivalence

We agree this is an important question, and we address this concern in Section 4.4 and Appendix D.3. We replaced SymPy with approximate textual and semantic equivalence checkers and retrained MSV with the resulting signals. With Jaccard or embedding similarity, MSV remains within 0.4 percentage points of the matched SymPy variant, showing that it can retain its advantage with approximate agreement signals.

As detailed in our response to Reviewer DMgB, our new MBPP+ and TL;DR experiments provide further evidence. Both experiments replace symbolic equivalence with a continuous agreement feature based on Jaccard similarity between candidate outputs. This allows MSV to model agreement among complete programs and free-form summaries, for which exact symbolic equivalence is unavailable. MSV16 with continuous agreement achieves the highest best-of-16 performance on both tasks.

Together, these results show that MSV does not depend on SymPy or exact answer equivalence. Its agreement mechanism can instead use approximate, domain-appropriate similarity measures for outputs with multiple valid forms.

We hope these address the critical weaknesses and that the reviewer will reconsider the assessment.

Dear reviewer idSY, we thank you again for the insightful questions and feedbacks. We have posted a response to your concerns and a revised version of our manuscript. We would greatly appreciate any further feedback or remaining concerns you might have.

Reviewer_DMgBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Multi-Sequence Verifier (MSV) to improve candidate answer selection and early stopping in parallel test-time scaling for large language models. Unlike conventional verifiers that evaluate each candidate answer independently, MSV uses a lightweight multi-mask Transformer to jointly model the hidden representations of multiple candidate answers, answer consistency, and inter-candidate relationships, thereby predicting the correctness probability of each candidate answer.

The authors apply MSV to best-of-N selection, confidence estimation, and early stopping in parallel generation. Experiments on multiple mathematical reasoning benchmarks show that MSV can improve answer selection accuracy and confidence calibration for best-of-64. Its streaming version also evaluates intermediate answers during generation and terminates all parallel sequences when the correctness probability of a candidate exceeds a threshold, thereby improving the trade-off between accuracy and inference latency.

Critical strengths
  1. The method design is sound. MSV jointly leverages the hidden representations of multiple candidate answers, answer consistency, and inter-candidate relationships, thereby preserving richer cross-sequence information than independent scoring or simple voting.

  2. The experiments are comprehensive and demonstrate clear practical effectiveness. The paper validates answer selection, confidence calibration, and parallel early stopping on multiple mathematical reasoning benchmarks. The results show that MSV can simultaneously improve best-of-N accuracy and improve the accuracy–latency trade-off.

Critical weaknesses
  1. The experimental scenarios are overly idealized and cannot support the claim of the method’s generality. All main experiments focus on mathematical reasoning datasets, where tasks typically have unique, short answers that can be automatically verified via rules or symbolic tools, making it possible to obtain large-scale, low-noise correctness labels and answer-equivalence relations at low cost. The paper does not evaluate on tasks where verification is genuinely difficult, such as code generation, open-ended question answering, or long-form text generation, and therefore has not yet demonstrated that MSV can address broader test-time scaling problems.

  2. The method relies heavily on answer extraction and equivalence judgment specific to mathematical tasks. MSV explicitly uses equivalence relations among candidate answers and consistency ratios, information that is easy to compute reliably in short-answer tasks such as MATH and AIME. In real-world tasks where answers have multiple valid expressions, cannot be accurately extracted, or lack clear equivalence criteria, these core inputs may be very noisy or even unavailable. Therefore, the current performance gains may mainly come from idealized answer-grouping signals rather than truly general cross-candidate reasoning ability; the paper lacks key validation without using equivalence information and on non-mathematical tasks.

Questions for authors

See the above weakness. My major questions is about the narrow experimental scenarios.

2 comments

We agree that the experiments on math datasets alone do not establish applicability to every domain. As such, we performed two additional experiments that address the concerns: MBPP+ (Austin et al., 2021; Liu et al., 2023), which is a coding task where outputs are Python functions, and TL;DR summarization (Völske et al., 2017; Stiennon et al., 2020), where the task is to summarize Reddit posts.

Meanwhile, we would also like to clarify that we used mathematical reasoning as our main benchmark because it is a widely established setting in the test-time scaling literature that permits controlled comparison using reliable correctness labels (Cobbe et al., 2021; Zhang et al., 2024; Snell et al., 2025; Toshniwal et al., 2025; Di et al., 2026; Wang et al., 2026). Nonetheless, we agree that including a more diverse suite of tasks would help establish the broad applicability of MSV, and thank the reviewer for this valuable feedback. We report the experimental setup and results on MBPP+ and TL;DR below.

Code generation. We evaluated best-of-16 program selection on MBPP+. We generated responses with Qwen2.5-Coder-1.5B-Instruct and used 108/39/224 problems for training, validation, and evaluation. Across the evaluation candidates, the median program contained 62 tokens, and the 90th percentile contained 140 tokens, compared with 2 and 9 tokens for MATH answers. Because functionally equivalent programs need not be symbolically identical, we replaced discrete answer groups with a continuous agreement feature based on Jaccard similarity.

Table R.3 Best-of-16 program-selection accuracy on MBPP+

MethodMBPP+ best-of-16 accuracy (%)
Pass@159.39
Probe64.96 ± 1.46
MSV$_1$62.54 ± 0.79
MSV$_{16}$66.15 ± 1.44

Free-form summarization. We further evaluated MSV on the TL;DR summarization task using Pythia-1B SFT (Huang et al., 2024). We used 448/224/224 training, validation, and test posts. A frozen preference-trained reward model (Huang et al., 2024) supplied summary-quality scores.

Table R.4 Best-of-16 mean reward on the TL;DR summarization task

MethodTL;DR best-of-16 mean reward
Random selection2.2057
Probe3.5973 ± 0.0155
MSV$_1$3.6213 ± 0.0063
MSV$_{16}$3.6391 ± 0.0210

In both experiments, we replace binary equivalence judgements with continuous Jaccard similarity scores. Since we can no longer perform hard grouping, we remove the equivalence class attention mask and equivalence class logit averaging. The similarity scores are used for the feature γk(n)\gamma^{(n)}_k (Section 3.2 Feature augmentation). Additionally, for the TL;DR task, we replace binary correctness labels with continuous scores from the frozen reward model. Probe and MSV are trained by regression to these scores, and the summary with the highest predicted reward is selected. Together, the results show that MSV can handle non-mathematical tasks with long, open-ended outputs and no symbolic equivalence checking.

We hope these address the critical weaknesses and that the reviewer will reconsider the assessment.

References:

  • Di et al., 2026. “Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scaling”.
  • Wang et al., 2026. “OptScale: Probabilistic Optimality for Inference-time Scaling”.
  • Austin et al., 2021. “Program Synthesis with Large Language Models”.
  • Liu et al., 2023. “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.”
  • Völske et al., 2017. “TL;DR: Mining Reddit to Learn Automatic Summarization”
  • Stiennon et al., 2020. “Learning to Summarize from Human Feedback.”
  • Huang et al., 2024. “The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization”.

Dear reviewer DMgB, we thank you again for the insightful questions and feedbacks. We have posted a response to your concerns and a revised version of our manuscript. We would greatly appreciate any further feedback or remaining concerns you might have.

Reviewer_mhwuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Multi-Sequence Verifier, which jointly evaluates multiple reasoning candidates using their hidden representations and answer agreement. It improves answer selection and confidence estimation while enabling earlier termination of parallel reasoning.

Critical strengths

1.Jointly verifying multiple reasoning trajectories is an interesting alternative to independent verification and majority voting.

2.The method combines answer selection and early stopping, addressing both accuracy and inference efficiency.

3.The experiments cover several mathematical benchmarks and base models, with useful ablations and comparisons against existing verifiers.

Critical weaknesses

1.The paper lacks a systematic evaluation of whether MSV can identify correct answers when most candidates agree on the same incorrect answer.

2.In streaming mode, earlier answers cannot be rescored using evidence from later candidates, potentially limiting selection accuracy.

3.The early-stopping rule relies on current confidence without considering whether unfinished trajectories could still produce better answers.

Questions for authors

1.When an incorrect answer dominates the candidate pool, how often can MSV recover the minority correct answer compared with majority and weighted voting?

2.Could selectively rescoring leading candidates as new answers arrive improve accuracy, and at what computational cost?

3.Among incorrectly answered early-stopped problems, how many could be corrected by continuing generation? Would keeping a few promising trajectories active offer a better tradeoff?

5 comments

We thank the reviewer for the constructive feedback and suggestions for additional analysis.

Q1. When an incorrect answer dominates the candidate pool, how often can MSV recover the minority correct answer compared with majority and weighted voting?

We conducted an analysis with N=16N=16, focusing on cases where some incorrect answer appears more than 8 times (i.e. more than half the time), while at least one correct candidate exists.

The following table reports correct-answer recovery rates, averaged over five training seeds.

Table R.1 Minority-correct-answer recovery rate under an incorrect majority answer

DatasetGroupsMajority VotingWeighted Voting w/ ProbeMSV16
MATH410%20.5%41.5%
OlympiadBench920%13.3%32.4%
AMC12300%38.0%56.7%
AIME720%46.7%66.1%
Omni-MATH870%14.5%31.3%

Majority voting always yields 0% recovery rate since it selects the incorrect answer that appeared more than 8 times. MSV also does consistently better than weighted voting at recovering correct answers despite the incorrect majority, substantiating the claim that weighted voting overly suppresses correct but rare answers, while MSV can recover those answers more successfully. We agree that adding this systematic evaluation will benefit our paper and thank the reviewer for the suggestion.

Q2. Could selectively rescoring leading candidates as new answers arrive improve accuracy, and at what computational cost?

We implemented a rescoring variant of MSV and evaluated it on all 448 AIME problems. The rescoring variant replaces the causal attention masks with full attention over the answers observed at the current point in generation. At every step of training, we randomly sample one cutoff and train the verifier to score every answer up to that cutoff, with full attention. Thus, the model learns to update an earlier answer’s score using candidates that arrived later, without accessing answers that had not yet been generated. We train the rescoring verifier under the same training wall-clock budget as the original MSV verifier.

At inference, whenever new answers arrive, we first score them and then selectively update earlier scores using all answers observed so far. At N=4N=4, we rescore the latest answer from each of the four sequences. At N=16N=16 and N=64N=64, we take the latest answer from each sequence and rescore the four with the highest current scores.

Although the verifier is trained with full attention over each sampled prefix, inference does not require recomputing the entire prefix. MSV applies only a single attention layer to fixed LLM representations, so we can cache the keys and values of all observed answers and recompute the attention outputs only for the selected candidates.

The figure below compares the resulting accuracy–token-position tradeoffs. Rescoring improves some shorter-budget operating points but does not consistently shift the tradeoff frontier outward.

Figure R.1 Accuracy–token-position tradeoff curve of original versus rescoring MSV

https://anonymous.4open.science/api/repo/iclr27-msv-discussion-E582/file/rescoring_tradeoff.png

At N=4,16,N=4, 16, and 6464, respectively, the total verifier runtime per problem was 1.54×, 1.64×, and 1.79× higher for selective rescoring compared to the original MSV.

We thank the reviewer for the suggestion and will include this experiment in the revised manuscript.

Q3. Among incorrectly answered early-stopped problems, how many could be corrected by continuing generation? Would keeping a few promising trajectories active offer a better tradeoff?

This is an interesting suggestion. In Section 4.3, we compare MSV with ST-BoN, which retains one trajectory for continued generation after stopping the others. Our work focuses primarily on improving the decision of when to stop parallel decoding, while combining MSV with selective continuation was left as a future research direction.

First, the potential for improved accuracy depends on the stopping threshold λ\lambda. For the lenient values (e.g. λ=0.96875\lambda=0.96875) that we used for the main headlines of our paper, accuracy barely degrades compared to full generation, making it unlikely that selective continuation helps improve the accuracy. On the other hand, for more aggressive values (e.g. λ=0.75\lambda=0.75) that sacrifice accuracy for even lower latency, it is possible for selective continuation to improve accuracy.

We evaluated two simple continuation policies. After MSV reaches its stopping point, Top-4 continues up to four unfinished trajectories with the highest latest MSV scores, while Random-4 continues up to four randomly selected unfinished trajectories.

Table R.2 Accuracy and estimated decoding latency with selective continuation on AIME

Stopping thresholdPolicyAccuracyEstimated latency
λ=0.96875\lambda=0.96875Early stop52.23%108.6 s
Random-4 continuation48.21%120.6 s
Top-4 continuation52.23%120.4 s
λ=0.75\lambda=0.75Early stop47.99%99.2 s
Random-4 continuation43.53%115.0 s
Top-4 continuation48.66%114.9 s
Full generation—52.90%139.6 s

Top-4 outperforms Random-4 at both thresholds λ\lambda, indicating that MSV scores can identify more promising unfinished trajectories. However, selective continuation does not improve the overall accuracy–latency tradeoff. At λ=0.75\lambda=0.75, Top-4 reaches 48.66% accuracy at 114.9 seconds, while early stopping with λ≈0.97\lambda\approx0.97 achieves higher accuracy of 52.23% at a lower latency of 108.6 seconds. Applying Top-4 after stopping with λ≈0.97\lambda\approx0.97 leaves accuracy unchanged while increasing latency.

We therefore find that our original early stopping policy with lenient threshold Pareto dominates selective continuation. Still, we think this is an interesting direction for future research that could further improve the accuracy-latency tradeoff, and thank the reviewer for the suggestion.

We hope these address the critical weaknesses and that the reviewer will reconsider the assessment.

Dear reviewer mhwu, we thank you again for the insightful questions and feedbacks. We have posted a response to your concerns and a revised version of our manuscript. We would greatly appreciate any further feedback or remaining concerns you might have.

Comments on the paper

1 comment

We thank the reviewers for the detailed feedback. We have uploaded a revised manuscript that incorporates the additional analyses and experiments discussed in our responses. All new material is shown in red.

The main changes are:

  1. Incorrect-majority recovery and candidate ordering (Section 4.2; Appendix B.2). We added a systematic analysis of groups in which an incorrect answer forms a strict majority while a correct candidate remains available. Across the five mathematical benchmarks, MSV recovers the minority correct answer in 31.3--66.1% of these cases, compared with 13.3--46.7% for Probe with weighted voting and 0% for majority voting. We also added the 100-permutation AIME analysis: permutation-level accuracy has a standard deviation of 0.24 percentage points, and different permutations select different answer classes on only 2% of problems on average.

  2. Recent test-time scaling baselines (Section 4.2; Appendix B.2). We added best-of-16 comparisons with CAPS and TrajSelector. MSV obtains the highest mean accuracy on all five mathematical benchmarks in this comparison.

  3. Selective rescoring and continuation (Section 4.3; Appendix B.3). We added experiments that update selected earlier scores as new answers arrive. Rescoring improves some short-budget operating points but does not consistently move the accuracy--token-position frontier outward, while increasing verifier runtime by 1.54x--1.79x. We also added Random-4 and Top-4 continuation experiments. Top-4 is better than random continuation, but the original MSV stopping rule with a lenient threshold remains Pareto-superior in the evaluated accuracy--latency comparison.

  4. Evaluation beyond mathematical reasoning (Section 4.6, Table 5; Appendix B.4). We added best-of-16 experiments on MBPP+ code generation and TL;DR free-form summarization. These settings replace symbolic equivalence with continuous Jaccard agreement; TL;DR additionally replaces binary correctness with frozen reward-model scores. MSV obtains the highest mean performance in both settings: 66.15% accuracy on MBPP+ and 3.6391 mean reward on TL;DR.

We also updated the abstract, introduction, related work, and conclusion to reflect the broader task evaluation and the new comparisons. We thank the reviewers for suggesting these analyses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.