- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a test-time scaling method for reasoning models that sharpens the marginal probability of the final answer rather than the joint probability of the reasoning trace and answer. Using the identity that the K-th power of the answer marginal m(a|x) equals the expected product of answer probabilities across K independent reasoning traces, the authors build a sequential importance sampler. The algorithm samples K reasoning traces in parallel up to a delimiter token, decodes a single answer from the normalized product of the K next-token distributions, and weights P such particles by the product of their per-token normalizers. Because these weights are bounded by 1, the authors can derive a finite-particle total variation bound relative to the sharpened marginal target. They test the method on math and code benchmarks across six small-to-medium language models and compare against temperature sampling, MCMC power sampling, and majority voting.
Critical strengths
(1) The multi-trace decomposition gives a fast, one-pass alternative to MCMC power sampling that scales to long reasoning traces. Decoding from the per-token product of K trace-conditioned distributions makes the incremental weight rho_t <= 1, so the sampler runs as one batched autoregressive pass with no lookahead. On MATH-500 with Qwen3-4B at T=8192 this cuts 75th-pct runtime from ~100 min per problem for power sampling to under 8 and raises accuracy from 69% to 84%.
(2) The particle scaling ablation in Figure 2 shows that the importance weights add real signal beyond the proposal. At fixed K=4, uniform selection over P particles would match P=1, so the 4 to 9 point gains from P=1 to P=8 come directly from resampling by
Critical weaknesses
(1) The exact-string marginal target in Eq 4 does not solve surface-form fragmentation on code, and the actual decoding step is very close to prior logit-aggregation methods that are not compared. Raising m(a|x) to a power K still concentrates on the single most likely string rather than pooling equivalent programs. What helps on code is the P=1 proposal, which sums next-token log-probs across K traces like Integrative Decoding [1] and ThinkMerge [2]. In Fig 3, K=32, P=1 with no importance weights is within 1 point of the best setup on four of five tasks, and the TV bound in Prop 3.1 exceeds 1 at K=8,P=4 and K=16,P=2 unless one string already has >80% base mass.
(2) The code baselines in Table 1 collapse on MBPP+, and an unstated detail on trace truncation in Algorithm 1 may confound the comparison. On Qwen2.5-7B, majority voting over 32 samples drops to ~22% versus 35% for a single sample. Voting on raw code strings is also an odd baseline here since the prompt includes public tests that could be used to filter or cluster programs by execution [3].
Also, Algorithm 1 samples each trace until the delimiter and then runs a seperate answer phase, but never says what happens if a trace loops until T as in App. C.2. If the delimiter is force-appended, marginal sharpening gets a budget-forcing rescue [4] while looping baseline runs fail extraction.
ref: [1] Cheng et al., Integrative Decoding: Improving Factuality via Implicit Self-Consistency, ICLR 2025.
[2] Wang et al., Think in Parallel, Answer as One: Logit Averaging for Open-Ended Reasoning, ICLR 2026...
[3] Shi et al., Natural Language to Code Translation with Execution, EMNLP 2022.
[4] Muennighoff et al., s1: Simple test-time scaling, arXiv:2501.19393, 2025.
Questions for authors
(1) In Algorithm 1, what happens when a sampled reasoning trace hits the length limit T without emitting the answer delimiter? Do you force-append the delimiter and still run the answer decoding loop, and how often does this happen on MBPP+ for Qwen2.5?
(2) How concentrated are the normalized particle weights for K=8, P=4 and K=16, P=2, eg effective sample size or how often the top particle takes >90% of the weight? If one particle almost always dominates, how does the method compare to verifier-free best-of-N rerankers like self-certainty [5]?
(3) Are Qwen2.5 7B and 14B in Table 1 the base or Instruct checkpoints? Please also compare against Integrative Decoding [1] or ThinkMerge [2] at K=32 and an execution-filtered vote using the public assert statements in the prompt.
ref: [5] Kang, Zhao, Song, Scalable Best-of-N Selection for Large Language Models via Self-Certainty, arXiv:2502.18581, 2025.
3 comments
Thanks for running extra experiments and being so transparent with the numbers. However, the new results reinforce my main concerns: 1. ESS of ~1.0 confirms that at P=2 or 4, the method isn't really sampling from the sharpened marginal distribution as the theory suggests, but simply doing argmax selection over a few ThinkMerge-style proposals. 2, on math it still trails simple Majority Vote, and on coding, once basic public-test filtering is applied, the large gap over Majority Vote mostly disappears. Given this mismatch between the theoretical framing and the practical results, I will keep my score
We thank the reviewer for this follow-up comment and respectfully disagree with this interpretation of our results.
1. Low ESS does not invalidate the target or the importance correction. The exact theoretical result concerns the multi-trace decomposition; the finite-particle sampler is explicitly an approximation. Concentrated weights suggest that this approximation may be coarse when using few particles. We agree that selection is then close to an argmax, but an argmax of the importance weights, which encode the target-to-proposal correction, rather than of the model probability. The correction is therefore decisive in determining which particle is selected. Crucially, Figure 2 shows that increasing P at fixed K improves the results, whereas uniform selection among particles would leave the output distribution unchanged. MS also outperforms both ThinkMerge and self-certainty in the requested comparisons. Together, these findings support the value of the proposed target and its importance correction beyond logit aggregation or alternative reranking, while showing that improving the particle approximation improves performance.
2. Execution feedback changes the setting being evaluated. Access to execution feedback can naturally level performance, but our goal is to improve inference without that feedback, the setting shared by all original baselines. On coding tasks, MS achieves the strongest results among the compared methods in this setting, including Power Sampling and temperature sampling. Adding execution feedback provides an additional capability; the resulting convergence in performance does not negate the improvement achieved without it. We already acknowledge majority voting’s strength on mathematics, and our theory makes no claim of universal superiority over voting, since both pursue closely related self-consistency objectives.
We thank the reviewer for recognizing the efficiency of the multi-trace construction and the evidence from particle scaling. Below we address the main questions:
1) Traces without an answer delimiter.
We do not force-append the answer delimiter. Instead, completions without a delimiter are excluded from the usable trace pool. Then each particle is constructed from K differnt traces that all contain a delimiter. In the MBPP+ source pools, 21.94% of Qwen2.5-7B completions and 8.55% of Qwen2.5-14B completions lacked the delimiter. If less then K traces contain a delimiter we reduced K as a fallback, but this only happened in 0.12% of the MBPP+ problems.
This matches the provided code, and we will clarify the procedure in the appendix.
2) P=1 proposal and ThinkMerge.
We would like to argue that the P=1 proposal is not the best method, as also mentioned in the critical strengths you identified. In Figure 3, K=8, P=4 is substantially better than K=32, P=1 on SxH. The importance weights are crucial for this improvement. At fixed K=4, Figure 2 shows gains of 4–9 percentage points as P increases from 1 to 8, highlighting the importance of the target and its importance weights beyond a strong proposal alone.
We compare MS with ThinkMerge on HumanEval+ and MBPP+ using Qwen2.5-14B and the same source pools of 32 responses. ThinkMerge aggregates up to K=32 usable traces with answer temperature 0.6. Results are averaged over five seeds. MS outperforms ThinkMerge on both benchmarks with both (K,P) configurations.
3) Weight concentration and self-certainty.
Indeed, the normalized particle weights are strongly concentrated. For (K,P)=(8,4) and (16,2) mean effective sample size ranges from 1.003 to 1.166, and the largest weight exceeds 90% in 78.3--99.7% of cases.
However, self-certainty did not outperform MS. We evaluated self-certainty on 32 saved base-model responses per MBPP+ problem, following the authors’ released scoring implementation. Self-certainty is computed under the base model, and the highest-scoring response is selected. Both methods use a nominal budget of 32 paths (N=32 for self-certainty, KP=32 for MS) and a maximum generation length of 4096 tokens.
4) Coding baselines and public tests.
We compare execution-filtered voting on MBPP+ using the same 32 candidates as ordinary voting. For a fair comparison with access to the same public tests, we also evaluate two MS variants: filtering final particles by setting failing particles' weights to zero before renormalizing and selecting, and filtering the initial 32 responses before using their reasoning traces in MS. The latter uses up to K usable passing traces per particle and ordinary weighted final selection. If no checks can be extracted or no usable candidate passes, each method falls back to its unfiltered procedure. As you can see from the table, MS with filter outperforms filtered voting. Moreover, original MS is still very competitive with filtered majority voting, despite not executing the public tests.
Not executing public tests during decoding is a deliberate choice. Our goal is to compare the models’ underlying reasoning abilities, rather than augment them with tools or execution feedback. This allows us to assess whether our method improves performance solely by changing the inference target. None of the original baselines executes public tests during decoding either, ensuring a fair comparison.
5) Qwen checkpoints.
Both are base checkpoints: Qwen/Qwen2.5-7B and Qwen/Qwen2.5-14B. We will include these exact identifiers in the experimental setup.
We thank the reviewer for the constructive feedback, which helped us clarify our contribution and strengthen the experimental comparisons. We hope these clarifications and additional results address your concerns, and kindly ask you to reconsider the score in light of them.