- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Parallel Power Tempering applies interacting replicas at different powers to sequence-level sharpened language-model distributions. A fixed-horizon construction corrects one-way shortening in prior samplers, while reversible suffix proposals and replica swaps preserve the intended stationary target. Experiments compare reasoning accuracy, sampling controls, communication schedules, and runtime/memory costs.
Critical strengths
- The fixed-horizon correction is technically important. The paper identifies a specific reverse-support failure and constructs proposals on a common record space, making the target-preservation argument materially stronger than an informal tempering analogy.
- The fixed-horizon single-chain, uncoupled-ladder, and token-budget-matched comparisons isolate useful parts of the empirical gain. The explicit memory and local-update costs also make the resource trade-off assessable.
Critical weaknesses
- Stationary-target preservation does not establish that the few refinement steps used in practice produce an approximately equilibrated output. Block extensions initialize a different distribution, and the paper itself separates finite-budget error from invariance. Stronger practical mixing evidence is needed to attribute accuracy gains specifically to sampling the intended sharpened target.
- The frontier-model comparison uses reported external scores rather than a matched inference protocol, while the strongest small-model settings use very large completion horizons and multiple internal proposals. Frontier comparability is therefore a limited benchmark observation rather than demonstrated resource-matched reasoning parity.
Questions for authors
- How do output statistics and accuracy change with longer refinement, different initializations, and replica round-trip diagnostics at the reported settings?
- Does the small-model advantage persist against matched latency/token-budget baselines under identical prompts and completion limits?
2 comments
Author Response to Reviewer zi5q (Part 3/3)
4. Longer refinement, initialization and round trips
Longer refinement. Table C varies the number of refinement steps per block, , from 0 (the initial trace, before any refinement) to 50, with Qwen3-8B. Accuracy rises quickly and then saturates. On MATH500, the seed-averaged accuracy increases within the first ten steps and then stays at about 88 all the way to , with the fluctuations on the plateau within seed noise. HumanEval follows the same pattern. Intuitively, the first few steps do most of the work because they correct the errors that are cheap to correct: an initial trace that went wrong through an unlucky low-probability step is quickly replaced by one that the sharpened target prefers. Once the chain has reached the high-probability region of its target, further steps only move among traces of comparable quality. The errors that remain are those where the model itself favors a wrong answer, which additional sampling from the same target cannot be expected to repair.
We draw two main conclusions: (i) a small budget captures most of the benefit, so the method does not rely on long chains; and (ii) extending the budget well beyond that point neither helps nor hurts. The absence of late degradation is what one expects from a sampler with a fixed target distribution, as opposed to an optimizer that keeps sharpening its output and eventually over-commits.
Initialization. We also examine how strongly the output depends on the initial sequence. We re-ran both sweeps with every chain's initial state drawn at temperature 1.0 (more diffuse than any rung) or 0.2 (sharper than any rung), instead of the rung's own . The target, local moves and swaps are unchanged. Tables E and F show that even when the replicas begin from deliberately diffuse or sharp distributions, their likelihood trajectories rapidly draw together. Refinement removes roughly 90% of the initial separation, with half of the diffuse start's gap to the default disappearing within just 2–3 steps. Most of the initial separation therefore disappears within the reported budget, showing substantial forgetting of the starting conditions. The different starts also produce similar final accuracy. Together, these results support substantial forgetting of initialization and are consistent with convergence toward a common output distribution. We visualize these tables in the revision (App. H.3).
Table E. Accuracy (%) vs. for three initializations, Qwen3-8B. mean ± std over 5 seeds.
| MATH500 default | MATH500 diffuse | MATH500 sharp | HumanEval default | HumanEval diffuse | HumanEval sharp | |
|---|---|---|---|---|---|---|
| 0 | 83.4 ± 0.7 | 83.0 | 81.2 | 86.0 ± 1.1 | 83.8 | 78.6 |
| 1 | 83.6 ± 1.4 | 81.6 | 81.0 | 85.6 ± 0.6 | 82.3 | 79.2 |
| 2 | 84.4 ± 1.6 | 82.8 | 82.2 | 86.2 ± 0.4 | 85.9 | 84.7 |
| 4 | 86.6 ± 0.8 | 82.6 | 80.2 | 89.8 ± 0.9 | 89.0 | 87.8 |
| 5 | 87.3 ± 0.6 | 83.0 | 81.8 | 91.5 ± 0.7 | 90.2 | 90.2 |
| 6 | 86.8 ± 0.8 | 84.8 | 84.2 | 90.2 ± 0.6 | 91.4 | 90.2 |
| 10 | 88.0 ± 0.6 | 86.4 | 84.6 | 92.2 ± 0.4 | 92.1 | 91.4 |
| 20 | 87.9 ± 0.8 | 88.0 | 87.6 | 94.9 ± 1.5 | 91.4 | 92.1 |
| 30 | 88.9 ± 0.6 | 87.8 | 87.2 | 93.7 ± 1.2 | 93.3 | 92.1 |
| 40 | 88.4 ± 0.4 | 87.4 | 86.8 | 94.1 ± 0.9 | 92.7 | 93.3 |
| 50 | 88.2 ± 0.6 | 88.0 | 86.8 | 94.3 ± 1.1 | 92.1 | 92.1 |
Table F. Per-token log-likelihood of the returned output vs. .
| MATH500 default | MATH500 diffuse | MATH500 sharp | HumanEval default | HE diffuse | HE sharp | |
|---|---|---|---|---|---|---|
| 0 | −0.1045 | −0.2038 | −0.0964 | −0.1644 | −0.2696 | −0.1264 |
| 1 | −0.1004 | −0.1690 | −0.0935 | −0.1535 | −0.2139 | −0.1201 |
| 2 | −0.0993 | −0.1509 | −0.0919 | −0.1466 | −0.1867 | −0.1154 |
| 4 | −0.0981 | −0.1401 | −0.0903 | −0.1439 | −0.1805 | −0.1153 |
| 5 | −0.0970 | −0.1281 | −0.0899 | −0.1402 | −0.1706 | −0.1159 |
| 10 | −0.0950 | −0.1149 | −0.0892 | −0.1372 | −0.1554 | −0.1168 |
| 20 | −0.0925 | −0.1065 | −0.0885 | −0.1312 | −0.1417 | −0.1158 |
| 30 | −0.0912 | −0.1030 | −0.0873 | −0.1286 | −0.1354 | −0.1147 |
| 40 | −0.0902 | −0.0996 | −0.0869 | −0.1256 | −0.1319 | −0.1140 |
| 50 | −0.0895 | −0.0975 | −0.0863 | −0.1241 | −0.1299 | −0.1123 |
Round trips. Table G gives the share of problems on which at least one replica completes a round trip on Qwen3-8B, with from 10 to 50 on every benchmark and all other settings as in Table 8. With 50 steps, almost all problems on every benchmark have a round trip.
Table G. Problems with at least one round trip (%), Qwen3-8B.
| MATH | AIME | GPQA-D | LCB v5 | HumanEval | |
|---|---|---|---|---|---|
| 10 | 17.4 | 32.1 | 27.0 | 36.9 | 54.5 |
| 20 | 38.5 | 48.7 | 69.3 | 79.7 | 68.1 |
| 30 | 73.5 | 83.0 | 77.2 | 92.6 | 86.6 |
| 40 | 99.0 | 97.4 | 95.7 | 100 | 98.3 |
| 50 | 100 | 100 | 100 | 100 | 100 |
References
[1] Spectral Gap of Replica Exchange Langevin Diffusion on Mixture Distributions.
Author Response to Reviewer zi5q (Part 1/3)
We truly appreciate the encouraging comments of the reviewer, as well as their useful remarks. In the following, we address the points raised by the reviewer.
1. Finite-budget mixing evidence
We thank the reviewer for the comment. Preserving the stationary target ensures the theoretical correctness of our algorithm. We agree that it does not imply that finite-budget samples have reached equilibrium, and we state this explicitly in the revision. We highlight that the accuracy gains over a single fixed-horizon chain are attributable to the parallel tempering mechanism of PPT. The mixing benefits of replica exchange, including exponential speedups for certain classes of multimodal distributions, have been widely studied [1]. We support this with two pieces of evidence.
Exact computation in a toy model. First, we devise the following example to isolate the mechanism by which exchange reduces finite-budget error from a non-target initialization:
Table A. TV distance to the target in the toy model.
The initializer places the output-rung marginal at TV 0.32 from the target, a concrete instance of the reviewer's point that block extension starts away from πα⋆. Exchange lowers the error by 2.6× after 5 rounds and by 4.7× after 10, round counts comparable to those we deploy (6–20 per block). Both arms make the same local proposals, and swaps reuse cached likelihoods, so the arms are matched in proposals, though not exactly in tokens, since suffix lengths depend on the state.
Exchange moves high-likelihood records to the cold rung, where they are sharpened. Second, since the distance to πα⋆ cannot be computed for an LLM, we measure the effect of exchange within the actual refinement budget: how often the cold chain holds the highest-likelihood state among the K replicas, with swaps enabled and disabled under matched compute and initialization. With swaps it does so in 89.2–97.5% of evaluations, compared with 29.0–54.8% without (Table B). Consistent with this, Figure 5 shows that PPT's returned traces concentrate at higher average log-likelihood than those of Power Sampling, GRPO and standard decoding (MATH500, Qwen3-8B). Concentrating on high-likelihood responses is how sharpening can improve final-answer accuracy.
Table B. Evaluations where the output rung holds the highest-likelihood record (%), Qwen3-8B.
2. Scope of the frontier-model comparison
We agree that the comparison should be described as an observation about reported benchmark scores. The frontier entries come from external evaluations, and we cannot establish comparable prompts, inference budgets, or latency. We will make this distinction clear in the caption of Figure 1 and in Section 4.1 of the revision.
The frontier entries in Figure 1 and Table 2 are scores reported by external evaluations. We did not re-run those models, so we cannot establish that their prompts, inference budgets, decoding settings, or latency match ours. Several reported configurations also do not disclose enough to align them. Therefore, the comparison is an observation about reported benchmark scores. It is not evidence of resource-matched reasoning parity, and we will state this explicitly. The point we want to make is that a 9B model, with no additional post-training, reward model, or verifier, and returning a single completion per problem, exceeds or comes within a few percentage points of the reported results of frontier models on GPQA-Diamond, AIME 24&25 and LiveCodeBench v5. We read this as evidence of how much reasoning capability is latent in a small model's distribution and how effectively PPT recovers it at inference time.