- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Pairwise-Guided Logit Adjustment (PGLA), a plug-and-play way to aggregate sampled reasoning paths. On top of standard voting, it builds a confidence based pairwise comparison matrix between competing answers (motivated by the Condorcet principle), folds that comparative advantage into a KL regularized objective, and shows the solution is just an additive logit offset on the base voting score. Across several model families and scales and a spread of math, knowledge, and code benchmarks, it reports a small but consistent average gain over voting baselines, plus calibration improvements and online sampling savings.
Critical strengths
The KL objective reducing to a clean closed form logit adjustment is elegant, and the proof plus the sorting and prefix sum acceleration make the method genuinely cheap and label free. Benchmark coverage is strong: multiple model families and scales, math, knowledge, and code, with means and standard deviations reported, which many aggregation papers skip. Nice honesty in the appendix: the ranking quality ablation directly ties gains to how well confidence separates correct from incorrect paths, and the two case framing gives a believable mechanism for when the method helps.
Critical weaknesses
-
The gains are small relative to variance, so the central claim is not yet statistically established. The headline improvement over voting baselines sits within the range of the reported standard deviations on several settings, and the paper offers no paired significance testing across seeds even though its matched sampled pools make such a test straightforward. This is consequential because the entire contribution is an incremental accuracy gain from a plug in term: if that gain cannot be distinguished from sampling noise, the empirical case for the method collapses, regardless of how clean the theory is. The pattern that gains concentrate on smaller and math specialized backbones and fade on the strongest model deepens this concern, since it suggests the effect may be an artifact of weaker base models rather than a durable property.
-
The evaluation never tests the regime where the method should fail, so the scope of the claim is unverified. Every benchmark is competition math, closed book STEM multiple choice, or code, all tasks where path confidence correlates with correctness. The method's own ranking quality analysis shows the gain tracks that confidence correctness correlation and can go negative when the correlation is weak. On open domain or multi-hop retrieval QA, the dominant failure mode is confident and wrong, exactly the regime where confidence based pairwise comparison should stop discriminating and the comparative term should degrade voting. By labeling GPQA alone as "knowledge reasoning" and omitting any retrieval benchmark, the paper claims broad applicability while testing only the favorable half of its own stated dependency. This is consequential because a plug in aggregation term that quietly hurts on a whole task family, without any guidance on when to turn it off, is risky to adopt as presented.
Questions for authors
1/Every benchmark here is competition math, closed book STEM multiple choice, or code, all tasks where path confidence tends to correlate with correctness. I was curious how PGLA behaves on open domain or multi-hop retrieval QA (HotpotQA, TriviaQA, Natural Questions), where the dominant failure mode is confident and wrong. Since the paper's own ranking quality analysis shows the gain tracks the confidence correctness correlation and can even go slightly negative when that correlation is weak, a single retrieval benchmark would test whether the comparative advantage term still helps or quietly degrades voting. It would also let the knowledge reasoning claim rest on an actual retrieval task rather than GPQA alone.
2/The headline average gain sits close to the reported standard deviations on several settings, with error bars that look like they overlap. It would strengthen the claim to add a paired significance test across seeds, since the matched sampled pools make a paired test natural. I was curious how much of the gain is just better confidence rather than the pairwise structure. A natural ablation: replace the pairwise term with a direct confidence weighted re-rank at the same compute, so readers can see what the Condorcet style comparison buys beyond the confidence signal itself.
3/The gains look concentrated on the smaller and math specialized backbones and shrink on the largest model. It would help to say plainly where PGLA stops helping as models get stronger, rather than reading the large model result as uniform support. 4/Correctness labeling is only lightly described (normalized answer match for math, execution for code). Since this is effectively the evaluation ground truth, how sensitive are the calibration numbers and the recoverable error rates to the matcher? A short robustness check with an alternative judge would reassure.
5/Confound worth addressing: is the comparative advantage partly tracking answer group size or path length rather than correctness? A control that regresses out block size would make the mechanism claim cleaner.
6/The online stopping results compare against voting based early stopping baselines, but the stability rule and warm up appear to differ slightly across methods. Could you confirm all methods see identical path prefixes so the savings are not a stopping artifact?
13 comments
Q2a. Can the reported gains be distinguished from sampling variation in settings where the gains are comparable to the reported standard deviations? What does a paired significance test across seeds show when each baseline and its PGLA counterpart are evaluated on the same sampled paths?
A2a. We evaluate each baseline and its PGLA version on the same sampled paths in each of 10 repetitions in the original paper. All 17 positive-gain comparisons reported below have a 95% confidence interval above zero and a raw -value below 0.05. We use SC as the baseline, with DeepConf for the PGLA adjustment on Qwen3 reasoning models and PPL on math-specialized models. The tables below cover the positive mean gains in 8 of 9 Qwen3 settings and all nine math-specialized settings.
- Paired SC comparisons on Qwen3 reasoning models using DeepConf for the PGLA adjustment.
| Model | Benchmark | SC (%) | SC + PGLA (%) | Mean gain (pp) | 95% CI (pp) | Raw -value |
|---|---|---|---|---|---|---|
| Qwen3-1.7B | AIME24 | 69.50 | 75.00 | +5.50 | [+3.63, +7.37] | 9.273e-05 |
| Qwen3-1.7B | AIME25 | 47.33 | 51.33 | +4.00 | [+2.60, +5.40] | 0.0001 |
| Qwen3-1.7B | HMMT25 | 27.00 | 30.00 | +3.00 | [+0.91, +5.09] | 0.0100 |
| Qwen3-4B | AIME24 | 80.00 | 83.00 | +3.00 | [+2.25, +3.75] | 8.538e-06 |
| Qwen3-4B | HMMT25 | 51.50 | 54.00 | +2.50 | [+0.31, +4.69] | 0.0299 |
| Qwen3-8B | AIME24 | 81.50 | 86.67 | +5.17 | [+3.53, +6.80] | 5.344e-05 |
| Qwen3-8B | AIME25 | 75.67 | 79.67 | +4.00 | [+1.35, +6.65] | 0.0077 |
| Qwen3-8B | HMMT25 | 55.94 | 58.67 | +2.72 | [+1.50, +3.94] | 0.0007 |
- Paired SC comparisons on math-specialized models using PPL for the PGLA adjustment.
| Model | Benchmark | SC (%) | SC + PGLA (%) | Mean gain (pp) | 95% CI (pp) | Raw -value |
|---|---|---|---|---|---|---|
| InternLM2-Math-Plus-1.8B | MATH | 36.48 | 39.77 | +3.28 | [+3.13, +3.44] | 4.611e-12 |
| InternLM2-Math-Plus-1.8B | MathOdyssey | 14.57 | 18.28 | +3.71 | [+3.09, +4.32] | 2.504e-07 |
| InternLM2-Math-Plus-1.8B | AIME | 2.66 | 3.37 | +0.71 | [+0.55, +0.87] | 3.270e-06 |
| InternLM2-Math-Plus-7B | MATH | 50.57 | 53.99 | +3.43 | [+3.31, +3.54] | 2.339e-13 |
| InternLM2-Math-Plus-7B | MathOdyssey | 28.30 | 35.30 | +7.00 | [+6.51, +7.48] | 1.225e-10 |
| InternLM2-Math-Plus-7B | AIME | 9.40 | 9.95 | +0.55 | [+0.32, +0.78] | 0.0005 |
| DeepSeek-Math-RL-7B | MATH | 53.33 | 53.52 | +0.19 | [+0.07, +0.31] | 0.0053 |
| DeepSeek-Math-RL-7B | MathOdyssey | 36.57 | 37.87 | +1.29 | [+0.70, +1.89] | 0.0008 |
| DeepSeek-Math-RL-7B | AIME | 9.42 | 10.02 | +0.60 | [+0.37, +0.84] | 0.0002 |
Details. The main results report mean accuracy and standard deviation across 10 sampling seeds. We apply a two-sided paired -test to the 10 accuracy differences under the null hypothesis of zero mean gain.
Q2b. I was curious how much of the gain is just better confidence rather than the pairwise structure. A natural ablation: replace the pairwise term with a direct confidence weighted re-rank at the same compute, so readers can see what the Condorcet style comparison buys beyond the confidence signal itself.
A2b. With the same sampled paths, confidence signals, and sampling budget, SC + PGLA outperforms both confidence weighting and direct confidence adjustment in all seven settings. It improves accuracy by 1.18–6.48 percentage points over Weighted SC and by 0.67–3.32 points over SC + Mean.
- Confidence weighting (Weighted SC). We sum the path weights within each answer group, as in
Eq. (4). We use inverse perplexity for PPL and the raw confidence scores for DeepConf and Self-Certainty. - Confidence adjustment (SC + Mean). We scale the path confidence scores to within each pool and average them within each answer group. This gives the mean-confidence adjustment for answer .
Thank you for your detailed comments on the evaluation and the scope of the method.
A1. We evaluated PGLA on all 7,405 questions in the HotpotQA Fullwiki validation set and all 3,610 questions in the NQ-Open original development set, using a confidence score that incorporates retrieved evidence. PGLA improves EM by 2.28–3.35 percentage points and F1 by 1.64–2.28 points across SC, Weighted SC, and RPC. We use Qwen3-1.7B (non-thinking) and run 10 matched repetitions. In each repetition, all methods use the same ten retrieved passages and 32 sampled paths per question.