- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether sharpening a model's distribution with Power Sampling helps self-consistency. Using Power weights on a shared pool of 8 base samples, the authors find that SC accuracy drops in 28 of 33 model/benchmark settings, even though on HARP the average correct mass goes up. They blame this on two things: sharpening can favor one strong wrong path over several weaker paths that agree on the right answer, and a fixed alpha changes some problems much more than others. Their fix, Relative-Rank SoftSat, weights samples by their likelihood rank in the pool and caps the gain at high ranks, optionally together with a same-model self-evaluation label. They test it on three open models and 11 benchmarks.
Critical strengths
I found the main observation useful. Many people assume that sharpening toward likely trajectories should help multi-sample methods, and the 33 settings show fairly clearly that it can hurt SC instead. The HARP analysis, where correct mass rises but the margin shrinks because the top trajectory takes most of the weight, makes the reason easy to see. The fix is also simple and cheap, since it only reweights samples that were already generated for SC, and the rank argument in Lemma 1 is clean.
Critical weaknesses
My main concern is that the title, the contributions and the conclusion talk about Power Sampling in general, but almost all experiments only reweight 8 base samples with p^(alpha-1). With alpha = 4 and long outputs, these weights put nearly all the mass on the most likely sample. On the HARP slice ESS is about one sample, and every alpha from 1.5 up gives almost the same scores as alpha = 4. So the "Power" baseline is close to taking the answer of the highest-likelihood sample, and the drop may come mostly from degenerate weights in a small pool. It may say little about the Power target itself. The only test with real Power-SMC is 9 single-seed settings. Those numbers do look lower than the Uniform results in Tables 1 and 2, which partly supports the paper, but they come from different runs and there is no Uniform baseline on the same seed.
Also, SoftSat on its own does not really improve over plain uniform SC. It wins and loses about equally against Uniform with the same average accuracy, so its main benefit is recovering from a baseline that the paper itself sets up. With a 75% uniform floor (eta = 0.25) it stays close to uniform voting by design, and on the HARP pool linear rank does a bit better, so I am not sure the saturation part matters much. The real gains come from self-evaluation, which needs an extra pass over every trajectory, and SE-only already beats Uniform in most settings. There is no comparison with weighted SC methods like CISC or RPC, even though they are cited. And the gaps between SoftSat, SelfEval and Uniform are mostly within the standard deviations, with no paired tests reported.
A smaller point on presentation: the results come from many different pools, seeds and slices, which made the paper hard to follow, and the writing is very dense.
Questions for authors
- Table 7 suggests that real Power-SMC with uniform SC also trails Uniform, but it is single-seed and from different runs. Can you run Power-SMC and Uniform SC on the same seeds and on more benchmarks? Without this, I read the results as being about importance weighting on 8 samples.
- How does SoftSat+SelfEval compare with confidence-weighted SC such as CISC or RPC? And how does it compare with simply drawing more samples using the extra compute that self-evaluation needs?
- Only beta is varied in the sensitivity study. How does SoftSat behave with a smaller uniform floor (larger eta)? I would like to know if it does anything beyond staying close to uniform.
- Did you try length-normalized log-likelihood, for both the Power weights and the ranks? Summed log-probs depend a lot on length, which may explain part of the collapse.