- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the computational inefficiency of fixed sampling budgets in inference-time alignment methods such as Best-of-N. It proposes RWSC, which combines reward model scores with answer-level self-consistency, and develops two sequential stopping methods: SPRT-RWSC and e-RWSC. In particular, e-RWSC uses per-answer e-processes and reward-informed Kelly betting to provide anytime-valid statistical guarantees while improving sampling efficiency. Experiments across multiple language models, reward models, and reasoning benchmarks demonstrate promising accuracy-efficiency trade-offs.
Critical strengths
1. A principled adaptive stopping mechanism with statistical guarantees.
The main strength is the application of anytime-valid e-processes to a dynamically expanding set of candidate answers. The proposed approach maintains error control even when the leading answer changes during sampling. Compared with heuristic stopping rules, this provides a stronger theoretical justification and is a key reason I view the paper positively.
2. A thoughtful integration of reward models with practical efficiency gains.
The reward-informed betting mechanism uses reward model predictions to accelerate evidence accumulation without directly affecting the validity of certification. This is an appealing design choice. The empirical results are also encouraging. For example, on MATH500 under a large sampling budget, e-RWSC improves accuracy by 8.2 percentage points while reducing average candidate generation by approximately 61.7% compared with Vanilla BoN.
Critical weaknesses
1. Proposition 1 appears to omit important conditions for its sampling complexity guarantee.
Proposition 1 assumes , whereas the actual algorithm restricts bets to , with . This weaker assumption does not ensure that the implemented strategy achieves the unconstrained Kelly-optimal growth rate. For example, when , , and , the optimal bet is 1.8, exceeding the implemented cap of 1.6.
Moreover, the claimed asymptotic sampling complexity is difficult to reconcile with a fixed finite , since must approach zero as . These issues suggest that Proposition 1 needs stronger assumptions and a clearer distinction between unconstrained certification time and budget-limited stopping.
2. The reliability evaluation and empirical superiority claims need further qualification.
Theorem 1 controls the probability of incorrectly certifying a non-dominant answer, rather than the correctness of the final response. Furthermore, when the sampling budget is exhausted, the algorithm returns an uncertified RWSC answer. Comparing early stopping decisions against a 512-sample consensus does not directly validate the theoretical certification event.
The empirical advantages are also not universal. For example, on MATH500 with Qwen2.5-3B and ArmoRM at , ConSol achieves 75.4% accuracy with 8.02 average candidates, compared with 74.4% and 15.59 for e-RWSC. While e-RWSC offers stronger statistical guarantees, its empirical accuracy-efficiency advantage should be stated more carefully.
Questions for authors
-
Could the authors revise Proposition 1 to account for the actual betting cap and distinguish unconstrained certification time from budget-limited stopping? Should the sampling complexity instead depend on the truncated betting strategy's achievable growth rate?
-
Could the authors report certification rates, budget exhaustion rates, and the accuracy of certified versus uncertified outputs across different and ? Could independent sampling estimates of answer probabilities also be used to assess the event controlled by Theorem 1?
-
How do the authors characterize e-RWSC's advantages in settings where ConSol or SPRT-RWSC achieves both higher accuracy and lower sampling costs? Confidence intervals, generated token counts, and end-to-end inference latency would help clarify the practical cost of obtaining statistical guarantees.
3 comments
We thank the reviewer for the constructive feedback. We have revised the manuscript with new results and clarifications, and we hope these, together with our responses, address your concerns. If any point remains unclear, please let us know; we would be glad to clarify further during the discussion period. If you feel your concerns have been resolved, we would be grateful if you would consider updating your assessment.
Comments on Rebuttal
Thank you for the detailed rebuttal and the substantial revisions. The authors have addressed most of my major concerns.
In particular, I appreciate the correction to Proposition 1, which now properly accounts for the constrained Kelly betting strategy and distinguishes the unbounded certification-time result from budget-limited stopping.
The new empirical evaluation is also a significant improvement. The authors now distinguish certified from uncertified outputs, report certification and budget exhaustion rates, and provide a more direct empirical assessment of the false-certification event controlled by Theorem 1. The comparison with MMC, together with the additional 24B model evaluation and computational overhead analysis, further strengthens the empirical study.
I particularly appreciate the analysis of certified accuracy, which better illustrates the reliability-efficiency trade-off offered by E-RWSC.
Some limitations remain. In particular, certification guarantees dominance under the generation distribution rather than factual correctness, and the empirical advantages over existing adaptive methods are not uniform across all benchmarks and budget settings. The applicability to genuinely open-ended tasks also remains to be empirically validated.
Nevertheless, I believe the authors have adequately addressed the main technical and experimental concerns raised in my review. The revised paper is substantially stronger, and I am updating my recommendation to Weak Accept (3).
Response to Reviewer Zwjk
Thank you for your feedback. We address the concerns below.
Proposition 1 statement: Thanks for catching this discrepancy in the Proposition 1 statement. We will revise the statement as: "λ⋆(θa⋆)≤c/p†, where c∈[0,1)..." Note that Step (i) of the proof already uses this condition, and the result will remain unchanged. Accordingly, in Lemma 1, we will state Lemma 1 for λ∈[0,c/p†] and modify the admissibility condition to λ⋆(θ)≤c/p†. The proof remains unchanged.
On Nmax, the reviewer is right that the α→0 limit concerns the unbudgeted certification time. We will state Proposition 1 for Nmax=∞, and state that the deployed rule stops at min{N,Nmax}. Accordingly, we will state Theorem 1 for any budget Nmax∈N∪{∞}, so truncation affects only whether certification occurs, never its validity. The proofs remain unchanged.
E-RWSC certification rates, budget exhaustion rates, and the accuracy of certified versus uncertified outputs: We report the results on MATH500 (Qwen2.5-3B base model + Armo reward model).
Scaling Across Budget N (α=0.10)
Sensitivity Across Error Boundary α (Nmax=512)
Empirical Assessment of Theorem 1: As per your suggestion, we evaluated Theorem 1 using independent empirical estimates θ^a⋆=N1∑i=1N1(yi∈a⋆) computed over the full N=512 independent Monte Carlo sample pool. Across all α∈[0.01,0.40], we observe that the empirical false certification rate (θ^a⋆≤p†) is zero, whereas for α=0.45,0.50, it is 0.4%. This verifies finite-sample error control under Theorem 1. We will replace the Figure in the main text with this new result.
E-RWSC Advantages vs. ConSol & SPRT-RWSC: ConSol performed better than our method on only one dataset (MATH500) for the Armo reward model, but E-RWSC maintains a consistently higher accuracy across datasets and models. Even on this benchmark, switching to a better reward model like Skywork-V2, E-RWSC achieves 78.6% accuracy compared to ConSol's 75.4%.
As you suggested, to further characterize e-RWSC's advantages, we compare Certified Accuracy, certification rate, and generated token counts on the MATH500 dataset (Qwen2.5-3B base model + Armo reward model).
We observe that when E-RWSC certifies, its accuracy is 94.5% spending only 13.1k tokens per query, compared to ConSol's 74.8% (9.1k tokens) and SPRT-RWSC's 76.8% (13.6k tokens). This shows that E-RWSC gives more reliable certified decisions.
While ConSol and SPRT-RWSC stop aggressively on small budgets, they suffer from premature convergence under compute scaling: ConSol halts at step ∼12 even with Nmax=512, plateauing at 74.8% accuracy. In contrast, E-RWSC adaptively allocates compute to difficult queries, thereby scaling overall accuracy to 78.0% (+3.2%).
Moreover, the CPU overhead of E-RWSC's e-process update is merely 0.014 ms per step, which is <0.0002% of a new candidate generation time on the GPU (7,144.3 ms), and thus negligible.