- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper compares nested truncation samplers. It gives a separation bound and a paired estimator.
Critical strengths
GSM8K tests show variance reductions of 1.3–3.8 times against prompt-level pairing. This improves measurement efficiency.
Critical weaknesses
Certificate and outcome runs use different stopping rules. This limits their connection. Sample requirements for other benchmarks rely on GSM8K estimates. These requirements need further validation.
Questions for authors
Do results hold with the same stopping rule? Does the sample analysis target fixed benchmarks or new prompts?
6 comments
Dear Reviewer ygCW,
Thank you again for your review and your two questions. As the discussion period closes on November 18, we wanted to check whether our response answered them.
In brief:
- In the chat format, with one stopping rule for both the certificate and the outcomes, the default pair keeps a 5.4× variance reduction, and its certificate (0.17) bounds every measured gap.
- The sample analysis targets new prompts. For fixed benchmarks, we now give each benchmark's requirement from its own variance.
We would be glad to clarify anything that remains. If our response resolves your concerns, we would be grateful if you would consider updating your assessment.
Dear Reviewer ygCW,
The discussion period closes on November 18 AoE, so we would like to ask you directly to check our rebuttal. Both of your questions now have direct answers:
- You asked whether the results hold with the same stopping rule. We ran exactly that experiment. With the certificate and the outcomes stopping at the same EOS, the default pair keeps a 5.4× variance reduction. Its certificate (0.17) bounds every measured gap, and Prop 7's guarantee holds (at least 2.1× guaranteed, 12.3× measured).
- You asked whether the sample analysis targets fixed benchmarks or new prompts. New prompts. For fixed benchmarks we now give the requirement directly, k = Cσ_w²/N.
- You noted that the requirements relied on GSM8K estimates. Each benchmark now uses its own variance: GSM8K 4.6, MATH-500 23 and AIME 2025 about 305 samples per prompt. The paper's conclusions hold.
Could you let us know before the deadline whether this answers your questions, or what remains? We will respond right away. If the new results resolve them, we ask that your assessment reflect this.
We thank the reviewer for finding the paper clear and easy to follow, and for highlighting the 1.3–3.8× variance reductions and the gain in measurement efficiency.
Below we quote each concern and add new evidence. Both questions led to new results.
"Certificate and outcome runs use different stopping rules" (Q1: "Do results hold with the same stopping rule?"): We reran with the chat template (thinking disabled), with every arm and the certificate stopping at the same EOS; 98% of generations end before the cap. For the default pair, the results hold:
Most of the certificate's drop comes from the chat format rather than from stopping: it is already 0.06 at T=64, before any generation stops. The two pairs whose certificates stay above 1/2 show smaller, non-significant reductions (1.19×, 1.25×); the theory guarantees nothing there. This closes the gap between the certificate and the outcome runs: with one stopping rule for both, the certificate bounds every measured gap, and Prop 7's variance guarantee holds on a real cross-family pair.
"Sample requirements for other benchmarks rely on GSM8K estimates": the paper states this. Sec. 5 notes that "this extrapolation assumes the τ² measured on GSM8K", and Table 3 is captioned "(default pair, GSM8K variance decomposition)". We now use each benchmark's own variance. For fixed benchmarks the conclusions hold:
GPQA-Diamond's estimate rests on 5 nonzero outcomes, and we say so. This validates the sample-size conclusions on each benchmark's own data: the larger benchmarks are resolved with tens of samples per prompt, and AIME 2025 needs hundreds.
Q2 ("Does the sample analysis target fixed benchmarks or new prompts?"): new prompts. We thank the reviewer for this question, it sharpens the paper. For a fixed benchmark, τ² drops out and k = Cσ_w²/N. For new prompts, the upper bounds from 100 and 30 prompts leave MATH-500 and AIME unresolved, and the camera-ready version will state this. This makes the sizing directly usable: for a fixed benchmark, every benchmark we study has a finite number of samples per prompt that resolves a one-point difference.
As we have addressed both weaknesses and both questions, and the new results strengthen our claims, we kindly ask the reviewer to consider an updated assessment.