- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Stratified Segment Policy Optimization (S-SPO), which replaces independent auxiliary continuations in SPO with equal-mass stratified arithmetic sampling. It characterizes how value-estimation errors propagate through local-credit construction into policy updates, establishing exact covariance-based guarantees for frozen linear updates and a first-order extension through normalization. Experiments demonstrate improved full-parameter gradient reliability and consistent GSM8K accuracy gains across three RhoMath training seeds, supplemented by one matched Llama comparison.
Critical strengths
-
A precise analysis of local-credit reliability. The paper explicitly accounts for covariance, evidence reuse, and update geometry, explaining why value-level error alone is insufficient to characterize gradient reliability. Its distinction between exact linear guarantees and conditional nonlinear improvements makes the theoretical contribution credible.
-
Carefully controlled empirical validation. The principal comparisons match the arithmetic decoder, continuation budget, and downstream SPO pipeline. Full-parameter gradient measurements, paired pool-level inference, and three positive RhoMath seed pairs on the official GSM8K test provide complementary evidence that the intervention improves both estimation and learning.
Critical weaknesses
-
The incremental contribution and practical informativeness of the theory remain limited. Stratified arithmetic sampling is already established in prior work, including QuasiMoTTo. The main addition is therefore its analysis and application within the local-credit pipeline. However, the normalized-update guarantee is only first-order, and its sufficient finite-budget condition is not evaluated at the experimental operating point. Although these limitations are acknowledged, the analysis provides limited quantitative guidance about when its guaranteed linear improvement translates into improved nonlinear updates.
-
The empirical study establishes effectiveness in a narrow configuration. All online experiments use GSM8K, the principal intervention comparison fixes , and the second model has only one matched run. Sensitivity to continuation budget and arithmetic token ordering—which directly affect the variance reduction—is not established. Comparisons with alternative dependent samplers within the same SPO pipeline are also absent, leaving the robustness and relative practical merit of this particular sampling design unresolved.
Questions for authors
-
In the frozen-gradient experiments, how accurately does the first-order risk predict the measured normalized-gradient error? Is the sufficient finite-budget improvement condition satisfied in any observed pools?
-
How sensitive are the reliability and accuracy gains to the continuation budget and token ordering? Can the existing measurements identify configurations in which stratification provides little benefit?
-
How does equal-mass stratification compare with another dependent sampler, such as randomized lattice sampling, under the same auxiliary-query budget and SPO pipeline?
2 comments
Thank you for the detailed response and clarifications. But two points remain unresolved:
-
Theoretical validation: Could the authors quantify how closely the first-order risk approximates the actual normalized-gradient error at , and whether the sufficient finite-budget improvement condition holds in any of the evaluated pools?
-
Sampling robustness: Could the authors evaluate how the gains from S-SPO vary across different continuation budgets and token orderings, particularly whether certain configurations yield negligible improvements over arithmetic-iid sampling?
Response to Reviewer nKBZ
We thank the reviewer for acknowledging the theoretical analysis and controlled empirical design. We clarify the contribution beyond classical stratification, the interpretation of normalized-gradient evidence, and the scope of our robustness claims.
We agree that stratification and its standard variance-reduction properties are established. Our contribution is not a new stratification primitive, but an analysis of how finite auxiliary-continuation noise affects policy updates through SPO's local-credit construction. Equation (3) shows that shared-boundary estimation errors enter adjacent credit differences with opposite signs and that their raw-gradient impact depends on adjacent segment-score differences. Theorem 1 connects evidence covariance and fixed boundary reuse to exact quadratic-risk guarantees for fixed linear updates. Crucially, lower value variance does not imply unconditional improvement after nonlinear normalization: Theorem 2 provides a first-order guarantee with an explicit remainder, and Appendix C gives a counterexample. These results identify the update-reliability problem, characterize its error-transport mechanism, and delineate where classical variance reduction does and does not suffice. S-SPO is a controlled intervention informed by this analysis, not the sole claimed contribution.
We agree that a first-order guarantee need not quantitatively predict realized nonlinear update risk. Theorem 2 bounds the discrepancy between first-order and actual risks using an explicit remainder; its sufficient condition for finite-sample improvement was not verified at N=9. However, our experiments directly measure full-parameter gradient direction against a high-budget Full128 reference rather than relying on that approximation. In 24 independent pools with matched N=9 sampling, gradient cosine rises from 0.8215 (M1) to 0.8466 (M2), with a paired 95% CI for the gain of [0.0091, 0.0431] and improvement in 20/24 pools. Using the exact unit-vector identity ||u-v||^2 = 2(1-cos(u,v)), these mean cosines correspond to mean squared unit-direction distances of 0.3570 and 0.3068, a 14.1% reduction relative to the same finite reference. Raw-gradient quadratic error also decreases by 12.5%. This is an algebraic restatement of the observed cosine, not an independent experiment, a comparison against an exact expected gradient, or a validation of Theorem 2's remainder. It directly supports improved measured directional agreement in the tested frozen contexts, without assuming first-order accuracy.
We distinguish theoretical guarantees from the size and generality of observed gains. Theorem 1 is not specific to N=9: its relative variance and fixed-linear-risk conclusions apply under the stated stratum-input laws, independence across primitive queries, and fixed reuse. Changing N or token ordering can alter how rewards map to strata, affecting the magnitude of improvement without by itself invalidating the conditional guarantee. Randomized lattice and other dependent samplers may also be effective, but have different dependence structures and were not compared; we make no optimality claim. Empirically, the 24-pool gradient study is complemented by three matched RhoMath training-seed pairs, giving mean gains of 2.75 percentage points on validation and 2.26 points on the official GSM8K test, all positive. One matched Llama-3.2-1B-Instruct pair adds a 2.85-point validation gain. Both online settings use GSM8K, and the Llama result has only one paired seed. Our conclusions therefore distinguish proven conditional guarantees, measured gains in the evaluated settings, and untested generalization across tasks, scales, and samplers.