- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Progressive Point Matching (PPM), which derives segment-level rewards from changes in the number of reference reasoning points reached by a generated prefix. A shortcutting mechanism credits intermediate points made obsolete by later progress. The paper analyzes optimal-policy equivalence under a perfect-reachability assumption and compares signal-to-noise ratios of several estimators in idealized reasoning graphs. Experiments cover controlled synthetic horizons and mathematical reasoning, including tasks with near-zero initial success and evaluation at longer generation budgets.
Critical strengths
-
The paper studies an important failure mode of outcome-only RL: useful partial progress can be common even when complete successes are extremely rare. The controlled horizon experiments and the distinction between trajectory-level and segment-level feedback make this issue concrete, rather than relying only on aggregate benchmark improvements.
-
Reference reasoning points and shortcutting provide a useful alternative to exact trajectory imitation. The hard-task experiments show that the approach can extract a training signal where the reported sparse-reward baseline makes essentially no progress, and the paper investigates several consequential implementation choices in Appendix C.
Critical weaknesses
-
The theoretical claims are not yet sufficiently connected to the actual update. Proposition 3.1 establishes equality of optimal policies only when an almost-surely successful policy exists and goal completion credits every point; it does not establish objective invariance within a restricted, imperfect policy class. More importantly, Appendix B's segment estimator Y3=Σv Iv xv/n omits the downstream score–reward terms present in the trajectory estimator Y2. In a sequential dependency graph, these estimators generally have different expectations, not just different variances. The paper needs to explain the optimized objective and the precise relationship between Y3, Algorithm 1's advantages, and standard reward-to-go credit assignment before the SNR comparison can substantiate the stated unbiasedness and learning-efficiency claims.
-
Practical validation of the central reward construction remains incomplete. The graph-generation prompt on p. 36 discourages a final answer from obsoleting substantive derivations, whereas the optimality argument requires goal completion to credit all reference points. It is unclear whether a separate closure rule enforces this requirement. Judge validation also uses a very small prompt set, and Tables 6 and 7 contain identical values despite evaluating different objects. These issues matter because approximate matching and shortcutting determine the reward actually optimized; plausible point descriptions alone do not validate the theoretical construction.
Overall assessment: 2 — Weak rejection. The problem and empirical direction are compelling, but the relationship between the analyzed segment estimator, the claimed objective properties, and the implemented reward construction is central and currently unresolved. A precise reconciliation could materially improve my assessment.
Questions for authors
-
What exact advantage is assigned to an early segment, including rewards from later segments? Please reconcile Appendix B's Y3 with the policy gradient of the stated cumulative PPM return. For the two-point chain I1=B1 and I2=B1B2 with separately parameterized Bernoulli actions, the gradient of E[I1+I2] includes an additional downstream contribution to the first action that is absent from I1x1+I2x2. Does the implementation include that contribution, or is it deliberately optimizing a different local surrogate?
-
Is full goal closure enforced independently of the generated obsoletion graph, including for correct answers obtained through an alternative derivation? Please specify the executed rule and clarify the graph-quality metric and identical entries in Tables 6 and 7.
-
Which POPE-hard results evaluate the same training problems at a larger token budget, and which evaluate held-out problems? Please distinguish this extrapolation result from new-problem generalization and, where available, report uncertainty and judge-generation cost alongside student training tokens.