- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RREDCoT, a segment-level reward redistribution method for reasoning models. It addresses credit assignment under delayed final-answer rewards during reinforcement learning fine-tuning. The method uses the model itself to estimate each segment’s contribution and redistribute rewards along the chain-of-thought trace. The authors evaluate the method on mathematical reasoning tasks and examine its correlations with Monte Carlo estimates and attribution methods.
Critical strengths
- The paper addresses fine-grained credit assignment under final-answer rewards. Building on RUDDER, it provides segment-specific training signals while reusing the generating model to reduce estimation costs relative to Monte Carlo sampling.
- Fine-tuning experiments of RREDCOT show improvements over GRPO on four of five evaluated benchmarks, providing preliminary evidence of the method’s effectiveness. The paper also analyzes Monte Carlo sampling costs and truncation effects.
Critical weaknesses
- The paper lacks controlled ablations of reference solution paths, hybrid segmentation, and normalization of . In particular, the benefit over using only changes in reference-answer probability remains unclear.
- The distinction from SPO (NeurIPS 2025), SSVPO (ICLR 2026), and TIPS (ICLR 2026) remains unclear. These methods already explore segment/step-level credit assignment or answer-probability-based rewards.
- The paper lacks controlled ablations of reference solution paths, hybrid segmentation, and weight normalization. In particular, the benefit over using only changes in correct-answer probability remains unclear.
- The experiments mainly compare against GRPO and Open-RS. The limited coverage of existing RL methods for mathematical reasoning makes it difficult to assess the proposed method’s relative effectiveness.
- There are also minor presentation issues, including inconsistent figure captions and typographical errors.
Questions for authors
- What estimator is used in the implementation? How are the normalization factors in Eqs. (9) and (22) consistent with the value definition in Eq. (4)?
- What additional benefit does the reference solution term provide beyond using only changes in correct-answer probability? Can the authors provide a controlled ablation?
- Could the authors include additional RL baselines for mathematical reasoning, such as SPO and SSVPO, to better assess the proposed method’s effectiveness?