- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DGPO, which constructs segment-level credit from binary outcome rewards. Sustained high entropy triggers tree branching, followed by semantic-similarity pruning. DIG multiplies normalized mutual information by a bounded log-odds polarity term; standardized global and same-prefix local scores augment trajectory advantages. Experiments cover four backbones and six mathematical reasoning benchmarks, with credit-estimator comparisons under a fixed branching procedure and a comparable-runtime comparison on Qwen3-4B-Base. The appendix expresses unsmoothed DIG as nonnegative reweighting of direct advantages and provides conditional improvement results for independent branch-policy updates on a fixed tree.
Critical strengths
Introducing mutual information into segment-level credit assignment is a meaningful and well-motivated contribution. By combining information gain with directional association, the proposed method offers a principled approach to extracting fine-grained learning signals from outcome-level rewards. The consistent improvements across multiple model backbones and benchmarks further highlight the potential of this direction.
Critical weaknesses
- Although Tables 2 and 5 provide evidence for the effectiveness of DIG, the contribution of the normalized information-gain factor remains unclear. Since the polarity term already captures both the direction and strength of segment–outcome associations, a controlled polarity-only ablation would help clarify the source of the improvements.
- As discussed in Appendix A.1, the availability of DIG corrections depends on the support threshold and the topology of the sampled reasoning trees. However, the paper does not report the proportion of supported edges or tokens receiving nonzero DIG corrections, nor does it examine sensitivity to the support threshold. These analyses would help clarify the practical coverage and robustness of the proposed credit assignment mechanism.
Questions for authors
- What is the independent contribution of the normalized information-gain factor?
- How frequently does DIG provide effective credit during training, and how sensitive is it to the support threshold?
2 comments
Follow-up on our response
Dear Reviewer gehj, thank you again for your thoughtful review and for recognizing the motivation behind DIG. In our response, we isolate the independent contribution of the normalized information-gain factor through a matched polarity-only ablation that removes while keeping the polarity, support masks, normalization, and global/local weights fixed; the information factor accounts for a 3.59-point gain. We also report how frequently DIG provides effective credit, measured on actual training trees across three training stages, and analyze sensitivity to the support threshold , where the default threshold achieves the best accuracy by balancing credit coverage against the reliability of each comparison. As the discussion period is nearing its end, we would appreciate it if you could take another look and let us know whether these additions address your concerns. We would be happy to clarify any remaining questions.
We thank the reviewer for recognizing the motivation behind DIG and for identifying the two questions about its information factor and practical coverage.
Q1. Contribution of the information factor
A1. The matched polarity-only ablation shows a 3.59-point contribution from the information factor. On Qwen3-4B-Base, seed 42, we remove G while retaining the smoothed polarity, DGPO tree-construction configuration, base trajectory advantage, global/local comparison sets, support masks, normalization, and mixture weights. Constant-outcome comparison sets retain zero raw scores.
Polarity already captures the direction and magnitude of the odds contrast. The information factor supplies a different weighting: the empirical outcome information carried by the occurrence split, normalized by segment-occurrence entropy. It therefore changes the relative credit across supported segments. This ablation isolates its contribution within the same credit-construction and normalization procedure.
Q2. Credit coverage and support threshold
A2. At the default threshold, 31.29% of token occurrences have a nonzero normalized global or local component, and 53.13% of trees contain at least one such component. We measured this on 768 actual training trees from Qwen3-4B-Base, seed 42, with 256 trees at each of steps 50, 250, and 450, using B=8, D=3, and τmin=2.
Edge statistics count each distinct edge once per tree. Token statistics count occurrences across complete paths, so a shared segment is counted once for each path containing it. Global and local coverage overlap. Unsupported edges remain zero after normalization.
For threshold sensitivity, we recomputed coverage on the same recorded trees at thresholds 1, 2, and 4. The accuracy measurements come from separate training runs using the corresponding threshold:
At τmin=1, comparisons based on a single path are admitted. Coverage increases, but accuracy falls; these comparisons are more sensitive to individual binary outcomes. At τmin=4, the eight-path budget permits only 4-versus-4 comparisons. The coverage audit leaves few tokens with segment corrections, and the corresponding training run scores close to the trajectory-only control under DGPO branching (48.29). The default threshold retains more credit coverage than four while requiring at least two paths in each comparison group.
We will incorporate the polarity-only ablation, coverage statistics, and threshold analysis into the revised manuscript.