- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ReSum, an RLVR method meant to teach a model to “self-summarize” during long reasoning. Two pilot studies motivate it: entropy tends to be high just before summary-like phrases and drops right after, and injecting such a phrase into truncated wrong rollouts makes regenerated continuations more accurate. Training builds rollout trees. From each initial rollout, branches are created either by masking a detected cue (Natural Points) or by injecting one at a random position (Artifact Points). Correct rollouts that contain cues get a small bonus reward, and advantages are computed against a cue group and a no-cue group and then summed.
Critical strengths
The question is a reasonable one: can a model learn when to consolidate its own reasoning, instead of relying on an external compressor? The masking/injection idea is a neat way to get counterfactual branches from the same prefix. The experimental coverage is decent for an RLVR paper, with four backbones, combinations with GPG/DAPO/GSPO, a multimodal setting, a rollout-budget study, and a timing table showing about 13% training overhead. Gains over GRPO and DGPO are consistent in direction across the main tables.
Critical weaknesses
-
The mechanism doesn’t do what’s claimed. The dual-reference advantage in Eq. (3) uses the same group statistics for every rollout. Adding the two terms therefore gives the same affine function of the reward for all rollouts of a query, which is just a rescaled GRPO advantage with no cue-specific credit. Table 5 also shows ReSum with J=1 (no branching at all) at B=4 already beating GRPO. So the gains seem to come mainly from the cue bonus in Eq. (2): about +0.2 for correct rollouts containing common words like “wait,” or “let me check,” which is closer to a format reward than learned self-summarization. Baselines with GRPO plus this bonus and GRPO at matched B are needed.
-
The evidence is weaker than the claims. The “long-horizon” framing doesn’t fit ~500–800-token outputs. The 18.6% length reduction is shown only against DGPO on one backbone. The pilot study lacks a neutral-phrase control, and Table 7 compares against an untrained base model rather than GRPO with the same prompt. Table 8 marks DeepSeek-Math-7B AIME24 as significant even though ReSum is lower there (1.66 vs. 1.98). The headline “4%” is the best backbone; across backbones the gain is 2.3–4.0.
Despite these issues, the empirical gains are consistent, so I lean toward acceptance if the authors reframe the mechanism and add the missing baselines.
Questions for authors
-
Can you add GRPO plus the cue bonus in Eq. (2) without trees (same B), and GRPO at B=4 to compare with ReSum (T=4, J=1)?
-
When computing the loss on a branch, are the inherited prefix tokens and the injected AP cue tokens included? The injected tokens weren’t sampled by the policy, so including them would effectively be supervised training on cue phrases.
-
How exactly is the 18.6% length reduction computed, and what is the comparison against GRPO and on other backbones?
-
For Fig. 1(b), which model and how many prefixes? What happens with a neutral injected phrase?
5 comments
Official Response to Reviewer YH76 (2/2)
W4. Length measurements.
Thank you for asking us to clarify this comparison. The approximately 18.6% reduction in Figure 4 is measured using the training rollout lengths at the final training step, step 230, on Qwen2.5-Math-7B. It is the relative decrease from DGPO to ReSum, computed as the difference in their mean rollout lengths divided by the DGPO mean. Revised Table 9 separately reports mean evaluation response lengths in tokens.
| Backbone | GRPO length | DGPO length | ReSum length | ReSum reduction vs. DGPO |
|---|---|---|---|---|
| Qwen2.5-Math-1.5B | 784.72 | 742.09 | 655.38 | 11.68% |
| Qwen2.5-Math-7B | 627.73 | 562.04 | 530.63 | 5.59% |
| Qwen2.5-3B | 671.95 | 664.02 | 581.45 | 12.43% |
| DeepSeek-Math-7B | 458.56 | 440.89 | 416.74 | 5.48% |
ReSum yields shorter mean evaluation responses than DGPO for every listed backbone. These evaluation averages are distinct from the Figure 4 dynamics. The revised Abstract specifies the backbone and DGPO comparison for the reported dynamics; Section 4.3 and Table 9 present the separate evaluation-length comparison. As Section 3.2 states, ReSum retains the preceding reasoning history and shapes the generated continuation.
W5. Loss masks and pilot details.
Thank you for asking about the loss computation. The injected AP cue tokens do not contribute to the policy loss, so their insertion does not train the model to reproduce those tokens as sampled actions. The inherited prefix is used as context for the branch and is not repeatedly updated as a separate loss contribution for each continuation. This prevents the shared history from receiving additional weight simply because it has multiple branches.
For Figure 1(b), we used Qwen2.5-3B and collected 200 incorrect rollouts. At each selected truncation position, we regenerated 100 continuations per rollout under each condition, with or without an injected summary cue. Each condition therefore used 20,000 continuations per position. This setup examines whether a cue helps the model recover from an otherwise incorrect reasoning trajectory.
To address your question about a neutral phrase, we added a training-level control in Section 4.3. Revised Table 8 compares neutral transitions with summary cues on Qwen2.5-Math-1.5B:
| Cue condition | Accuracy (%) | Mean response length |
|---|---|---|
| No cue | 29.39 | 784.72 |
| Neutral transition | 32.56 | 725.20 |
| Summary | 33.07 | 655.38 |
The neutral transition improves on the no-cue baseline, indicating that a generic transition can also be useful. The summary-cue configuration achieves higher accuracy and shorter responses than the neutral-phrase configuration in this experiment. This comparison supports a benefit beyond adding an arbitrary transition, while also showing that a neutral phrase is a meaningful control. It complements the prefix-level pilot rather than constituting a neutral-phrase arm of Figure 1(b).
Thank you for pointing out the importance of comparing against a trained model with the same summarization prompt. Table 10 examines a different setting. It applies prompt refinement to the untrained backbone and reports the resulting performance alongside RL-trained ReSum. We have clarified this setup in the table caption and the discussion in Appendix E. The comparison shows how the methods perform in their reported settings, but it does not isolate the effect of prompting on a GRPO-trained model. A comparison using the same prompt on a trained GRPO checkpoint would be useful for answering that question directly.
Thanks again for your patience and expertise on our manuscript! We appreciate your valuable feedback, which helps us improve the quality of our paper.
Dear Reviewer YH76,
Thank you once again for your time and for your valuable feedback on our manuscript.
We have carefully considered all of your comments and have provided a comprehensive response. The corresponding revisions have been made in the manuscript, and we believe these changes have significantly improved our work.
We sincerely hope that our responses and the revised manuscript will resolve your concerns. We are happy to engage in any further discussion.
Best regards,
Authors
Official Response to Reviewer YH76 (1/2)
Thank you for the detailed review and for recognizing the consistency of the empirical gains. The revised manuscript clarifies the mechanism and includes the comparisons needed to assess its contribution. All table references below refer to the revised manuscript.
W1. Advantage and the role of branching.
Thanks for this opinion. Your algebra is correct. For a question with both reference groups nonempty, Eq. (3) is A^(τ)=cqR(τ)−dq, with cq>0 shared by the rollouts for that question. Section 3.3 now states this explicitly, and the Abstract and Section 3.4 distinguish local interventions from the rollout-level training signal. However, the algebra alone does not establish that the gain comes mainly from the cue bonus. ReSum uses prefix-local cue interventions to construct continuations, whose sequence rewards then enter the rollout-level advantage. The analytical paired gain Gi characterizes the local intervention rather than serving as a directly optimized term. The controls below test the reward-only explanation directly.
W2. Matched-budget and reward controls.
Thanks for this opinion. We appreciate your idea. Revised Table 5 in the new manuscript compares rollout budgets on Qwen2.5-3B. The table is also shown in the following:
The matched-budget rows make it possible to compare the methods at the same number of rollouts. At the smaller budget, the J=1 configuration spends the budget on initial rollouts, whereas the J=2 configuration allocates part of it to branching. Both improve on the GRPO result at that budget. This comparison shows the performance of the reported configurations without attributing the entire gain to branching alone. At the larger budget, the balanced ReSum configuration also improves on GRPO. The (T,J) notation describes ReSum's tree configuration. GRPO instead samples a group of independent rollouts, so the tree-configuration field is not applicable to it.
Revised Table 7 further separates the cue reward from the full method. Accuracy is again averaged across the six mathematical benchmarks.
The no-tree GRPO control directly addresses the suggestion that the cue bonus mainly explains the gains. Adding that bonus to GRPO does not match full ReSum on either reported backbone. The reward-free ReSum variant provides the complementary comparison within the ReSum design. Full ReSum performs better than both controls on each backbone, and the single-group variant also trails the full configuration. The token figures offer a separate view of generation cost for these configurations. A matched rollout count and a matched generated-token budget answer different questions, so we report them separately.
W3. Scope and numerical corrections.
Thanks for this opinion. We agree that the original headline conflated the best-backbone gain with the cross-backbone range. The revised Abstract and Introduction now describe multi-step reasoning and report a range rather than presenting the strongest result as the cross-backbone average. The table below gives the averages from revised Tables 1 and 2. Accuracy is averaged across the six mathematical benchmarks.
The backbone-specific results establish the scope of the gain. We also agree that a significant difference should not be described as an improvement when its direction favors DGPO. The revised significance table in Appendix E reports p-values and directs readers to the main accuracy tables for the direction. The relevant accuracy values from Table 2 are shown below.
The original implication of an improvement in this cell was our error. Revised Table 11 reports the p-value, while Table 2 gives the corresponding accuracy values and their direction.