acceptodds
Under review as a conference paper at ICLR 2027

ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a central technique for improving long-horizon reasoning in Large Language Models (LLMs). Yet, reward-driven reasoning can produce unnecessarily long rollouts, repeated verification, and accumulated errors that reduce coherence and exhaust the generation budget. Existing long-context organization methods commonly rely on external compressors, cache mechanisms, or auxiliary modules. We propose ReSum, an RLVR framework that internalizes self-summarization and related reasoning-control behavior to organize subsequent reasoning within the policy. Our pilot studies reveal that summary-like cues occur around high-uncertainty states, are followed by lower token-level entropy, and can improve recovery from incorrect rollout prefixes. Building on these observations, ReSum constructs adaptive contrastive rollout trees around two complementary events: Natural Points (NPs), where a model-generated cue is masked, and Artifact Points (APs), where a cue is injected at a selected non-cue position. ReSum then applies a dual-reference, cue-aware advantage that evaluates each continuation against cue-containing and cue-free rollout distributions, providing state-local credit assignment for beneficial reasoning-control behavior. Across six mathematical benchmarks, multiple backbone sizes and families, and a multimodal geometry benchmark, ReSum consistently improves reasoning accuracy by an average of 4% while reducing rollout length by 18.6%. These results demonstrate a practical RLVR approach for achieving a stronger accuracy–length trade-off in verifiable long-horizon reasoning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LLM Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_YH76OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReSum, an RLVR method meant to teach a model to “self-summarize” during long reasoning. Two pilot studies motivate it: entropy tends to be high just before summary-like phrases and drops right after, and injecting such a phrase into truncated wrong rollouts makes regenerated continuations more accurate. Training builds rollout trees. From each initial rollout, branches are created either by masking a detected cue (Natural Points) or by injecting one at a random position (Artifact Points). Correct rollouts that contain cues get a small bonus reward, and advantages are computed against a cue group and a no-cue group and then summed.

Critical strengths

The question is a reasonable one: can a model learn when to consolidate its own reasoning, instead of relying on an external compressor? The masking/injection idea is a neat way to get counterfactual branches from the same prefix. The experimental coverage is decent for an RLVR paper, with four backbones, combinations with GPG/DAPO/GSPO, a multimodal setting, a rollout-budget study, and a timing table showing about 13% training overhead. Gains over GRPO and DGPO are consistent in direction across the main tables.

Critical weaknesses
  1. The mechanism doesn’t do what’s claimed. The dual-reference advantage in Eq. (3) uses the same group statistics for every rollout. Adding the two terms therefore gives the same affine function of the reward for all rollouts of a query, which is just a rescaled GRPO advantage with no cue-specific credit. Table 5 also shows ReSum with J=1 (no branching at all) at B=4 already beating GRPO. So the gains seem to come mainly from the cue bonus in Eq. (2): about +0.2 for correct rollouts containing common words like “wait,” or “let me check,” which is closer to a format reward than learned self-summarization. Baselines with GRPO plus this bonus and GRPO at matched B are needed.

  2. The evidence is weaker than the claims. The “long-horizon” framing doesn’t fit ~500–800-token outputs. The 18.6% length reduction is shown only against DGPO on one backbone. The pilot study lacks a neutral-phrase control, and Table 7 compares against an untrained base model rather than GRPO with the same prompt. Table 8 marks DeepSeek-Math-7B AIME24 as significant even though ReSum is lower there (1.66 vs. 1.98). The headline “4%” is the best backbone; across backbones the gain is 2.3–4.0.

Despite these issues, the empirical gains are consistent, so I lean toward acceptance if the authors reframe the mechanism and add the missing baselines.

Questions for authors
  1. Can you add GRPO plus the cue bonus in Eq. (2) without trees (same B), and GRPO at B=4 to compare with ReSum (T=4, J=1)?

  2. When computing the loss on a branch, are the inherited prefix tokens and the injected AP cue tokens included? The injected tokens weren’t sampled by the policy, so including them would effectively be supervised training on cue phrases.

  3. How exactly is the 18.6% length reduction computed, and what is the comparison against GRPO and on other backbones?

  4. For Fig. 1(b), which model and how many prefixes? What happens with a neutral injected phrase?

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer YH76 (1/2)

Thank you for the detailed review and for recognizing the consistency of the empirical gains. The revised manuscript clarifies the mechanism and includes the comparisons needed to assess its contribution. All table references below refer to the revised manuscript.

W1. Advantage and the role of branching.

Thanks for this opinion. Your algebra is correct. For a question with both reference groups nonempty, Eq. (3) is A^(τ)=cqR(τ)−dq\hat A(\tau)=c_qR(\tau)-d_q, with cq>0c_q>0 shared by the rollouts for that question. Section 3.3 now states this explicitly, and the Abstract and Section 3.4 distinguish local interventions from the rollout-level training signal. However, the algebra alone does not establish that the gain comes mainly from the cue bonus. ReSum uses prefix-local cue interventions to construct continuations, whose sequence rewards then enter the rollout-level advantage. The analytical paired gain GiG_i characterizes the local intervention rather than serving as a directly optimized term. The controls below test the reward-only explanation directly.

W2. Matched-budget and reward controls.

Thanks for this opinion. We appreciate your idea. Revised Table 5 in the new manuscript compares rollout budgets on Qwen2.5-3B. The table is also shown in the following:

MethodRollout budget BBReSum configuration (T,J)(T,J)Accuracy (%)
GRPO4N/A23.28
ReSum4(2, 2)26.75
ReSum4(4, 1)27.46
GRPO16N/A25.47
ReSum16(4, 4)28.81

The matched-budget rows make it possible to compare the methods at the same number of rollouts. At the smaller budget, the J=1J=1 configuration spends the budget on initial rollouts, whereas the J=2J=2 configuration allocates part of it to branching. Both improve on the GRPO result at that budget. This comparison shows the performance of the reported configurations without attributing the entire gain to branching alone. At the larger budget, the balanced ReSum configuration also improves on GRPO. The (T,J)(T,J) notation describes ReSum's tree configuration. GRPO instead samples a group of independent rollouts, so the tree-configuration field is not applicable to it.

Revised Table 7 further separates the cue reward from the full method. Accuracy is again averaged across the six mathematical benchmarks.

ConfigurationQwen2.5-Math-1.5B accuracy (%)Qwen2.5-Math-7B accuracy (%)Generated tokens (M/step)
GRPO29.3937.61101.02
GRPO + RFR_F, no tree29.6537.00103.79
ReSum, RF=0R_F=029.3140.1190.93
ReSum, single-group normalization32.8240.5487.04
ReSum, neutral-phrase branching32.5639.6388.04
Full ReSum33.0741.6483.26

The no-tree GRPO control directly addresses the suggestion that the cue bonus mainly explains the gains. Adding that bonus to GRPO does not match full ReSum on either reported backbone. The reward-free ReSum variant provides the complementary comparison within the ReSum design. Full ReSum performs better than both controls on each backbone, and the single-group variant also trails the full configuration. The token figures offer a separate view of generation cost for these configurations. A matched rollout count and a matched generated-token budget answer different questions, so we report them separately.

W3. Scope and numerical corrections.

Thanks for this opinion. We agree that the original headline conflated the best-backbone gain with the cross-backbone range. The revised Abstract and Introduction now describe multi-step reasoning and report a range rather than presenting the strongest result as the cross-backbone average. The table below gives the averages from revised Tables 1 and 2. Accuracy is averaged across the six mathematical benchmarks.

BackboneGRPO accuracy (%)ReSum accuracy (%)Gain over GRPO (percentage points)
Qwen2.5-Math-1.5B29.3933.073.68
Qwen2.5-Math-7B37.6141.644.03
Qwen2.5-3B25.4728.813.34
DeepSeek-Math-7B14.9117.322.41

The backbone-specific results establish the scope of the gain. We also agree that a significant difference should not be described as an improvement when its direction favors DGPO. The revised significance table in Appendix E reports pp-values and directs readers to the main accuracy tables for the direction. The relevant accuracy values from Table 2 are shown below.

Backbone and benchmarkDGPO accuracy (%)ReSum accuracy (%)
DeepSeek-Math-7B, AIME241.981.66

The original implication of an improvement in this cell was our error. Revised Table 11 reports the pp-value, while Table 2 gives the corresponding accuracy values and their direction.

AuthorsOpenReview ↗

Official Response to Reviewer YH76 (2/2)

W4. Length measurements.

Thank you for asking us to clarify this comparison. The approximately 18.6% reduction in Figure 4 is measured using the training rollout lengths at the final training step, step 230, on Qwen2.5-Math-7B. It is the relative decrease from DGPO to ReSum, computed as the difference in their mean rollout lengths divided by the DGPO mean. Revised Table 9 separately reports mean evaluation response lengths in tokens.

BackboneGRPO lengthDGPO lengthReSum lengthReSum reduction vs. DGPO
Qwen2.5-Math-1.5B784.72742.09655.3811.68%
Qwen2.5-Math-7B627.73562.04530.635.59%
Qwen2.5-3B671.95664.02581.4512.43%
DeepSeek-Math-7B458.56440.89416.745.48%

ReSum yields shorter mean evaluation responses than DGPO for every listed backbone. These evaluation averages are distinct from the Figure 4 dynamics. The revised Abstract specifies the backbone and DGPO comparison for the reported dynamics; Section 4.3 and Table 9 present the separate evaluation-length comparison. As Section 3.2 states, ReSum retains the preceding reasoning history and shapes the generated continuation.

W5. Loss masks and pilot details.

Thank you for asking about the loss computation. The injected AP cue tokens do not contribute to the policy loss, so their insertion does not train the model to reproduce those tokens as sampled actions. The inherited prefix is used as context for the branch and is not repeatedly updated as a separate loss contribution for each continuation. This prevents the shared history from receiving additional weight simply because it has multiple branches.

For Figure 1(b), we used Qwen2.5-3B and collected 200 incorrect rollouts. At each selected truncation position, we regenerated 100 continuations per rollout under each condition, with or without an injected summary cue. Each condition therefore used 20,000 continuations per position. This setup examines whether a cue helps the model recover from an otherwise incorrect reasoning trajectory.

To address your question about a neutral phrase, we added a training-level control in Section 4.3. Revised Table 8 compares neutral transitions with summary cues on Qwen2.5-Math-1.5B:

Cue conditionAccuracy (%)Mean response length
No cue29.39784.72
Neutral transition32.56725.20
Summary33.07655.38

The neutral transition improves on the no-cue baseline, indicating that a generic transition can also be useful. The summary-cue configuration achieves higher accuracy and shorter responses than the neutral-phrase configuration in this experiment. This comparison supports a benefit beyond adding an arbitrary transition, while also showing that a neutral phrase is a meaningful control. It complements the prefix-level pilot rather than constituting a neutral-phrase arm of Figure 1(b).

Thank you for pointing out the importance of comparing against a trained model with the same summarization prompt. Table 10 examines a different setting. It applies prompt refinement to the untrained backbone and reports the resulting performance alongside RL-trained ReSum. We have clarified this setup in the table caption and the discussion in Appendix E. The comparison shows how the methods perform in their reported settings, but it does not isolate the effect of prompting on a GRPO-trained model. A comparison using the same prompt on a trained GRPO checkpoint would be useful for answering that question directly.

Thanks again for your patience and expertise on our manuscript! We appreciate your valuable feedback, which helps us improve the quality of our paper.

Dear Reviewer YH76,

Thank you once again for your time and for your valuable feedback on our manuscript.

We have carefully considered all of your comments and have provided a comprehensive response. The corresponding revisions have been made in the manuscript, and we believe these changes have significantly improved our work.

We sincerely hope that our responses and the revised manuscript will resolve your concerns. We are happy to engage in any further discussion.

Best regards,

Authors

Reviewer_VHbjOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReSum, a reinforcement learning framework for self-summarization of reasoning trajectories. ReSum constructs paired reasoning branches using natural summarization points and artificially introduced summarization points , and introduces Summarization-aware Group Policy Optimization to learn the effectiveness of summarization and determine when it should be performed. On Qwen2.5-Math-7B, ReSum achieves an average accuracy of 41.64% across six mathematical reasoning benchmarks, outperforming GRPO (37.61%) and DGPO (39.79%), while reducing reasoning trajectory length by approximately 18.6%. The authors further validate the effectiveness of the proposed method across different model backbones, through component ablations, in combination with existing policy optimization methods, and on multimodal geometric reasoning tasks. Overall, this work presents a promising RLVR optimization mechanism with practical potential. However, the independent contribution of the summarization behavior to the observed performance gains warrants further investigation.

Critical strengths
  1. Consistent Empirical Performance. ReSum achieves strong results across multiple mathematical reasoning benchmarks and different model backbones, improving both accuracy and generation length. In particular, compared with GRPO, ReSum yields a 4.03 percentage-point improvement in average accuracy for Qwen2.5-Math-7B. These results suggest that ReSum not only alters the model's generation behavior but also has the potential to improve its practical reasoning performance.

  2. Compatibility and Applicability. The paper integrates ReSum with several policy optimization methods, including GPG, DAPO, and GSPO, and further evaluates it on multimodal geometric reasoning tasks. These experiments suggest that the self-summarization mechanism can be incorporated into diverse optimization frameworks and has potential for broader applications. Moreover, introducing summarization as an inductive prior to guide rollout branching is intuitive and well aligned with common practices in complex reasoning. By encouraging the model to consolidate intermediate reasoning before exploring alternative continuations, this design provides a natural mechanism for structuring the search process and potentially improving the efficiency of exploration.

Critical weaknesses
  1. The Role of Summarization and Its Relationship to Context Compression Require Further Clarification.

ReSum constructs reasoning branches by inserting or masking summarization phrases to guide subsequent reasoning, while retaining the original reasoning history preceding the summarization point. Consequently, the current experiments do not sufficiently disentangle the effect of the summarization semantics themselves from that of generic prompt interventions on subsequent generation. Nor do they directly demonstrate that the proposed mechanism reduces the amount of historical context actually processed by the model. I encourage the authors to introduce neutral transition prompts as controls under matched branching positions and computational budgets, and to clarify the distinction between reductions in generated trajectory length and actual input context compression.

  1. Potential Ambiguity in the Implementation of the Format Reward.

According to Algorithm 1 in the appendix, the authors determine (N_{\mathrm{NP}}) by counting the occurrences of phrases matching a candidate phrase set in each rollout, while setting (N_{\mathrm{AP}}=1) for branches with artificially introduced summarization points. However, the AP phrase is itself selected from the same candidate set and inserted into the rollout. It is therefore unclear whether this artificially inserted phrase is also counted toward (N_{\mathrm{NP}}), potentially contributing to both (N_{\mathrm{NP}}) and (N_{\mathrm{AP}}). If the implementation explicitly excludes such occurrences from (N_{\mathrm{NP}}), the authors should clarify the corresponding counting rules to ensure consistency between the reward formulation and its actual implementation.

Questions for authors

Question 1: Does ReSum reduce the effective context length or primarily modify subsequent generation behavior?

In the AP branch, the summary phrase is inserted into the existing reasoning prefix before subsequent tokens are generated, while the preceding reasoning history appears to remain intact. Does ReSum explicitly compress or replace the preceding history during training or inference, or does it primarily use summary prompts to influence subsequent generation? How should the reported 18.6% reduction in rollout length be distinguished from a reduction in the context length processed by the model? Clarifying this distinction would help assess the mechanism underlying the reported efficiency gains.

Question 2: How are NP and AP counts computed in the implementation of the format reward?

In Algorithm 1, (N_{\mathrm{NP}}) is derived from the number of matches to the candidate phrase set (\mathcal{P}), while (N_{\mathrm{AP}}=1) is assigned to an AP branch. Since the artificially inserted AP phrase is also selected from (\mathcal{P}), is this phrase excluded when computing (N_{\mathrm{NP}})? Please clarify whether the counts are computed before or after inserting the AP phrase, and how the implementation avoids potential double-counting. This clarification is important for verifying that the implemented reward matches the stated objective.

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer VHbj (1/2)

Thank you for the constructive review and for recognizing ReSum's performance and compatibility with multiple policy objectives. The revised manuscript provides further evidence for the effect of summary cues and clarifies how we measure efficiency. All table references below refer to the revised manuscript. We wish to address all of your concerns.

W1. Reasoning organization and generated length.

Thanks for this opinion. ReSum retains the preceding reasoning history and uses cue-conditioned interventions to shape subsequent generation. Its efficiency result concerns generated response length rather than replacement or compression of the historical input. Section 3.2 already specifies that the preceding history is retained, and the revised discussion in Section 4.3 now makes the generated-length interpretation explicit. The approximately 18.6% reduction reported in Figure 4 compares generated rollout length with DGPO on Qwen2.5-Math-7B at the final training step, step 230.

Revised Table 9 separately reports mean evaluation response lengths and average accuracies across the six mathematical benchmarks.

BackboneMethodAccuracy (%)Response length
Qwen2.5-Math-1.5BGRPO29.39784.72
Qwen2.5-Math-1.5BDGPO30.71742.09
Qwen2.5-Math-1.5BReSum33.07655.38
Qwen2.5-Math-7BGRPO37.61627.73
Qwen2.5-Math-7BDGPO39.79562.04
Qwen2.5-Math-7BReSum41.64530.63
Qwen2.5-3BGRPO25.47671.95
Qwen2.5-3BDGPO27.19664.02
Qwen2.5-3BReSum28.81581.45
DeepSeek-Math-7BGRPO14.91458.56
DeepSeek-Math-7BDGPO16.53440.89
DeepSeek-Math-7BReSum17.32416.74

The table reports higher average accuracy and shorter mean evaluation responses for ReSum than for both baselines on every backbone. These are complementary measurements of the reported accuracy–length trade-off. The evaluation-length means are distinct from the training and evaluation dynamics shown in Figure 4.

The distinction from context compression is important. At an intervention point, ReSum keeps the available reasoning prefix and conditions the continuation on the selected cue. The model can therefore use earlier steps when it generates the rest of the solution. A shorter generated response reflects a change in the continuation produced by the policy; it should not be interpreted as a direct measurement of fewer input-context tokens being processed.

W2. Neutral and cue-category controls.

Thanks for this opinion. Revised Table 7 compares full ReSum with neutral-phrase branching at both scales. Accuracy is averaged over the six mathematical benchmarks.

InterventionQwen2.5-Math-1.5B accuracy (%)Qwen2.5-Math-7B accuracy (%)
Neutral-phrase branching32.5639.63
Full ReSum33.0741.64

Thank you for raising this question. It is important to understand whether the improvement comes from the content of the cue or simply from inserting a phrase during branching. To investigate this, we added a neutral-phrase control and its discussion to Section 4.3. The control uses a neutral transition in place of the summary-oriented intervention, allowing us to examine whether the choice of phrase makes a difference.

As shown above, full ReSum performs better on both backbones, although the difference is smaller on the 1.5B model. The neutral-phrase configuration is also effective, so the results do not suggest that generic transitions have no value. Rather, this comparison provides evidence that the reported gains are not fully explained by inserting a neutral phrase alone. The comparison is at the training level and does not establish what happens at every individual reasoning step.

We view summarization as a broad way of organizing reasoning rather than a behavior with a sharp semantic boundary. Reviewing progress can overlap with checking earlier steps or reconsidering an approach, so ReSum embraces these related summary-like cues. Their presence marks a candidate intervention point; it does not guarantee a particular behavior in the continuation.

AuthorsOpenReview ↗

Official Response to Reviewer VHbj (2/2)

Revised Table 8 compares groups of cue wordings on Qwen2.5-Math-1.5B. The grouping helps examine their effects without requiring the underlying behaviors to be mutually exclusive. Mean response length is measured in tokens.

Cue categoryAccuracy (%)Mean response length
No cue29.39784.72
Neutral transition32.56725.20
Verification / repair32.39809.55
Strategy switch28.09797.08
Summary33.07655.38

Summary cues achieve the best accuracy and shortest responses among the reported categories. The neutral transition provides a comparison in which a phrase is introduced without explicitly asking for a summary. The other categories show how alternatives such as verification or strategy switching behave in the same reported analysis. These training-level controls complement the original prefix-based pilot in Figure 1(b), which compares continuations with and without a summary cue. The cue lists appear in Appendix H.

W3. NP and AP counting.

Thanks for this opinion. In revised Algorithm 1, NNPN_{\rm NP} counts natural cue matches excluding the injected AP phrase. An AP branch sets NAP=1N_{\rm AP}=1; otherwise NAP=0N_{\rm AP}=0. The stated counting rule therefore assigns the injected event to the AP count. The missing else branch was an error in the original pseudocode. We corrected it and made the exclusion rule explicit in Algorithm 1 in Appendix F. We will confirm the corresponding implementation detail against the runs that produced the reported results.

W4. Practical setting.

Thanks for this opinion. ReSum combines the cue interventions with a correctness-gated reward and dual-reference optimization. Revised Table 7 reports generated tokens per training step for the listed configurations.

ConfigurationGenerated tokens (M/step)
GRPO101.02
ReSum, neutral-phrase branching88.04
Full ReSum83.26

We understand the concern that constructing branches could make ReSum impractical by requiring substantially more generation during training. However, when we keep the rollout budget the same for both GRPO and ReSum, ReSum still maintains a performance advantage. The results in Table 7 proves this: in the reported setting, full ReSum generates fewer tokens per training step than GRPO and the neutral-phrase variant. This suggests that the additional branches do not necessarily translate into a larger generated-token count per step.

Generated tokens alone, however, do not capture all of the computational work involved. Branching and managing the rollout tree can also affect elapsed time. For this reason, Appendix E reports a separate timing comparison with DGPO on Qwen2.5-Math-1.5B. ReSum has a higher reported training-step time in that comparison, despite the lower token count shown above against the configurations in Table 7. We therefore present token use and timing as complementary measures, rather than using one to claim an overall reduction in training cost.

Thanks again for your patience and expertise on our manuscript! We appreciate your valuable feedback, which helps us improve the quality of our paper.

Dear Reviewer VHbj,

Thank you once again for your time and for your valuable feedback on our manuscript.

We have carefully considered all of your comments and have provided a comprehensive response. The corresponding revisions have been made in the manuscript, and we believe these changes have significantly improved our work.

We sincerely hope that our responses and the revised manuscript will resolve your concerns. We are happy to engage in any further discussion.

Best regards,

Authors

Reviewer_gv9aOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Reinforcement Learning with Verifiable Rewards (RLVR) has substantially enhanced the long-chain reasoning capabilities of large language models (LLMs). However, reward-driven optimization can favor excessively long Chain-of-Thought (CoT) sequences, leading to overthinking. As reasoning trajectories grow longer, models may lose track of earlier steps, revisit resolved subproblems, or exhaust their reasoning budgets through repeated retries and redundant self-verification. Existing approaches to managing long reasoning trajectories often rely on external compressors, caching mechanisms, or auxiliary modules. To address these limitations, ReSum proposes an RLVR framework that internalizes self-summarization and reasoning control within the policy, enabling more efficient organization of subsequent reasoning.

Critical strengths
  1. ReSum introduces a novel RLVR training framework that enables models to proactively summarize, reflect, or adjust their strategies at appropriate reasoning steps, thereby reducing redundant reasoning and improving answer accuracy.
  2. The framework incorporates two key innovations: Summarization-based Rollout Expansion and SGPO.
Critical weaknesses
  1. The paper does not establish that RLVR necessarily leads to longer reasoning trajectories, nor does it demonstrate that longer reasoning inherently results in overthinking.
  2. The proposed advantage function aims to evaluate the current rollout relative to trajectories with or without cues. However, linearly combining the two advantages conflates their effects. A more appropriate approach would be to directly compare the rewards of cue-conditioned versus cue-free trajectories.
  3. How does the method ensure that phrases in the cue pool trigger summarization rather than other reasoning behaviors? For example, phrases such as "wait" or "let me check" are included in the cue pool, yet they may induce reflection or verification instead of summarization.
  4. The design of the RFR_F reward implicitly assumes that summarization improves reasoning performance. However, summarization may not always be beneficial. Under the current formulation, trajectories with unhelpful summarization may still receive higher rewards, potentially encouraging unnecessary summarization.
Questions for authors
  1. Figure 1(a) does not clearly illustrate the differences. A zoomed-in view of the entropy changes before and after summarization would make them more apparent.
14 comments
AuthorsOpenReview ↗

Official Response to Reviewer gv9a (1/2)

Thank you for your precious opinions. The revised manuscript clarifies how ReSum shapes reasoning and provides further evidence for the effectiveness of its design. All table references below refer to the revised manuscript. We hope to address your concerns.

W1. Motivation and empirical setting.

Thanks for this opinion. We appreciate your concern and agree that our motivation needs clarification. We do not suggest that RLVR inevitably makes reasoning trajectories longer. Nor do we regard a long response alone as evidence of overthinking. Additional reasoning can be valuable when a problem requires it. Our concern is that a model may sometimes revisit steps it has already resolved or repeat the same verification. It may also pursue alternatives that do not help solve the problem. In these cases, generating more tokens does not necessarily lead to a better solution.

ReSum is motivated by the possibility of helping the model organize intermediate progress and continue reasoning more effectively. Our aim is therefore not simply to make every response shorter, but to improve the balance between solution accuracy and generation length. We have revised the Abstract to describe the setting as multi-step verifiable reasoning. To make the accuracy–length trade-off clear, Section 4.3 compares the methods across four backbones in Table 9. The table reports accuracy averaged over six mathematical benchmarks alongside mean evaluation response length in tokens.

BackboneMethodAccuracy (%)Response length
Qwen2.5-Math-1.5BGRPO29.39784.72
Qwen2.5-Math-1.5BDGPO30.71742.09
Qwen2.5-Math-1.5BReSum33.07655.38
Qwen2.5-Math-7BGRPO37.61627.73
Qwen2.5-Math-7BDGPO39.79562.04
Qwen2.5-Math-7BReSum41.64530.63
Qwen2.5-3BGRPO25.47671.95
Qwen2.5-3BDGPO27.19664.02
Qwen2.5-3BReSum28.81581.45
DeepSeek-Math-7BGRPO14.91458.56
DeepSeek-Math-7BDGPO16.53440.89
DeepSeek-Math-7BReSum17.32416.74

Across the reported backbones, ReSum combines higher average accuracy with shorter generated responses than both comparison methods. This directly illustrates the accuracy–length trade-off that motivates our approach.

W2. Dual-reference advantage.

ReSum constructs cue-conditioned interventions at shared reasoning prefixes and uses the resulting rollout rewards for optimization. Your observation about Eq. (3) is correct. With fixed question-level group statistics, the sum of the two normalized terms is an affine function of the rollout reward, A^(τ)=cqR(τ)−dq\hat A(\tau)=c_qR(\tau)-d_q with cq>0c_q>0. Section 3.3 now states this explicitly. The prefix-matched expected return difference GiG_i characterizes the intervention theoretically; the training objective uses the rollout-level dual-reference advantage.

The single-group control in revised Table 7 evaluates the normalization choice directly. Accuracy is averaged over the six mathematical benchmarks.

NormalizationQwen2.5-Math-1.5B accuracy (%)Qwen2.5-Math-7B accuracy (%)
Single-group normalization32.8240.54
Dual-reference normalization33.0741.64

The affine simplification is a valid observation about the estimator, but it does not imply that the choice of reference distributions has no effect on training. The full dual-reference configuration improves on the single-group alternative at both scales. In the full method, each rollout is evaluated relative to the reward distributions of both cue-containing and cue-free samples. This control tests the normalization choice within the overall ReSum design. The shared prefix provides a way to characterize how a cue changes the continuation, while the training update incorporates rollout outcomes through the question-level advantage. We have made this distinction explicit in the revision.

AuthorsOpenReview ↗

Official Response to Reviewer gv9a (2/2)

W3. About cue semantics.

Thank you for raising this question. We fully appreciate this. We view summarization in reasoning as a broad process with no sharp semantic boundary. A model may organize its progress by reviewing what it has established, checking an earlier step, or reconsidering its approach. These behaviors can overlap in practice and help the model decide how to continue. For this reason, ReSum embraces a broad pool of summary-like cues rather than limiting the intervention to one form of wording. As described in Section 3.2 and Appendix D, the pool also includes phrases associated with verification, repair, and strategy switching. A cue provides a candidate intervention point; its presence alone does not guarantee that the subsequent text performs a particular behavior.

Although the semantic boundaries between these categories are not always clear, during the rebuttal we made our best effort to collect separate cue lists for each category. We then compared them in Table 8 of Section 4.3 to examine how different cue wordings affect performance. The results below use Qwen2.5-Math-1.5B, with response length measured in tokens.

Cue categoryAccuracy (%)Mean response length
No cue29.39784.72
Neutral transition32.56725.20
Verification / repair32.39809.55
Strategy switch28.09797.08
Summary33.07655.38

The summary-labeled cues achieve the highest accuracy and shortest responses among the reported categories. Neutral transitions provide a control that introduces a phrase without explicitly directing the model to review its progress. The verification and repair group also improves on the no-cue baseline, while producing longer responses. These differences show how the choice of wording relates to performance in the reported setting. They do not require us to draw a rigid semantic boundary between the underlying behaviors. The category analysis is in Section 4.3, and the phrase lists are in Appendix H.

W4. Correctness-gated cue reward.

Thanks for this opinion. The auxiliary reward is activated only when the final answer is correct, and Eq. (2) bounds its contribution by 0.2. Revised Table 7 compares the full configuration with a reward-free variant and with GRPO using the same auxiliary reward without a tree.

ConfigurationQwen2.5-Math-1.5B accuracy (%)Qwen2.5-Math-7B accuracy (%)
GRPO + RFR_F, no tree29.6537.00
ReSum, RF=0R_F=029.3140.11
Full ReSum33.0741.64

A cue is not assumed to improve every continuation. The auxiliary reward is conditioned on a correct final answer, and the experiments assess the resulting method rather than treating the presence of a cue as proof of helpful summarization. The full configuration outperforms both controls at both scales. The reward-free variant helps assess the auxiliary reward within ReSum, while the no-tree GRPO variant shows the result of adding the reward to a different rollout design. Since the auxiliary term is gated by answer correctness, it favors cue-bearing solutions within the set of correct responses. These controls support considering the reward together with rollout construction and normalization.

W5. Concerning the Figure 1(a).

Thank you for the suggestion!! We agree that the entropy change around summary-like cues is easier to see at a larger scale. We have added an enlarged view of the entropy profile around the cue positions in Appendix Figure 6. This view complements the full profile in Figure 1(a) and makes the decrease after the cues more visible.

Thanks again for your patience and expertise on our manuscript! We appreciate your valuable feedback, which helps us improve the quality of our paper.

Dear Reviewer gv9a,

Thank you once again for your time and for your valuable feedback on our manuscript.

We have carefully considered all of your comments and have provided a comprehensive response. The corresponding revisions have been made in the manuscript, and we believe these changes have significantly improved our work.

We sincerely hope that our responses and the revised manuscript will resolve your concerns. We are happy to engage in any further discussion.

Best regards,

Authors

Reviewer_xRBfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReSum, an RLVR framework that encourages self-summarization and related reasoning-control behaviors. It constructs rollout branches by removing naturally occurring cues at Natural Points and injecting cues at Artifact Points. Training combines a correctness-gated auxiliary cue reward with an advantage normalized against both cue-containing and cue-free rollout groups. Experiments across six mathematical benchmarks, four text backbones, and a multimodal geometry benchmark show consistent average accuracy improvements. The paper also reports shorter generated responses and modest additional training overhead.

Critical strengths
  1. The paper addresses a relevant problem: improving reasoning accuracy while reducing redundant generation. The method requires neither an additional process reward model nor a dedicated inference-time module.

  2. The empirical improvements extend across multiple model sizes and families. ReSum improves average accuracy over both GRPO and DGPO on all four evaluated text backbones, and its combinations with other policy optimization objectives also show gains.

  3. The paper provides component ablations, branching-budget analysis, and efficiency measurements. The substantial degradation after removing Natural Points suggests that branching around naturally occurring reasoning-control events is a useful direction.

Critical weaknesses
  1. The claimed state-local credit assignment is not sufficiently supported by the advantage formulation. In Sec. 3.4, the paper states that “the matched branch exposes the value of proceeding without that intervention” (L252–253), and defines the local intervention gain as Gi=Rcue(i)−Rnocue(i)G_i=R_{\mathrm{cue}}(i)-R_{\mathrm{nocue}}(i) for continuations from the same prefix (L260–261). However, Eq. 3 uses statistics pooled across cue-containing and cue-free rollouts for the same question, without explicitly retaining this prefix-matched comparison. When both groups are nonempty, the resulting advantage simplifies to A^(τ)=cqR(τ)−dq,\hat A(\tau)=c_qR(\tau)-d_q, where cqc_q and dqd_q are shared across all rollouts for question qq. Consequently, rollouts with identical total rewards receive identical advantages regardless of their local intervention gains. Moreover, this rollout-level advantage “is assigned to all generated tokens” (L235). While the branching strategy provides local interventions and the normalization may improve training, these properties do not by themselves establish state-local credit assignment. Could the authors clarify how the local gain GiG_i enters the actual optimization, and provide evidence distinguishing this mechanism from sequence-level reward shaping and normalization?

  2. The experiments do not sufficiently isolate the proposed mechanism. ReSum changes rollout sampling, reward shaping, and advantage normalization simultaneously. Table 6 provides useful component ablations, but does not isolate the auxiliary cue reward through comparisons with ReSum without RFR_F and standard GRPO with the same RFR_F. A matched branching baseline without cue intervention would also help determine whether the gains primarily arise from additional exploration. Furthermore, the cue pool includes verification, repair, and strategy-switching expressions, so the results do not establish the specific contribution of summarization. The paper acknowledges this broader scope, but a comparison between consolidation-only and other cue categories would substantially strengthen the interpretation. Table 7 also compares prompt refinement on an untrained backbone against RL-trained ReSum, rather than testing the same prompting strategy on an RL-trained baseline.

  3. Comparisons with the closest rollout-based methods are missing. The paper discusses TreeRL and Tree-GRPO, but the main comparisons emphasize policy optimization objectives rather than alternative branching and credit-assignment strategies. Since tree-based rollout construction is central to ReSum, a comparison against a closely related tree-based method under matched generation-token or compute budgets is important for establishing its incremental contribution. Matching the number of completed rollouts alone does not establish equivalent computational cost. The reported timing experiment is useful, but covers only one 1.5B configuration.

  4. Important implementation details remain unclear. The paper should specify whether injected cue tokens contribute to the policy loss, whether shared prefixes receive repeated updates across branches, and what masking an NP entails. Algorithm 1 also appears inconsistent with the stated procedure: branching is executed after the initialization block even when j=1j=1, potentially overwriting the initial rollout, and NAPN_{\mathrm{AP}} is unconditionally reset to zero after being set to one for an AP branch. These may be pseudocode errors rather than implementation errors, but they affect reproducibility. Finally, the value of the advantage stabilizer ξ\xi is unspecified. This matters because a reference group with zero reward variance can produce large cross-group advantages.

Questions for authors
  1. How does the prefix-specific intervention gain GiG_i enter the implemented training objective? Is the claimed local credit assignment attributable to the sampling procedure, the advantage estimator, or both?
  2. What are the results for ReSum without the auxiliary cue reward, standard GRPO with the same reward, and matched branching without cue intervention?
  3. Can the authors provide a comparison with a closely related tree-based RL method under a matched generation-token or compute budget?
  4. Please clarify the loss masks, shared-prefix handling, NP masking operation, and ξ\xi, and reconcile Algorithm 1 with the implementation used for the experiments.
3 comments
AuthorsOpenReview ↗

Official Response to Reviewer xRBf (1/2)

Thank you for the careful assessment and for recognizing ReSum's cross-backbone gains and practical design! The revised manuscript clarifies how the method constructs interventions and provides further experimental evidence for its design. All table references below refer to the revised manuscript. We hope these would address your concerns.

W1. Local interventions and the advantage.

Thanks for this opinion. ReSum constructs interventions at a shared reasoning prefix. At a natural point, it masks the detected cue and regenerates the continuation; at an artifact point, it injects a cue and regenerates from the selected prefix. These interventions shape rollout construction, while the resulting rewards enter the two reference distributions.

Your derivation of Eq. (3) is correct. When both reference groups are nonempty, the advantage reduces to A^(τ)=cqR(τ)−dq\hat A(\tau)=c_qR(\tau)-d_q, where cq=∑g(σg+ξ)−1>0c_q=\sum_g(\sigma_g+\xi)^{-1}>0 and dq=∑gμg/(σg+ξ)d_q=\sum_g\mu_g/(\sigma_g+\xi). Section 3.3 now states this explicitly. The paired gain GiG_i characterizes the local intervention theoretically; the implemented objective uses the rollout-level advantage. The local comparison therefore arises from the branching procedure rather than an explicit per-prefix term in the advantage. We corrected the compressed description in the Abstract and clarified this distinction in Sections 3.3 and 3.4. The algebraic simplification does not make the branching design redundant. It changes the continuations and reference distributions used for training, which is why the component controls below are important.

W2. Component and cue controls.

We appreciate this idea. In the revised maniscript, Table 7 reports the requested component comparisons. Accuracy is the average across the six mathematical benchmarks.

ConfigurationQwen2.5-Math-1.5B accuracy (%)Qwen2.5-Math-7B accuracy (%)
GRPO29.3937.61
GRPO + RFR_F, no tree29.6537.00
ReSum, RF=0R_F=029.3140.11
ReSum, single-group normalization32.8240.54
ReSum, neutral-phrase branching32.5639.63
Full ReSum33.0741.64

These controls address the concern that the gain could be explained by the cue reward or by an arbitrary branching phrase alone. Full ReSum leads both the no-tree reward baseline and neutral-phrase branching on the two backbones. The single-group comparison also favors the full normalization. In addition to these new controls, the original component ablation already examined the roles of the two types of branching points. We retain those results in revised Table 6 and discuss them in Section 4.3.

ConfigurationQwen2.5-Math-1.5B accuracy (%)Qwen2.5-Math-7B accuracy (%)
Without natural points30.1138.94
Without artifact points31.2140.73
Full ReSum33.0741.64

Both removals reduce average accuracy. In particular, the natural-point ablation supports the usefulness of branching at model-generated reasoning-control events, alongside the new reward and phrase controls.

We view summarization in reasoning as a broad process rather than a behavior with a sharp semantic boundary. Reviewing progress, checking an earlier step, or reconsidering an approach can be related ways to reorganize reasoning before continuing. ReSum therefore includes these summary-like cues in its broader cue pool. A cue marks a possible intervention point, rather than guaranteeing that a particular behavior occurs.

To examine how different wordings affect performance, we added a cue-category comparison in Table 8 of Section 4.3. The categories group phrases for this analysis; they are not intended as mutually exclusive definitions of the underlying reasoning behaviors. Results are on Qwen2.5-Math-1.5B, and response length is measured in tokens.

Cue categoryAccuracy (%)Mean response length
No cue29.39784.72
Neutral transition32.56725.20
Verification / repair32.39809.55
Strategy switch28.09797.08
Summary33.07655.38

Summary cues achieve the strongest accuracy and shortest responses among the reported categories. The neutral-transition condition helps distinguish the effect of summary-oriented wording from an intervention that simply introduces a transition. Verification and repair cues also improve on the no-cue baseline, but their reported responses are longer. The category comparison therefore gives a more detailed picture of how the wording used at a reasoning point relates to subsequent performance. The phrase lists are provided in Appendix H.

AuthorsOpenReview ↗

Official Response to Reviewer xRBf (2/2)

W3. Rollout budget and related methods.

Thanks for this opinion. Revised Table 5 compares ReSum and GRPO at the same rollout counts on Qwen2.5-3B. Accuracy is the average across the six mathematical benchmarks.

MethodRollout budget BBReSum configuration (T,J)(T,J)Accuracy (%)
GRPO4N/A23.28
ReSum4(2, 2)26.75
ReSum4(4, 1)27.46
GRPO16N/A25.47
ReSum16(4, 4)28.81

From the table, we can find that both ReSum configurations at the smaller budget outperform GRPO at the same rollout count. The configuration with branching provides a direct view of how cue-conditioned continuations perform under that budget, while the configuration with initial rollouts shows the result when the budget is allocated differently. At the larger budget, the balanced tree configuration also exceeds GRPO. Here (T,J)(T,J) describes ReSum's tree structure; the entries for GRPO are not applicable because GRPO uses a group of independently sampled rollouts.

Moreover, matching the number of rollouts is useful for understanding how each method uses its samples, while generated-token cost provides an additional view of the computational budget. Revised Table 7 reports the following measurements per training step for its listed configurations.

ConfigurationGenerated tokens (M/step)
GRPO101.02
GRPO + RFR_F, no tree103.79
ReSum, neutral-phrase branching88.04
Full ReSum83.26

In this measurement, full ReSum generates fewer tokens per step than both GRPO variants. Appendix E also reports training-step timing for the Qwen2.5-Math-1.5B setting. Together, these results give the rollout-count and generation-cost context for the comparisons above; they are not a substitute for a direct comparison with another tree-based method.

We agree that TreeRL is a relevant baseline for this question and are working on reproducing it in our mathematical reasoning setting. Adapting TreeRL to our training framework and tuning its hyperparameters require additional time beyond the rebuttal period, so the reproduction is still in progress. We plan to include the results in the final version, with the same task and evaluation protocol and a matched generation-token budget. Nevertheless, we have already cited and discussed TreeRL in the Related Work section, where we distinguish its uncertainty-based branching from ReSum's cue-conditioned interventions.

For Tree-GRPO, it addresses a different setting. Its tree nodes are built around ReAct thought–action steps in an agentic RL workflow that interacts with an environment, whereas our mathematical reasoning experiments use solution rollouts without that interaction structure. A direct comparison under the present protocol would therefore require adapting the task and rollout framework to supply meaningful ReAct actions and observations. For this reason, we focus the requested tree-method comparison on reproducing TreeRL for our setting.

W4. Algorithm and reproducibility.

We agree that the two pseudocode cases were incorrect, and we have corrected them in Algorithm 1 in Appendix F. The initialization and branching cases now have an explicit if/else, as does the AP-count assignment. The revised algorithm defines NNPN_{\rm NP} as natural cue matches excluding an injected AP cue, and sets NAP=1N_{\rm AP}=1 for an AP branch and zero otherwise.

For an AP branch, the injected cue tokens are excluded from the policy loss. The inherited prefix provides the context for generating the branch; it is not repeatedly updated as a separate loss contribution for each branch. This avoids treating the injected phrase as a policy-generated action or giving the shared history extra weight simply because it has multiple continuations.

At an NP, we use longest matching subsequence to locate a summary cue in the original rollout. We then retain the prefix preceding the matched cue and regenerate from that point, without including the original cue in the branch input. The advantage stabilizer is set to ξ=10−6\xi=10^{-6}.

W5. Prompt refinement.

Thank you for pointing out this distinction. In Table 10, prompt refinement is applied to the backbone before RL training, whereas ReSum is evaluated after RL training. We have clarified this setup in the table caption and the discussion in Appendix E.

The comparison shows the performance of the two approaches in their respective settings. It does not, however, determine how much an explicit summarization prompt would help a GRPO-trained model. Applying the same prompt to a trained GRPO checkpoint would provide a more direct comparison. Our component and cue-category controls in Tables 7 and 8 address a related but different question: how the reward, branching design, and choice of cue affect the reported training configurations.

Thanks again for your patience and expertise on our manuscript!

Dear Reviewer xRBf,

Thank you once again for your time and for your valuable feedback on our manuscript.

We have carefully considered all of your comments and have provided a comprehensive response. The corresponding revisions have been made in the manuscript, and we believe these changes have significantly improved our work.

We sincerely hope that our responses and the revised manuscript will resolve your concerns. We are happy to engage in any further discussion.

Best regards,

Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thanks for your constructive feedback.

Dear PCs, SACs, ACs and Reviewers:

We thank the reviewers for their constructive feedback. We have uploaded the revised manuscript, with changes marked in blue. Below we summarize the main revisions; all table and figure numbers refer to the revised manuscript. Further details are in our individual responses.

Mechanism (Reviewers xRBf, gv9a, and YH76; Sections 3.3–3.4). As xRBf and YH76 observed, the dual-reference advantage is affine in rollout reward when both groups are nonempty. We agree and now state this in Section 3.3. For gv9a, we clarify that cue interventions construct continuations from a shared prefix, while optimization uses a rollout-level advantage. The paired gain helps us analyze the effect of a cue at a particular reasoning point, but it is not directly optimized during training. The branches still matter because they provide different continuations and rewards for the training process.

Component and cue controls (Reviewers xRBf, gv9a, VHbj, and YH76; Section 4.3, Tables 6–8; Appendix H). To address concerns about the auxiliary reward, unhelpful cues and generic branching or phrase insertion, Table 7 compares the following configurations. Accuracy is averaged over six mathematical benchmarks; Table 6 retains the NP and AP ablations.

ConfigurationQwen2.5-Math-1.5BQwen2.5-Math-7B
GRPO + auxiliary reward, no tree29.6537.00
ReSum without auxiliary reward29.3140.11
ReSum with single-group normalization32.8240.54
ReSum with neutral-phrase branching32.5639.63
Full ReSum33.0741.64

Full ReSum outperforms these controls on both backbones, supporting consideration of reward, branching and normalization together. In response to xRBf and gv9a's questions about cue semantics, Table 8 compares cue wordings, with lists in Appendix H. We view summarization broadly, overlapping with verification, repair and strategy switching. The summary-oriented configuration achieves the highest accuracy and shortest responses among those compared.

Budget, efficiency and related methods (Reviewers xRBf and YH76; Section 4.3, Tables 5 and 7; Appendix E; Section 2). For YH76, Table 5 adds GRPO at the smaller rollout budget; both ReSum configurations outperform it at the same rollout count. As xRBf noted, equal rollout counts do not imply equal cost. Table 7 reports generated tokens per step, while Appendix E reports timing; fewer tokens do not necessarily mean lower wall-clock time.

For xRBf's request for tree-based comparisons, Section 2 discusses TreeRL. Its reproduction is ongoing, and we plan a matched comparison in the final version. Tree-GRPO uses ReAct-style agentic interactions; direct comparison requires adapting the task and rollout framework.

Scope and length (Reviewers gv9a, VHbj, and YH76; Abstract, Introduction, Section 4.3 and Appendix E). For gv9a, we clarify that RLVR does not necessarily lengthen every trajectory and that length alone does not establish overthinking. Following YH76's concerns, we focus on multi-step verifiable reasoning, distinguish the strongest backbone gain from the cross-backbone average, and correct the Appendix E significance table where accuracy favors DGPO.

For VHbj, we clarify that ReSum retains the preceding history and changes the continuation rather than explicitly compressing context. For YH76, Figure 4's 18.6% reduction compares training rollout lengths against DGPO on Qwen2.5-Math-7B at step 230. Table 9 separately reports evaluation accuracy and mean response length across four backbones. Following gv9a's suggestion, Appendix Figure 6 enlarges the entropy profile around the cues in Figure 1(a).

Algorithm and reproducibility (Reviewers xRBf, VHbj, and YH76; Appendix F and individual responses). For xRBf, we corrected two pseudocode errors in Algorithm 1; for VHbj, we clarified NP and AP counting. Our responses to xRBf and YH76 specify the loss treatment of injected AP tokens and inherited prefixes; xRBf's response also explains NP regeneration and the advantage stabilizer.

For YH76, we provide Figure 1(b)'s model and sampling details and distinguish Table 8's training-level neutral-phrase control from a separate pilot arm. As xRBf and YH76 noted, prompt refinement was not applied to a GRPO-trained checkpoint. Appendix E and Table 10 clarify that it uses the backbone before RL training; the comparison therefore does not isolate prompting on a GRPO-trained model.

Thanks again for your patience and expertise on our manuscript!

Author of Submission14776

AuthorsOpenReview ↗

A summary of the rebuttal. We would like to thank everyone for their efforts.

We thank the Program Chairs, Senior Area Chair, Area Chair, and reviewers for their continued engagement. Changes in the revised manuscript are marked in blue. Below we summarize where each concern now stands; further detail is in our individual responses.

Mechanism and credit assignment (Reviewers xRBf, gv9a, and YH76; Sections 3.3–3.4). The derivation that the dual-reference advantage is affine in the rollout reward is correct, and Section 3.3 now states it as A^(τ)=cqR(τ)−dq\hat A(\tau)=c_qR(\tau)-d_q. We separate the prefix-matched gain GiG_i, which characterizes the intervention, from the rollout-level advantage used in the update. The single-group control in Table 7 tests this choice, at 32.82% and 40.54% against 33.07% and 41.64%.

Component, reward, and phrase controls (Reviewers xRBf, gv9a, VHbj, and YH76; Tables 6–8). GRPO with the same auxiliary reward but no tree reaches 29.65% and 37.00%, and ReSum without the reward 29.31% and 40.11%, neither matching full ReSum. Neutral-phrase branching (32.56% and 39.63%) and single-group normalization (32.82% and 40.54%) also trail. Table 8 compares cue categories on Qwen2.5-Math-1.5B, where summary cues give the highest accuracy and the shortest responses (33.07%, 655.38 tokens). The auxiliary reward is gated by answer correctness and bounded by 0.2, so it cannot raise the reward of an incorrect solution.

Budget, cost, and related methods (Reviewers xRBf and YH76; Tables 5 and 7, Appendix E). At matched rollout budgets on Qwen2.5-3B, ReSum exceeds GRPO at B=4B{=}4 (26.75% and 27.46% against 23.28%) and at B=16B{=}16 (28.81% against 25.47%). Equal rollout counts do not imply equal cost: generated tokens per step are 83.26M against 101.02M, while step time is higher for ReSum than DGPO on the 1.5B model. Reproducing TreeRL under a matched budget is ongoing.

Scope, length, and numerical corrections (Reviewers gv9a, VHbj, and YH76; Abstract, Section 4.3, Table 9). We do not claim that RLVR inevitably lengthens reasoning or that a long response is itself overthinking; the concern is revisiting resolved steps, repeating verification, and pursuing alternatives that do not advance the solution. ReSum retains the preceding history and changes the continuation, so the efficiency result concerns generated length rather than processed input context. The 18.6% reduction is training rollout length against DGPO on Qwen2.5-Math-7B at step 230. Appendix Figure 6 enlarges the entropy profile, and the Appendix E significance marking and prompt-refinement baseline are corrected.

Implementation and figures (Reviewers xRBf, gv9a, VHbj, and YH76; Appendix F, Figures 1(b) and 3). Algorithm 1 now has explicit branches for initialization, branching, and the AP count, with NNPN_{\mathrm{NP}} excluding the injected phrase. Injected AP tokens are excluded from the policy loss, and the inherited prefix serves as context rather than a separate per-branch contribution. At a natural point we retain the prefix before the cue and regenerate without it, so the model may produce the cue again; trajectories are classified after the whole tree is generated. Figure 1(b) used Qwen2.5-3B with 200 incorrect rollouts and 100 regenerations per condition at each truncation position. In Figure 3 the training curves start at step 2 and the evaluation curves at step 10 for legibility, both from the same base model, with unsmoothed values provided. Artifact points are chosen at random, and a position-wise training study would need a separate model per position, which the remaining period does not allow.

Reasoning model. On DeepSeek-R1-Distill-Qwen-7B, trained for 230 steps under the same rollout budget and evaluated on all 500 MATH-500 problems, accuracy is 85.60%, 87.20%, and 88.40% for the base, GRPO, and ReSum models, with mean generated lengths of 3292.5, 2857.7, and 2527.6 tokens. Split by correctness, ReSum shortens correct trajectories from 1388.9 to 728.0 tokens and incorrect ones from 2028.4 to 1209.0, against final DGPO means of 1517.3 and 1945.2. Post-hoc drift at steps 100, 200, and 230 is 0.00012, 0.00034, and 0.00045 nats per token for ReSum, against 0.00022, 0.00111, and 0.00092 for GRPO.

We are grateful to the Program Chairs, the Senior Area Chair, the Area Chair, and all reviewers for the time they have given to this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.