acceptodds
Under review as a conference paper at ICLR 2027

ReAPO: Retry-Aware Policy Optimization for Efficient Exploration in LLM Reasoning

Abstract

RLVR improves language-model reasoning, but commonly relies on independent stochastic rollouts for exploration, which can repeatedly sample similar failed strategies. Prior work has seen success with approaches that guide exploration by introducing dependence into the sampling process by including incorrect solutions in the context for subsequent attempts. Still, these methods introduce the additional question of how to assign credit to each attempt. We propose to view the credit assignment through the lens of *Stochastic Shortest Paths*, a framework in classical reinforcement learning that seeks to minimize the expected cumulative cost to reach a terminal state (in our case, a verified successful response). We show that, under suitable simplifying assumptions, this perspective naturally leads to a simple modification to GRPO, which we call **Retry-Aware Policy Optimization (ReAPO)**. ReAPO allocates additional generation to failed trajectories, allowing successful retries to be reinforced without rewarding preceding failures, and assigns credit in a theoretically principled way to each attempt. Across Qwen3 models from 0.6B to 4B and eight mathematical and scientific reasoning benchmarks, ReAPO improves first-attempt performance over GRPO as well as natural recent baseline algorithms, while additional ablations demonstrate the utility of both serial sampling and ReAPO's credit assignment. ReAPO also outperforms GRPO with larger group size, with gains preserved under token- and wall-clock-normalized comparisons. Taken together, our results suggest that ReAPO is a simple yet effective approach for improving LM reasoning performance through improved exploration and retry-aware credit assignment.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, large language models, reasoning, exploration, verifiable rewards, policy optimization, self-correction, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_39G9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReAPO, which lets the model retry failed answers using the previous answer and feedback. It computes advantages separately for each attempt, so a successful retry does not give the earlier failure the same reward. The authors derive this design from an objective that minimizes failures before success. Experiments on Qwen3 0.6B, 1.7B, and 4B show improvements in first-attempt accuracy and training efficiency on math and science reasoning tasks.

Critical strengths
  1. The method is simple and addresses a practical problem: independent samples can repeat the same mistakes. The gradient derivation and the analyses of truncation and normalization help explain how credit is assigned to each attempt.

  2. The experiments include multiple independent training seeds, comparisons of generated tokens and training time, and a Multi-Attempt GRPO baseline that shares reward across the full trajectory. These checks support the method's effectiveness in the tested settings.

Critical weaknesses
  1. The analysis does not adequately account for the information gained from failed attempts. A failure can help the model reach a correct answer later, so penalizing it based only on its own correctness may discourage useful exploration.

  2. The experiments are limited to small Qwen3 models, with two attempts in the main setting and a three-attempt ablation. These short retry settings provide limited evidence for stronger models or tasks that require longer trial and error.

Questions for authors
  1. If a failed first attempt provides useful information for a successful second attempt, why should it receive a negative advantage? Can the authors distinguish helpful from unhelpful failures and explain when the theoretical assumptions hold in practice?
3 comments
AuthorsOpenReview ↗

Response to Review

We thank the reviewer for their review. We respond to individual concerns below.

W1/Q1: Why should a useful failed attempt receive a negative advantage?

We agree that a failed attempt can carry information that helps a later retry succeed, and this mechanism is used by prior work like CAL-GRPO to maximize the probability of a correct answer out of multiple serial attempts, which only makes sense when the verifier is available at inference time. ReAPO does not make this assumption, which is why we penalize the first failed attempt. To reiterate, the model must get the answer on its first attempt during evaluation so incorrect early attempts should not be rewarded. Conditional retries are therefore a training-time mechanism for exploration and credit assignment. Our objective, the expected number of failures before success, reflects this perspective: a failed attempt remains a failure, even if it later enables a correct retry. The useful information in that failure isn’t discarded, since it is credited to the successful retry, whose positive advantage reinforces the reasoning that recovered from the mistake (Table 7, page 26, and Table 8, page 27). We have also directly tested the alternative the reviewer suggests. Our Multi-Attempt ablation (Figure 5; page 9) assigns the whole trajectory the same advantage, so a failed first attempt followed by a successful retry receives a positive advantage. This variant performs substantially worse than ReAPO in first-attempt accuracy. Similarly, CAL-GRPO, which down-weights the first attempt when later retries are likely to succeed, underperforms ReAPO in first-attempt accuracy (Figure 2; page 6). Regarding when the theoretical assumptions hold, our derivation assumes that the success probability of attempt m depends only on the fact that earlier attempts failed, not on the specific content of the failed attempt (Appendix E; line 1492-1496). This assumes each failed attempt for a question is equally as helpful. This is a simplification, since the content of a failure could influence the retry. However, it does not affect the direction of the credit assignment: failed attempts are still penalized, and successful retries are still rewarded. Our analysis of truncation and normalization (Appendix E; line 1587–1690) characterizes the remaining bias. We agree that distinguishing helpful from unhelpful failures is an interesting direction for future work.

W2: Limited to small Qwen3 models

We agree that larger models would strengthen the paper. However, our experimental scale aligns with recent work proposing RL algorithms for LLM reasoning. MaxRL (Tajwar et al., 2026; ICML 2026 oral) trains Qwen3-1.7B-Base and Qwen3-4B-Base on POLARIS-53K, the setup we used. CAL-GRPO (Ildiz et al., 2026; ICML 2026) uses a single model, Qwen3-4B-Instruct. In comparison, we evaluate four models across two families (Qwen3-0.6B, 1.7B, 4B and OLMo-2-1B) and several seeds on eight benchmarks. Training Qwen3-4B-Base took over a week per seed, and we trained multiple seeds per model, so scaling beyond 4B was not feasible within our compute budget. Since submission, we have extended ReAPO to OLMo-2-0425-1B-SFT. ReAPO outperforms GRPO in Mean@32 and Pass@32 throughout training, including in token and wall-clock comparisons (Figure 3; page 6). Additionally, we have also scaled ReAPO experiments to K=3 attempts and a context limit of 8192. This result also shows improvements over K=2, suggesting that allowing for longer trial and error leads to stronger performance. In sum, while more experiments are always helpful, our settings align or surpass that of closely related work, and are significantly more rigorous in controlling seed-to-seed variance during training.

Thanks for the authors' response, since the authors did not present border experiments with larger models (at least newer models with 8/9 B parameters would make some progress), I have to say sorry that I maintain my original score.

We thank the reviewer for their response. We note that the original review did not specify an 8–9B threshold. Our scale matches closely related work (MaxRL, ICML 2026 oral: CAL-GRPO, ICML 2026), and we added a second model family (OLMo-2-1B). Additionally, we accounted for variance across training seeds, which many existing papers do not. Training a baseline and ReAPO on a 4B model for one seed takes over a week on 4 B200 GPUs. Training multiple seeds for an 8B+ model would have taken well over a month, which was not feasible. We also hope our response resolved the concern about penalizing failed attempts, as it was not raised again.

Reviewer_eZFnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReAPO, which lets a model retry after a wrong answer by including its failed attempt in the next prompt. It assigns credit to each attempt separately, so successful attempts are reinforced while failed attempts are not. Experiments show improved first-attempt accuracy over GRPO on several math and science benchmarks.

Critical strengths
  • ReAPO allows the model to retry after an incorrect attempt by keeping the failed answer in context. This gives the model a chance to explore a different approach and learn from its previous mistake, while allocating more generation to harder problems.
  • The paper’s main contribution is its attempt-level credit assignment: it rewards successful attempts without directly rewarding the preceding failed attempts. The authors also provide a theoretical derivation of this approach.
  • The experiments cover Qwen3 models at several math and science reasoning benchmarks. ReAPO improves first-attempt performance over GRPO and several baselines, and the paper also reports gains in comparisons that account for token and wall-clock budgets.
Critical weaknesses
  • The most important baseline CAL-GRPO (Ildiz et al., 2026) is missing. CAL-GRPO also employs "verifier-conditioned retry + attempt-level credit assignment", and different from ReAPO in this paper, it conduct normalization within each attempt turns, and weighted by the subsequent failure probabilities.
  • This paper proves that ReAPO outperforms vanilla GRPO and other relatively unrelated baselines (e.g., MaxRL), but the most closely related retry-based GRPO baseline is ignored. Also, methods feature "self-correction + multi-attempt RL" in related work section are also not considered as baselines. More declaration or experiments are required.
  • It seems that "ReAPO (Different prompt)" is better than "ReAPO (K=2)", especially in terms of pass@32, as shown in Figure4. Did the authors try K=3 under this setup?
  • It is surprising that K=3 is worse than K=2, which is not intuitive. I'm afraid the reason stems from insufficient model capability of Qwen3-0.6B-Base and the simplicity of GSM-8K-Platinum. The authors are suggested to test the performance gain of K>3 on harder benchmarks (e.g., the hard data subset of deepmath-103k).
  • It might be overclaim to use "Stochastic Shortest Paths". Based on my understanding, all retry-based methods should investigate how to get correct solution under minimal number of attempts.
Questions for authors

Why not incorporate SCoRe-stage2 into the ablation study of figure4? Is its efficiency significantly lower than that of the other methods because it evaluates the second attempt? But given that "unconditional retry" is considered, ScoRe-stage2 should also be considered.

4 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for their review. We respond to individual concerns below.

W1/W2: CAL-GRPO Baseline

We agree with the reviewer that CAL-GRPO is the most closely related baseline. We previously discussed it in our related works section (Appendix A; line 873-885). However, following the reviewer’s suggestion, we have implemented CAL-GRPO and run it using the same settings as the other experiments. These results show that ReAPO beats CAL-GRPO on both Mean@32 and Pass@32, as shown in Figure 2 (page 6) and Table 5 (page 25) of the revision. As the reviewer notes, the key differences are CAL-GRPO's per-attempt normalization and its weighting by the subsequent failure probabilities. This weighting reduces the gradient on earlier attempts as later retries become more accurate, which is optimal for CAL-GRPO's objective (“Ver@K”). This de-emphasizes first-attempt accuracy, which is what matters when no verifier is available at inference. According to the results from Ildiz et al., their baseline achieves higher first-attempt success than CAL-GRPO on MATH; to reiterate, we see the same phenomenon in our experiments described above. Regarding other self-correction and multi-attempt RL methods, we compared ReAPO with two of them: SCoRe Stage II (Kumar et al., 2024; Figure 2; page 6) and Multi-Attempt GRPO (Chung et al., 2025; Figure 5; page 9), which assigns a single advantage to the full trajectory. Of the remaining related methods, Self-Refine (Madaan et al., 2023) is an inference-time method without RL training, and SkillFactory (Sprague et al., 2026) is a self-distillation method, both of which we judged to be out of scope, as the purpose of ReAPO is to design a general purpose RL algorithm for training.

W3: "ReAPO (Different prompt)" and K=3 under that prompt

The different-prompt variant replaces our default retry instruction ("Your answer is wrong. Reflect on your previous answer and try something else.") with the shorter "Try again." We deliberately did not tune the retry prompt: we fixed the default prompt before running experiments and used it throughout, to avoid selecting the prompt that gives the best results. The different-prompt variant performs comparably to the default, with a modest advantage in Pass@32. If anything, it suggests that prompt tuning could improve ReAPO further. We have not yet run K=3 with this prompt, and we are happy to run additional experiments on it and provide the results if the reviewer remains concerned, subject to sufficient notice before the discussion period ends.

W4: K=3 experiments

We agree with the reviewer that the limited gain from increasing K from 2 to 3 may be due to the simplicity of GSM8K. By the end of training, more than 90% of trajectories succeed within the first two attempts, so fewer than 10% of prompts reach the third attempt, providing little learning signal. Since the submission deadline, we tested this by running K=3 on the 1.7B model with a maximum response length of 4096 and 8192. We still used Polaris 53k, which is a much harder dataset than GSM8K-Platinum. The results in Figure 10 (page 24) show that K=3 for the 1.7B model outperforms K=2 on most benchmarks in steps and tokens generated. We expect this gain to carry over to the 4B model, which is more capable of learning from failed attempts.

W5: "Stochastic Shortest Paths" as an overclaim

We would appreciate clarification on which part of the SSP framing the reviewer finds overclaimed, so that we can address it directly. SSPs are a classical RL formulation (Bertsekas & Tsitsiklis, 1991) for minimizing the expected cost of reaching a terminal state. Our contribution is framing multi-attempt reasoning in this context: the terminal state is a verified correct answer, and each failed attempt incurs a cost. This yields the objective E[τ − 1]. We respectfully note that not all retry-based methods minimize the expected number of attempts to a correct solution. CAL-GRPO maximizes Ver@K, the probability that at least one of K attempts succeeds, so it is indifferent between success on the first and the K-th attempt, and its calibration weights explicitly down-weight the first attempt when retries are reliable. For example, the CAL-GRPO advantage function completely ignores the first attempt when all second attempts are correct. SCoRe optimizes the reward of exactly 2 attempts. The SSP objective is what distinguishes ReAPO, and our ablations (Figure 5; page 9) and new CAL-GRPO comparison (Figure 2; page 6) show that this difference is essential for first-attempt accuracy.

AuthorsOpenReview ↗

Response 2/2

Q1: SCoRe Stage II in the ablation figure

We agree that SCoRe Stage II is an important comparison. We included it in the main efficiency comparison (Figure 2; page 6) because we view it as a baseline from prior work rather than an ablation of ReAPO. Since SCoRe always generates a second attempt unconditionally, we report its second-attempt accuracy, which favors SCoRe. As the reviewer noted, its unconditional second attempt makes it substantially less token-efficient: first-attempt accuracy is above 75% for most of training. Hence, most of SCoRe's second attempts are on problems already solved, which ReAPO avoids. This is the same inefficiency we observe in our Unconditional Retry ablation, and it illustrates a benefit of verifier-conditioned retries: additional compute is allocated only to problems the model currently fails.

Thank you for the detailed rebuttal and additional experiments. The CAL-GRPO comparison and the harder K=3 experiments address several of my concerns. However, given the close relationship to existing retry-based methods, I remain unconvinced that the methodological distinction and demonstrated advantages warrant a higher score. I will maintain my original score.

We thank the reviewer for acknowledging that the new experiments addressed several of their concerns. We note that the point about novelty compared to the baseline was not in the original review. Existing methods optimize eventual success (Multi-Attempt GRPO), a fixed two attempts (SCoRe), or Ver@K (CAL-GRPO), while ReAPO minimizes the expected number of failures before success. Thus, ReAPO’s loss favors the first-attempt accuracy, which is the setting most commonly considered in practice and does not require access to the verifier during inference at evaluation. The theory is reflected in our experiments, where we find that, as expected, ReAPO substantially outperforms the other retry methods.

Reviewer_MT6aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReAPO, a simple extension of GRPO that allows models to retry after failed attempts, get feedbacks, and assigns separate advantages to each attempt. The method improves reasoning performance across several models and benchmarks.

Critical strengths
  1. The problem is important and the method is simple, intuitive, and easy to implement.
  2. The experiments are fairly solid, with multiple seeds, different model sizes, and compute-matched comparisons.
Critical weaknesses
  1. The source of improvement is not fully isolated. It would be useful to compare against a baseline that retries after failures but samples independently, without seeing previous failed responses.

  2. The idea is somewhat incremental, given existing work on RL with feedback and self-correction.

Questions for authors
  1. Could the authors compare ReAPO with a baseline that retries after failure but does not include the previous failed response in the context? or other experiments to isolate the improvement.

  2. Could you further clarify the differences between ReAPO and existing multi-attempt RL methods, like CAL-GRPO

  3. Can you explain why the 4B model shows performance degradation later in training?

2 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for their review. We respond to individual concerns below.

W1/Q1: Insufficient Isolation of Improvement

We certainly agree with the reviewer that it is important to run careful ablations to isolate the cause of the benefit of ReAPO. We would like to respectfully ask for further clarification on

“It would be useful to compare against a baseline that retries after failures but samples independently, without seeing previous failed responses.”

As we understand this question, running retries of samples without seeing the previous failed response is the same as sampling only the prompt as context, which is an independent sample. This is equivalent to dynamically increasing the group size on wrong answers, with an upper bound being G=32. In terms of the performance relative to the number of steps this is strictly dominated by doubling the group size for GRPO, which we found in Figure 2 to underperform relative to ReAPO. While we agree that the suggested approach might be more token efficient than doubling the group size, our results on the final performance of GRPO with 2x group size, which should upper bound the quality of the proposed approach, suggest that this will still be worse than ReAPO.

If the reviewer is unsatisfied with this and our understanding of the question is correct, we are happy to run the additional experiment and include it, subject to sufficient notice before the discussion period ends.

On the question of ablations, in addition to doubling the group size (and number of steps) of GRPO, we currently consider in Figure 5 (page 9):

  • Ablating the credit assignment. What we call “Multi-Attempt” uses our retry strategy but instead assigns credit uniformly in a trajectory. It, unsurprisingly, underperforms ReAPO when evaluating only the first attempt, which is the studied setting in our paper.
  • Ablating the retries. What we call “Unconditional Retry” always samples twice, which leads to inefficiencies hurting training. Moreover, because we end up rewarding both positive answers in a hypothetical rollout where both are correct, this advantage estimate is not principled in the way that ReAPO is (as motivated by SSPs) and thus underperforms even when considering number of gradient steps instead of wall clock time or number of generated tokens.
  • In Figure 3 (page 6) we also consider ReAPO on a different model family and still see performance gains relative to GRPO.

W2: Novelty

We agree that ReAPO builds on top of existing RL methods with feedback and self correction, which we have discussed in Section 5 (page 10; line 525-535). ReAPO uses a different objective: minimizing the expected number of attempts before a correct answer, E[τ−1]\mathbb{E}[\tau - 1], which we derive from the classical stochastic shortest path (SSP) formulation. Existing methods optimize an eventual success after K attempts. None of these objectives explicitly distinguishes success on the first attempt from success after retries. Hence, we believe the contribution is not incremental. While other RL methods use feedback and self-correction, ours aims to minimize the expected number of retries until a correct answer. Existing multi-attempt methods optimize eventual success (Multi-Attempt GRPO), the reward of a fixed number of attempts (SCoRe), or the probability that at least one of K attempts succeeds (CAL-GRPO). This difference is especially important during inference, when a verifier may not be available. The most important difference is that during inference, when a verifier or the correct answer is not known, the first attempt’s performance is what matters, which ReAPO outperforms the other methods on. Please see related work in Appendix A (page 16-17; line 852-889) for a more detailed discussion.

AuthorsOpenReview ↗

Response 2/2

Q2: Difference with CAL-GRPO

We agree that CAL-GRPO is an important reference and discussed it in detail in the related work section in Appendix A (line 873-885). To summarize, CAL-GRPO is different from ReAPO in its objective and credit assignment.

  • Objective: CAL-GRPO aims to maximize “Ver@K”, which is the probability at least one of the first K attempts are correct. ReAPO minimizes the expected number of attempts until a correct answer. Under CAL-GRPO’s objective, each of the first K attempts are preferred equally, while ReAPO heavily prefers success on the early attempts.
  • Credit Assignment: CAL-GRPO normalizes the rewards within each attempt stage. For example, all second attempts within a group are normalized independently of the first attempts. It then multiplies the advantage of each attempt stage by an estimate of the probability the later attempt fails. Under this calculation, if the second attempts are fully correct in a group, the first attempt’s advantage is multiplied by 0, effectively ignoring all first attempts within the group, even if they are incorrect. For ReAPO, all attempts within a group are normalized, so even under the last scenario, the incorrect attempts are still penalized.

Additionally, following the reviewers’ suggestions, we added CAL-GRPO as a baseline on Qwen3-0.6B-Base to Figure 2 (page 6) and Table 5 (page 25) to the revision. Our results show that ReAPO obtains a higher first-attempt accuracy compared to CAL-GRPO, as expected.

Q3: 4B model performance

We thank the reviewer for raising this point, which we discuss in Appendix B (page 18; line 942-950). Both ReAPO and GRPO degrade after roughly 500–600 steps on Qwen3-4B-Base. We attribute this to two factors:

  • Response length truncation: Because all attempts share the same context budget, over 50% of ReAPO’s responses end up getting truncated, compared with under 10% for GRPO. The increase in response length truncation also happened at roughly the same time as the performance degradation, which we noted in Appendix Section D (line 1231-1237).
  • Hyperparameter: For comparability, we adopted the same hyperparameters as the MaxRL paper (Tajwar et al., 2026) for our experiments, which reported extensive tuning of the baseline GRPO and of their proposed algorithm in order to ensure that our baselines were competitive and our results are not artifacts of a poorly tuned baseline. These hyperparameters included a constant learning rate. Because we did not have enough compute to run multiple experiments to tune the hyperparameters, a constant learning rate can make training a bit unstable after long training runs.

We expect that, with a well tuned learning rate scheduler and high maximum number of generated tokens, the performance of the 4B model will be more stable.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank all reviewers for their feedback. We have uploaded a revised version with the requested experiments as well as some additional results that help illustrate the effect of ReAPO. The most relevant changes to the draft are in red. We summarize them below.

New experiments

  • CAL-GRPO baseline (MT6a, eZFn). We implemented CAL-GRPO (Ildiz et al., 2026) in the same training setup as ReAPO on Qwen3-0.6B-Base. ReAPO achieves higher first-attempt Mean@32 and Pass@32 (Figure 2, page 6; Table 5, page 25).
  • A second model family (39G9). On OLMo-2-0425-1B-SFT, ReAPO outperforms GRPO in Mean@32 and Pass@32 throughout training, including in token- and wall-clock-normalized comparisons (Figure 3, page 6).
  • K=3 on a larger model and harder data (eZFn, 39G9). On Qwen3-1.7B-Base trained on Polaris-53K, with maximum response lengths of 4,096 and 8,192 tokens, K=3 outperforms K=2 on most benchmarks (Appendix D.5; Figure 10 and Table 4, page 24).

Clarifications

  • Objective. Existing methods optimize eventual success (Multi-Attempt GRPO), the reward after two attempts (SCoRe), and the probability that at least one of K attempts succeeds (CAL-GRPO). ReAPO minimizes the expected number of failures before a success, E[τ−1]\mathbb{E}[\tau - 1]. This favors succeeding on earlier attempts, which is often what matters at inference time because a verifier is typically unavailable. Hence, the model will only receive one attempt. This is why in our experiments, we track the accuracy only of the first attempt during evaluation.
  • Ablations isolating ReAPO’s components. Figure 5 (page 9) compares ReAPO with Multi-Attempt GRPO, which assigns one advantage to the entire trajectory, and with Unconditional Retry, which always samples a second attempt regardless of the first attempt’s correctness. Figure 2 also compares ReAPO against GRPO with twice the group size and training steps. ReAPO outperforms all of these.

We respond to each reviewer’s concerns below and are happy to provide more clarification or run more experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.