- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReAPO, which lets the model retry failed answers using the previous answer and feedback. It computes advantages separately for each attempt, so a successful retry does not give the earlier failure the same reward. The authors derive this design from an objective that minimizes failures before success. Experiments on Qwen3 0.6B, 1.7B, and 4B show improvements in first-attempt accuracy and training efficiency on math and science reasoning tasks.
Critical strengths
-
The method is simple and addresses a practical problem: independent samples can repeat the same mistakes. The gradient derivation and the analyses of truncation and normalization help explain how credit is assigned to each attempt.
-
The experiments include multiple independent training seeds, comparisons of generated tokens and training time, and a Multi-Attempt GRPO baseline that shares reward across the full trajectory. These checks support the method's effectiveness in the tested settings.
Critical weaknesses
-
The analysis does not adequately account for the information gained from failed attempts. A failure can help the model reach a correct answer later, so penalizing it based only on its own correctness may discourage useful exploration.
-
The experiments are limited to small Qwen3 models, with two attempts in the main setting and a three-attempt ablation. These short retry settings provide limited evidence for stronger models or tasks that require longer trial and error.
Questions for authors
- If a failed first attempt provides useful information for a successful second attempt, why should it receive a negative advantage? Can the authors distinguish helpful from unhelpful failures and explain when the theoretical assumptions hold in practice?
3 comments
Thanks for the authors' response, since the authors did not present border experiments with larger models (at least newer models with 8/9 B parameters would make some progress), I have to say sorry that I maintain my original score.
We thank the reviewer for their response. We note that the original review did not specify an 8–9B threshold. Our scale matches closely related work (MaxRL, ICML 2026 oral: CAL-GRPO, ICML 2026), and we added a second model family (OLMo-2-1B). Additionally, we accounted for variance across training seeds, which many existing papers do not. Training a baseline and ReAPO on a 4B model for one seed takes over a week on 4 B200 GPUs. Training multiple seeds for an 8B+ model would have taken well over a month, which was not feasible. We also hope our response resolved the concern about penalizing failed attempts, as it was not raised again.
Response to Review
We thank the reviewer for their review. We respond to individual concerns below.
W1/Q1: Why should a useful failed attempt receive a negative advantage?
We agree that a failed attempt can carry information that helps a later retry succeed, and this mechanism is used by prior work like CAL-GRPO to maximize the probability of a correct answer out of multiple serial attempts, which only makes sense when the verifier is available at inference time. ReAPO does not make this assumption, which is why we penalize the first failed attempt. To reiterate, the model must get the answer on its first attempt during evaluation so incorrect early attempts should not be rewarded. Conditional retries are therefore a training-time mechanism for exploration and credit assignment. Our objective, the expected number of failures before success, reflects this perspective: a failed attempt remains a failure, even if it later enables a correct retry. The useful information in that failure isn’t discarded, since it is credited to the successful retry, whose positive advantage reinforces the reasoning that recovered from the mistake (Table 7, page 26, and Table 8, page 27). We have also directly tested the alternative the reviewer suggests. Our Multi-Attempt ablation (Figure 5; page 9) assigns the whole trajectory the same advantage, so a failed first attempt followed by a successful retry receives a positive advantage. This variant performs substantially worse than ReAPO in first-attempt accuracy. Similarly, CAL-GRPO, which down-weights the first attempt when later retries are likely to succeed, underperforms ReAPO in first-attempt accuracy (Figure 2; page 6). Regarding when the theoretical assumptions hold, our derivation assumes that the success probability of attempt m depends only on the fact that earlier attempts failed, not on the specific content of the failed attempt (Appendix E; line 1492-1496). This assumes each failed attempt for a question is equally as helpful. This is a simplification, since the content of a failure could influence the retry. However, it does not affect the direction of the credit assignment: failed attempts are still penalized, and successful retries are still rewarded. Our analysis of truncation and normalization (Appendix E; line 1587–1690) characterizes the remaining bias. We agree that distinguishing helpful from unhelpful failures is an interesting direction for future work.
W2: Limited to small Qwen3 models
We agree that larger models would strengthen the paper. However, our experimental scale aligns with recent work proposing RL algorithms for LLM reasoning. MaxRL (Tajwar et al., 2026; ICML 2026 oral) trains Qwen3-1.7B-Base and Qwen3-4B-Base on POLARIS-53K, the setup we used. CAL-GRPO (Ildiz et al., 2026; ICML 2026) uses a single model, Qwen3-4B-Instruct. In comparison, we evaluate four models across two families (Qwen3-0.6B, 1.7B, 4B and OLMo-2-1B) and several seeds on eight benchmarks. Training Qwen3-4B-Base took over a week per seed, and we trained multiple seeds per model, so scaling beyond 4B was not feasible within our compute budget. Since submission, we have extended ReAPO to OLMo-2-0425-1B-SFT. ReAPO outperforms GRPO in Mean@32 and Pass@32 throughout training, including in token and wall-clock comparisons (Figure 3; page 6). Additionally, we have also scaled ReAPO experiments to K=3 attempts and a context limit of 8192. This result also shows improvements over K=2, suggesting that allowing for longer trial and error leads to stronger performance. In sum, while more experiments are always helpful, our settings align or surpass that of closely related work, and are significantly more rigorous in controlling seed-to-seed variance during training.