- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve LLMs' self-refinement ability for competitive programming (CP) and math reasoning tasks. The authors point out that existing self-correction methods usually rely on public test cases to decide whether a generated solution needs further refinement. However, passing all public tests does not necessarily mean the solution is correct, as it may still fail on hidden test cases. To address this issue, the authors propose RefineRL, which mainly consists of two parts. The first is Skeptical-Agent, which iteratively refines generated solutions based on execution feedback. When a solution fails on public tests, the agent uses error-driven feedback to guide the correction. Even when all public tests are passed, it still uses skepticism-driven feedback to encourage the model to recheck its solution. The second part is Self-Refinement RL, which uses self-generated error-driven and skepticism-driven feedback to construct additional refinement training samples and trains the model with GRPO. The authors also introduce a squared-incentive reward, which provides partial rewards based on the number of tests passed before the first failure. The experiments use around 4.5K original CP problems and 4.2K generated refinement samples to train Qwen3-4B and Qwen3-4B-2507. The results show clear improvements on LiveCodeBench v5/v6 and USACO. When combined with Skeptical-Agent, the trained 4B models can also achieve performance close to the single-attempt results of some much larger models. The authors further show improvements on AIME 2024/2025/2026, suggesting that the learned self-refinement ability may also transfer to math reasoning tasks.
Critical strengths
- Existing self-correction methods usually stop refining once the generated code passes all public tests, but this does not necessarily mean the code is actually correct. The idea of continuing to refine the solution even after passing the public tests is quite reasonable, especially for competitive programming where public tests are often limited. I think this is an interesting direction to explore.
- The authors use the model's own error-driven and skepticism-driven feedback to construct refinement training data, so that the model can learn how to improve its previous solutions through RL. I think this is a reasonable way to explicitly train the model's self-refinement ability.
- The experimental results are promising. The authors show improvements on LiveCodeBench, USACO, and AIME with two 4B models. In particular, RefineRL-4B-2507 with Skeptical-Agent achieves 75.97% and 67.82% on LCB-v5 and LCB-v6, respectively. I also appreciate that the authors provide several comparisons with different inference-time strategies, such as Random Sampling, Rejection Sampling, and Reflexion. They also include ablations on the agent hyperparameters, reward functions, self-refinement training data, and model scale, which help support the effectiveness of the method.
Critical weaknesses
- The reward is calculated based on the number of test cases passed before the first failure, rather than the total number of passed tests. This means that even if two solutions pass exactly the same number of tests, they may receive very different rewards depending on the ordering of the test cases. Also, the authors argue that this reward can discourage the model from overfitting to easy tests and encourage it to pass later, more rewarding tests. But I am not sure why the later test cases should necessarily be harder or more important than the earlier ones. I think it would be helpful to see some experiments on how sensitive the results are to the test case ordering, for example by randomly shuffling the test cases during training.
- The authors compare it with RejSamp and Reflexion, and also show that increasing M2 from 1 to a larger value improves performance. However, Skeptical-Agent continues to generate new solutions even after passing all public tests, while other baselines may stop earlier. So I wonder if part of the improvement simply comes from using more actual inference compute. I think a more direct comparison would be to let a standard self-refinement method continue generating solutions after passing all public tests, but without using the proposed skepticism-driven prompt, under the same number of actual model calls or token budget. Also, Table 3 shows that the average response length increases quite a lot after RL training, which makes me wonder how much of the performance gain comes from longer reasoning. I think these comparisons would help better understand the actual contribution of skepticism-driven feedback.
- Another concern is about the comparison between Self-Refinement RL and Standard RL in Table 5. It is good that the authors use the same squared-incentive reward for both methods. However, Standard RL only uses around 4.5K original CP problems, while Self-Refinement RL additionally includes around 4.2K generated refinement samples, almost doubling the training data size. So I feel it is still a bit unclear whether the improvement mainly comes from the self-refinement training strategy, or simply from having more training samples. It would be helpful to compare them under the same number of training samples and similar training compute, for example by adding more regular CP training samples to Standard RL. This could better show how much benefit comes specifically from the refinement data.
Questions for authors
One minor suggestion is that I think the paper could benefit from a more concrete case study to explain how the skepticism-driven mechanism actually works. For example, it would be interesting to see a real CP problem where the initial solution passes all public tests but still fails on some hidden tests, and then see how the skepticism-driven feedback helps the model identify the potential issue and fix the solution step by step. I think such an example would make the proposed mechanism much easier to understand and give readers a better intuition of why continuing refinement after passing public tests can be helpful.