acceptodds
Under review as a conference paper at ICLR 2027

RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learning

Abstract

While large language models (LLMs) have demonstrated strong performance on complex reasoning tasks such as competitive programming (CP), existing methods predominantly focus on single-attempt settings, overlooking their capacity for iterative refinement. In this paper, we present RefineRL, a novel approach designed to unleash the self-refinement capabilities of LLMs for CP problem solving. RefineRL introduces two key innovations: (1) Skeptical-Agent, an iterative self-refinement agent equipped with local execution tools to validate generated solutions against public test cases of CP problems. This agent always remains skeptical of its own outputs and thereby enforces rigorous self-refinement even when validation suggests correctness. (2) A reinforcement learning (RL) solution to incentivize LLMs to self-refine with only standard RLVR data (i.e., problems paired with their verifiable answers). Extensive experiments on Qwen3-4B and Qwen3-4B-2507 demonstrate that our method yields substantial gains: after our RL training, these compact 4B models integrated with the Skeptical-Agent not only outperform much larger 32B models but also approach the single-attempt performance of 235B models. Notably, although our models are trained solely on CP coding data, RefineRL also substantially improves their performance on mathematical reasoning benchmarks, suggesting that the learned self-refinement behavior can transfer beyond coding-specific tasks. These findings highlight self-refinement as a promising direction for scaling LLMs' general reasoning ability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Code Generation, Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PCDWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to improve LLMs' self-refinement ability for competitive programming (CP) and math reasoning tasks. The authors point out that existing self-correction methods usually rely on public test cases to decide whether a generated solution needs further refinement. However, passing all public tests does not necessarily mean the solution is correct, as it may still fail on hidden test cases. To address this issue, the authors propose RefineRL, which mainly consists of two parts. The first is Skeptical-Agent, which iteratively refines generated solutions based on execution feedback. When a solution fails on public tests, the agent uses error-driven feedback to guide the correction. Even when all public tests are passed, it still uses skepticism-driven feedback to encourage the model to recheck its solution. The second part is Self-Refinement RL, which uses self-generated error-driven and skepticism-driven feedback to construct additional refinement training samples and trains the model with GRPO. The authors also introduce a squared-incentive reward, which provides partial rewards based on the number of tests passed before the first failure. The experiments use around 4.5K original CP problems and 4.2K generated refinement samples to train Qwen3-4B and Qwen3-4B-2507. The results show clear improvements on LiveCodeBench v5/v6 and USACO. When combined with Skeptical-Agent, the trained 4B models can also achieve performance close to the single-attempt results of some much larger models. The authors further show improvements on AIME 2024/2025/2026, suggesting that the learned self-refinement ability may also transfer to math reasoning tasks.

Critical strengths
  • Existing self-correction methods usually stop refining once the generated code passes all public tests, but this does not necessarily mean the code is actually correct. The idea of continuing to refine the solution even after passing the public tests is quite reasonable, especially for competitive programming where public tests are often limited. I think this is an interesting direction to explore.
  • The authors use the model's own error-driven and skepticism-driven feedback to construct refinement training data, so that the model can learn how to improve its previous solutions through RL. I think this is a reasonable way to explicitly train the model's self-refinement ability.
  • The experimental results are promising. The authors show improvements on LiveCodeBench, USACO, and AIME with two 4B models. In particular, RefineRL-4B-2507 with Skeptical-Agent achieves 75.97% and 67.82% on LCB-v5 and LCB-v6, respectively. I also appreciate that the authors provide several comparisons with different inference-time strategies, such as Random Sampling, Rejection Sampling, and Reflexion. They also include ablations on the agent hyperparameters, reward functions, self-refinement training data, and model scale, which help support the effectiveness of the method.
Critical weaknesses
  • The reward is calculated based on the number of test cases passed before the first failure, rather than the total number of passed tests. This means that even if two solutions pass exactly the same number of tests, they may receive very different rewards depending on the ordering of the test cases. Also, the authors argue that this reward can discourage the model from overfitting to easy tests and encourage it to pass later, more rewarding tests. But I am not sure why the later test cases should necessarily be harder or more important than the earlier ones. I think it would be helpful to see some experiments on how sensitive the results are to the test case ordering, for example by randomly shuffling the test cases during training.
  • The authors compare it with RejSamp and Reflexion, and also show that increasing M2 from 1 to a larger value improves performance. However, Skeptical-Agent continues to generate new solutions even after passing all public tests, while other baselines may stop earlier. So I wonder if part of the improvement simply comes from using more actual inference compute. I think a more direct comparison would be to let a standard self-refinement method continue generating solutions after passing all public tests, but without using the proposed skepticism-driven prompt, under the same number of actual model calls or token budget. Also, Table 3 shows that the average response length increases quite a lot after RL training, which makes me wonder how much of the performance gain comes from longer reasoning. I think these comparisons would help better understand the actual contribution of skepticism-driven feedback.
  • Another concern is about the comparison between Self-Refinement RL and Standard RL in Table 5. It is good that the authors use the same squared-incentive reward for both methods. However, Standard RL only uses around 4.5K original CP problems, while Self-Refinement RL additionally includes around 4.2K generated refinement samples, almost doubling the training data size. So I feel it is still a bit unclear whether the improvement mainly comes from the self-refinement training strategy, or simply from having more training samples. It would be helpful to compare them under the same number of training samples and similar training compute, for example by adding more regular CP training samples to Standard RL. This could better show how much benefit comes specifically from the refinement data.
Questions for authors

One minor suggestion is that I think the paper could benefit from a more concrete case study to explain how the skepticism-driven mechanism actually works. For example, it would be interesting to see a real CP problem where the initial solution passes all public tests but still fails on some hidden tests, and then see how the skepticism-driven feedback helps the model identify the potential issue and fix the solution step by step. I think such an example would make the proposed mechanism much easier to understand and give readers a better intuition of why continuing refinement after passing public tests can be helpful.

Reviewer_56WZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RefineRL for improving self-refinement in competitive programming. Its Skeptical-Agent continues to inspect and revise solutions even after they pass public tests, aiming to catch errors that those tests miss. The authors construct feedback-conditioned training examples from model-generated solutions and train with GRPO using the squared fraction of passed tests as the reward. Experiments report improvements in both single-attempt and iterative code generation, along with transfer to mathematical reasoning despite training only on programming tasks.

Critical strengths
  1. The paper addresses a practical limitation of code refinement: passing public tests does not guarantee correctness on hidden tests.
  2. Experiments show gains in both single-attempt and iterative code generation, with additional improvements on mathematical reasoning tasks.
  3. The paper includes ablations on training data, reward functions, and inference settings.
Critical weaknesses
  1. The methodological contribution is limited. The approach combines continued self-refinement after passing public tests, feedback-conditioned GRPO training, and a squared test-pass reward. The paper demonstrates benefits from this combination, but does not clearly establish what new learning principle it introduces beyond existing self-correction and feedback-based training methods. The squared reward is a simple transformation of test-pass rate, and its effectiveness alone does not establish a substantial methodological advance.

  2. The evidence for transfer to mathematical reasoning is mixed, and the contribution of training is not fully isolated. Without the agent, RefineRL-4B slightly underperforms Qwen3-4B on AIME24 and AIME26, although the 4B-2507 variant improves on all three AIME benchmarks. The larger gains are obtained with Skeptical@16. Comparisons with the corresponding untrained backbones using the same refinement procedure, and with budget-matched sampling baselines, would help distinguish transfer from programming training from the benefit of additional test-time computation.

  3. The comparisons with much larger models do not establish an efficiency advantage. The headline results compare 4B models using up to 16 refinement attempts with larger models evaluated in a single-attempt setting. Although the paper includes inference-strategy baselines and timing measurements, these do not establish the claimed practical advantage over larger models under comparable total inference budgets. This is particularly relevant because training substantially increases response length.

  4. The evidence for learning skeptical refinement is incomplete. The training data include original problems and feedback-conditioned examples, while inference combines error-driven correction, skepticism-driven revision, restarts, and stopping rules. The existing data and reward ablations show that the overall recipe helps, but do not cleanly isolate the contribution of skepticism-specific training from ordinary error-correction training and additional feedback-conditioned data. A matched comparison separating these training components would better support the paper’s central claim.

Questions for authors
  1. How much of the improvement comes specifically from skepticism-conditioned training? Could you compare error-feedback-only, skepticism-feedback-only, and combined training under matched data and compute budgets?

  2. How does RefineRL compare with larger models when total inference compute or latency is matched, including all refinement attempts and code execution?

  3. Does the squared test-pass reward improve correction of solutions that pass public tests but fail hidden tests? An analysis separating these cases from ordinary test failures would help explain its contribution.

Reviewer_cLsGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to improve the performance of large language models on Competitive Programming (CP) tasks through self-refinement. The authors propose RefineRL, which consists of two components: Skeptical-Agent and Self-Refinement RL.

Skeptical-Agent iteratively revises generated code using execution feedback from public test cases. Even when a program passes all public tests, the model continues to examine potential errors. Self-Refinement RL constructs training data from self-generated feedback-conditioned trajectories and trains the model using GRPO and a Squared-incentive Reward.

Experimental results show that the proposed method improves competitive programming performance and achieves some transfer gains on mathematical reasoning tasks.

Critical strengths
  1. A practically meaningful research problem.

By continuing skeptical refinement even after all public tests are passed, the authors attempt to improve the model's ability to identify latent errors, which has practical value.

Critical weaknesses
  1. Limited methodological novelty and insufficient differentiation from existing feedback-based self-correction training methods.

Using previous attempts and execution feedback to generate revised solutions, and subsequently improving the model's self-correction ability through RL or self-distillation, has already been explored in works such as SCoRe, RLTF-SD, SDPO, and TaPR. Although these methods differ from RefineRL in their specific optimization objectives and training procedures, the paper does not sufficiently clarify the core novelty of Skeptical-Agent and Self-Refinement RL relative to these approaches. In particular, direct comparisons with closely related self-correction training methods under matched computational budgets are missing.

  1. Insufficient theoretical and empirical justification for the Squared-incentive Reward.

The proposed reward is calculated based on the number of consecutive test cases passed before the first failure, rather than the actual pass rate across all test cases. Consequently, two programs that pass the same number of test cases may receive substantially different rewards depending on the ordering of the test cases.

Although the Squared-incentive Reward outperforms the Binary and Linear Rewards in the experiments, the current ablations do not establish the validity of this order-dependent reward design.

  1. More rigorous compute-matched comparisons are needed to establish the benefits of Self-Refinement RL.

Standard RL uses approximately 4.5K original CP training samples, while Self-Refinement RL uses approximately 8.7K samples, including feedback trajectories. The training curves in Appendix B.1 also indicate that the final models are trained for different numbers of GRPO steps.

Questions for authors

Q1. What is the core algorithmic novelty of RefineRL compared with existing feedback-based self-correction training methods such as RLTF-SD, SDPO, and TaPR? Could the authors provide direct comparisons under matched training and inference budgets?

Q2. How sensitive is the Squared-incentive Reward to test-case ordering? Why should the reward depend on the position of the first failed test case rather than the actual proportion of passed test cases?

Q3. Could the authors compare Standard RL and Self-Refinement RL under matched GRPO steps and generated tokens to further isolate the contribution of feedback trajectories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.