acceptodds
Desk rejected at ICLR 2027

Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern code generation models exhibit longer outputs, accelerated capability growth, and changed training dynamics, rendering traditional training methodologies, algorithms, and datasets ineffective for improving their performance. To address these training bottlenecks, we propose MicroCoder-GRPO, an improved Group Relative Policy Optimization approach with two original innovations and one inherited improvement: conditional truncation masking to improve long output potential while maintaining training stability, diversity-determined temperature selection to maintain and encourage output diversity, and removal of KL loss with high clipping ratios following DAPO to facilitate solution diversity. MicroCoder-GRPO achieves up to 17.6% relative improvement over strong baselines on LiveCodeBench v6, with more pronounced gains under extended context evaluation. Additionally, we release MicroCoder-Dataset, a more challenging training corpus that achieves 3x larger performance gains than mainstream datasets on LiveCodeBench v6 within 300 training steps, and MicroCoder-Evaluator, a robust framework that reduces disagreement with human labels from 10.8% to 1.4% and around 40% faster execution through parallel processing. Through comprehensive analysis across more than thirty controlled experiments, we reveal training insights across seven main aspects, demonstrating that properly trained models can achieve competitive performance with larger counterparts.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reinforcement Learning, Code Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6qs2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates performance bottlenecks in reinforcement learning for code generation and proposes MicroCoder-GRPO, while also introducing MicroCoder-Dataset and MicroCoder-Evaluator. The paper presents a systematic analysis of long-output training, sampling diversity, and training stability. Experimental results show that the proposed approach outperforms standard GRPO and DAPO.

Critical strengths

The research problem is practically meaningful and clearly motivated. The authors introduce two modifications to GRPO: conditional truncation masking and dynamic temperature selection, while inheriting DAPO’s strategy of removing the KL loss with high clipping ratios. The paper presents comprehensive and systematic experiments and analyses.

Critical weaknesses
  1. The clarity regarding the algorithmic novelty and incremental contribution relative to existing work is not sufficiently clear. The method appears to extend GRPO with only two modifications, which may appear incrementally limited.
  2. Insufficient theoretical justification for dynamic temperature adjustment. Does the proposed dynamic temperature control strategy have a clear theoretical foundation? The paper adjusts the temperature based on a diversity metric and an empirical power-law relationship. Could the authors clarify the substantive technical differences between this approach and existing adaptive sampling temperature or entropy control methods?
  3. Training experiments are conducted only on Qwen3 models. It would strengthen the paper to evaluate on newer model families, such as Qwen3.5.
Questions for authors

Please refer to the weaknesses above.

Reviewer_4i7gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies several practical issues in reinforcement learning for code generation, including response truncation, diversity collapse, training data difficulty, and reward evaluation. The proposed MicroCoder-GRPO combines conditional truncation masking and diversity-based temperature control with KL removal and a higher upper clipping threshold. The paper also introduces a curated dataset and a code evaluator. Experiments on LiveCodeBench v6 show improvements over the reported GRPO and DAPO baselines across several models. The appendices include three-seed results, component ablations, experiments on another training dataset, and evaluator validation against human annotations.

Critical strengths
  1. The experiments make a useful effort to separate different sources of improvement. Table 1 compares datasets under fixed algorithms, while Table 2 compares algorithms under fixed data. Appendix C further evaluates the algorithm on DeepCoder data. Appendix B includes both adding and removing components, as well as ablations of the masking conditions and temperature controller. These experiments provide more convincing evidence than a comparison of final systems alone.
  2. The reported gains are practically relevant and supported by multiple runs. For example, with 4K training and 8K testing, the 1.7B model improves from 20.4 to 24.0, and the 4B model improves from 36.3 to 38.7 over the reported DAPO baseline. Appendix A reports standard deviations across three training seeds. The authors also state that all test results use the official LiveCodeBench evaluator, which helps separate improvements in test performance from changes in the training reward.
Critical weaknesses
  1. The paper repeatedly describes DAPO as KL removal with higher clipping, and Table 4 explicitly labels this combination as DAPO. However, the original DAPO recipe also includes dynamic sampling, token-level loss, and overlong reward handling. The last component is particularly relevant to this paper. If these components are absent, the results establish an improvement over a simplified variant, rather than the full baseline. This affects both the strength of the empirical comparison and the assessment of novelty. The contribution of conditional masking should also be distinguished clearly from the existing practice of masking truncated samples.
  2. Appendix F.2.3 uses model success rates on LiveCodeBench v6 to calibrate difficulty boundaries for subsequent data filtering. It is unclear whether this is only an illustrative analysis or whether these thresholds were used to construct the final training dataset. The paper also does not clearly describe an independent validation set for checkpoint and design selection. In addition, the 16-gram filtering and aggregate cosine similarity results do not establish the absence of translated, paraphrased, or cross-platform duplicates. I am not claiming that contamination occurred, but the current evidence is insufficient to support the strong claim of zero overlap. This matters because AtCoder and LeetCode are subsets of the same LiveCodeBench evaluation, rather than independent benchmarks.
Questions for authors
  1. Does the DAPO implementation include dynamic sampling, token-level loss, and overlong reward handling? Please provide the exact configuration differences between the methods. If these components were omitted, how does MicroCoder-GRPO compare with the full recipe?
  2. Were the difficulty thresholds calibrated on LiveCodeBench v6 used to construct the final training dataset? Which data were used for hyperparameter tuning and checkpoint selection? Is there a test split that did not participate in these decisions?
  3. How are evaluator normalization rules restricted by task semantics? For example, "007" and "7" are not equivalent for every string-output task, and converting large integers to floating point can lose information. Please specify the numerical tolerances and the conditions under which type conversion or set comparison is enabled. The near-miss results are useful, but concrete test cases and implementation details are needed to assess these risks.
  4. What is the total training cost when temperature calibration is included? The reported calibration requires three runs of 300 steps, compared with 520 steps for a main run. Please report measured GPU-hours or token costs under comparable tuning budgets. The claimed 40–50% computational saving should also be supported by end-to-end measurements.
  5. Can the authors provide an accessible anonymous code/data release, the exact deduplication procedure, and an audit of potential translated or cross-platform duplicates?
Reviewer_gYP3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MicroCoder-GRPO, combining conditional truncation masking and diversity-controlled sampling temperature with DAPO-style changes. It also contributes a curated coding dataset and a more flexible training evaluator. Controlled experiments report improved coding accuracy and training stability across several model sizes on LiveCodeBench v6.

Critical strengths
  • Table 2 and Appendices A–C provide algorithm-controlled comparisons, three-seed uncertainty, and component ablations. The gains also persist on DeepCoder data, supporting an algorithmic contribution beyond dataset replacement.
  • The evaluator analysis includes human-label agreement and adversarial near-miss checks, while reported test accuracies use the official evaluator (§5). This makes the practical training improvements more credible.
Critical weaknesses
  • The efficiency case is incomplete: §5 requires three 300-step temperature-calibration runs per configuration, whereas §6's 40–50% compute-saving claim relies on attention-length scaling. Actual total training cost, including calibration and changing rollout lengths, is needed to assess the tradeoff.
  • The headline test results all use LiveCodeBench v6; AtCoder and LeetCode are subsets of that benchmark (§5). The broad experimental analysis is useful, but independent evidence beyond this evaluation setting remains limited.
Questions for authors
  1. What measured GPU-time or token-cost comparison includes calibration, and when can that calibration be reused without additional runs?
  2. Which method and hyperparameter choices were fixed before examining the 200 test problems used for the temperature analysis in §5?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.