acceptodds
Desk rejected at ICLR 2027

MupCoder: Mitigating Reward Sparsity in Code Generation via Multi-Perspective Token Level Reinforcement Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

The demands of syntactic correctness and logical precision in code generation remain challenging for off-the-shelf large language models, motivating post-training alignment as a common approach. However, current approaches face critical constraints: Supervised Fine-Tuning (SFT) is prone to in-distribution overfitting and struggles with out-of-distribution generalization, while Reinforcement Learning (RL) suffers from reward sparsity. Therefore, we propose MupCoder (Multi-perspective Coder) and a token-level RL framework that provides dense process supervision without requiring auxiliary reward models. Rather than relying solely on sparse outcome feedback, MupCoder equips the model with multi-perspective hints, such as reference demonstrations and test cases. Conditioned on these contexts, the model acts as its own teacher via on-policy self-distillation, deriving dense token-level rewards from the sampled-token log-probability gap between the standard student branch and hint-conditioned teacher branches. To capture long-horizon dependencies in code generation, we further accumulate these dense signals through a future-aware discounted return, which propagates downstream alignment information to earlier tokens. This dense shaping signal is combined with sparse execution feedback to anchor functional correctness. Experiments on multiple code generation benchmarks show that MupCoder consistently improves generation accuracy over representative baselines.

Primary area:
reinforcement learning
Keywords:
Code Generation, Reinforcement Learning, Token-Level Rewards, Reward Sparsity, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Pqe2OpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on the post-training, and it points out that SFT may suffer from the in-distribution overfitting, and RL may suffer from the sparse rewards. Therefore, it propose the MupCoder, which includes a token-level RL framework that provides dense process supervision without requiring auxiliary reward models. This approach equips the model with multi-perspective hints, such as reference demonstrations and test cases, and trains LLMs via the on-policy self-distillation. It shows the empirical improvements on diverse code generation benchmarks than other baselines.

Critical strengths
  1. The paper is well-written and easy to follow.
  2. The empirical experiments are conducted over models in different sizes and evaluated on diverse benchmarks, which is convincing.
  3. The ablations are also sufficient.
Critical weaknesses
  1. One concern might be the novelty, since I think this approach looks like the combination of existing methods and pipelines. For example, the hint information is similar to the privileged information in value model, which has been studied before. and the reward design is just reasonable.
  2. I am also wondering if it is possible to put all three hints into a whole prompt? Will it be better or not?
Questions for authors

Please refer to weakness.

Reviewer_GtpqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MupCoder, a code-generation post-training framework that combines hint-conditioned on-policy self-distillation with execution rewards. The same model evaluates student-generated tokens under demonstration, pseudocode, and test-case contexts. The averaged teacher–student log-probability difference supplies a dense reward, which is accumulated through a discounted return and combined with a binary execution reward. Experiments cover several model sizes and code-generation benchmarks, with component ablations and comparisons against OPD and OPSD.

Critical strengths
  1. The framework offers a practical way to obtain dense supervision without a separately maintained teacher or process reward model. The reported accuracy and training-cost results make this a useful engineering contribution.

  2. The evaluation goes beyond a single headline comparison: it includes hint ablations, hybrid-reward ablations, and an explicit OPSD comparison in Appendix J. These results support the effectiveness of the complete recipe.

Critical weaknesses
  1. Most of the method is a combination of hint-conditioned self-distillation, execution rewards, and a standard discounted return. The results are useful, but I do not yet see a clear methodological advance over these existing ingredients.

  2. The multi-perspective component adds relatively little over the strongest single hint: tests alone score 66.4, compared with 67.1 for all three hints. The paper also needs to explain why its “future-aware” return offers a new credit-assignment insight rather than a standard way to accumulate rewards.

Questions for authors
  1. Relative to the OPSD formulation discussed in Appendix J, what is the precise methodological novelty of MupCoder? Please distinguish the contribution of hint-conditioned self-distillation itself from the contributions of multiple contexts, execution feedback, and the return estimator.

  2. Equation (6) uses a conventional discounted reward-to-go. What makes this a code-specific credit-assignment contribution beyond applying this standard estimator to dense distillation rewards? In particular, what in the existing analysis distinguishes downstream distributional alignment from attribution of functional errors to earlier tokens?

  3. Table 2 shows that tests alone reach 66.4 average pass@1, compared with 67.1 for all three hints. What does combining the perspectives reveal that cannot be obtained from the strongest single context or a conventional ensemble of teacher contexts? Please clarify the distinctive role of the multi-perspective construction using the existing ablations.

Reviewer_oVytOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper has the same policy score each sampled token under the original prompt and under three hint-conditioned contexts, takes the log-probability gap as a dense reward, and combines it with test execution results and a discounted future return for training code generation models. On Qwen2.5-Coder 1.5B/3B/7B, the method exceeds the listed baselines on average pass@1 across HumanEval+, MBPP+, and LiveCodeBench-V6, and reports transfer results on other languages and SWE-bench Verified.

Critical strengths

1.The comparison surface is broad and has targeted ablations: three model sizes, public benchmarks, hint categories, discount factors, dense/sparse rewards, and OPSD comparisons are all reported. The 7B average pass@1 of 67.1% shows a consistent edge over PRLCoder's 65.1% and GRPO's 62.0% (Tables 1–2, Appendix J). 2.Training cost and generalization are not supported by a single leaderboard alone. The authors report 22.4 hours on the same eight H100s versus 31.1 hours for GRPO, and disclose three additional same-model scoring passes. SWE-bench Verified is 38/500 versus 32/500 for the closest baseline (Appendix C, F). Removing target-solution pseudocode still yields 66.7%, easing concerns about dependence on reference answers (Appendix E).

Critical weaknesses

1.The dense signal itself only measures the "hint-conditioned same model" preference relative to the unhinted branch; it has not been shown that its token-level localization corresponds to functional errors. The hints may emphasize the problem statement or directly provide the algorithm/test answer. The γ=0 vs. γ=.99 difference supports the discounted return's effectiveness, but it does not by itself substantiate the "downstream correctness credit assignment" mechanism. Please provide correspondence between token-level signals and independent error annotations/repairable error locations, and a more direct comparison with test-hint supervision of equal informativeness (Equations 4–8, Table 2). 2.Fairness and statistical uncertainty remain at the acceptance boundary: GRPO uses 8 rollouts, MupCoder uses 1, OPD uses an external 32B teacher—the computational structures differ across baselines. The main table gives point estimates only, and the five-run training variance in Figure 4 has not been converted into difference intervals for the main table. The 1–2 point advantages near baselines and the 6-task gap on SWE-bench need seed-paired confidence intervals and fixed-compute or fixed-execution-budget comparisons (Table 1, Figure 4, Appendices C/F).

Questions for authors

1.Please provide multi-seed means and paired intervals for each model/benchmark in the main table, and include GRPO and OPSD+execution-reward baselines under equal GPU-hours/equal-executions, as well as a direct ablation without future accumulation. 2.For samples that completely fail before training, does the token-level log-probability gap correlate with human-annotated error tokens or execution-based localization signals? What is the performance when all target solutions and public test examples are removed, leaving only non-answer hints?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.