acceptodds
Withdrawn from ICLR 2027

RLPBE: Reinforcement Learning with Adaptive Multi-Aspect Rewards for Programming-by-Example

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Programming-by-Example (PBE) enables the automatic synthesis of programs from input–output examples, allowing users to implement their intended programs without requiring programming expertise. Classic PBE methods mainly depend on domain-specific languages (DSL) and search strategies to generate programs, but are limited by DSL expressiveness. Recent approaches based on large language models (LLMs) reduce this dependency and synthesize programs from examples through prompting or supervised fine-tuning (SFT). However, the lack of fine-grained supervision signals often leads LLMs to generate programs that deviate from intended behaviors or contain redundant code. To address these limitations, we introduce RLPBE, a reinforcement learning framework designed for PBE tasks. RLPBE incorporates multi-aspect reward metrics covering four aspects: function usage, code summarization, pass rates, and code conciseness. These rewards are aggregated using our proposed performance-guided adaptive reward strategy and utilized to train LLMs through Group Relative Policy Optimization (GRPO). Experimental results on representative PBE tasks demonstrate the effectiveness of our method. For example, on the List transformation task, RLPBE outperforms the strongest baseline by up to 6.43% on Pass@1 and 7.45% on Pass@5.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Programming-by-Example, Reinforcement Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nnZYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RLPBE, a reinforcement learning framework for improving LLM program synthesis on programming-by-example (PBE) tasks. Its main contribution is a reward that measures how close a generated program is to a reference program along two axes, in addition to execution correctness:

  • Function-usage reward: overlap in the functions the two programs use, a proxy for syntactic and structural similarity.

  • Code-summarization reward: similarity between natural-language descriptions of the programs’ code blocks, a proxy for semantic similarity.

The paper also introduces a performance-guided adaptive rewarding mechanism that acts as a curriculum over the reward components. Early in training, when the pass rate is low, the function-usage and logic resemblance reward carries more weight. That weight decreases as training progresses and the pass rate rises. Across several benchmarks and model backbones, RLPBE outperforms SFT and standard RLVR, and the resulting small models perform on par with frontier LLMs.

Critical strengths
  1. Strong results against frontier models. RLPBE-trained small models match or exceed frontier LLMs on PBE tasks across benchmarks.

  2. Consistent gains over RLVR. RLPBE generally outperforms pass-rate-only RLVR across backbones. This supports the paper’s main claim that rewarding properties of a program beyond pass rate helps when the binary signal is sparse.

  3. Thorough ablations. The paper examines the contribution of each reward component and of the adaptive weighting mechanism separately.

  4. Adequate experimental coverage. The selection of benchmarks (Lists, SyGuS, MBPP, PROSE) and backbones is broad enough to support the paper’s claims.

Critical weaknesses

W1. Missing comparison with process-reward approaches to PBE. The paper does not compare against PRM-PBE (PRM-PBE: Process Reward Model for Reinforcement Learning in Programming-by-Example, 2026). PRM-PBE also pays a one-time cost: it builds a process-supervised dataset for PBE, trains a PRM with preference learning, and uses that PRM to supervise the synthesizer during RL. Based on the reported numbers, PRM-PBE outperforms RLPBE on every benchmark, for instance across Qwen-3, Llama-3 backbones. One plausible explanation is that step-level process rewards give the synthesizer denser, more localized feedback than RLPBE’s sequence-level, reference-similarity rewards. The authors should discuss this comparison directly and explain why and when RLPBE should be preferred.

W2. Missing related work on RL rewards anchored to reference programs. Several earlier code-generation methods outside PBE already reward similarity to a reference program, beyond execution, and the paper should position its reward components against them:

  • RLCF (Jain et al., 2023, Coarse-Tuning Models of Code with Reinforcement Learning Feedback) uses an LLM-based comparison of generated and reference programs. This is conceptually close to RLPBE’s code-summarization reward.

  • PPOCoder (Shojaee et al., 2023, Execution-based Code Generation using Deep Reinforcement Learning) rewards the AST subtrees and data-flow-graph edges a generated program shares with the reference. This is close to RLPBE’s function-usage reward.

Comparing against these would clarify what is new about RLPBE’s reward design.

W3. Possible train–test overlap. The training data pipeline uses tasks from MBPP, Lists, and PROSE, which are also evaluation benchmarks. What steps were taken to ensure that the intents of the test tasks do not overlap with the generated training tasks? Please describe the deduplication procedure and report the overlap rate, for example by exact match, n-gram overlap, or embedding similarity of task descriptions and I/O examples.

W4. Cost of LLM-based reward computation. Computing the function-extraction and code-summarization rewards requires calls to a frontier LLM for every generated program, in every GRPO group, for every training task. This makes training expensive. What is the latency cost added by these calls. Can a smaller open model replace the frontier LLM without hurting performance. This affects both cost and reproducibility, since frontier API models change over time.

W5. Anchoring to a single reference penalizes valid alternatives. From conceptual point of view, the rewarding mechanism of RLPBE during initial course of training could be pulling generations toward one reference program that may penalize correct but structurally different solutions. Can you explain if and how RLPBE is avoiding this?

Questions for authors

See Weaknesses section.

Reviewer_CYdiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduced a new reinforcement learning framework called RLPBE for the Programming-by-Example task. By aggregating four scoring dimension (namely function usage reward, code summarization reward, pass-rate reward and code conciseness reward) into one reward function, the authors demonstrate that optimizing these rewards using GRPO leads to consistent performance improvements across multiple language models on four PBE benchmarks. Ablation studies further support the effectiveness of the individual reward components and the proposed adaptive weighting strategy.

Critical strengths
  1. The paper is well organized and easy to follow. Combining functional correctness, semantic similarity, and structural feedback provides an intuitive approach to improving program synthesis.
  2. The ablation studies are generally informative, demonstrating the contributions of individual reward components and examining the effects of different weighting strategies, reward shaping functions, and training data scales.
Critical weaknesses
  1. The main components of the proposed framework, including GRPO, code similarity metrics, and execution-based rewards, are well-established concepts in the existing literature. The primary contribution of combining these signals through a weighting formula appears incremental.
  2. The function matching, semantic similarity and conciseness rewards rely heavily on reference implementations. However, obtaining reliable reference programs can be challenging in practical PBE settings, and synthesized reference implementations may vary on quality and introduce unintentional biases toward particular implementations.
Questions for authors

The experiements used GPT-5.2 as the annotation model during the RL training, which is a very strong model compared with the training bases. How sensitive is RLPBE's performance to the choice of annotation model? In particular, could the authors provide some results using a smaller open-source model, such as the corresponding backbone model itself, for reward annotation?

Reviewer_oU7gOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the programming-by-example (PBE) problem, which learns to program given examples of input–output assertions. Previous llm-based methods mainly depend on SFT or RL with varifiable rewards (success/failure). This paper proposes to use richer reward signals, including pass-rate, code conciseness, code summarization, etc, to assistant RL for PBE. The proposed algorithm is empirically validated on four benchmarks and shows improved performance.

Critical strengths

The paper is well-written and easy to follow. The proposed algorithm is simple and easy to apply in practice. Improved performance is shown in training.

Critical weaknesses
  1. The algorithm design is more engineered than principled. Beyond the four chosen criteria, there exist many other perspectives one can evaluate a program, e.g., time/space complexity, structural simplicity, readability, robustness in edge cases, etc. It is unclear whether the four chosen criteria are enough to reveal the essence of the reference code, or other reward terms could also help the RL.

  2. Critical training details are missing. Appendix B mentioned how RLPBE is trained using SFT with 10k samples and RL with 40k data samples. However, it is not clear how many samples the SFT baselines are trained with. Specifically, are they trained with 10k samples or 50k samples? These details are critical for a fair comparison.

  3. Critical evaluation details are missing. Firstly, it remains unclear whether the evaluations of the generated program are performed in assertions included in the problem statement, or are also done on unseen assertions. If it is the former case, the evaluation results can not support the claim of the paper of "generalizing to unseen instances". If it is the latter case, could the authors provide more details on how the test assertions are generated? Secondly, it is unclear whether the seed samples (mentioned in Appendix D) are included or excluded from the evaluation.

Questions for authors

Please see the "critical weaknesses" section for main questions. I also have an additional question:

The conciseness reward Rc seems to encourage code with similar length to the reference. It will penalizes programs the achieve the same goal with minimal length. This seems to have negative influence during training especially when the reference code is not optimal in length.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.