- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes RLPBE, a reinforcement learning framework for improving LLM program synthesis on programming-by-example (PBE) tasks. Its main contribution is a reward that measures how close a generated program is to a reference program along two axes, in addition to execution correctness:
-
Function-usage reward: overlap in the functions the two programs use, a proxy for syntactic and structural similarity.
-
Code-summarization reward: similarity between natural-language descriptions of the programs’ code blocks, a proxy for semantic similarity.
The paper also introduces a performance-guided adaptive rewarding mechanism that acts as a curriculum over the reward components. Early in training, when the pass rate is low, the function-usage and logic resemblance reward carries more weight. That weight decreases as training progresses and the pass rate rises. Across several benchmarks and model backbones, RLPBE outperforms SFT and standard RLVR, and the resulting small models perform on par with frontier LLMs.
Critical strengths
-
Strong results against frontier models. RLPBE-trained small models match or exceed frontier LLMs on PBE tasks across benchmarks.
-
Consistent gains over RLVR. RLPBE generally outperforms pass-rate-only RLVR across backbones. This supports the paper’s main claim that rewarding properties of a program beyond pass rate helps when the binary signal is sparse.
-
Thorough ablations. The paper examines the contribution of each reward component and of the adaptive weighting mechanism separately.
-
Adequate experimental coverage. The selection of benchmarks (Lists, SyGuS, MBPP, PROSE) and backbones is broad enough to support the paper’s claims.
Critical weaknesses
W1. Missing comparison with process-reward approaches to PBE. The paper does not compare against PRM-PBE (PRM-PBE: Process Reward Model for Reinforcement Learning in Programming-by-Example, 2026). PRM-PBE also pays a one-time cost: it builds a process-supervised dataset for PBE, trains a PRM with preference learning, and uses that PRM to supervise the synthesizer during RL. Based on the reported numbers, PRM-PBE outperforms RLPBE on every benchmark, for instance across Qwen-3, Llama-3 backbones. One plausible explanation is that step-level process rewards give the synthesizer denser, more localized feedback than RLPBE’s sequence-level, reference-similarity rewards. The authors should discuss this comparison directly and explain why and when RLPBE should be preferred.
W2. Missing related work on RL rewards anchored to reference programs. Several earlier code-generation methods outside PBE already reward similarity to a reference program, beyond execution, and the paper should position its reward components against them:
-
RLCF (Jain et al., 2023, Coarse-Tuning Models of Code with Reinforcement Learning Feedback) uses an LLM-based comparison of generated and reference programs. This is conceptually close to RLPBE’s code-summarization reward.
-
PPOCoder (Shojaee et al., 2023, Execution-based Code Generation using Deep Reinforcement Learning) rewards the AST subtrees and data-flow-graph edges a generated program shares with the reference. This is close to RLPBE’s function-usage reward.
Comparing against these would clarify what is new about RLPBE’s reward design.
W3. Possible train–test overlap. The training data pipeline uses tasks from MBPP, Lists, and PROSE, which are also evaluation benchmarks. What steps were taken to ensure that the intents of the test tasks do not overlap with the generated training tasks? Please describe the deduplication procedure and report the overlap rate, for example by exact match, n-gram overlap, or embedding similarity of task descriptions and I/O examples.
W4. Cost of LLM-based reward computation. Computing the function-extraction and code-summarization rewards requires calls to a frontier LLM for every generated program, in every GRPO group, for every training task. This makes training expensive. What is the latency cost added by these calls. Can a smaller open model replace the frontier LLM without hurting performance. This affects both cost and reproducibility, since frontier API models change over time.
W5. Anchoring to a single reference penalizes valid alternatives. From conceptual point of view, the rewarding mechanism of RLPBE during initial course of training could be pulling generations toward one reference program that may penalize correct but structurally different solutions. Can you explain if and how RLPBE is avoiding this?
Questions for authors
See Weaknesses section.