acceptodds
Under review as a conference paper at ICLR 2027

SEER: Meta-Reinforcement Learning for Exploratory Search in Code LLM Post-Training

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a powerful paradigm for improving the reasoning performance of large language models. Yet its improvement is constrained by a simple bottleneck: RL can directly reinforce only reasoning trajectories that the policy has already discovered. Increasing rollout budgets can expose rare successes, but through increasingly expensive, undirected sampling; meanwhile, objective-level interventions reshape the policy distribution without specifying where or how the model should explore. Consequently, useful solution strategies may remain effectively inaccessible under practical rollout budgets. We address this limitation with SEER, a meta-reinforcement learning framework for Code LLM post-training targeting useful solutions rarely exposed under practical rollout budgets. SEER organizes attempts into meta-trajectories, where prior code, verifier feedback, and reflections guide search. Credit extends across attempts: an attempt is valued for immediate correctness and for enabling later success, so failed but productive attempts receive downstream credit. A meta-GRPO objective combines direct and cross-attempt signals. Yet search-conditioned discovery alone is insufficient: behaviors found under search context may remain inaccessible from the original prompt. SEER therefore transfers discoveries to the original-prompt policy via a distillation-augmented objective, enabling context-free inference. SEER improves mean Pass@1 by 3.58–5.40 percentage points over the baseline average; it reaches 33.8% coverage on previously unseen cases and gains +5.88 points in meta-search over StepCoder, supporting structured exploration and reintegration as an path for extending RLVR.

Primary area:
reinforcement learning
Keywords:
Code LLMs, RLVR, Meta-Reinforcement Learning, Exploratory Search, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2jbFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SEER connects multiple code-generation attempts through test feedback and reflection. It uses rewards from individual attempts and the overall search to give credit to failed attempts that help later ones succeed. It also trains on successful programs found during search, paired with the original problem, to improve direct generation without the search history. Experiments cover several code benchmarks and models from 3B to 14B, with additional model families evaluated in the appendix.

Critical strengths
  1. The paper evaluates both whether search finds correct programs and whether these discoveries improve direct generation. This distinction is useful and makes the role of distillation easier to assess.

  2. The experiments cover several model sizes and code benchmarks, with relevant baselines and useful ablations. They provide reasonable support for the method's effectiveness.

Critical weaknesses
  1. The method appears to combine LaMer-style learning across attempts with supervised training on successful outputs. Both ideas have been studied before, and the paper does not clearly explain what is new beyond their combination.

  2. The reward design relies on several heuristics without sufficient justification. The limited search analysis leaves it unclear how sensitive the results are to these choices or whether they generalize to more complex coding tasks.

Questions for authors
  1. What accounts for SEER's additional gains over LaMer-style training followed by supervised learning on the same successful programs? Does the successful-sample replay baseline match the source of the training examples, how often they are reused, and the total training budget?

  2. How does reintegration relate to On-Policy Self-Distillation (OPSD)? The objectives differ, but both aim to transfer abilities available with extra context to a policy given only the original problem. Why use supervised learning on successful programs, and what does this add relative to existing distillation methods?

4 comments

We thank the reviewer. We added reward/advantage ablations, replay, harder-task/hidden-test, and OPSD experiments to Appendices B, K, and L. Scores are %; H = hidden; runs differ.

W1/Q1. Contribution and Matched Replay

We agree cross-attempt learning and successful-output SFT are not new individually. SEER targets coding agents' first-answer ability and recovery after failure. SEER is motivated by two successive bottlenecks in the RLVR code.i) With limited rollouts, RL reinforces sampled solutions; ii) more rollouts increase search volume, not how to search. SEER links attempts via verifier feedback/reflection and credits productive failures. Search-discovered solutions may remain unlikely from the original prompt, so SEER reintegrates the first verified discovery under it.

On 12 matched HumanEval29 cases, SEER recovers 75.0% vs. 66.7% for StepCoder; resetting history or removing verifier feedback gives 66.7% (Fig. 7).

APPS-hard Direct@1 / Selected@1,@4

TestSEER DLaMer DSEER S@1/4LaMer S@1/4
APPS3002623.6721.25/36.0020.17/34.33
HE16479.2775.6173.02/93.9072.56/92.07

Long context, replay, or immediate credit alone cannot reproduce SEER:

ControlDirectH-Sel.H-Oracle
Long ctx + replay24.3314.7117.65
Long ctx, no replay23.6713.7318.63
Immediate credit2511.7617.65
Full SEER27.3315.6920.59

Matched LaMer→SFT: same no-distill step-300 checkpoint; 247 identical successes/160 problems, 4 replays each (988 exposures), 247 updates, 140,232 target tokens. Evaluation uses only the original prompt.

MethodDirectH-Sel.H-Oracle
LaMer credit + matched SFT2510.7818.63
Full SEER26.3314.7121.57
Gain (pp)1.333.932.94

With successes fixed, context matters:

ReintegrationAPPS300Source-160
No distillation25.6747.5
Original prompt → SFT27.3353.13
Search context → SFT24.3349.38
Context dropout2650

Original-prompt SFT beats search-context SFT by 3.00 pp; on source-160 it recovers 9/84 failures, loses 0/76 successes (net +9/160). SFT/SEER replay: APPS 980/139,400 vs. 980/139,392; MBPP 1,158/131,202 vs. 1,158/131,198 (exposures/target tokens).

W2. Reward Sensitivity and Harder Tasks

Reward and advantage ablations are in Appendix K.

RewardSolved@1
R0: verifier only21.33
R1: + full-correct bonus22.67
R2: + rare search bonus25
R3: + early-success25.33
R4: + rare first-answer bonus27

Advantage terms target first-answer quality, cross-attempt credit, and trajectory optimization:

AdvantagePass@1Pass@4
Full SEER24.6736.67
w/o (AtA_t)23.0834.33
w/o (AeA_e)22.5832.67
w/o (A0A_0)22.7533.67

On harder APPS (300 Competition), shaped rewards improve hidden-test Oracle over strict 0/1 rewards:

k0/1FullΔ pp
17.8412.754.9
213.7316.672.94
317.6522.554.9
420.5922.551.96

Normalization lowers length–token-weighted-credit correlation 0.646→0.556; long-first-attempt recovery (≥256 tokens) rises 4.6%→11.6%.

Q2. OPSD and Why SFT

SEER SFTs verifier-confirmed code under the original prompt; OPSD-KL uses a privileged-context teacher for on-policy outputs. Paired methods share fixed successes, timing, and replay; Full SEER retains dynamic discovery. Qwen2.5-Coder-3B-Instruct, 300 updates, 1 seed; APPS+: 4,063/300 train/val; MBPP: 374/90 train/val, 224 tests.

EvaluationPaired SFTOPSD-KLFull SEER
APPS30026.3326.3327.33
HumanEval16476.8379.8881.1
MBPP90*56.6755.5660
MBPP22467.367.4169.42
MBPP+22457.4857.9260.83
MBPP→MBPP+ drop (pp)9.829.498.59

*MBPP90 is step 200. OPSD-KL gains 3.05 pp on HumanEval164 over SFT, ties on APPS300, and gains only 0.11/0.44 pp on MBPP/MBPP+. Full SEER performs best, including dynamic discovery gains.

Cost (OPSD vs. SFT)APPS+MBPP
Forward-token ratio2.42×3.05×
Extra student tokens0.163M0.186M
Total output-token increase0.10150.3695
Training time18.73→34.92 h8.04→15.57 h
Wall-clock ratio1.86×1.94×

Forward-token ratios exclude generation, backpropagation, and KL (not FLOPs); wall-clock is end-to-end. Verified code provides exact targets without extra student sampling/full-vocabulary KL. OPSD-KL offers no consistent gain at ~1.9× cost; SFT is simpler and cheaper.

Thank you again for your constructive feedback. In our rebuttal, we added budget- and data-matched LaMer→SFT comparisons, controlled reintegration experiments, and reward/advantage ablations, further clarifying SEER’s contributions beyond combining existing techniques.

We also included harder-task hidden-test evaluations and direct comparisons with OPSD, including performance and computational costs. We would greatly appreciate your thoughts on whether these additional results adequately address your concerns.

Thanks for the authors' response and the extended experiments, I therefore raised my score.

We sincerely thank the reviewer for their thoughtful feedback and for taking the time to evaluate our revisions. We are glad that our additional experiments and clarifications addressed your concerns. We greatly appreciate your consideration and your decision to raise the score.

Reviewer_HrW8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies exploration under limited computational budgets in code LLM post-training. SEER organizes code generation, execution feedback, and reflection into meta-trajectories containing multiple attempts. It trains a search policy through cross-attempt credit assignment and distills successful programs discovered during search back into the original problem context to improve direct generation (Sec. 3). Experiments cover three sizes of Qwen2.5-Coder-Instruct and four code generation benchmarks. Relative to the mean performance of three baselines, the paper reports average Pass@1 improvements of 3.58–5.40 percentage points (Table 1). Ablations, training dynamics, and search-budget analyses further examine the method's behavior.

Critical strengths
  1. The paper addresses an important problem in code LLM post-training by improving exploration under limited sampling budgets.
  2. SEER combines feedback-guided exploration, cross-attempt credit assignment, and distillation into a unified framework for improving both search and direct generation.
  3. The method achieves consistent improvements across multiple benchmarks, model scales, and model families, supported by extensive experimental analyses.
Critical weaknesses
  1. Training–test mismatch. SEER trains the policy to solve problems through multiple attempts conditioned on previous failures, verifier feedback, and reflections, while direct evaluation requires generating correct solutions from the original prompt alone. Its methods may not fully bridge this gap, as the model could become dependent on iterative feedback rather than learning to solve problems directly. The paper does not sufficiently investigate the impact of this mismatch on direct-generation performance.
  2. SEER discovers successful solutions conditioned on previous attempts, but reintegration distills them using only the original prompt. This creates a mismatch between the contexts used for discovery and training, introducing misaligned supervision when solutions depend on information obtained during search. The paper does not sufficiently investigate this issue.
  3. SEER combines several established techniques, including discounted returns, group-normalized advantages, reward shaping, and successful-sample distillation. However, the paper does not sufficiently isolate the contributions of its three advantage terms or the two scarcity bonuses. The existing ablations support the overall framework but do not establish the necessity of these specific algorithmic designs over simpler alternatives.
Questions for authors
  1. Does training with multi-attempt reflection encourage the model to rely on iterative correction rather than solving problems directly? How do first-attempt success rates evolve during training? To what extent do the first-attempt advantage and reintegration mitigate this issue?
  2. How do the average code-generation length, reflection length, and output truncation rate change during training? Does SEER produce substantially longer responses than GRPO under the same inference settings due to the model's awareness of available reflection chances during training?
  3. How do your method compare with using a longer context length?
4 comments

We thank reviewer; additional experiments and analyses are in the Appendix.

W1/Q1. Does search training create dependence on iterative feedback rather than improve direct solving?

SEER separates history-conditioned discovery from original-prompt deployment. Fig. 1(b): vs. StepCoder, LaMer gains +2.94 pp Meta-search Solved@1 but loses 0.74 pp Direct Pass@1; SEER gains +5.88/+2.22 pp. Fig. 6: removing distillation costs 3.92/0.98 pp Direct@1/Solved@3; immediate rather than cross-attempt credit costs 1.96/4.90 pp. Final Direct@1/Meta Solved@1: LaMer 23.67/28.00, w/o distillation 25.67/29.58, Full SEER 27.33/29.67. Distillation adds 1.66 pp direct but only 0.09 pp search.

MethodFirst-code success S0→S100Direct@1 S0→S100
Full SEER20.58→22.92 (+2.33)21.67→27.33 (+5.67)
w/o distillation20.58→22.83 (+2.25)22.67→25.67 (+3.00)
Long-context GRPO, no replay20.58→21.42 (+0.83)—

On the original prompt, removing A₀, distillation, or both lowers Direct@1 to 26.00%, 25.67%, 25.00%, vs. 27.33% for Full SEER.

W2. Does original-prompt distillation misalign supervision with search-discovered solutions?

Search enables discovery; distillation targets deployment. All variants share the step-300 checkpoint and 247 successful programs from 160 problems (replayed four times), and use the original prompt for evaluation.

Distillation inputRecovery / LossNet gainAPPS300Source-160
No distillation0 / 0%0.000 pp25.6747.5
Original-only10.71 / 0%+5.625 pp27.3353.13
Search-context7.14 / 3.95%+1.875 pp24.3349.38
Context-dropout7.14 / 2.63%+2.500 pp2650

Recovery/Loss use 84 initially failed / 76 initially solved problems; Net gain = (recovered − forgotten)/160. Original-only gives the best net gain with zero forgetting. APPS* hidden-direct scores: 5.88/6.86/7.84/6.86 (none/original/search/dropout), so Original-only does not dominate every split.

W3. Are the advantage terms and shaped rewards necessary?

Combined advantage: A₀, Aₑ, and Aₜ.

VariantDirect@1Direct@4
Full SEER24.6736.67
w/o A₀22.7533.67
w/o Aₑ22.5832.67
w/o Aₜ23.0834.33
w/o A₀ & distillation2231.67

Without Aₑ, recovery falls 20.99%→11.90% (−9.09 pp). Aₜ lowers length/absolute-credit correlation 0.646→0.556 and 1,024–2,047-token trajectory credit 1,051.09→949.07.

RewardAdded componentSolved@1
R0Verifier reward21.33
R1plus full-correctness bonus22.67
R2+ search-success scarcity (bₛ)25
R3plus early-success term25.33
R4+ first-answer scarcity (b₀)27
R4 w/o bₛRemove bₛ only24.67

Adding bₛ improves R1→R2 by 2.33 pp, b₀ improves R3→R4 by 1.67 pp, and removing bₛ from R4 costs 2.33 pp. Hidden-test Selected/Oracle, strict 0/1 vs. shaped:

MetricStrict 0/1 Selected/OracleShaped reward Selected/Oracle
@17.84/7.8412.75/12.75
@211.76/13.7313.73/16.67
@313.73/17.6515.69/22.55
@415.69/20.5916.67/22.55

Q2. Does reflection cause output-length inflation?

No. SEER's training code length falls 251→177 tokens, P95 512→351, truncation 7.13%→0.76% (GRPO: 240→196.4; 8.85%→0.42%); reflection length falls 89→84 tokens, with cap rate 7–13%→3–7%.

Under history-free APPS300 decoding (512-token cap), mean/P95 are 210.6/422 (GRPO), 179.4/389 (long-context + replay), 178.4/360 (immediate-credit SEER), and 171.7/360 (Full SEER); no-EOS-at-cap: 3.00% vs. 2.67% (GRPO/Full). Full SEER's direct output is 18.5% shorter.

Q3. Does SEER merely benefit from longer context?

Hidden: Selected/Coverage; cost: input+output tokens/verifier calls.

MethodDirect@1Solved@4Hidden S/CTokens (k) / Calls
Long-context GRPO23.6747.0610.78/18.631,459.9 / 1,037
Long-context + replay24.3351.9614.71/20.591,443.9 / 1,054
Immediate-credit SEER24.6751.9614.71/20.591,445.0 / 1,055
Full SEER27.3355.8817.65/22.551,453.8 / 1,052

Vs. long-context GRPO + replay, Full SEER gains +3.00 pp Direct@1, +3.92 pp Solved@4, +2.94 pp hidden Selected, and +1.96 pp hidden Coverage, using 0.69% more tokens and two fewer verifier calls. Immediate-credit SEER underperforms despite reward shaping and distillation, isolating cross-attempt credit.

Cross-attempt credit improves discovery; original-prompt reintegration enables direct use.

Thank you again for your thoughtful feedback. Our rebuttal adds first-attempt success analyses, controlled distillation comparisons, and ablations of the advantage terms and scarcity bonuses. We also report response-length and truncation trends.

We further compare SEER with long-context GRPO under similar token and verifier budgets, including direct, search, and hidden-test results. We’d appreciate your thoughts on whether these additions address your concerns and welcome remaining questions.

Thank you for the rebuttal. While the additional experiments address some concerns, three issues remain:

  1. Training stability. Repeated context reconstruction introduces context inconsistencies and spurious references to previous attempts in the model's cot, compromising stability and scalability in extended RL training.
  2. Limited gains. Most components contribute only 1–2 pp improvements, which is limited relative to the complexity of the proposed designs.
  3. Evaluation inconsistency. Direct@1 and First-code success both measure initial-attempt correctness, yet differ substantially, raising concerns about the reliability of the results.

Overall, I appreciate the additional evidence but will maintain my original rating.

We thank the reviewer for raising concerns about training stability, methodological complexity, and evaluation consistency. We clarify these points.

1. Training Stability

We agree that context consistency matters in iterative RL, especially when reflections may incorrectly reference prior attempts. SEER uses explicit reflection memory, not an ever-growing concatenated history. Each independent trial starts with empty memory (m0=∅m_0=\emptyset) and maintains separate history. Code generation conditions on the original problem and memory. After failure, reflection summarizes the current code and verifier feedback, guides the next attempt, and updates memory. Execution state resets between complete-program episodes, while memory is preserved.

Our configuration uses five independent trials per problem, each with at most three code episodes and two reflection stages; thus, our evidence is limited to a bounded search horizon.

During training, average code length decreases from 251 to 177 tokens, P95 from 512 to 351, and truncation from 7.13% to 0.76%; average reflection length decreases from approximately 89 to 84 tokens. This indicates no sustained output-length inflation. However, stable lengths do not establish semantic consistency or rule out erroneous references. Our results support SEER's effectiveness in the current search horizon; we will clarify its configuration and discuss possible extensions.

2. Methodological Complexity and Performance Gains

Although some individual ablation gains are modest, SEER combines complementary mechanisms:

  • A0A_0 preserves first-attempt solving and limits reliance on subsequent search.
  • AeA_e propagates downstream success to productive earlier attempts. Removing it reduces recovery from 20.99% to 11.90%, demonstrating the value of cross-attempt credit.
  • AtA_t regulates trajectory-level credit; the correlation between trajectory length and absolute credit decreases from 0.646 to 0.556.
  • Distillation reintegrates successful discoveries into the original-prompt policy, transferring exploration gains to direct solving.

Against Long-context GRPO + Replay under approximately matched budgets, Full SEER improves Direct@1 by 3.00 pp, Solved@4 by 3.92 pp, Hidden Selected Accuracy by 2.94 pp, and Hidden Oracle Coverage by 1.96 pp, using only 0.69% more input/output tokens and two fewer verifier calls. These gains are associated with cross-attempt credit assignment, rather than simply longer contexts or successful-sample replay.

The components address complementary challenges in exploration and knowledge transfer. Their combination improves performance under nearly matched computational budgets.

3. First-code Success versus Direct@1

We thank the reviewer for flagging this ambiguity. It reflects unclear descriptions of the decoding protocols, not unreliable results. We apologize for the confusion.

First-code success measures whether the first code in each independent meta-search trial passes the tests. Each trial starts with empty reflection memory, so this generation receives no prior feedback. Answers use stochastic sampling (typically T=0.8T=0.8 with top-p sampling); the metric averages success across initial samples:

First-code success = Number of successful first-code answers / (Number of problems × Number of independent search trials per problem).

Direct@1 evaluates one answer per problem, generated from the original prompt without search history or reflection feedback, using greedy decoding:

Direct@1 = Number of problems solved by greedy decoding / Total number of problems.

The metrics need not coincide: First-code success averages multiple stochastic initial answers, whereas Direct@1 evaluates one deterministic greedy answer per problem. Training may concentrate probability mass on high-probability canonical solutions, improving greedy decoding, while stochastic sampling may still produce lower-probability errors.

For Full SEER, First-code success rises from 20.58% to 22.92% (+2.33 pp), while Direct@1 rises from 21.67% to 27.33% (+5.67 pp). This difference reflects the distinct protocols; the larger Direct@1 gain is consistent with transferring search discoveries to the original-prompt policy through successful-sample distillation.

Finally, S0→S100S_0\rightarrow S_{100} in Table 3 denotes an experiment with only 100 parameter updates, due to limited resources and time. It illustrates trends under a limited budget, not performance after full convergence.

We acknowledge that the original manuscript did not clearly distinguish the metrics' sampling procedures, decoding strategies, and denominators. We thank the reviewer for helping clarify SEER's mechanisms, scope, and evaluation protocols.

Reviewer_z57gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets RLVR’s finite-budget exploration bottleneck and proposes SEER, which has two parts. First, it organizes successive code attempts, code, verifier feedback, and reflections, into meta-trajectories and trains them with a meta-GRPO objective that combines first-answer, episode-return, and trajectory-level advantages, so that failed but productive attempts receive downstream credit. Second, it distills first successful search discoveries back into the original-prompt policy, so inference needs no search history. Experiments use Qwen2.5-Coder 3B/7B/14B plus two other backbones on APPS+, MBPP+, TACO, and LCB, and report direct-generation and meta-search gains, ablations, and a reachability-stratified analysis.

Critical strengths
  • The framing is clear. The paper separates discoverable under search from generatable from the original prompt, the search–policy gap, and addresses each with cross-attempt credit and distillation, respectively. Appendix B is convincing on this point. Discovery coverage is similar with and without distillation, yet direct performance differs, which supports reintegration as the mechanism.
  • The gains are consistent in direction across 3 scales, 3 backbones, and multiple benchmarks.
Critical weaknesses
  • The paper is motivated by extending effective reachability, yet Fig. 4 shows the gap essentially vanishes at Pass@128 (54.00 / 53.67 / 53.33). The direct-policy gains therefore look like sample efficiency rather than an expanded reachable set. “Unseen@4” only means 0/4 successes under Instruct, which is a very noisy stratification and should not be equated with “never discovered.”
  • Figs. 1 and 19 report public-test solve rates, but search uses those same public tests as its verifier and stops when they pass, which inherently favors search methods. Hidden-test results appear only in Appendix F and are not compared against baselines.
Questions for authors
  • Can you add a “GRPO + success-sample distillation” baseline to separate the contribution of meta-RL credit assignment from self-distillation?
3 comments

We thank the reviewer and address each concern below.

W1. Effective reachability and Unseen@4

We agree that convergence at Pass@128 does not support the stronger claim that SEER expands asymptotic solution support. As stated on p.2, our focus is effective reachability under practical rollout budgets, not strict distributional-support expansion.

Direct Pass@128SEERStepCoderGRPO
Accuracy (%)5453.6753.33

Figure 4 shows that SEER's discovery advantage narrows as independent-sampling budgets increase. We revised the main-text wording, replacing asymptotic/support-expansion claims with finite-budget effective reachability and discovery efficiency.

Unseen@4 does not mean unreachable: the split is defined by four public-test samples from the Instruct reference policy, and 0/4 means only that no success was observed within this limited budget. We retain this finite-budget interpretation. This distinction concerns observed sampling outcomes, not whether a solution has zero probability under the policy.

On this subset, SEER discovers more hidden-test-correct programs:

Unseen Solved@4GRPOLaMerSEER
Hidden Selected1.4111.2711.27
Hidden Oracle2.8212.6815.49

Our revised claim is better finite-budget discovery of low-probability correct solutions, not asymptotic support expansion. We added this clarification to the main text and moved supporting hidden-test analyses to Appendix L.

W2. Public-test bias and hidden-test evaluation

We agree public-test solve rate alone is insufficient because public tests guide search and early stopping. We therefore re-evaluated all stored APPS* candidates on independent hidden tests never used for generation, reflection, or stopping, adding all baselines under the same protocol.

Each cell reports Hidden Selected Accuracy / Hidden Oracle Coverage (%):

MethodSolved@1Solved@2Solved@3Solved@4
Instruct6.86/7.848.82/8.827.84/9.808.82/9.80
GRPO4.90/5.884.90/7.846.86/9.808.82/11.76
StepCoder12.75/13.739.80/13.7312.75/13.7312.75/13.73
ReflexiCoder8.82/13.7311.76/14.7112.75/16.6711.76/19.61
LaMer15.69/17.6514.71/17.6515.69/18.6317.65/18.63
SEER15.69/19.6115.69/21.5715.69/22.5517.65/23.53

Selected evaluates the single answer chosen by the public verifier; Oracle asks whether any generated candidate passes hidden tests. SEER has the highest Hidden Oracle Coverage at all four budgets, exceeding LaMer by +1.96/+3.92/+3.92/+4.90 pp.

Selected is not always highest because selection still uses public tests. However, higher Oracle coverage shows that SEER discovers more genuinely correct programs. We added this full hidden-test baseline comparison to Appendix F and will distinguish public-verifier performance from hidden-test correctness in the main text.

Q1. GRPO + successful-sample distillation

We apologize that Fig. 6 did not make this comparison explicit. Success-replay is exactly the reviewer-requested “GRPO + successful-sample distillation” baseline. We clarified this in the main text and added supporting experiments in Appendix B.

VariantChangeAPPS Direct Δ*APPS Search Δ*HE Direct ΔHE Search Δ
No-distillRemove distillation−3.92−0.98−5.49−2.44
Immediate-creditCross-attempt → immediate credit−1.96−4.90−1.22−9.15
Success-replayGRPO + success distillation−1.96−2.94−2.44−4.27
Reflection-offRemove reflection−3.92−0.98−6.71−0.61

Removing distillation mainly hurts direct generation, while replacing cross-attempt credit mainly hurts meta-search. GRPO + successful-sample distillation still trails Full SEER by 2.94/4.27 pp in search, showing distillation alone cannot explain SEER's gain.

This separation also appears in direct generation:

Direct Pass@1LaMerSEERGain
APPS30023.67262.33
HumanEval75.6179.273.66

Finally, we fixed the no-distill checkpoint, 247 successful targets, replay counts, and updates, varying only the distillation input:

Distillation inputAPPS300 Direct
No-distill25.67
Original prompt27.33
Search context24.33
Context dropout26

Original-prompt distillation beats search-context distillation by +3.00 pp, paired 95% CI [+0.67, +5.67]. Together, these results show that cross-attempt meta-RL improves discovery, while original-prompt distillation reintegrates discoveries into direct generation.

Thank you again for your constructive feedback. In our rebuttal, we clarified SEER’s finite-budget reachability claim, added hidden-test comparisons with all baselines, and reported Success-Replay (GRPO + successful-sample distillation) with ablations separating meta-RL credit assignment from distillation.

We would greatly appreciate your thoughts on whether these additions address your concerns. We remain happy to clarify any remaining questions during the discussion.

Dear Reviewer z57g:

As the discussion period is drawing to a close, we sincerely believe that a constructive and thorough discussion is important for ensuring that our work and the additional experimental results are fully understood. We hope that our responses and the newly added experiments have adequately addressed your concerns and clarified the points raised.

If any aspects remain unclear or if you have further questions about the experiments or our explanations, we would be more than happy to discuss them and provide additional clarification, especially as the discussion period is drawing to a close. We truly value this opportunity for further exchange and appreciate your time and thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.