acceptodds
Under review as a conference paper at ICLR 2027

Analyzing Refinement in Data-Free Evolution of Code-Generation LLMs

Abstract

Data-free evolution methods reduce reliance on human-curated data: a proposer generates tasks, and a solver learns to solve them. The solver's performance is determined by the quality of its generated tasks, which is often improved through a refinement operation that filters or selects important tasks for solver training. In this paper, we investigate when refinement actually helps the solver, from two specific perspectives. First, from the solver-gradient perspective: the refinement affects the solver only through the alignment between the solver gradient and the downstream task performance. Then, we investigate the proposer-support perspective: the refinement can only select tasks that the proposer generates, so the solver performance is limited by the capability of the proposer. Empirically, on code generation benchmarks, existing refinement operators do not significantly outperform the non-refinement baseline. Furthermore, using a stronger proposer can produce more diverse tasks, which tends to improve the solver's performance even without refinement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Data-free evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1BkQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper analyzes task selection in a single-round proposer–solver pipeline for code generation. It evaluates learnability- and IFD-based filtering, finding no statistically distinguishable aggregate improvement over the unrefined pool in the main experiments, and investigates proposer diversity.

Critical strengths
  1. Separating selection from proposer updates addresses whether filtering adds value beyond task generation. The multi-model comparisons are informative, and the discussion appropriately acknowledges statistical-power limits and confounding in the diversity analysis.
Critical weaknesses
  1. Lemma 1 does not hold for the effective support defined in §3. For a two-logit softmax with rewards (1,0)(1, 0), an on-policy expected-reward gradient step of size 0.10.1 moves p=0.099p = 0.099 above ε=0.1\varepsilon = 0.1, to approximately 0.1006030.100603. Finite sampling also does not ensure D⊆{P>ε}D \subseteq \{P > \varepsilon\}. The containment chain in Proposition 2 is therefore unjustified: selection stays within a realized pool, but this does not establish the claimed restriction to the base proposer’s effective support.
  2. The IFD result concerns a different selection protocol from the cited method. Li et al. (NAACL 2024, §2.2) exclude scores above 11; this paper selects the highest scores and reports selected means of 1.211.21, 1.241.24 and 1.641.64 in Table 2. This may be a reasonable RL adaptation, but its negative result does not establish the performance of the cited filtered procedure. The distinction matters for interpreting one of the two tested refinement operators.
Questions for authors
  1. What sampling or update assumptions would make Lemma 1 valid, and what nontrivial support conclusion remains under them?
  2. Was IFD>1\mathrm{IFD} > 1 excluded before selection, using the same score reported in Table 2? If omitted, does the result persist with that filter under a comparable training budget?
1 comment

We sincerely thank you for your careful examination and review.

Q1 / W1: Effective support and finite sampling.

What sampling or update assumptions would make Lemma 1 valid, and what nontrivial support conclusion remains under them?

Response. We agree that Lemma 1 does not hold for the stated effective-support definition. In your example, the update increases the success probability from 0.0990.099 to approximately 0.1006030.100603, crossing the threshold 0.10.1. A task with probability 0<p≤ϵ0<p\le\epsilon can also appear in NN independent draws with probability 1−(1−p)N>01-(1-p)^N>0.

Selection guarantees F(Dz)⊆DzF(D_z)\subseteq D_z. Containment in a fixed set SzS_z additionally requires sampling to remain restricted to SzS_z, for example through a fixed hard mask; ordinary on-policy RLVR does not provide this guarantee. We will correct the support statement. Pool containment constrains the tasks refinement can select, but does not bound solver generalization.

Q2 / W2: IFD threshold and rerun.

Was IFD > 1 excluded before selection, using the same score reported in Table 2? If omitted, does the result persist with that filter under a comparable training budget?

Response. We agree that the reported arm differs from the original IFD procedure. Li et al. exclude scores above 1 before ranking [1], whereas our selected means of 1.211.21, 1.241.24, and 1.641.64 are incompatible with that exclusion under the reported score definition. We will label this as an IFD ranking adaptation to RL and clarify the implementation.

The current results concern only the tested IFD ranking adaptation. Whether they persist with thresholded selection remains open. Adding the threshold alone would still not reproduce the original instruction-tuning pipeline, which also uses a pre-experienced scoring model.

Reference

[1] From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning

Reviewer_qvw2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an analysis that studies whether task-level refinement, which filters model-generated tasks before training the solver, actually helps in data-free self-evolution for code generation. Using a simplified single-round setup with a frozen proposer, the authors argue that (1) refinement affects the solver only through the alignment between the solver gradient and the downstream task performance, and that (2) proposer’s support fundamentally limits the achievable solver performance. Empirically, they find that common refinement rules do not significantly outperform training on all generated tasks, while a stronger proposer tends to help.

Critical strengths
  1. The core research question is well motivated and cleanly isolated. Refinement is usually evaluated within complex self-evolution systems where its effect is tangled with an adapting proposer, so freezing the proposer and varying only the selection rule is a reasonable and rigorous way to ask whether selection itself really matters.

  2. The empirical reporting is also careful and honest. The authors run significance tests, state clearly that their null result could reflect low power rather than a true absence of effect, and openly report confounds and unstable runs. Several findings are also useful to the community. For example, each refinement rule substantially shifts the property it targets (e.g., roughly doubling the average learnability of the selected tasks), but solver performance barely moves, suggesting that commonly used selection criteria may not capture what makes a task useful for training. The results also indicate that where the tasks come from matters more than which ones are selected.

Critical weaknesses
  1. The main findings and insights are largely expected. That refinement cannot go beyond what the proposer generates follows directly from refinement being a selection step. Similarly, that a stronger proposer tends to yield a better solver, and that filtering a few hundred tasks has little effect in a single short LoRA round, are what most readers would anticipate. The paper would be more compelling if it could identify when refinement would be more useful, or offer a non-obvious, actionable principle for designing better selection rules or proposers.

  2. The experimental setup is too narrow to support the main conclusion. Training in this analysis is a single round with a few hundred tasks, a 50% filter, a short low-rank LoRA update, and three seeds - the authors themselves describe this as "close to the least favorable setting for seeing a refinement benefit". In existing works, refinement is often used in adaptive, multi-round loops where its effects can compound, so a null result here says little about those more complex systems that motivate the paper. The comparison also lacks a random-subset control: refinement keeps only 50% of the generated tasks while the solver trains for the same number of steps, so a null result here could mean either that selection has no effect, or that a real benefit from selection is canceled out by the cost of training on less diverse data.

Questions for authors
  1. How reliable are the generated tasks? It seems each task has only a few self-generated unit tests, and the reference solution is checked only against those tests. If a substantial fraction of tasks are wrong or weakly tested, the solver might receive misleading rewards, which could outweigh the small differences between selection rules. Faulty tests also distort learnability itself, since it is computed from pass rates on those same tests.
2 comments

We sincerely thank you for your thoughtful review and constructive feedback.

W1: Contribution and implications.

The main findings and insights are largely expected. That refinement cannot go beyond what the proposer generates follows directly from refinement being a selection step. Similarly, that a stronger proposer tends to yield a better solver, and that filtering a few hundred tasks has little effect in a single short LoRA round, are what most readers would anticipate. The paper would be more compelling if it could identify when refinement would be more useful, or offer a non-obvious, actionable principle for designing better selection rules or proposers.

Response. We study refinement separately from proposer adaptation, which is usually part of the same learning loop. The solver-gradient analysis distinguishes the score used to select tasks from their performance alignment. Empirically, the comparison asks whether the tested refinement operators improve solver performance over non-refinement in this setting.

For Qwen3-8B, learnability increases from 0.0720.072 to 0.1450.145, while average pass@1 changes from 57.257.2 to 56.456.4; Qwen3-4B instead changes from 54.654.6 to 55.655.6. Increasing the selection score is expected by construction. A higher selection score alone is not evidence of downstream benefit, and the reported comparisons do not establish a benefit or harm. The analysis distinguishes selection scores from performance alignment, but does not establish a general rule for when refinement helps.

W2: Experimental scope and random selection.

The experimental setup is too narrow to support the main conclusion. Training in this analysis is a single round with a few hundred tasks, a 50% filter, a short low-rank LoRA update, and three seeds - the authors themselves describe this as "close to the least favorable setting for seeing a refinement benefit". In existing works, refinement is often used in adaptive, multi-round loops where its effects can compound, so a null result here says little about those more complex systems that motivate the paper. The comparison also lacks a random-subset control: refinement keeps only 50% of the generated tasks while the solver trains for the same number of steps, so a null result here could mean either that selection has no effect, or that a real benefit from selection is canceled out by the cost of training on less diverse data.

Response. Freezing the proposer removes co-adaptation, but selecting half the tasks also changes coverage and reuse. Without a same-size random control, the current comparison does not isolate ranking quality. The current evidence is limited to one round, the reported task pools, 50% retention, rank-32 LoRA, and three seeds. It does not establish behavior in multi-round curricula or that proposer changes are more effective than refinement. Proposer diversity and capability remain confounded.

Q1: Reliability of generated tasks and rewards.

How reliable are the generated tasks? It seems each task has only a few self-generated unit tests, and the reference solution is checked only against those tests. If a substantial fraction of tasks are wrong or weakly tested, the solver might receive misleading rewards, which could outweigh the small differences between selection rules. Faulty tests also distort learnability itself, since it is computed from pass rates on those same tests.

Response. The reference solution is checked against four generated tests, alongside format and topic checks. This establishes internal consistency, not correctness against the task specification. Errors can affect both rewards and learnability, and refinement may change their prevalence even when all methods use the same source pool.

These errors have not been quantified independently. We limit the empirical claim to the current self-verification pipeline and do not claim that reward noise is small or balanced across methods.

Thanks for the clarifications. It seems the authors are largely acknowledging my concerns raised in the review by narrowing the scope of their claims, instead of using empirical evidence to address my concerns. So I'm maintaining my original assessment score.

Reviewer_kTupOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

PAPER SUMMARY: Setting. Data-free evolution for code generation: a proposer generates coding tasks, a solver is trained on them by RLVR, with hidden as the verifier. Refinement is a parameter-free rule that selects/filters the generated pool before solver training. The paper asks a narrow, sharply scoped question: in a decoupled, single-round version of this loop (proposer reward depends only on validity/topic-adherence/reference self-consistency and never on the solver; proposer trained to convergence and then frozen; one round), does refinement actually help the solver?

CONTRIBUTIONS:

  1. Framing refinement as a first-order alignment problem on the solver gradient, yielding a crisp necessary-and-sufficient condition for when a selection operator can help.
  2. A support-limitation argument that redirects attention from "which selector" to "which proposer" — refinement cannot escape the base proposer's effective support.
  3. An empirical isolation study: refinement vs non-refinement with everything else held fixed (same proposer, same solver budget), across some LLMs in the appendix.
  4. A null/negative finding, deliberately stated as "not distinguishable at this sample size" rather than "no effect", plus a proposer-swap experiment suggesting proposer-side diversity is the more promising lever — and an honest caveat that diversity alone does not predict solver quality.

VALIDATION: Design. LoRA r=32, Dr.GRPO, 500 proposer steps / 500 solver steps, lr 1e-5, batch 8, group size 8, 8k context, 40 Codeforces topic tags, 500 valid tasks per seed, refinement = top 50%, 5 benchmarks , pass@1 via unbiased estimator with 8 samples/problem, 3 fully independent seeds end-to-end.

Results. (a) Generated data is competitive with human-curated data. (b) Refinement gains are small and sign-flipping: 4B 54.6→55.6/55.1 and 14B 58.3→58.6/59.1 (better), 1.7B 43.1 vs 42.9/42.6 and 8B 57.2 vs 56.4/56.8 (worse). (c) Welch t on 3-seed summaries: |t| ≤ 1.84, p ≥ 0.15 for all ten operator-vs-non-refinement comparisons, CIs spanning zero (half-widths ~1 to ~12 points). (d) Proposer swaps: 8B solver 57.2→55.9 with a 1.7B proposer (p=0.21); 1.7B solver 43.1→43.5 with an 8B proposer (p=0.47). (e) Diversity metrics move in the predicted direction (mean NN 0.628→0.530, [email protected] 93.5→98.5) but are explicitly not claimed causal.

Where validation is thin . No random-50% control, so "selection quality" is confounded with halved pool size and doubled repetition within a fixed 500-step budget; the training volume is roughly one epoch ,which is close to the least favorable regime for any selection effect; ∇M is not well-defined for a discrete pass@1 functional, so A ≡ 0 almost everywhere under the paper's own definition; n=3 with summary-statistic t-tests ignores that benchmark items are shared across seeds; and Appendix B promises per-operator implementation and refined-set construction but delivers only a hyperparameter table and prompts. Also, the one place refinement visibly rescues a collapsing arm is explicitly discounted by the authors.

Critical strengths

STRENGTH 1 — A negative result delivered with calibrated, self-auditing inference

The paper's core deliverable is a claim of absence, and claims of absence are where papers in this area usually fail: an underpowered null gets sold as "X doesn't work". These authors do the opposite. They report interval estimates for every operator-vs-baseline comparison, note that the sign of the gap flips across models, and explicitly write the conclusion as "refinement is not distinguishable from non-refinement at this sample size, not refinement has no effect" — then carry that hedge consistently into the abstract and conclusion rather than letting the abstract drift past the evidence. They also surface their own failure modes and flag the mechanism that could invalidate their own headline .

Why this is consequential: for a "when does refinement help?" paper, the inference quality is the contribution — there is no method to fall back on. Because the authors have already conceded precisely the objections a reviewer would otherwise raise , the remaining defects read as design limitations with a clear fix rather than as misrepresentation. It moves the paper from "an unsupported null that should be rejected" to "a scoped, honest null whose fix is well-specified". This single property is what keeps the paper in the discussion at all, given a theory section I regard as close to vacuous.

STRENGTH 2 — Genuine causal isolation, plus a transfer test that probes the ceiling claim from the other direction

Two design decisions deserve credit. First, the decoupled single round: the proposer's reward never touches solver performance and the proposer is trained to convergence and frozen before the solver is touched. That is a real identification move — it removes the co-adapting proposer that makes refinement effects uninterpretable in Absolute Zero / R-Zero-style systems, and the authors name the scope cost themselves rather than hiding it. Second, the proposer-swap experiment: hold the solver fixed, retrain it on tasks from a weaker and a stronger proposer, and add diversity metrics . This tests the support-ceiling argument empirically from the input side rather than only asserting it, and it produces the paper's most interesting finding: the Llama-3.1-8B proposer is as diverse as Qwen3-8B's yet yields a worse solver at both scales, which the authors use to refuse a causal diversity story they could easily have sold.

Why this is consequential: it gives the paper a substantive, transferable claim — the binding constraint is the proposer's output, not the selector over it — that survives even if every individual operator comparison is noise, and it comes with the pattern of a paper that tests its own thesis adversarially Combined with strength 1, the acceptance case is: a well-scoped, honestly reported protocol plus one directional finding that redirects the field's attention, at the cost of a theory section that should be rewritten or demoted to a discussion. That is a weak accept on contribution clarity, conditional on the authors fixing the ill-defined ∇M and adding the matched-pool control — not an accept on the strength of the analysis as submitted.

Critical weaknesses

WEAKNESS 1 — The analytical core is ill-defined in the paper's own setting, and nothing in the experiments tests it

Section 4 is nominally the paper's contribution: "refinement affects the solver only through the alignment A(D′)". But A is defined as ⟨∇_θ M, g⟩, and M is expected pass@1 on a held-out benchmark under discrete correctness — a piecewise-constant functional of the parameters. Its gradient is zero almost everywhere, so A ≡ 0 by construction, Prop. 1's first-order identity is vacuous, and Cor. 1's "iff" condition is a statement about 0 > 0. No smoothed surrogate is introduced anywhere. Prop. 1/Cor. 1 are also algebraic restatements of the definition of A rather than predictions, Prop. 2 is Wu et al. 2025 re-lettered, and the quantity the theory says everything flows through is never measured — Section 5.3 concedes this and defers it. Remark 2, the only result offered in justification of learnability, rests on an independence assumption the authors themselves say should be tested in future work.

Given a null empirical result, the analysis is the only thing that could carry the paper. A vacuous analysis means the contribution collapses to "we ran an experiment and saw nothing", which is not an ICLR paper. This is the decisive weakness: it is not a gap to be filled in revision, it is the paper's stated main contribution failing to hold at the definitional level, and no amount of additional seeds repairs it — the theory must be rebuilt on a differentiable performance surrogate or demoted from "analysis" to "discussion", which changes what the paper claims to be.

WEAKNESS 2 — The experimental design cannot identify the effect it reports a null for

The comparison is not controlled. Refined arms train on 250 tasks for the same 500 steps that the baseline uses for 500 tasks, so selection quality is perfectly confounded with pool size and with repetition count. The one arm that would separate them — a random 50% subset at matched pool size — is absent, and it is the control that determines whether the entire paper means what its title says. Compounding this, 500 steps × batch 8 ≈ 4000 samples over a 250–500 task pool is roughly one epoch, so the design is near the least favorable point for any curriculum-style selection effect; the authors acknowledge this and then still place the null in the abstract and conclusion. Statistics do not rescue it: three seeds, Welch tests reconstructed from means and stds rounded to one decimal, treating seeds as independent although benchmark items are shared across them, with no power analysis or equivalence test to make the null interpretable. And the authors' own hypothesis — that LoRA rank 32 may leave the optimizer insensitive to task selection generally — if true invalidates the flat Qwen3 results as evidence about refinement at all.

This matters more than a list of smaller omissions because the null is the paper. Everything else (proposer swap, diversity table, honest hedging) is supporting material for a headline that the design cannot attribute: what has been demonstrated is closer to "under a thin update budget, a 250-task subset behaves like a 500-task set", which is a much weaker and less interesting claim than "existing refinement operators do not outperform the non-refinement baseline". Combined with weakness 1, the paper has neither a working theory nor an identifying experiment, and its own scope section says as much while the abstract does not.

Questions for authors

CLARIFICATION REQUESTS

  1. How is ∇_θ M defined in your setting? M is pass@1 on a finite benchmark with discrete correctness, i.e. piecewise constant in θ, which makes ∇M = 0 a.e. and A ≡ 0 under Definition 2. Do you have an explicit smooth surrogate (expected likelihood of correct completions, sampling-correctness probability, or similar), and if so, is Prop. 1 stated with respect to it? If A is identically zero in your experiments, what claim survives from Section 4?

  2. Was a random-50% subset arm ever run at matched pool size, on any model, at any point (including informally or in a pilot)? If yes: the numbers, per benchmark. If no: do you agree that selection quality and pool size are currently confounded, and can you add it in revision?

  3. Exactly how was solver compute allocated per arm — equal steps (500 for all arms), equal tokens, or equal epochs? How many distinct tasks did each arm train on, and how many times was each task repeated on average?

SUGGESTIONS FOR FUTURE WORK

  1. Rerun in the closed-loop, multi-round regime the introduction actually motivates (proposer co-adapting, several rounds, larger solver budget), where refinement has room to compound. Your Section 5.3 already identifies this as the regime where the null may not transfer, and I agree — which is also why the current scoping keeps the paper's claim narrow.

  2. Capability-matched proposers: your diversity result is confounded with capacity, and the Llama control shows diversity alone does not predict solver quality. A proposer sweep at fixed capability with diversity manipulated independently is the experiment that could establish the proposer-ceiling claim you gesture at.

1 comment

We sincerely thank you for your careful review and constructive comments.

Q1 / W1: Performance gradient.

How is ∇_θ M defined in your setting? M is pass@1 on a finite benchmark with discrete correctness, i.e. piecewise constant in θ, which makes ∇M = 0 a.e. and A ≡ 0 under Definition 2. Do you have an explicit smooth surrogate (expected likelihood of correct completions, sampling-correctness probability, or similar), and if so, is Prop. 1 stated with respect to it? If A is identically zero in your experiments, what claim survives from Section 4?

Response. Section 3 defines MM as the solver's expected pass rate. For mm benchmark tasks, decoding distribution qθq_\theta, and binary correctness indicator cic_i,

M(\theta)=\frac1m\sum_i\mathbb E_{q_\theta}[c_i(y)], \nabla M(\theta)=\frac1m\sum_i\mathbb E_{q_\theta}[c_i(y)\nabla\log q_\theta(y\mid x_i)]. \end{aligned}$$ Expectations are over $y\sim q_\theta(\cdot\mid x_i)$. The derivative acts on output probabilities, so binary correctness does not imply a zero gradient. The eight-sample evaluation estimates $M$, not its gradient. With bounded generation length, the expansion requires locally fixed top-$p$ support and a locally Lipschitz gradient; support changes carry no such guarantee. **Alignment.** Proposition 1 is a local first-order characterization. For binary-reward score-function updates with $0<p_\tau<1$, $$\alpha(\tau)=p_\tau(1-p_\tau)\langle\nabla M,\Delta_\tau\rangle,$$ Here $\Delta_\tau$ is the difference in conditional mean log-probability gradients between successful and unsuccessful outputs. Learnability captures the first factor, but not the sign or magnitude of the second. Conditional optimality requires equal positive mean residual alignment across tasks given all pool scores, and only concerns fixed-size selectors using those scores. Neither alignment nor this assumption was tested. The factorization does not automatically extend to Dr. GRPO with LoRA. We will restrict Corollary 1 to first order, retain the finite-step remainder, and correct effective-support containment. ## Q2 / W2: Random-50% control. > Was a random-50% subset arm ever run at matched pool size, on any model, at any point (including informally or in a pilot)? If yes: the numbers, per benchmark. If no: do you agree that selection quality and pool size are currently confounded, and can you add it in revision? **Response.** No, random-50% result, including a pilot result, is available in this response. Our experiments measure filtering relative to the full pool at the same nominal update budget. Ranking quality remains confounded with coverage and reuse. ## Q3 / W2: Training budget and reuse. > Exactly how was solver compute allocated per arm — equal steps (500 for all arms), equal tokens, or equal epochs? How many distinct tasks did each arm train on, and how many times was each task repeated on average? **Response.** Table 5 reports 500 solver steps, batch size 8, and group size 8. This matches nominal steps, not necessarily tokens or epochs. If each step presents eight tasks, a 500-task pool gives 8 presentations per task on average, versus 16 after 50% selection, rather than one epoch. Actual exposure requires batch, group, accumulation, and sampling logs. **W2: Uncertainty and instability.** The ten main-table aggregate comparisons use three seeds and rounded summaries. Their intervals include zero; this does not establish equivalence. A shared fixed benchmark does not itself make independent runs dependent. Exact run pairing and unrounded outputs are needed for seed-level inference; item-level uncertainty is separate. DeepSeek reaches $47.0\pm1.1$ with learnability versus $44.4\pm5.1$ without refinement. A possible benefit remains unresolved even though neither exceeds the base. The seed spread covers the full pipeline. LoRA insensitivity remains a hypothesis. **Implementation and scope.** We will clarify the scoring checkpoint, score computation, invalid-score handling, subset construction, and budget accounting. Multi-round training, larger budgets, and capability-matched diversity interventions are outside the scope of this work, and we will investigate in future work. Similar diversity but different solver outcomes for Llama and Qwen3 proposers caution against a diversity-only explanation.

Comments on the paper

1 comment

Dear Reviewers,

A friendly reminder to check the authors’ responses and the other reviewers’ comments. It would be helpful to hear your thoughts on the differing assessments and whether they affect your evaluation.

Thank you for your time and effort!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.