Analyzing Refinement in Data-Free Evolution of Code-Generation LLMs

Data-free evolution methods reduce reliance on human-curated data: a proposer generates tasks, and a solver learns to solve them. The solver's performance is determined by the quality of its generated tasks, which is often improved through a refinement operation that filters or selects important tasks for solver training. In this paper, we investigate when refinement actually helps the solver, from two specific perspectives. First, from the solver-gradient perspective: the refinement affects the solver only through the alignment between the solver gradient and the downstream task performance. Then, we investigate the proposer-support perspective: the refinement can only select tasks that the proposer generates, so the solver performance is limited by the capability of the proposer. Empirically, on code generation benchmarks, existing refinement operators do not significantly outperform the non-refinement baseline. Furthermore, using a stronger proposer can produce more diverse tasks, which tends to improve the solver's performance even without refinement.