acceptodds
Under review as a conference paper at ICLR 2027

Steerable Instruction-Following Code Synthesis via Actor-Schema Co-Evolution

Abstract

Interpreting and following human instructions is a critical capability of large language models (LLMs) in automatic programming. However, synthesizing large-scale coding data paired with fine-grained, verifiable instructions remains largely unexplored, particularly when ensuring logical compatibility among multiple constraints. This setting reflects real-world user specifications, e.g., coding standards, dependency restrictions, and interface contracts. In this study, we propose IFCodeEvolve, an actor-schema co-evolution framework for instruction-following coding data generation. By representing each instruction as a parametric function schema, we construct a library that covers the vast constraint space via dynamic instantiation. Building upon this, a Monte Carlo Tree Search (MCTS) sampler is applied to efficiently navigate this space, utilizing actor model feedback as a dynamic termination signal. Furthermore, to progressively explore challenging problems, we introduce a co-evolving paradigm that iteratively advances both the actor model and the schema library based on sampler statistics. Empirical results demonstrate that IFCodeEvolve significantly boosts base model performance, with our 32B model achieving parity with proprietary SOTA models. Besides, we contribute IFCodeBench, a comprehensive human-verified benchmark equipped with solutions and robust AST-based verification.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Instruction following, synthetic dataset, coding agent, self-evolving agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_m1yQOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IFCodeEvolve, a framework for synthesizing instruction-following (IF) coding data in which each instruction is a parametric schema paired with an AST-based verifier, and constraints are added to seed problems one at a time by an MCTS sampler, with a generator LLM producing a witness solution to guarantee solvability (proof-by-construction). An actor model decides which problems are hard enough to keep, and the actor and schema library co-evolve across rounds through GRPO post-training, schema composition and schema mutation.

Critical strengths

Existing coding-data synthesis methods mostly ask an LLM to brainstorm problems or rewrite queries, which gives no guarantee that multiple constraints are mutually compatible or that the accompanying checks are correct. IFCodeEvolve makes each constraint an explicit parametric schema with a deterministic AST verifier and accepts a new constraint only when an adapted solution passes both the unit tests and all accumulated verifiers. This turns constraint synthesis into a controllable search over a structured space, which is what makes MCTS sampling, schema composition and schema mutation possible in the first place.

Critical weaknesses

The main comparison does not isolate the contribution of the proposed synthesis mechanism. OSS-Instruct and Code Evol-Instruct generate general coding queries without explicit verifiable constraints, so on IF benchmarks the large gap mostly reflects having constraint-annotated data at all rather than the advantage of MCTS sampling and co-evolution. The component ablations support this concern: replacing MCTS with uniform random schema sampling lowers IFEvalCode Inst. only from 0.8701 to 0.8542 and CodeIF Inst. from 0.8435 to 0.8388 for Qwen2.5-Coder-7B, and removing schema evolution or actor evolution changes most metrics by about 1–3 points (Table 10), all from single runs. A comparison against an IF-specific synthesis baseline (e.g., AutoIF-style instruction and verifier generation, Dong et al., 2024, which the paper cites but does not compare against) under the same seed data, generator, RL recipe and budget is necessary, together with variance across seeds for the ablations. The paper should also state how the baselines were trained, in particular which reward was used for data that have no IF verifiers.

The claim that the 32B model "achieves parity with proprietary SOTA models" is not supported by the reported numbers. After training, Qwen2.5-Coder-32B reaches 0.7095 Prompt on IFEvalCode and 0.4147 on CodeIF (Table 3), compared with 0.8571 and 0.6667 for GPT-5.2 and 0.7952 and 0.5489 for Seed-2.0-thinking (Table 1). In addition, the paper reports only IF metrics and never checks general coding ability. Training with a reward in which instruction compliance carries as much weight as functional correctness (Eq. 6) may degrade plain functional correctness, so results on a standard benchmark such as HumanEval(+), MBPP(+) or LiveCodeBench before and after training are needed to show that the gains do not come at the cost of general coding performance.

Questions for authors

The routing threshold is set to λ=1\lambda = 1 (Eq. 5), so a problem is finalized as "difficult" as soon as the functional pass rate and the all-instruction pass rate over NN samples are both below 100%, i.e., a single failed sample on each criterion suffices. This makes the difficulty criterion very lenient. What value of NN is used, and how sensitive are the corpus composition and downstream results to λ\lambda?

An MCTS "win" is credited to the newly added schema whenever the indicator in Eq. 5 fires, even though the functional failures it includes may be unrelated to that schema. Combined with mutation that explicitly tightens constraints until the actor's solution fails, this objective rewards constraints that are hard rather than realistic, e.g., requiring every variable name to contain a digit. How do the authors ensure that evolved constraints reflect meaningful user requirements, and is any filter applied to remove unnatural or overly restrictive ones?

Instructions of finalized problems are paraphrased by an LLM after verification. Paraphrasing can change the meaning of a constraint (e.g., "at least" vs. "exactly") while the verifier stays unchanged, which would produce mismatched training signals. Is consistency between the paraphrased text and its verifier checked, and how often does it fail?

IFEvalCode's rule-based checkers are described as unreliable, and the paper states that the judge is validated against IFCodeBench verifiers. Which judge produces the numbers in Tables 1–3? If it is Seed-1.6 or another Seed model, the same family serves as generator, benchmark filter and judge, which may bias both the main results and the strong Seed-model scores in Table 7.

IFCodeBench is curated from the pipeline's own output, 75.5% of its problems come from MBPP, and its constraints follow the same schema library. It is therefore not independent of IFCodeEvolve and mainly reflects easy algorithmic problems. What fraction of its instructions come from schemas that are absent from the training library, and how do the authors expect it to complement IFEvalCode and CodeIF?

Reviewer_aLYsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IFCodeEvolve, a framework for synthesizing instruction-following (IF) coding data with verifiable constraints. Each constraint is a parametric schema paired with an AST-based checker. An MCTS sampler adds constraints to seed problems one at a time. For each new constraint, a generator LLM must rewrite the reference solution so that it passes all checkers and the original unit tests (proof-by-construction). An actor model decides when a problem is hard enough. After each round, the actor is post-trained with GRPO, and the schema library is updated by composing and mutating schemas based on sampling statistics. On IFEvalCode and CodeIF, the method outperforms OSS-Instruct and Code Evol-Instruct. The authors also release IFCodeBench, a benchmark of 530 human-verified problems.

Critical strengths
  1. The paper is clearly written and well structured, and the method is easy to follow.
  2. The paper evolves the synthesis mechanism itself (the schema library) together with the target model, not only the model. I find this an interesting view.
  3. Each added constraint must come with a witness solution. This is a simple way to guarantee solvability, and it avoids the unsolvable-problem issue that is common in adversarial self-play.
  4. The analysis covers ablations, diversity, held-out schemas, contamination and a Java extension, and the compute cost is reported. IFCodeBench may be a useful resource for the community.
Critical weaknesses
  1. Proprietary generator. The main results and IFCodeBench rely on Seed-1.6, a closed model, which limits reproducibility. The generator ablation also shows that performance depends strongly on how capable the generator is. This makes it hard to tell how much of the gain comes from the framework and how much from distilling a stronger model.

  2. Baselines. The baselines (OSS-Instruct, Code Evol-Instruct) are 2023 methods built for general code generation. They are also re-implemented inside the authors’ pipeline rather than run from their released data or official code, so the comparison is less convincing. More relevant methods for verifiable instruction following, such as AutoIF and IF-RLVR, are cited but not compared.

  3. Base models and benchmarks. The trained models are dated, and no recent open models or reasoning models are included. Gains already shrink on the stronger actor. The evaluation uses only two IF-coding benchmarks and no general coding benchmarks to check for regressions. The transfer experiments also lack baseline comparisons.

  4. Data correctness. Beyond the automatic checks, the training data gets no human audit and no error rate is reported. The seed unit tests are weak, and instructions are paraphrased after verification. In some cases the instruction text and its checker do not agree.

  5. The claim of parity with proprietary models is somewhat overstated. The evaluation protocol on the public benchmarks (LLM judge vs. official checkers) should be described more clearly.

Questions for authors

Please refer to weaknesses.

Reviewer_q7YPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes IFCODEEVOLVE for synthesizing instruction-following code data. It represents instructions as parameterized schemas with AST-based verifiers, uses an MCTS-style sampler to add constraints, and adapts reference solutions to verify satisfiability. Across rounds, the actor is trained on generated data, while schemas are composed or mutated. Experiments on IFEvalCode and CodeIF report improvements over OSS-Instruct and Code Evol-Instruct. The paper also introduces the 530-problem IFCODEBENCH.

Critical strengths
  1. Verifiable data synthesis. Each schema has an AST-based verifier, and the adapted solution must pass both unit tests and accumulated instruction checks (Section 3.3). This provides stronger validity control than unconstrained LLM-based data generation.

  2. Broad evaluation within the chosen setting. Tables 1, 3, 6, and 10 provide baseline comparisons, cross-scale transfer, sampler comparisons, and component ablations. Appendix D also includes a Java extension and held-out-schema evaluation.

Critical weaknesses
  1. The code-specific significance is unclear. Most schemas in Table 8 concern local syntactic properties, such as variable naming, loop counts, imports, and required built-ins. The sampling and co-evolution framework appears applicable to general constraint following, with code mainly providing convenient AST-based verifiers. At the same time, the method does not address behavioral or repository-level constraints that distinguish realistic software engineering. The limitations section explicitly leaves such constraints to future work. The results therefore demonstrate improved compliance with narrow function-level constraints, but not clearly improved software-engineering ability.

  2. The framework relies on assumptions that do not extend to realistic software-engineering tasks. First, proof-by-construction requires an existing correct reference solution that can be modified into a new witness. Extending this procedure to repository-level tasks would require seed tasks with validated reference patches and executable tests, whose availability and quality may limit scalability. Second, the framework assumes that constraints can be checked locally using AST-based verifiers, whereas requirements such as backward compatibility, consistency across modules, or reuse of existing abstractions require repository-level reasoning. Third, passing a finite test suite does not establish behavioral correctness because hidden behaviors, regressions, and environment-dependent effects may remain uncovered. Finally, composing repository-level constraints may require coordinated changes to callers, tests, configuration, and documentation, making witness adaptation and validation substantially more complex. These requirements limit the demonstrated scope to function- and class-level synthesis; the paper does not establish whether the framework remains effective and scalable in settings such as SWE-bench.

  3. Actor failure is a limited measure of difficulty. Section 3.3 evaluates the actor through independently sampled, non-interactive solutions and uses their pass rates as the routing and MCTS signal. However, software-engineering tasks require iterative exploration, codebase navigation, test execution, and debugging. A one-shot failure cannot distinguish insufficient model capability from missing context or an unsuccessful exploration trajectory. Even within the paper’s setting, Equation 5 uses λ\lambda=1, so a problem is labeled difficult if at least one sample fails functional tests and at least one—potentially a different sample—fails full instruction compliance. This can occur even when the remaining N-1 samples satisfy both requirements. The paper does not report sensitivity to this definition.

Questions for authors
  1. What part of IFCODEEVOLVE is specific to code instruction following beyond the availability of AST-based and execution-based verifiers? Would the same framework apply largely unchanged to format-constrained natural-language generation?

  2. Does the generated data improve performance on behavioral or repository-level coding tasks? What changes to witness adaptation and verification would be needed to extend the framework to SWE-style tasks?

  3. What value of N and which sampling parameters are used in Equation 5? How do the results change with different (λ\lambda) values or a stricter criterion based on the fraction of samples jointly passing both functional and instruction checks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.