- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes IFCodeEvolve, a framework for synthesizing instruction-following (IF) coding data in which each instruction is a parametric schema paired with an AST-based verifier, and constraints are added to seed problems one at a time by an MCTS sampler, with a generator LLM producing a witness solution to guarantee solvability (proof-by-construction). An actor model decides which problems are hard enough to keep, and the actor and schema library co-evolve across rounds through GRPO post-training, schema composition and schema mutation.
Critical strengths
Existing coding-data synthesis methods mostly ask an LLM to brainstorm problems or rewrite queries, which gives no guarantee that multiple constraints are mutually compatible or that the accompanying checks are correct. IFCodeEvolve makes each constraint an explicit parametric schema with a deterministic AST verifier and accepts a new constraint only when an adapted solution passes both the unit tests and all accumulated verifiers. This turns constraint synthesis into a controllable search over a structured space, which is what makes MCTS sampling, schema composition and schema mutation possible in the first place.
Critical weaknesses
The main comparison does not isolate the contribution of the proposed synthesis mechanism. OSS-Instruct and Code Evol-Instruct generate general coding queries without explicit verifiable constraints, so on IF benchmarks the large gap mostly reflects having constraint-annotated data at all rather than the advantage of MCTS sampling and co-evolution. The component ablations support this concern: replacing MCTS with uniform random schema sampling lowers IFEvalCode Inst. only from 0.8701 to 0.8542 and CodeIF Inst. from 0.8435 to 0.8388 for Qwen2.5-Coder-7B, and removing schema evolution or actor evolution changes most metrics by about 1–3 points (Table 10), all from single runs. A comparison against an IF-specific synthesis baseline (e.g., AutoIF-style instruction and verifier generation, Dong et al., 2024, which the paper cites but does not compare against) under the same seed data, generator, RL recipe and budget is necessary, together with variance across seeds for the ablations. The paper should also state how the baselines were trained, in particular which reward was used for data that have no IF verifiers.
The claim that the 32B model "achieves parity with proprietary SOTA models" is not supported by the reported numbers. After training, Qwen2.5-Coder-32B reaches 0.7095 Prompt on IFEvalCode and 0.4147 on CodeIF (Table 3), compared with 0.8571 and 0.6667 for GPT-5.2 and 0.7952 and 0.5489 for Seed-2.0-thinking (Table 1). In addition, the paper reports only IF metrics and never checks general coding ability. Training with a reward in which instruction compliance carries as much weight as functional correctness (Eq. 6) may degrade plain functional correctness, so results on a standard benchmark such as HumanEval(+), MBPP(+) or LiveCodeBench before and after training are needed to show that the gains do not come at the cost of general coding performance.
Questions for authors
The routing threshold is set to (Eq. 5), so a problem is finalized as "difficult" as soon as the functional pass rate and the all-instruction pass rate over samples are both below 100%, i.e., a single failed sample on each criterion suffices. This makes the difficulty criterion very lenient. What value of is used, and how sensitive are the corpus composition and downstream results to ?
An MCTS "win" is credited to the newly added schema whenever the indicator in Eq. 5 fires, even though the functional failures it includes may be unrelated to that schema. Combined with mutation that explicitly tightens constraints until the actor's solution fails, this objective rewards constraints that are hard rather than realistic, e.g., requiring every variable name to contain a digit. How do the authors ensure that evolved constraints reflect meaningful user requirements, and is any filter applied to remove unnatural or overly restrictive ones?
Instructions of finalized problems are paraphrased by an LLM after verification. Paraphrasing can change the meaning of a constraint (e.g., "at least" vs. "exactly") while the verifier stays unchanged, which would produce mismatched training signals. Is consistency between the paraphrased text and its verifier checked, and how often does it fail?
IFEvalCode's rule-based checkers are described as unreliable, and the paper states that the judge is validated against IFCodeBench verifiers. Which judge produces the numbers in Tables 1–3? If it is Seed-1.6 or another Seed model, the same family serves as generator, benchmark filter and judge, which may bias both the main results and the strong Seed-model scores in Table 7.
IFCodeBench is curated from the pipeline's own output, 75.5% of its problems come from MBPP, and its constraints follow the same schema library. It is therefore not independent of IFCodeEvolve and mainly reflects easy algorithmic problems. What fraction of its instructions come from schemas that are absent from the training library, and how do the authors expect it to complement IFEvalCode and CodeIF?