acceptodds
Under review as a conference paper at ICLR 2027

MoldCoder: Automated Construction of Code Instruction Data Using Real Test Execution as Functional Supervision

Abstract

High-quality code instruction-tuning data are an important foundation for improving the code generation capabilities of pretrained language models. Beyond data scale, task complexity, and data diversity, code instruction-tuning data also require that the code correctly implement the functional requirements specified by its corresponding instruction. Many execution-based data construction methods use model-generated tests to verify synthetic code, with execution outcomes guiding sample filtering or code repair. However, the reliability of this functional supervision remains constrained by the quality of the generated tests. A central question is therefore how to obtain more reliable functional supervision and leverage it more effectively. To this end, we introduce Mold-Instruct, an automated code instruction data construction method that uses real-test execution as functional supervision: it first recovers repository-dependent real tests into standalone executable test–code pairs, then uses functional requirements from the tests to guide instruction construction and execution outcomes for verification and heuristic capability diagnosis, thereby constructing target-specific training data. By adjusting its extensible configurations, Mold-Instruct constructs 46.7K high-quality test–code pairs from open-source code repositories, which are then used to construct training data for two base models and fine-tune the corresponding MoldCoder models. MoldCoder improves all nine evaluated metrics on Qwen3-4B-Base and all seven on SmolLM3-3B-Base, increasing their respective average scores from 45.7 to 51.6 and from 33.1 to 36.9. Further controlled experiments show that real tests outperform target-model-generated tests in our setting, functional constraints and execution verification each provide gains, and capability diagnosis guides supervision selection that further improves performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Code Instruction Tuning, Code Instruction Data Construction, Code Generation, Real Test Execution, Functional Supervision, Heuristic Capability Diagnosis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QfWtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Mold-Instruct, a pipeline that builds code instruction data using real tests from open-source repositories as functional supervision. It extracts those tests into standalone executable test–code pairs, uses them to constrain instruction synthesis and to verify the generated code, and adds a cross-model diagnosis step that picks extra supervision based on the target model's weaknesses. Fine-tuning two small base models on this data gives consistent gains, and the results are competitive with existing datasets.

Critical strengths

Using real repository tests as the source of functional supervision is a well-motivated and practical idea. Most prior execution-based pipelines depend on model-generated tests, whose reliability is the bottleneck. The engineering needed to turn repository-bound tests into self-contained executable pairs is substantial and non-trivial, and the released pipeline could be useful to the community on its own.

Critical weaknesses

The real-versus-generated test comparison uses tests generated by the small target model, which is arguably a weak baseline. A more convincing comparison would use tests generated by a stronger model. The scope is also narrow: Python only, 3–4B models. That makes it hard to judge how general the conclusions are.

Questions for authors

How do real tests compare with tests generated by a stronger model, for example the same stronger model used for diagnosis, under the same matched setting? This seems necessary to support the claim that real tests give more reliable supervision than generated tests in general.

Do the findings hold for larger models or other languages? Even a limited experiment on a larger backbone would help.

Reviewer_riMNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Mold-Instruct, a pipeline for constructing code instruction-tuning data using developer-written repository tests. It recovers standalone test–code pairs through code localization, dependency and resource recovery, execution, and refinement. The recovered tests constrain instruction synthesis and verify regenerated code. For cases where a target model fails but a stronger model succeeds, cross-model regeneration provides a heuristic diagnosis that guides the selection of additional supervision. The pipeline produces 46.7K executable test–code pairs and target-specific training datasets for Qwen3-4B-Base and SmolLM3-3B-Base. The resulting MoldCoder models improve their respective average evaluation scores from 45.7 to 51.6 and from 33.1 to 36.9. Controlled experiments examine execution filtering, test-informed instructions, test sources, and diagnosis-based data selection.

Critical strengths
  1. Recovering executable supervision from repository tests is a practical contribution. The pipeline handles cross-file code, imports, and file resources, and filters tests that do not meaningfully depend on the implementation. The resulting 46.7K test–code pairs provide a useful foundation for subsequent data construction.

  2. The analysis of test quality is well targeted. The authors compare test sources on matched code examples and execute the same mutants against both sources. The higher mutation score of real tests provides evidence beyond downstream model accuracy. A size-matched random-selection control also supports the benefit of execution filtering.

  3. The evaluation includes eight public datasets under a unified training setup on two backbones, together with several useful controls. Experiments on direct stronger-model augmentation, cross-file supervision, and code source and style help distinguish the proposed pipeline from simply adding teacher outputs or reformatting repository code.

Critical weaknesses
  1. Limited methodological novelty. Prior work such as UnitCoder and CodeI/O has already incorporated test-based code validation and filtering, while methods such as AutoCodeGen and SelfCodeAlign have explored synthesizing instructions from seed code. The novelty of this work therefore appears to lie primarily in combining and organizing existing techniques, rather than introducing new insights. Furthermore, although the data are constructed from repositories, the evaluation remains focused on standalone programming tasks such as HumanEval and MBPP. I would expect an evaluation on repository-level tasks, such as SWE-bench, to better establish the need for recovering tests from repository contexts.

  2. Limited empirical gains. The proposed data do not consistently outperform existing methods and datasets across the evaluated benchmarks. Given that many of these datasets were introduced nearly two years ago, the results weaken the case for the effectiveness and necessity of the proposed pipeline. The use of small models for data synthesis does not fully address this concern. At a minimum, an experiment using a stronger model as the primary data generator would help establish whether the pipeline delivers more substantial benefits. Related to the first concern, I also find the evaluation benchmarks insufficiently challenging to adequately characterize current LLM coding capabilities.

Questions for authors

Please see weaknesses.

Reviewer_wnh2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a pipeline for creating code instruction-tuning data using repository tests as functional supervision called Mold-Instruct. The method by first mining Python code with provided tests and recovering components needed for execution such as relative imports, cross-file inlining and execution checks. It then follows with refining code-test pairs with steps like boundary repair, filtering and deduplication to construct 46.7K high-quality test-code pairs from open-source repositories. The data is validated by fine-tuning Qwen3-4B-Base and SmolLM3-3B-Base models and improving their scores from 45.7 to 51.6 and from 33.1 to 36.9 respectively. Additional controlled experiments are performed to show that real tests outperform target-model-generated tests among other ablations.

Critical strengths
  1. The pipeline is clearly demonstrated and practically useful. The components of the pipeline like cross-file inlining, relative dependencies and boundary repair are intuitive and clearly described for example as seen in Figure 2 and in the Method section. Additionally, the work serves as an easy to follow and useful recipe for producing instruction-tuning data and compares against a substantial number of common approaches as seen in Table 1.

  2. The ablations in the work are thorough and examine valuable questions like “What Makes Real-Test Functional Supervision Effective?” which are central to the proposed pipeline. Section 5.3 answers this question explicitly by providing comparisons of real and generated tests on a subset of matched examples in addition to a paired mutation study. Other controls are carefully designed to isolate different factors in the pipeline such as the execution-verification study which shows gain is not only due to data-quantity. The appendix also includes ablations such as a code-matched control.

Critical weaknesses
  1. The main results in the paper are competitive but not substantial compared to other approaches. MoldCoder outperforms the best alternative approach by 0.2 points over the average of benchmarks for Qwen3-4B-Base while underperforming the best approach by 1.4 points with the SmolLM3-3B-Base model. Looking at the subsets of the benchmarks in both cases MoldCoder is better than alternative approaches on 1 out of the 9 (for Qwen3-4B-Base) and 7 (for SomlLM3-3B-Base) benchmark subsets.

  2. Although the paper reports averages over multiple seeds it does not include statistical significance or variability analysis. This is especially important as MoldColder’s results vary from the best approaches by in some cases <0.5 points and certain benchmarks can be low in sample count leading to substantial variability. This adds to the difficulty in comparing the approach to other pipelines.

  3. The results do not isolate the advantage of the data construction pipeline and introduce potential confounders. In Table 1 MoldCoder uses 32K/27K examples while other baselines use random 75K subsets of their available datasets and achieve comparable performance. Additionally, baseline datasets use different models for solution generation leading to confounders of teacher model capability and dataset size when trying to compare the different data construction pipelines. Although it’s difficult to fully control all of these factors, a smaller number of size and teacher matched comparisons would provide notable clarity to the success of the approach.

Questions for authors
  1. Can you provide any size or teacher matched comparisons of previous competitive approaches to MoldCoder?
  2. Can you report the standard deviations or confidence intervals for the runs in Tables 1-4?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.