- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Mold-Instruct, a pipeline that builds code instruction data using real tests from open-source repositories as functional supervision. It extracts those tests into standalone executable test–code pairs, uses them to constrain instruction synthesis and to verify the generated code, and adds a cross-model diagnosis step that picks extra supervision based on the target model's weaknesses. Fine-tuning two small base models on this data gives consistent gains, and the results are competitive with existing datasets.
Critical strengths
Using real repository tests as the source of functional supervision is a well-motivated and practical idea. Most prior execution-based pipelines depend on model-generated tests, whose reliability is the bottleneck. The engineering needed to turn repository-bound tests into self-contained executable pairs is substantial and non-trivial, and the released pipeline could be useful to the community on its own.
Critical weaknesses
The real-versus-generated test comparison uses tests generated by the small target model, which is arguably a weak baseline. A more convincing comparison would use tests generated by a stronger model. The scope is also narrow: Python only, 3–4B models. That makes it hard to judge how general the conclusions are.
Questions for authors
How do real tests compare with tests generated by a stronger model, for example the same stronger model used for diagnosis, under the same matched setting? This seems necessary to support the claim that real tests give more reliable supervision than generated tests in general.
Do the findings hold for larger models or other languages? Even a limited experiment on a larger backbone would help.