- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a training recipe for teaching small language models to write code in low-resource programming languages, where “low-resource” means that no gold-standard programs in the target language are available for training. The recipe relies on the model’s ability to transfer coding skills learned on high-resource languages to an unfamiliar language once it is shown that language’s syntactic rules. It replaces the usual SFT + RL pipeline with two self-distillation stages:
- Self-distillation SFT (SD-SFT). A teacher, which is the same model conditioned on a syntax sheet for the target language, generates programs. Programs that pass verification are used to fine-tune a student that does not see the sheet. This moves the language knowledge from context into the weights.
- SDPO. In the RL stage, the teacher’s context contains the syntax sheet, environment feedback, and a correct program for the problem. The teacher scores the student’s rollouts and provides dense, token-level rewards.
SD-SFT alone raises Gemma 4’s pass@1 on Ballerina by 55.4 points, surpassing frontier models; the full pipeline reaches gains of up to 62.7 points. The recipe is evaluated on Gemma 4 and Qwen 3 across Ballerina, OCaml, and Fortran.
Critical strengths
-
Realistic and practically relevant problem setting. Assuming poor off-the-shelf performance and no gold programs in the target language reflects the real-world situation for many niche or proprietary, or newly released languages. The setting extends LLM code generation to languages that standard pipelines cannot reach.
-
Large empirical gains. The recipe improves pass@1 by up to 62.7 points on Ballerina and outperforms frontier models with a small model. This is a notable demonstration that knowledge supplied in context can be distilled into weights.
-
Gains also hold where the base model's performance is non-trivial. On OCaml, where the base model already has non-zero zero-shot performance, the training framework still yields clear improvements. So it is also useful for languages the model knows something about.
-
Selective distillation is well motivated. The paper distills only what is needed: the problem-solving constructs of the target language, mined from a program in corresponding reference language. The choice is justified both conceptually and by ablation, with its limitations discussed.
Critical weaknesses
W1. No comparison against an oracle with gold programs. It is unclear whether this recipe should replace collecting ground-truth programs and running standard SFT + RL, or whether it is mainly a fallback when such data cannot be obtained. The authors should report Gemma 4 / Qwen 3 performance when gold programs in the target language are available and standard SFT followed by SDPO is used. This gives an upper bound and shows how much of the gap self-distillation closes. A data-efficiency curve would be even more useful: how many gold programs are needed to match the self-distilled model? That would tell practitioners when collecting data is worth it.
W2. Effect on other languages (catastrophic forgetting). The paper does not report how performance on high-resource languages such as Python or JavaScript changes after the model is optimized for a target language like Ballerina. For a recipe meant to add a language, preserving existing ability is an essential property. It would be interesting to train on several target languages at once, to see whether they interfere with each other.
W3. Missing ablation on the teacher’s SDPO context. It is unclear why the teacher needs a passing program in its context during SDPO. Can the teacher score student rollouts adequately with only the syntax sheet and environment feedback? Please report results with , i.e., without a correct solution. This matters for two reasons. First, it shows how much of the dense reward comes from comparing against a known solution rather than from the teacher’s own understanding of the language. Second, as W4 explains, it affects training cost.
W4. Cost and coverage of teacher resampling. This follows from W3. When the student fails to produce a correct solution, the teacher must be sampled again to obtain one, which adds training cost. Please report: (a) the fraction of training problems that required teacher resampling; (b) the fraction of problems discarded because the teacher also failed to find a correct solution. Can you also describe how the discarded set is distributed by difficulty. If hard problems are systematically filtered out, the learned policy may be biased toward easy constructs.
W5. Missing ablation on the discard threshold . There is no experiment on how the threshold for discarding problems affects final performance. This ablation would answer whether hard problems, where the teacher found few or no correct solutions, still provide useful learning signal during RL. One would expect them to: the dense reward depends on the teacher’s ability to localize errors in incorrect student programs, not on its ability to generate correct programs itself, and verification is often easier than generation. This intuition goes back to the complexity-theoretic P vs. NP distinction (Cook, 1971).
W6. How “unseen” are the target languages? Can you further quantify base-model familiarity using for e.g. perplexity on target-language code?
Questions for authors
See Weaknesses section.
6 comments
(2/2) Authors response to reviewer Vx5n
W5. SDPO selection threshold. This is a design principle, not a tuned hyperparameter: we train on a problem only if at least one candidate program passes its tests. We did not ablate this condition because our early experiments showed that training on data not grounded in real-world signals often leads to performance collapse at unexpected points. For example, in initial experiments we also rewarded compilation. When no rollout passed the tests, we used a program that merely compiled as the training target, which made the target suboptimal. The model then learned to aim for compilation alone, producing trivial "hello world" programs that compiled but solved nothing.
Although verification is easier than generation, our method relies on the compiler and tests to verify programs and ground training in a concrete signal. Without a passing program, the teacher has no verified solution on which to ground its feedback.
W6. Perplexity across foreign languages. We selected ~200 human-written programs from each language's documentation and measured perplexity by replaying each program through the model. The results are below.
| Model | Language | Bits per byte | Bytes per token | Perplexity |
|---|---|---|---|---|
| Gemma 4 E4B | Ballerina | 0.913 | 3.66 | 10.1 |
| Gemma 4 E4B | Python | 0.921 | 3.00 | 6.8 |
| Qwen3-8B | Ballerina | 0.575 | 4.11 | 5.2 |
| Qwen3-8B | Python | 0.489 | 3.36 | 3.1 |
We hope we have answered all of your concerns and provided sufficient clarity. If you need further information or clarification, please let us know. Thank you again for the insights.
I thank the authors for addressing my concerns.
-
Regarding W1. I understand that training only on gold programs without reasoning traces could break the model's own ability to reason. What I wanted to understand is whether, if an LLM encounters an unknown or low-resource language on which it performs poorly, the effort should be directed towards collecting ground-truth data for it or using an approach like SD-SFT?
-
Regarding W2. Resolved
-
Regarding W3. Resolved
-
Regarding W4. It seems nearly 20% of problems are being dropped. Please add the results of this to the main body of the paper, either as a limitation or as future work that you can address to further improve performance.
(1/2) Authors response to reviewer Vx5n
We thank the reviewer for the positive feedback and the questions. We address all concerns raised below.
W1. A gold-program oracle. No gold corpus exists for these problems, but that is not the main obstacle. Even with a gold program for every problem, SFT on gold programs is not an option for these models, because it either destroys the reasoning or breaks the policy. Both base models reason before they answer, so a training example is reasoning followed by a program, and a gold program has no reasoning. That leaves two choices, and both fail. This is not a question of data efficiency.
Self-distillation avoids both: every example is the model's own reasoning and its own program, kept only if the program passes, and the new knowledge enters through the sheet.
RL, the other standard route, transfers no new knowledge: a reward can only reinforce programs the model already writes, and the base model writes no passing Ballerina program (0.0).
The remaining option is GRPO after stage one. We report the results below (Table 3), which show that SDPO works better in that setting.
We will include this explanation in the discussion.
W2. Forgetting. We ran a Python control on HumanEval (164 problems, three samples per problem) using models fine-tuned on Ballerina. The results are below; the difference is within sampling noise.
W3. Why does the teacher see a passing program? The "passing program" is not a gold or reference solution. It is a program the model wrote in the same round (a sibling rollout, or one of the teacher's own gated samples) that compiled and passed the hidden input/output tests, which we call agnostic tests. Correctness is decided only by execution (§2.1).
To separate the two sources of signal, we re-scored the same failed rollouts with and without the passing program in the teacher's context:
With the passing program, the learning signal is stronger (divergence is higher), and the share of successful completions rises from 36% to 100%.
W4. Cost and coverage. The statistics below come from the SDPO runs and show how the method handles problems where the student cannot pass a single rollout.
For 75% of problems, the teacher adds no cost.
Only syntactic load differs significantly: 27% of problems needing more than six constructs are dropped, against 15% of those needing four or fewer. Length and rating do not differ. No construct is lost: every construct a dropped problem needs also appears in kept problems.