acceptodds
Under review as a conference paper at ICLR 2027

SWE-OPD: Enhancing Agentic Coding via On-Policy Distillation

Abstract

In this paper, we present our practical experience with on-policy distillation for enhancing agentic coding capabilities. Specifically, we leverage a large and capable teacher model and successfully transfer its agentic coding capabilities to a smaller student model through vanilla on-policy distillation. Unlike prior work that primarily focuses on mathematical reasoning and relatively simple agentic tasks such as web search, we extend the application of on-policy distillation to the more challenging domain of agentic coding. We conduct extensive experiments on various datasets and consistently demonstrate its effectiveness on various SWE benchmark. For example, on-policy distillation improves the performance of Qwen3.5-9B on SWE-bench Verified from 57.9% to 62.8%, around 5% enhancement under the supervision of Qwen3.6-27B. Additionally, to alleviate the substantial GPU memory overhead, we develop an offline variant of on-policy distillation that substantially reduces memory consumption with only a slight performance degradation. Finally, we show some intriguing findings during our training and evaluation and make our code available, hoping to further push the development of on-policy distillation on agentic coding with community.

Primary area:
reinforcement learning
Keywords:
software engineering, agentic coding, on-policy distillation, machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FpUBOpenReview ↗
Rating
1(−1)
Confidence
4(+1)
Informativeness
1(−1)
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces SWE-OPD, applying on-policy distillation (OPD) to software engineering tasks. It compares different KL-based objectives and proposes both online and offline training variants. The experiments demonstrate consistent improvements across multiple SWE benchmarks with favorable computational efficiency. In addition, the paper analyzes several phenomena and failure modes observed in OPD, providing useful empirical insights into the behavior of OPD for agentic coding.

Critical strengths

The paper studies both online and offline variants of SWE-OPD, with relatively comprehensive experiments and ablations, including comparisons between the RKL and ST objectives.

Applying OPD to software engineering tasks is meaningful and provides a useful empirical baseline for future research on OPD in long-horizon agentic settings.

Critical weaknesses

Lack of baselines. The baseline comparisons in the paper are not comprehensive enough. It doesn't compare against strong existing methods, such as rejection-sampling SFT, RL, or other recent OPD variants (e.g., agentic OPD methods like EOPD/AdaOPD, or hybrid OPD-RL approaches).

Limited performance improvements. The overall gains are modest. For instance, some results only improve by 0.1 percentage points, which is not very convincing.

Questions for authors

Is OPD truly a necessary or promising training paradigm for agentic coding? Given that we already have access to a strong teacher model and an executable environment with verifiable test outcomes, wouldn't standard SFT on filtered/successful teacher trajectories (e.g., rejection-sampling SFT) be a much simpler and potentially more effective approach? Considering the high engineering complexity of OPD and the modest gains shown in the paper, I am not fully convinced that formulating agentic distillation as an OPD problem is actually necessary. Could the authors justify why OPD should be preferred over a well-curated SFT pipeline in practice?

Reviewer_hTfbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies OPD on student-generated repository interactions, comparing top-K and sampled-token with online and fixed-trajectory training. The authors report improved resolve rates, lower costs with offline supervision, and a GLM repetition collapse. They also propose a support-escape mechanism.

Critical strengths
  1. Writing is easy to follow and understand.
  2. This paper reports negative results and benchmark-dependent accuracy–cost tradeoffs.
Critical weaknesses
  1. The contribution seems to be incremental and limited. Firstly, the pipeline largely applies existing OPD and OPD on SWE-tasks is not new. Seems "Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation" already uses fixed initial-student rollouts and cached teacher scores and this paper only provides some findings in its evaluations.
  2. The collapse mechanism (GLM case) remains unverified. No fixed-prefix measurement or corrective intervention links support escape to GLM’s repetition. The toy example demonstrates worsening alignment, not repetition. ST also uses ratio clipping, adding a confound to the stability comparison.
  3. To me, the SFT baseline is insufficient (in Table 1). It imitates unfiltered teacher trajectories. verified-trajectory SFT and dense distillation on teacher-generated prefixes would better test the value of student-visited histories, separately from supervision format.
  4. Cost-saving findings lack sufficient reasons. Both variants nominally generate one trajectory per task over one epoch, so repeated reuse across epochs does not explain the measured savings. Online stage costs, token counts, and utilization are needed to distinguish computational savings from deployment and scheduling effects.
Questions for authors
  1. What are the teachers’ resolve rates under the same scaffold and evaluation budgets?
Reviewer_FCBVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper applies OPD to transfer teacher capabilities to student models, evaluated on SWE-type tasks. It proposes an offline variant that precomputes teacher scores on fixed trajectories, decoupling rollout collection, teacher scoring, and student optimization so that the three stages can share a single GPU pool, which improves end-to-end training efficiency. In addition, the authors compare different supervision objectives in OPD, such as the top-K reverse-KL loss and the sampled-token loss, and their analysis of the resulting training behaviors provides useful empirical insights into OPD for agentic coding.

Critical strengths

The paper demonstrates capability transfer through OPD across two teacher–student pairs, two training sources, and three SWE benchmarks, while showing that offline supervision can reduce end-to-end GPU-hours by approximately 59–73%, providing valuable evidence for more efficient coding-agent training. Another key strength is its diagnosis of decreasing top-K reverse-KL loss alongside deteriorating agent behavior, supported by gradient analysis identifying support escape as a potential mechanism, which offers useful insights into the reliability of distillation objectives.

Critical weaknesses
  1. The methodological novelty is limited. OPD has already been integrated into training pipelines that consolidate RL-tuned coding/SWE capabilities, e.g., MiMo-V2-Flash [1] and KAT-Coder-V2 [2]. The paper mostly adapts these existing recipes to a cross-scale SWE setting without clear new insight.
  2. The top-K analysis largely overlaps with prior findings [3, 4]. Compared with these existing works, the paper brings no new gain in this analysis, and the SWE setting does not seem to offer additional insight.

References

[1] Xiaomi LLM-Core Team. MiMo-V2-Flash Technical Report. arXiv preprint arXiv:2601.02780, 2026.

[2] Li, F., Zhang, H., Huang, H., et al. KAT-Coder-V2 Technical Report. arXiv preprint arXiv:2603.27703, 2026.

[3] Li, Y., Zuo, Y., He, B., et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv preprint arXiv:2604.13016, 2026.

[4] Zhu, S., Ye, X., Lu, H., Shi, W., and Liu, G. The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes. arXiv preprint arXiv:2605.11182, 2026.

Questions for authors

Could the authors clarify the distinct contributions of this work relative to the studies discussed above, in terms of methodological novelty and new empirical findings?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to AC and Reviewers

Dear Reviewers and AC,

Thank you for your time and efforts in reviewing our paper.

We believe that some aspects of our work may not have been sufficiently clarified in the original submission, potentially leading to misunderstandings. Therefore, we provide further explanations below to better clarify our methodology and contributions, along with additional experimental results.

In this work, we present a simple and controlled study of On-Policy Distillation (OPD) for software engineering (SWE) agents. Specifically, a smaller student model (GLM-4.5-Air, 100B) interacts directly with executable code repositories, while a larger teacher model (GLM-5.2 / DeepSeek-V4-Flash, 800B / 300B) provides supervision over the sequence prefixes generated by the student throughout its interactions.

Importantly, our approach requires no reinforcement learning (RL), no task-success-based trajectory filtering, and no curated trajectories. We deliberately retain the original student-generated trajectories and apply vanilla OPD, keeping the training pipeline as simple and controlled as possible.

Additional Experiment: Industrial-Scale Cross-Tokenizer Capability Transfer

To further examine whether OPD can generalize beyond models within the same family, we extend our experiments to a cross-tokenizer setting, pairing GLM-4.5-Air as the student with DeepSeek-V4-Flash-0731 as the teacher.

Unlike conventional same-family distillation, the two models use different tokenizers, meaning their vocabularies and token boundaries do not match. We address this challenge using a naive GOLD-style span alignment strategy, merging tokens when their character spans differ across tokenizations. Notably, we introduce neither explicit cross-vocabulary mapping nor modifications to the original distillation objective.

Despite this minimal alignment strategy, offline ST alone achieves substantial improvements, yielding the largest gains in our study on SWE-Pro (+3.9) and SWE-Multilingual (+4.6).

Industrial-Scale Cross-Tokenizer Capability Transfer (GLM-4.5-Air ← DeepSeek-V4-Flash-0731)

MethodVerifiedΔProΔMultilingualΔ
Student base (GLM-4.5-Air)59.9–35.8–41.6–
Offline SWE-OPD (ST)62.5+2.639.7+3.946.2+4.6

Δ denotes the improvement over the GLM-4.5-Air baseline.

These results demonstrate that OPD is not restricted to same-family teachers or shared tokenizers: a stronger teacher from a different model family can effectively transfer coding capabilities to the student, even with a simple token-span alignment strategy in industrial-scale scenarios.

Three Key Highlights of Our Work.

To the best of our knowledge, our work is the first to successfully demonstrate industrial-scale large-to-small OPD for long-horizon agentic coding, with three key highlights:

Highlight 1: OPD for Industrial-Scale, Long-Horizon Agentic Coding.

We move beyond conventional distillation on short-form text and static datasets, bringing OPD to real-world SWE agent tasks involving multi-turn interactions, tool use, and long-horizon reasoning.

Highlight 2: Vanilla OPD — No RL, No Trajectory Filtering, No Additional Curation, Keep original chain-of-thought (CoT).

We preserve the student's original generation trajectories and chain-of-thought (CoT), without filtering trajectories based on task success, performing additional data curation, or introducing sophisticated training tricks. Our principle is simple: the student explores and generates; the teacher provides supervision.

Highlight 3: Industrial-Scale Models, Cross-Tokenizer Transfer, and Large-Scale Compute.

We conduct experiments across industrial-scale models, including DeepSeek-V4-Flash and GLM-5.2, and further demonstrate cross-family, cross-tokenizer capability transfer. Training and validation are powered by 32 NVIDIA B300 GPUs.

We sincerely hope that these additional clarifications and experimental results help address the reviewers' concerns and provide a more complete picture of our contributions. We respectfully ask the AC and reviewers to reconsider their evaluations in light of these points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.