- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LoEC, a three-stage post-training framework for repository-level issue reproduction. It first distills the expert's initial planning behavior into the student, then uses execution-based progress signals to identify intervention states in student trajectories. Successful expert takeovers provide supervision for recovery-plan distillation and masked SFT on selected corrective segments. Experiments train a Qwen3.5-4B student with a Qwen3.5-27B expert on SWE-Gym and evaluate on SWT-Bench Verified. LoEC improves reproduction success over the base student and OEC/SCoRe baselines while using fewer supervised tokens than the full-trajectory and full-suffix imitation variants.
Critical strengths
- Improving issue reproduction through targeted expert correction is a practically relevant direction. The use of execution-based verification to select recoverable states provides a concrete connection between supervision and task success.
- The paper reports meaningful improvements over the base student and evaluates on repositories separate from the training set. The comparison with teacher-selected intervention points provides useful evidence for the proposed progress-guided localization.
Critical weaknesses
- The methodological contribution appears incremental relative to existing expert-correction approaches. OEC [1], SCoRe [2], and SRC [3] already train agents using expert intervention and corrections on student-generated trajectories. LoEC's contribution lies mainly in its task-specific localization rules, plan-level OPD, and supervision selection. The localization ablation provides useful support, but the value of the other additions is less clear. Table 2 changes both the supervision scope and learning objective during initial alignment. The correction comparison also removes recovery-plan OPD while replacing masked SFT with full-suffix SFT. Thus, the results do not establish whether selecting expert segments offers an advantage over simply combining recovery-plan OPD with full-suffix SFT. This limits the evidence for the distinctive contribution beyond the existing correction framework.
- The empirical support remains limited to one student–expert pair and one benchmark. The advantage over the strongest baseline, SCoRe-RL, is 2.59 percentage points, whereas the reported SEM measures environment-induced variation across five runs on the same issues. It does not characterize uncertainty over the evaluated issue distribution. Results on another student model, together with per-repository comparisons and paired issue-level confidence intervals, would make the advantage more convincing, particularly since Django accounts for nearly half of the benchmark (Appendix F).
[1] Imitation Learning for Multi-turn LM Agents via On-policy Expert Corrections.
[2] Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs.
[3] Speculative Rollback Correction for Quality-Diverse Web Agent Imitation.
Questions for authors
- Does selecting expert steps for supervision improve performance beyond recovery-plan OPD itself? To isolate this, how does recovery-plan OPD followed by full-suffix SFT compare with LoEC, using the same takeover trajectories and SFT settings?
- How applicable is the three-stage framework beyond issue reproduction? The current implementation uses task-specific execution feedback and gold-patch information to guide intervention. What feedback would be needed to identify stalled states and select corrective segments in other agent tasks, and which components would require redesign?
- The reported 7.59× FLOP ratio measures training-update costs rather than end-to-end compute. Could the authors report GPU-hours and wall-clock costs for LoEC and the baselines, including rollout collection, expert generation and scoring, and test execution? In particular, how much compute does localized SFT save over full-suffix SFT when masked context tokens still require processing?
- Appendix D reports a three-epoch training budget for localized SFT, but the evaluated model uses the checkpoint at update 16. How was this checkpoint selected, and was a separate validation set used? Since Table 2 reports supervised-token counts under the configured budget, could the authors also report the supervised-token counts up to the evaluated checkpoints for LoEC and the baselines?
10 comments
Author Rebuttal (2)
W2. The empirical support remains limited to one student–expert pair and one benchmark. The advantage over the strongest baseline, SCoRe-RL, is 2.59 percentage points, whereas the reported SEM measures environment-induced variation across five runs on the same issues. It does not characterize uncertainty over the evaluated issue distribution. Results on another student model, together with per-repository comparisons and paired issue-level confidence intervals, would make the advantage more convincing, particularly since Django accounts for nearly half of the benchmark (Appendix F).
Answer: Thank you for this suggestion. We compare LoEC against SCoRe-RL under comparable GPU budgets and further provide paired issue-level confidence intervals and per-repository comparisons. The results provide stronger evidence for the advantage of LoEC over SCoRe-RL across the evaluated issue distribution.
Under a comparable GPU budget, LoEC performs one additional correction round and achieves 35.06% Pass@1, outperforming SCoRe-RL (29.65%) while using fewer GPU-hours (407.58 vs. 421.16). To characterize uncertainty beyond the five-run SEM, we perform 200,000 paired issue-level bootstrap resamples, preserving both methods' five outcomes per issue. The resulting 95% confidence interval of [2.22, 8.59] percentage points.
Across 12 repositories, LoEC outperforms SCoRe-RL in eight. Excluding Django, LoEC maintains a 4.52 percentage point advantage, while equal weighting across repositories yields a 3.72 percentage point improvement.
| Repository | Issues | LoEC (%) | SCoRe-RL (%) | Difference (pp) |
|---|---|---|---|---|
| Django | 216 | 23.61 | 17.31 | +6.30 |
| SymPy | 73 | 53.97 | 51.51 | +2.47 |
| Matplotlib | 32 | 43.75 | 45.62 | -1.88 |
| Sphinx | 28 | 15.71 | 9.29 | +6.43 |
| Scikit-learn | 24 | 65.83 | 50.00 | +15.83 |
| Astropy | 17 | 55.29 | 43.53 | +11.76 |
| Xarray | 15 | 69.33 | 52.00 | +17.33 |
| Pytest | 15 | 28.00 | 40.00 | -12.00 |
| Pylint | 6 | 3.33 | 0.00 | +3.33 |
| Requests | 4 | 50.00 | 45.00 | +5.00 |
| Seaborn | 2 | 0.00 | 10.00 | -10.00 |
| Flask | 1 | 100.00 | 100.00 | 0.00 |
As detailed in our subsequent Author Rebuttal (Cross-Student Model Evaluation), LoEC improves Pass@1 on SWT-Bench Verified from 5.78% to 11.79% for Qwen3-4B-Instruct-2507, achieving a 2.04× improvement. Meanwhile, SCoRe-RL exhibits repetitive command execution, resulting in poor reproduction performance.
Q1. Does selecting expert steps for supervision improve performance beyond recovery-plan OPD itself? To isolate this, how does recovery-plan OPD followed by full-suffix SFT compare with LoEC, using the same takeover trajectories and SFT settings?
Answer: Thank you for the suggestion. We have added a controlled comparison in which full-suffix SFT and selective SFT matched training settings. Compared with recovery-plan OPD alone, selective SFT improves Pass@1 by 1.99 percentage points (30.25% to 32.24%), outperforming full-suffix SFT by 1.06 percentage points while using 47.32% fewer SFT supervised tokens. These results demonstrate that selective expert supervision provides additional benefits beyond recovery-plan OPD and full-suffix SFT.
| Method | Supervised Tokens ↓ | Pass@1 ↑ |
|---|---|---|
| Recovery-plan OPD Only | 284,073 | 30.25% |
| + Full-Suffix SFT | 1,259,369* | 31.18% |
| + Selective SFT (LoEC) | 797,824* | 32.24% |
Note. * Supervised-token counts include recovery-plan OPD (284,073 tokens) and the subsequent SFT stage.
Author Rebuttal (3)
Q2. How applicable is the three-stage framework beyond issue reproduction? The current implementation uses task-specific execution feedback and gold-patch information to guide intervention. What feedback would be needed to identify stalled states and select corrective segments in other agent tasks, and which components would require redesign?
Answer: Thank you for this question. We examined LoEC's adaptation to SWE-bench Verified and Terminal-Bench to assess which components require redesign. The three-stage framework remains applicable, while task-specific progress evaluation and crucial-action selection require adaptation.
| Component | SWE-bench Verified | Terminal-Bench |
|---|---|---|
| Progress-Guided Localization | ||
| Test-Edit Inactivity | ✅ | ✅ |
| Progress Plateau | ✅ | ❌ |
| Incorrect Submission | ✅ | ✅ |
| Selective Supervision | ||
| Crucial Edits | ✅ | ❌ |
| Test Submit | ✅ | ✅ |
| Submit Recovery | ✅ | ✅ |
Note: ✅ No substantial redesign required; ❌ Task-specific redesign required.
For SWE-bench Verified, inactivity can track code modifications, progress plateaus can use test failures and code coverage, and incorrect submissions can be identified through repair verification. Crucial edits can be selected from verified repair patches. In a preliminary analysis of 101 intervention windows from 25 instances, only 16 showed substantive progress.
For Terminal-Bench, inactivity and incorrect submission can be detected through execution patterns and task outcomes without substantial redesign. However, progress-plateau detection requires task-specific intermediate feedback, as terminal tasks lack a unified progress metric. For example, artifact-generation tasks can measure progress by the number of input records successfully converted into the target artifact. Similarly, crucial-action selection requires task-specific evidence, since successful outcomes may depend on generated artifacts, configuration changes, or runtime states rather than code patches. In contrast, Test Submit and Submit Recovery can largely reuse task-completion verification and unsuccessful termination signals.
Author Rebuttal (1)
Answer: Thank you for this suggestion. We have added controlled comparisons to isolate the contributions of plan-level OPD and selective SFT. The results show that Plan-Level OPD improves initial alignment beyond plan-only supervision, while selective SFT consistently outperforms full-suffix SFT under matched settings.
For initial alignment, Plan-Level OPD achieves 27.25% Pass@1, outperforming Plan-Only SFT (17.09%) and Full-Trajectory BC (25.64%). Plan-Only SFT uses the same supervision scope as Plan-Level OPD and the same training settings as Full-Trajectory BC.
For expert correction, we compare selective SFT and full-suffix SFT using identical starting checkpoints, verified expert trajectories, and SFT settings within each comparison. Selective SFT improves Pass@1 by 1.62 percentage points without recovery-plan OPD and 1.06 percentage points with recovery-plan OPD, demonstrating its contribution independently of recovery-plan OPD.