acceptodds
Under review as a conference paper at ICLR 2027

Beyond Full-Trajectory Imitation: Localized Expert Supervision for Issue Reproduction

Abstract

Recent imitation learning methods improve code agents by learning from complete expert rollouts and expert continuations collected during student execution. However, effectively using such supervision requires identifying both where the student needs expert intervention and which expert behaviors are useful for correcting it. In this work, we present LoEC, a three-stage post-training framework that selectively allocates expert supervision for repository-level issue reproduction. LoEC first shapes the initial strategy through plan-level on-policy distillation, and then uses issue-reproduction progress signals to localize stalled or invalid states that require expert intervention. For successful expert recoveries, it further narrows supervision through intervention-level plan shaping and masked SFT over selected segments, including effective test edits retained in the final test. On SWT-Bench Verified, LoEC improves Pass@1 from 21.11% to 32.24%, and outperforms two expert-takeover imitation baselines. Compared with full-trajectory BC for initial alignment and full-suffix SFT for correction, LOEC achieves better performance while reducing supervised-token usage by over 91% and over 18%.

Primary area:
reinforcement learning
Keywords:
Agent Trajectory, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VBEDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LoEC, a three-stage post-training framework for repository-level issue reproduction. It first distills the expert's initial planning behavior into the student, then uses execution-based progress signals to identify intervention states in student trajectories. Successful expert takeovers provide supervision for recovery-plan distillation and masked SFT on selected corrective segments. Experiments train a Qwen3.5-4B student with a Qwen3.5-27B expert on SWE-Gym and evaluate on SWT-Bench Verified. LoEC improves reproduction success over the base student and OEC/SCoRe baselines while using fewer supervised tokens than the full-trajectory and full-suffix imitation variants.

Critical strengths
  1. Improving issue reproduction through targeted expert correction is a practically relevant direction. The use of execution-based verification to select recoverable states provides a concrete connection between supervision and task success.
  2. The paper reports meaningful improvements over the base student and evaluates on repositories separate from the training set. The comparison with teacher-selected intervention points provides useful evidence for the proposed progress-guided localization.
Critical weaknesses
  1. The methodological contribution appears incremental relative to existing expert-correction approaches. OEC [1], SCoRe [2], and SRC [3] already train agents using expert intervention and corrections on student-generated trajectories. LoEC's contribution lies mainly in its task-specific localization rules, plan-level OPD, and supervision selection. The localization ablation provides useful support, but the value of the other additions is less clear. Table 2 changes both the supervision scope and learning objective during initial alignment. The correction comparison also removes recovery-plan OPD while replacing masked SFT with full-suffix SFT. Thus, the results do not establish whether selecting expert segments offers an advantage over simply combining recovery-plan OPD with full-suffix SFT. This limits the evidence for the distinctive contribution beyond the existing correction framework.
  2. The empirical support remains limited to one student–expert pair and one benchmark. The advantage over the strongest baseline, SCoRe-RL, is 2.59 percentage points, whereas the reported SEM measures environment-induced variation across five runs on the same issues. It does not characterize uncertainty over the evaluated issue distribution. Results on another student model, together with per-repository comparisons and paired issue-level confidence intervals, would make the advantage more convincing, particularly since Django accounts for nearly half of the benchmark (Appendix F).

[1] Imitation Learning for Multi-turn LM Agents via On-policy Expert Corrections.

[2] Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs.

[3] Speculative Rollback Correction for Quality-Diverse Web Agent Imitation.

Questions for authors
  1. Does selecting expert steps for supervision improve performance beyond recovery-plan OPD itself? To isolate this, how does recovery-plan OPD followed by full-suffix SFT compare with LoEC, using the same takeover trajectories and SFT settings?
  2. How applicable is the three-stage framework beyond issue reproduction? The current implementation uses task-specific execution feedback and gold-patch information to guide intervention. What feedback would be needed to identify stalled states and select corrective segments in other agent tasks, and which components would require redesign?
  3. The reported 7.59× FLOP ratio measures training-update costs rather than end-to-end compute. Could the authors report GPU-hours and wall-clock costs for LoEC and the baselines, including rollout collection, expert generation and scoring, and test execution? In particular, how much compute does localized SFT save over full-suffix SFT when masked context tokens still require processing?
  4. Appendix D reports a three-epoch training budget for localized SFT, but the evaluated model uses the checkpoint at update 16. How was this checkpoint selected, and was a separate validation set used? Since Table 2 reports supervised-token counts under the configured budget, could the authors also report the supervised-token counts up to the evaluated checkpoints for LoEC and the baselines?
10 comments
AuthorsOpenReview ↗

Author Rebuttal (1)

W1. The methodological contribution appears incremental relative to existing expert-correction approaches. OEC [1], SCoRe [2], and SRC [3] already train agents using expert intervention and corrections on student-generated trajectories. LoEC's contribution lies mainly in its task-specific localization rules, plan-level OPD, and supervision selection. The localization ablation provides useful support, but the value of the other additions is less clear. Table 2 changes both the supervision scope and learning objective during initial alignment. The correction comparison also removes recovery-plan OPD while replacing masked SFT with full-suffix SFT. Thus, the results do not establish whether selecting expert segments offers an advantage over simply combining recovery-plan OPD with full-suffix SFT. This limits the evidence for the distinctive contribution beyond the existing correction framework.

Answer: Thank you for this suggestion. We have added controlled comparisons to isolate the contributions of plan-level OPD and selective SFT. The results show that Plan-Level OPD improves initial alignment beyond plan-only supervision, while selective SFT consistently outperforms full-suffix SFT under matched settings.

For initial alignment, Plan-Level OPD achieves 27.25% Pass@1, outperforming Plan-Only SFT (17.09%) and Full-Trajectory BC (25.64%). Plan-Only SFT uses the same supervision scope as Plan-Level OPD and the same training settings as Full-Trajectory BC.

MethodPass@1 ↑
Full-Trajectory BC25.64%
Plan-Only SFT17.09%
Plan-Level OPD (Ours)27.25%

For expert correction, we compare selective SFT and full-suffix SFT using identical starting checkpoints, verified expert trajectories, and SFT settings within each comparison. Selective SFT improves Pass@1 by 1.62 percentage points without recovery-plan OPD and 1.06 percentage points with recovery-plan OPD, demonstrating its contribution independently of recovery-plan OPD.

SettingMethodPass@1 ↑
Without Recovery-plan OPDFull-Suffix SFT27.02%
Selective SFT28.64%
With Recovery-plan OPDRecovery-plan OPD Only30.25%
+ Full-Suffix SFT31.18%
+ Selective SFT (LoEC)32.24%
AuthorsOpenReview ↗

Author Rebuttal (2)

W2. The empirical support remains limited to one student–expert pair and one benchmark. The advantage over the strongest baseline, SCoRe-RL, is 2.59 percentage points, whereas the reported SEM measures environment-induced variation across five runs on the same issues. It does not characterize uncertainty over the evaluated issue distribution. Results on another student model, together with per-repository comparisons and paired issue-level confidence intervals, would make the advantage more convincing, particularly since Django accounts for nearly half of the benchmark (Appendix F).

Answer: Thank you for this suggestion. We compare LoEC against SCoRe-RL under comparable GPU budgets and further provide paired issue-level confidence intervals and per-repository comparisons. The results provide stronger evidence for the advantage of LoEC over SCoRe-RL across the evaluated issue distribution.

Under a comparable GPU budget, LoEC performs one additional correction round and achieves 35.06% Pass@1, outperforming SCoRe-RL (29.65%) while using fewer GPU-hours (407.58 vs. 421.16). To characterize uncertainty beyond the five-run SEM, we perform 200,000 paired issue-level bootstrap resamples, preserving both methods' five outcomes per issue. The resulting 95% confidence interval of [2.22, 8.59] percentage points.

Across 12 repositories, LoEC outperforms SCoRe-RL in eight. Excluding Django, LoEC maintains a 4.52 percentage point advantage, while equal weighting across repositories yields a 3.72 percentage point improvement.

RepositoryIssuesLoEC (%)SCoRe-RL (%)Difference (pp)
Django21623.6117.31+6.30
SymPy7353.9751.51+2.47
Matplotlib3243.7545.62-1.88
Sphinx2815.719.29+6.43
Scikit-learn2465.8350.00+15.83
Astropy1755.2943.53+11.76
Xarray1569.3352.00+17.33
Pytest1528.0040.00-12.00
Pylint63.330.00+3.33
Requests450.0045.00+5.00
Seaborn20.0010.00-10.00
Flask1100.00100.000.00

As detailed in our subsequent Author Rebuttal (Cross-Student Model Evaluation), LoEC improves Pass@1 on SWT-Bench Verified from 5.78% to 11.79% for Qwen3-4B-Instruct-2507, achieving a 2.04× improvement. Meanwhile, SCoRe-RL exhibits repetitive command execution, resulting in poor reproduction performance.

Q1. Does selecting expert steps for supervision improve performance beyond recovery-plan OPD itself? To isolate this, how does recovery-plan OPD followed by full-suffix SFT compare with LoEC, using the same takeover trajectories and SFT settings?

Answer: Thank you for the suggestion. We have added a controlled comparison in which full-suffix SFT and selective SFT matched training settings. Compared with recovery-plan OPD alone, selective SFT improves Pass@1 by 1.99 percentage points (30.25% to 32.24%), outperforming full-suffix SFT by 1.06 percentage points while using 47.32% fewer SFT supervised tokens. These results demonstrate that selective expert supervision provides additional benefits beyond recovery-plan OPD and full-suffix SFT.

MethodSupervised Tokens ↓Pass@1 ↑
Recovery-plan OPD Only284,07330.25%
+ Full-Suffix SFT1,259,369*31.18%
+ Selective SFT (LoEC)797,824*32.24%

Note. * Supervised-token counts include recovery-plan OPD (284,073 tokens) and the subsequent SFT stage.

AuthorsOpenReview ↗

Author Rebuttal (3)

Q2. How applicable is the three-stage framework beyond issue reproduction? The current implementation uses task-specific execution feedback and gold-patch information to guide intervention. What feedback would be needed to identify stalled states and select corrective segments in other agent tasks, and which components would require redesign?

Answer: Thank you for this question. We examined LoEC's adaptation to SWE-bench Verified and Terminal-Bench to assess which components require redesign. The three-stage framework remains applicable, while task-specific progress evaluation and crucial-action selection require adaptation.

ComponentSWE-bench VerifiedTerminal-Bench
Progress-Guided Localization
Test-Edit Inactivity✅✅
Progress Plateau✅❌
Incorrect Submission✅✅
Selective Supervision
Crucial Edits✅❌
Test Submit✅✅
Submit Recovery✅✅

Note: ✅ No substantial redesign required; ❌ Task-specific redesign required.

For SWE-bench Verified, inactivity can track code modifications, progress plateaus can use test failures and code coverage, and incorrect submissions can be identified through repair verification. Crucial edits can be selected from verified repair patches. In a preliminary analysis of 101 intervention windows from 25 instances, only 16 showed substantive progress.

For Terminal-Bench, inactivity and incorrect submission can be detected through execution patterns and task outcomes without substantial redesign. However, progress-plateau detection requires task-specific intermediate feedback, as terminal tasks lack a unified progress metric. For example, artifact-generation tasks can measure progress by the number of input records successfully converted into the target artifact. Similarly, crucial-action selection requires task-specific evidence, since successful outcomes may depend on generated artifacts, configuration changes, or runtime states rather than code patches. In contrast, Test Submit and Submit Recovery can largely reuse task-completion verification and unsuccessful termination signals.

Reviewer_UkAYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LoEC post-trains an issue-reproduction agent through initial-plan on-policy distillation, retrospective localization of stalled student trajectories, and correction from verified expert takeovers. Correction combines recovery-plan distillation with masked SFT over selected expert segments. With a Qwen3.5-4B student and Qwen3.5-27B expert, mean Pass@1 across five evaluation runs on 433 SWT-Bench Verified tasks increases from 21.11% to 32.24%, compared with 29.65% for SCoRe-RL. Stage-wise comparisons examine performance and supervised-token usage.

Critical strengths
  1. The method connects expert intervention to the structure of issue reproduction. Progress-aware localization, execution-verified takeovers, and selection of edits retained in the final test patch provide a concrete approach to targeting supervision. The paper clearly identifies localization as retrospective selection of training states.

  2. The complete system substantially improves reproduction success over the base model, alongside coverage and applicability. Replacing progress-guided localization with teacher-selected intervention points reduces Pass@1 from 32.24% to 27.94%, supporting the proposed localization procedure. The supervised-token accounting is useful and appropriately distinguished from end-to-end computational cost. The repeated evaluations also characterize environment-induced variability; they should not be interpreted as independent training runs.

Critical weaknesses

The correction-stage comparison does not isolate the benefit of selective supervision. Localized correction combines recovery-plan OPD with masked SFT, whereas the full-suffix control applies SFT without that OPD stage. The 32.24% versus 28.41% comparison therefore supports the combined recipe but does not establish how much of the gain comes from masking. The initial-stage comparison similarly changes both the objective and the supervised region. The discussion of Plan-Only SFT provides some additional evidence, but no numerical result is reported for that control. Consequently, the interpretation that restricting supervision explains the gains is less firmly established than the full system's empirical improvement.

Questions for authors

Is a masked-versus-full-suffix comparison available after the same recovery-plan OPD checkpoint, using the same takeover trajectories and matched training settings? This would clarify the mask's contribution. Otherwise, narrowing the component attribution to the combined recovery-plan-distillation and selective-SFT recipe would better match the evidence.

2 comments
AuthorsOpenReview ↗

Author Rebuttal

W1. The correction-stage comparison does not isolate the benefit of selective supervision. Localized correction combines recovery-plan OPD with masked SFT, whereas the full-suffix control applies SFT without that OPD stage. The 32.24% versus 28.41% comparison therefore supports the combined recipe but does not establish how much of the gain comes from masking. The initial-stage comparison similarly changes both the objective and the supervised region. The discussion of Plan-Only SFT provides some additional evidence, but no numerical result is reported for that control. Consequently, the interpretation that restricting supervision explains the gains is less firmly established than the full system's empirical improvement.

Answer: Thank you for the suggestion. We have added controlled experiments for both initial alignment and expert correction. The results show that Plan-Level OPD substantially outperforms Plan-Only SFT under the same supervision scope, while selective SFT consistently outperforms full-suffix SFT under matched settings.

  • Initial Plan-Level OPD substantially outperforms Plan-Only SFT.

    We additionally report the numerical results for Plan-Only SFT, which supervises only expert-generated initial plans, matching the supervision scope of Plan-Level OPD. Plan-Level OPD achieves 27.25% Pass@1, outperforming Plan-Only SFT (17.09%) by 10.15 percentage points. These results demonstrate the effectiveness of Plan-Level OPD over SFT when supervision is restricted to the initial planning turn.

    MethodPass@1 ↑
    Plan-Only SFT17.09%
    Plan-Level OPD27.25%
  • Selective SFT consistently outperforms full-suffix SFT in expert correction.

    Using identical expert takeover trajectories and SFT settings, selective SFT outperforms full-suffix SFT by 1.62 percentage points without recovery-plan OPD and 1.06 percentage points with recovery-plan OPD. Recovery-plan OPD alone also improves Pass@1 from 27.25% to 30.25%. These results confirm that both recovery-plan OPD and selective supervision contribute to the effectiveness of LoEC.

    SettingMethodPass@1 ↑
    Without Recovery-plan OPDPlan-Level OPD Only27.25%
    + Full-Suffix SFT27.02%
    + Selective SFT28.64%
    With Recovery-plan OPDPlan-Level OPD Only → Recovery-plan OPD27.25% → 30.25%
    + Full-Suffix SFT31.18%
    + Selective SFT (LoEC)32.24%

Q1. Is a masked-versus-full-suffix comparison available after the same recovery-plan OPD checkpoint, using the same takeover trajectories and matched training settings? This would clarify the mask's contribution. Otherwise, narrowing the component attribution to the combined recovery-plan-distillation and selective-SFT recipe would better match the evidence.

Answer: Thank you for the suggestion. We have added a controlled comparison to isolate the contribution of selective supervision. Both methods start from the same recovery-plan OPD checkpoint (30.25% Pass@1), using identical expert takeover trajectories and SFT settings. Selective SFT achieves 32.24% Pass@1, outperforming full-suffix SFT (31.18%) by 1.06 percentage points, while using 47.32% fewer supervised tokens. The results show that selective SFT improves reproduction performance over full-suffix SFT while substantially reducing supervised tokens.

MethodSFT Supervised Tokens ↓Pass@1 ↑
Recovery-plan OPD Only—30.25%
+ Full-Suffix SFT975,29631.18%
+ Selective SFT (LoEC)513,75132.24%
AuthorsOpenReview ↗

Reviewer–Author Discussion: Just Over Two Days Remaining

Dear Reviewer UkAY,

Thank you for your thoughtful feedback on our submission. We have provided detailed responses and additional experimental evidence to address your concerns. With just over two days remaining in the discussion period, we would greatly appreciate it if you could revisit our responses and share any remaining questions or concerns. We would be happy to provide further clarification before the discussion closes.

Your effort and time in reviewing our submission are sincerely appreciated.

Best regards,

Author(s)

Reviewer_wjVpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LOEC, a three-stage post-training framework that selectively allocates expert supervision for repository-level issue reproduction. Instead of imitating complete expert trajectories, it first performs initial plan shaping, which applies plan-level on-policy distillation (OPD). traces task-specific progress failures back to intervention states near the onset of stalled or invalid behaviour. For intervention states where expert takeovers successfully reproduce the target issue, the framework combines recovery-plan OPD with masked supervised fine-tuning (SFT) on selected corrective segments. Experiments apply a 4B model as student and a 27B model as a expert, with training instances adapted from SWE-Gym and evaluation on SWT-Bench Verified. LOEC improves average Pass@1 from 21.11% to 32.24%, outperforming the evaluated expert-correction baselines.

Critical strengths

Task-aware and coherent expert supervision design The framework connects task progress, intervention-state selection, verified expert recovery, and selective supervision. Its three triggers target recognizable problems in issue reproduction: prolonged test-edit inactivity, test revisions without progress, and incorrect submission.

Encouraging results from selective supervision The stage-wise comparisons suggest that effective agent training does not necessarily require supervising every action in a successful expert trajectory. LOEC achieves higher Pass@1 while using considerably fewer supervised tokens in both initial alignment and correction, motivating further investigation into how expert supervision should be allocated.

Critical weaknesses

The ablation comparisons do not fully isolate the effects of supervision scope and optimization method. Table 2 uses the same starting checkpoint and training instances within each stage but changes several factors together. Full-Trajectory BC vs. Initial Plan OPD changes the supervised positions, target distribution, and training objective. During correction, Localized Correction adds recovery-plan OPD while replacing full-suffix SFT with masked SFT. The corresponding Table 3 ablation makes the same joint change. These results support the combined method, but do not establish the contribution of selective SFT. The numerical result and configuration for Plan-Only SFT are also missing.

The stall detection rules and intervention-state selection are not sufficiently validated. Prior work such as OEC and SCoRe has explored expert correction on student trajectories, using random switch points or expert-identified error states. Therefore, a key contribution of LOEC is its task-specific progress-guided intervention localization and retrospective selection of takeover states. However, the paper does not sufficiently validate these design choices.

LOEC uses three triggers. However, the paper does not report how many training examples each trigger contributes or how much each trigger improves final performance. It is also unclear whether the chosen rules identify states that need expert intervention. For example, no test edits over 20 actions or no improvement across three test revisions does not always indicate that the student is stuck.

Although Table 3 shows that the overall localisation method outperforms teacher-selected intervention points, it does not establish the contribution of each trigger or whether the chosen retrospective intervention states are better than alternative positions. The paper also lacks sensitivity analysis for the thresholds H = 20 and K = 3 in Appendix B.2. Ablations of the three triggers and comparisons of different intervention positions would help validate these design choices.

The expert takeover success rate and training-data selection process are insufficiently characterised. LOEC retains expert takeovers only when they successfully produce a verified issue reproduction. This filtering is important because both recovery-plan OPD and localized SFT depend on successful expert recoveries. Section 4.1 reports 1,174 identified stall states from which expert continuations were collected over five rounds (Line 287). However, Appendix D reports only 306 states for recovery-plan OPD and 269 trajectories for localized SFT. The paper does not provide a complete breakdown of how these training sets were obtained, particularly the number of expert takeovers that passed verification and the number of examples removed during filtering or deduplication. These quantities are important for understanding the effectiveness of intervention-state selection and the cost of collecting usable supervision.

Questions for authors

Q1. Could you compare full-suffix and masked SFT both with and without recovery-plan OPD, holding the intervention states, verified expert suffixes, starting checkpoint, and SFT settings fixed? This would separate the contributions of recovery-plan OPD and masking. Please also report the numerical results and training configuration for Plan-Only SFT.

Q2. For the 1,174 intervention states, how many yielded expert takeovers that passed verification? Please provide the filtering and deduplication counts leading to the 306 OPD states and 269 SFT trajectories, including unique issue counts and a breakdown by collection round and trigger. What fraction of expert tokens is retained for each trigger, and what evidence supports the chosen thresholds and retrospective takeover positions?

7 comments
AuthorsOpenReview ↗

Author Rebuttal (1)

W1. The ablation comparisons do not fully isolate the effects of supervision scope and optimization method. Table 2 uses the same starting checkpoint and training instances within each stage but changes several factors together. Full-Trajectory BC vs. Initial Plan OPD changes the supervised positions, target distribution, and training objective. During correction, Localized Correction adds recovery-plan OPD while replacing full-suffix SFT with masked SFT. The corresponding Table 3 ablation makes the same joint change. These results support the combined method, but do not establish the contribution of selective SFT. The numerical result and configuration for Plan-Only SFT are also missing.

Answer: Thank you for this suggestion. We have added three controlled comparisons to better isolate the contributions of supervision scope and training strategy.

  • Plan-Level OPD improves initial alignment beyond plan-only SFT.

    We compare Plan-Level OPD with Plan-Only SFT using expert-generated plans. For Plan-Only SFT, we use the same training configuration as Full-Trajectory BC. Both methods supervise only initial plans but adopt different training strategies. Plan-Level OPD achieves 27.25% Pass@1, outperforming Plan-Only SFT (17.09%) by 10.15 percentage points, demonstrating the benefit of our on-policy plan shaping.

    MethodSupervised RegionPass@1 ↑
    Plan-Only SFTInitial plans17.09%
    Plan-Level OPDInitial plans27.25%
  • Selective SFT provides additional gains beyond recovery-plan OPD. Starting from the recovery-plan OPD checkpoint, adding selective SFT improves Pass@1 from 30.25% to 32.24%, yielding a 1.99 percentage point improvement. This confirms the additional contribution of selective SFT to LoEC.

    MethodSFT SupervisionPass@1 ↑
    Recovery-plan OPD OnlyNone30.25%
    + Localized Correction (LoEC)Selected corrective steps32.24%
  • Selective SFT consistently outperforms full-suffix SFT under matched settings. Using identical settings, selective SFT outperforms full-suffix SFT by 1.62 percentage points without recovery-plan OPD and 1.06 percentage points with recovery-plan OPD. These results demonstrate the effectiveness of supervising crucial corrective steps.

    SettingMethodPass@1 ↑
    Without Recovery-plan OPDFull-Suffix SFT27.02%
    Localized Correction28.64%
    With Recovery-plan OPDFull-Suffix SFT31.18%
    Localized Correction (LoEC)32.24%

W2. The stall detection rules and intervention-state selection are not sufficiently validated. Prior work such as OEC and SCoRe has explored expert correction on student trajectories, using random switch points or expert-identified error states. Therefore, a key contribution of LOEC is its task-specific progress-guided intervention localization and retrospective selection of takeover states. However, the paper does not sufficiently validate these design choices.

Answer: Thank you for this suggestion. We have expanded our intervention-state selection ablations to validate progress-guided localization and retrospective selection.

  • Progress-guided localization improves intervention effectiveness.

    We compare LoEC with Random Selection and Expert-Guided Selection. LoEC achieves 32.24% Pass@1, outperforming Random Selection (25.17%) and Expert-Guided Selection (27.94%) by 7.07 and 4.30 percentage points, respectively. This demonstrates the benefit of using task-specific progress signals to guide intervention localization.

    Selection StrategyPass@1 ↑
    Random Selection25.17%
    Expert-Guided Selection27.94%
    LoEC (Ours)32.24%
  • Retrospective selection improves interaction efficiency.

    We introduce Direct Stall Points, which initiates expert intervention at the detected stall states without backtracking. LoEC achieves comparable Pass@1 (32.24% vs. 31.64%) while reducing average interaction turns from 51.12 to 42.55 (16.76%). This shows that retrospective selection enables more efficient issue reproduction while maintaining reproduction performance.

    Selection StrategyPass@1 ↑Turns ↓
    Direct Stall Points31.64%51.12
    LoEC (Ours)32.24%42.55
AuthorsOpenReview ↗

Author Rebuttal (2)

W3.1. LOEC uses three triggers. However, the paper does not report how many training examples each trigger contributes or how much each trigger improves final performance.

Answer: Thank you for this suggestion. We have added three ablations to quantify the contribution of each intervention trigger. The results show that all three triggers provide useful supervision, and removing any trigger reduces Pass@1. The table below reports the retained training examples (with removed examples in parentheses) and the resulting effectiveness. Removing test-edit inactivity, test-quality plateau, and incorrect submission reduces Pass@1 by 1.52, 0.83, and 1.29 percentage points, respectively.

VariantOPD StatesSFT Traj.Pass@1 ↑
w/o Test-Edit Inactivity106 (−200)80 (−189)30.72%
w/o Test-Quality Plateau278 (−28)244 (−25)31.41%
w/o Incorrect Submission228 (−78)214 (−55)30.95%
LoEC (Full Design)30626932.24%

W3.2. It is also unclear whether the chosen rules identify states that need expert intervention. For example, no test edits over 20 actions or no improvement across three test revisions does not always indicate that the student is stuck.

Answer: Thank you for raising this concern. We have conducted a manual evaluation of the states identified by our intervention triggers.

  • Most identified states exhibit no meaningful task progress.

    We randomly sample 78 test-edit inactivity and 43 test-quality plateau triggers from 406 and 76 candidates, respectively, using a 95% confidence level and a 10% margin of error. Two experienced software engineering researchers independently assess task progress and resolve disagreements through discussion. The results show that 92.31% of inactivity triggers and 79.07% of plateau triggers exhibit no meaningful progress.

    TriggerSample SizeNo Meaningful Progress95% CI
    Test-Edit Inactivity7872/78 (92.31%)84.98% to 96.55%
    Test-Quality Plateau4334/43 (79.07%)69.74% to 85.53%
  • Students rarely recover during subsequent execution.

    We further examine all 121 original student trajectories to determine whether students make progress after the triggers are confirmed. Only 6/78 (7.69%) test-edit inactivity cases subsequently produce executable test edits reaching issue-relevant code, while 2/43 (4.65%) test-quality plateau cases improve beyond their previous best quality level.

    TriggerSample SizeSubsequent Self-Recovery
    Test-Edit Inactivity786/78 (7.69%)
    Test-Quality Plateau432/43 (4.65%)
    Overall1218/121 (6.61%)
AuthorsOpenReview ↗

Author Rebuttal (3)

W4.1. Although Table 3 shows that the overall localisation method outperforms teacher-selected intervention points, it does not establish the contribution of each trigger or whether the chosen retrospective intervention states are better than alternative positions.

Answer: Thank you for this suggestion. We have added trigger-wise ablations and intervention-state selection comparisons. The results show that all three triggers contribute to performance, while progress-guided localization and retrospective selection improve intervention effectiveness and efficiency. Removing test-edit inactivity, test-quality plateau, or incorrect submission reduces Pass@1 by 1.52, 0.83, and 1.29 percentage points, respectively. Under the same correction procedure, LoEC outperforms random and expert-guided selection by 7.07 and 4.30 percentage points. Compared with Direct Stall Points, retrospective selection improves Pass@1 from 31.64% to 32.24% while reducing interaction turns by 16.76%.

MethodPass@1 ↑
Full LoEC32.24%
w/o Edit Inactivity30.72%
w/o Quality Plateau31.41%
w/o Incorrect Submission30.95%
Selection StrategyPass@1 ↑Turns ↓
Random Selection25.17%42.27
Expert-Guided Selection27.94%41.76
Direct Stall Points31.64%51.12
LoEC (Retrospective Selection)32.24%42.55

W4.2. The paper also lacks sensitivity analysis for the thresholds H = 20 and K = 3 in Appendix B.2.

Answer: Thank you for this suggestion. We have added a sensitivity analysis of the inactivity threshold H and plateau threshold K. LoEC maintains comparable performance under moderate threshold variations. For H=10, K=2, and K=4, successful takeovers remain close to the default, with largely unchanged intervention positions and comparable Pass@1 (31.64%, 32.24%, and 31.41%, respectively, vs. 32.24%). In contrast, increasing H to 30 reduces successful takeovers from 310 to 140 and Pass@1 to 30.48%, indicating that overly conservative stall detection limits available supervision.

SettingSuccessful TakeoversPosition AgreementPass@1 (%) ↑
H20/K3 (Default)310—32.24
H10/K3305281/305 (92.13%)31.64
H30/K3140140/140 (100%)30.48
H20/K2302302/302 (100%)32.24
H20/K4288288/288 (100%)31.41

W5. The expert takeover success rate and training-data selection process are insufficiently characterised. LOEC retains expert takeovers only when they successfully produce a verified issue reproduction... Section 4.1 reports 1,174 identified stall states from which expert continuations were collected over five rounds (Line 287). However, Appendix D reports only 306 states for recovery-plan OPD and 269 trajectories for localized SFT. The paper does not provide a complete breakdown of how these training sets were obtained, particularly the number of expert takeovers that passed verification and the number of examples removed during filtering or deduplication...

Answer: Thank you for this question. We have clarified the expert takeover verification and training-data filtering process. Among 1,174 intervention states, 310 (26.4%) yield verified expert takeovers, producing 879 successful trajectories. After filtering overlong prompts, we retain 306 states for recovery-plan OPD. For localized SFT, eligibility filtering, issue-level deduplication, and sequence-length filtering yield 269 trajectories from 269 unique issues. The detailed breakdown is provided below.

StageFiltering StepRemovedRemaining
Recovery-plan OPDVerified takeover states—310
Overlength prompts4306
Localized SFTVerified takeover trajectories—879
No eligible supervision steps130749
Deduplication by issue478271
Overlength sequences2269

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their careful reading and constructive feedback. Our responses and revisions include three main improvements:

  • Stronger performance at comparable GPU cost. To ensure a fair comparison with SCoRe-RL, we perform an additional LoEC correction round to match their GPU budgets more closely. Under comparable GPU costs (407.58 vs. 421.16 GPU-hours), LoEC achieves 35.06% Pass@1, outperforming SCoRe-RL (29.65%). Per-repository comparisons and paired issue-level confidence intervals further support this advantage.
  • Additional validation of our methodological contributions. We add controlled comparisons to clarify the contributions of plan-level OPD and selective SFT. Additional ablations, manual evaluation, and sensitivity analyses further validate progress-guided localization, retrospective selection, and individual intervention triggers.
  • Improved experimental transparency. We provide detailed takeover verification and data-filtering statistics, clarify checkpoint selection, and report GPU-hours, test-execution time, and supervised-token counts. We also discuss the adaptations needed to extend LoEC to other agent tasks and investigate its generalizability across student models.
  • In addition to the source code already made available in our paper, we have publicly archived the final trained LoEC model and evaluation trajectories on Zenodo to support reproducibility and long-term access.

We appreciate the opportunity to strengthen our work. We welcome any further questions or concerns and would be happy to provide additional clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.