- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses reference patches to provide guidance of varying strength before rollouts, after a failed attempt, or during interaction, then trains a software-engineering student on successful trajectories. Recovering useful training data from tasks that unguided sampling repeatedly fails to solve is a worthwhile objective. The experiments also show that producing more successful trajectories need not yield a better student. However, matched collection costs and training budgets are still needed to establish the contribution of guidance itself.
Critical strengths
-
Sections 3-4 organize the experiments around when guidance is introduced and how much information it provides. This offers a clear, interpretable framework for studying the use of privileged reference information.
-
Table 5 reports both data yield and student performance, showing that directly revealing a reference patch can produce more successful trajectories while reducing their learning value.
-
Section 7 examines whether guidance should remain in the training input and how gains vary across categories of issue information. This broadens the empirical analysis of how the choice of training inputs affects learning from trajectories.
Critical weaknesses
-
The central claim about the value of moderate guidance is confounded by collection and training budgets. Sections 5-6 compare guided sampling against only three unguided attempts per task, without additional unguided retries at the same call or token cost. Section 5.2 and Appendix A.3 also use two SFT epochs, so guidance strength changes task coverage, data mixture, and the number of training updates simultaneously. These comparisons do not isolate the benefit of guidance information from additional computation and training.
-
Table 2 draws training tasks from multiple software-engineering sources, but the paper does not describe deduplication against SWE-bench Verified and Multilingual at the issue, commit, or near-duplicate-patch level. Independence between training and evaluation therefore remains unconfirmed when reference patches are used.
-
The Pre/Post Keep-Remove comparisons in Table 3 explicitly use different experimental batches, and the historical Intra results differ from the defaults in the current visibility controls. The general conclusion that offline guidance should be retained therefore requires validation under fully matched conditions.
Questions for authors
-
At the same total collection-token cost, how does moderate guidance compare with additional unguided retries and direct patch distillation?
-
Does the nonmonotonic Utility trend persist when the number of added examples, task set, and training-token or update budget are matched across guidance levels?
-
Can you provide the deduplication rules and overlap statistics for training and evaluation, together with results after excluding any overlapping tasks?
-
Can you repeat the Pre/Post Keep-Remove comparisons with the same trajectory source, initialization, and evaluation batch to distinguish training variability from the effect of guidance visibility?
2 comments
Thanks again for your feedback
Dear Reviewer dcgc,
As the discussion phase wraps up, we wanted to express our sincere thanks again for your careful reading and helpful comments. We hope that our response above and the General Response have addressed the points you raised for your final evaluation.
Thanks for your review!
Thank you for the clear review and for highlighting our framework.
Weakness 1 and Question 1 (Additional unguided vs. guided rollout pass and token cost; Section 7.1, Table 4 & General Response #1): Across the 2,527 twice-failed tasks (35.45% Supply, 34.21% Utility), total token consumption across one additional unguided pass and guided Pre/Post L3 and L4 passes varies by less than 5% (5.07B to 5.32B tokens, including <0.8% hint overhead). Even so, a third unguided pass lifts Supply only to 37.00% and Utility to 35.09%, whereas moderate guidance lifts Supply to 47.18% to 49.48% and Utility to 40.48% (+ Pre L3) and 43.61% (+ Post L3); more explicit + Post L4 reaches 57.63% Supply yet drops to 39.97%, matching the L6 patch-distillation drop in our main sweep (34.71% in Post and 30.58% in Intra). Within a <5% token spread, moderate guidance clearly outperforms both extra unguided retries and overly explicit supervision.
Weakness 1 and Question 2 (Nonmonotonic Utility at matched trajectories and SFT updates; Appendix B.2, Table 8 & General Response #2): Training each unique trajectory for 2 epochs evaluates the balance between extra Supply and trajectory artificiality, whereas repeating the 1,480 baseline trajectories equates to 2.56 and 2.68 epochs. When we hold added trajectories (414 and 506), prioritized task overlap, and update steps (238 and 250) fixed, downsampled L5 drops to 37.09% in Pre and 32.96% in Post, clearly below Pre L3 (39.22%) and Post L3 (37.72%), while upsampled Baseline (38.97% / 36.84%) also stays below 2-epoch L3. This separates guidance quality from update count and confirms that the nonmonotonic Utility trend persists under matched SFT budgets.
Weakness 2 and Question 3 (Deduplication check; General Response #3): We audited all 4,000 training tasks against SWE-bench Verified (500) and Multilingual (298) using normalized repository names, PR/issue numbers, and normalized reference patches, confirming zero overlap between our training tasks and the evaluation benchmarks.
Weakness 3 and Question 4 (Keep/Remove matching and Table 5 Intra fix): Both points arose from draft mistakes that we have now corrected. First,
"cross-batch"in the caption and"historical/current"labels were editing errors: all 10 Pre/Post Keep/Remove pairs shared identical trajectories,Qwen3.5-9B-Baseinitialization, SFT settings, and evaluation setup (Keep wins 10/10 pairs, +1.88 to +6.77 pp, sign test p = 0.002). Second, Table 3 (now Table 5) had inadvertently listed an older deprecated run in theIntra Removed (default)column; updating L1 to L4 to our main Intra checkpoints (39.72%, 38.47%, 36.97%, 39.72%) aligns all tables. Our conclusion that task-specific offline hints should remain in the SFT input is therefore supported under matched conditions, distinct from level-dependent online reminder insertion.We hope these controlled experiments and manuscript corrections address all the confounding factors you raised, and we would be very grateful if you would reconsider your rating.