- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes XRepoSkill, an LLM-based pipeline that extracts rules from successful–failed coding-agent trajectories, scores their association with historical success, merges recurring rules across repositories, and injects selected rules into the agent prompt. Experiments report improvements on SWE-bench Pro and DeepSWE.
Critical strengths
The evaluation uses repository-disjoint benchmarks and identical underlying agents across methods. The repeated DeepSWE experiment suggests that the final rule prompt can improve agent performance.
Critical weaknesses
-
The claimed rule verification is observational and heavily confounded by backbone capability. Successful and failed trajectories are generated by different models, with successful trajectories intentionally selected from stronger models. Historical association therefore does not establish that a rule causes improved resolution.
-
Rule generation, predicate construction, effect estimation, clustering, and evidence scoring reuse the same trajectory pool. There is no genuinely held-out rule-validation set or correction for adaptive multiple testing. Furthermore, gain and z-score are not used to reject ineffective rules, despite the paper describing retained rules as verified.
-
The evaluation lacks the most important simple baseline: a manually written or automatically selected generic software-engineering checklist with matched prompt length and inference calls. The most frequently selected rules are standard practices such as reproducing the failure and inspecting tests. Without this baseline, the necessity of XRepoSkill’s elaborate learning pipeline is unclear.
Questions for authors
- Does XRepoSkill outperform a compact generic SWE checklist under matched prompt length and selection-call budgets?
- Do rule effects remain after controlling for backbone identity and evaluating predicates on entirely held-out issues?
- Are the improvements over the strongest baseline statistically significant, particularly the 2.1-point gain on SWE-bench Pro?
3 comments
[Q3] Are the improvements over the strongest baseline statistically significant, particularly the 2.1-point gain on SWE-bench Pro?
Six of the eight comparisons of XRepoSkill against mini-SWE-agent and the three skill-learning baselines across both benchmarks remain statistically significant at the 5% level after applying Holm–Bonferroni correction to the p-values from the eight McNemar tests.
XRepoSkill achieves the highest resolution rates on both benchmarks, with statistically significant improvements over both leading skill-learning baselines on at least one benchmark each. Specifically, XRepoSkill significantly outperforms STAIR on DeepSWE and Trace2Skill on SWE-bench Pro. Its gains over STAIR on SWE-bench Pro (+2.1 pp) and Trace2Skill on DeepSWE (+5.0 pp) do not reach statistical significance.
Furthermore, gain and z-score are not used to reject ineffective rules, despite the paper describing retained rules as verified.
Sorry for being unclear. We clarify that gain and z-scores quantify observational associations rather than establish statistical significance or causality. XRepoSkill deliberately uses these scores for issue-adaptive rule selection, not as hard retention thresholds, because a rule may be useful in specific contexts despite a negative average estimated gain.
To validate this design choice, we added an ablation that removes rules with negative gains, reducing DeepSWE resolution from 58.4% to 54.9% with DeepSeek as the backbone. Separately, removing cross-repository generalization reduces resolution to 51.3% (Table 4). These results support retaining rules based on cross-repository recurrence and using evidence scores for issue-adaptive prioritization. We will clarify that “verification” refers to empirical behavioral evidence and cross-repository support, rather than formal statistical or causal verification.
Thank you for the detailed response. All my concerns have been addressed, and I will raise my score accordingly.
Thank you for your constructive feedback. We have addressed all your questions as follows.
Yes. Following your suggestion, we have added a generic SWE checklist baseline on DeepSWE with DeepSeek, matching both prompt length and selection-call budget.
We prompted the skill-learning LLM to generate 36 generic SWE practices from its prior knowledge, without access to any trajectories. For each issue, the DeepSeek backbone selects up to three items using the same Stage 3 selection procedure as XRepoSkill, with the injected guidance matched in length to XRepoSkill's average.
The generic checklist achieves 49.6% resolution, compared with 46.0% for mini-SWE-agent and 58.4% for XRepoSkill. Thus, XRepoSkill outperforms the generic checklist by 8.8 percentage points under matched prompt length and selection-call budgets, demonstrating the added value of trajectory-learned, issue-adaptive guidance beyond generic SWE best practices.
Thank you for the question. We conducted two complementary analyses to examine whether rule associations remain robust after controlling for backbone identity and generalize to held-out issues.
(1) Controlling for backbone identity. We re-estimated each rule's association with issue resolution using both issue and backbone fixed effects, with standard errors clustered by issue:
yt=βrpr(t)+αissue(t)+γbackbone(t)+εt.
Here, βr captures the association between predicate satisfaction and issue resolution after accounting for issue difficulty and backbone identity. We used the adjusted estimates and their z-scores in place of the original rule evidence scores.
Table R3: Rule evidence before and after controlling for backbone identity.
After adjustment, the median estimated association is +3.9 percentage points, and 15 of the 19 most frequently selected rules have positive estimates. Importantly, using backbone-adjusted evidence preserves downstream resolution performance (57.5% vs. 58.4%).
(2) Held-out issue validation. We randomly held out 100 of the 500 issues and reconstructed the rule library using only the remaining 400 issues, including rule discovery, predicate construction, evidence estimation, and cross-repository clustering. The learned rules and predicates were frozen before evaluation on the held-out issues.
The reconstructed library contains 29 transferable rules. Of the 20 rules with positive training-set gains, 18 (90.0%) remain positive on held-out issues, including all seven rules with training gains above +10 percentage points. Overall, 24/29 rules (82.8%) preserve the sign of their estimated associations.
Together, these results support the robustness of the learned rule associations to backbone adjustment and issue holdout. While these associations are observational rather than causal, our downstream evaluation independently demonstrates the effectiveness of the resulting skill guidance on unseen repositories.