- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ReCAP, a framework designed to extract and reuse problem-level repair experience from entangled pull-request (PR) histories for automated continuous integration (CI) repair. To resolve the lack of explicit alignment between individual CI problems and developer edits, ReCAP performs bidirectional change attribution by reconciling a backward endpoint view (from the final patch) and a forward development view (across commit trajectories). The extracted records are abstracted into three hierarchical levels (L1 concrete edits, L2 contextual strategies, and L3 transferable patterns) and dynamically retrieved to guide repair agents sequentially. Evaluations on CI-REPAIR-BENCH across multiple backbones and agents show consistent Pass@1 improvements over baselines.
Critical strengths
-
Well-written paper and easy to follow. The problem formulation of learning from unaligned, noisy PR histories is practical and addresses a realistic gap in current coding agent memory systems.
-
The complementary design of combining backward endpoint patches and forward commit histories is intuitive and well-supported by instance-level overlap analyses.
-
Compelling empirical evidence showing that naive reuse of unaligned historical repair plans degrades performance below the no-experience baseline, validating the need for problem-level attribution.
Critical weaknesses
-
While the dynamic reuse stage introduces dependency expansion and recurring-problem clustering to predict hidden downstream failures, there are no ablation studies for these components. The current ablations in Tables 3 and 4 only test the two views and abstraction levels, leaving it unclear whether guessing unobserved problems actually helps or introduces false-positive edits.
-
The paper reports inference-time token costs in detail, but completely omits the offline overhead of building the memory bank. Extracting experience from historical PRs relies on a heavy, multi-stage prompting pipeline across intermediate commits, and there is also no direct evaluation of the intermediate attribution accuracy beyond end-to-end Pass@1.
-
Because the dataset split puts the earliest 30% of multi-instance repos into the history bank while leaving single-instance repos entirely in the test set, the evaluation mixes in-repository reuse with cold-start repositories. Reporting only aggregate metrics makes it unclear whether the highest-level abstractions truly generalize to unseen repositories without prior repo-specific history.
Questions for authors
-
How does the method perform if the hidden-problem prediction modules (dependency expansion and recurring problems) are removed so the agent only repairs directly observed failures? Also, what is the total token and monetary cost of running the offline extraction pipeline on the historical set?
-
Could the authors break down the test results between repositories that have prior instances in the history bank versus cold-start repositories that rely purely on cross-repository retrieval?
3 comments
Dear Reviewer, The author rebuttal period closes soon. We have submitted a detailed response addressing all your concerns and hope you have had a chance to review it.
If you have any remaining questions, we would be glad to clarify them.
Thank you for your time and consideration.
Thank you for your reply. My concern has been resolved and I have decided to increase my score to 3
We thank the reviewer for the constructive feedback. We address the questions below.
Q1. Hidden-problem prediction ablation and offline extraction cost.
Excluding hidden-problem prediction (dependency expansion and recurring-problem recall) drops Pass@1 by 7.2 pp (53.6% vs. 46.4%) on 110 evaluation instances.
Hidden-problem prediction. We select 110 evaluation instances and disable dependency expansion and recurring-problem recall, restricting the agent to directly observed CI problems. All other components (experience retrieval, applicability filtering, repair ordering) remain unchanged. Both configurations use mini-SWE-agent with DeepSeek-V4-Flash under identical execution budgets.
The 7.2 pp gap shows that hidden-problem prediction helps in cases with multiple CI problems, where dependent or recurring failures are not visible in CI logs until earlier problems are resolved, and historical experience from similar repairs guides their identification in advance.
Offline extraction cost. The total one-time preprocessing cost across the full 565-instance benchmark is $15.96, covering CI log analysis for all 565 instances and experience reconstruction for the 157 historical instances. This cost does not scale with the number of evaluation instances at inference time.
Q2. Repository-history availability breakdown.
ReCAP consistently outperforms No-Experience across both repository groups and both backbone models, confirming that problem-level repair experience transfers effectively even without target-repository history.
We partition the 408 evaluation instances into History-Available (375 instances, at least one historical repair in memory) and Cold-Start (33 instances, no target-repository history). Cold-Start instances rely exclusively on cross-repository retrieval. We report existing evaluation outcomes for both models without re-running experiments.
Gains persist in the Cold-Start setting (+15.1/+9.1 pp) even without any target-repository history, but the smaller Cold-Start subset (33 instances) makes it difficult to draw strong conclusions. The consistent gains across both models show that Transferable Repair Patterns can generalize to unseen repositories.