acceptodds
Under review as a conference paper at ICLR 2027

ReCAP: Learning Problem-Level Repair Experience from Entangled Pull-Request Histories

Abstract

Large language model (LLM) agents increasingly reuse experience from prior tasks, but existing methods generally assume that each experience is already aligned with a known problem and solution. Real software development histories lack this alignment: a completed pull request (PR) may contain multiple continuous integration (CI) problems, intermediate repair attempts, reverted edits, and unrelated changes. The final passing revision shows that the CI workflow passes after all changes are applied, but not which changes resolved each problem. We introduce ReCAP, which learns problem-level repair experience from such histories. ReCAP attributes developer changes to CI problems through two views: the endpoint view reasons backward from changes retained in the passing revision, while the development view traces forward through commit history to capture how repairs evolved. It reconciles the two views using CI execution evidence and represents each repair at three abstraction levels, from concrete changes to transferable repair patterns. For a new CI failure, ReCAP retrieves and filters problem-specific experience to guide repair. We evaluate ReCAP on CI-REPAIR-BENCH, containing 565 PR-level repairs from 101 repositories across 12 failure categories. With mini-SWE-agent, ReCAP improves Pass@1 from 19.6% to 31.9% using MiniMax-M2.5 and from 23.3% to 32.8% using DeepSeek-V4-Flash; with Codex, it improves Pass@1 from 15.5% to 27.5% on a matched subset. Combining endpoint and development evidence consistently outperforms either view alone, showing that recovering problem–change alignment allows noisy development histories to be reused for CI repair.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM Agents, Experience Learning, Task-Structure Learning, Software Repair, Continuous Integration, Agent Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_u2PrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ReCAP, a framework designed to extract and reuse problem-level repair experience from entangled pull-request (PR) histories for automated continuous integration (CI) repair. To resolve the lack of explicit alignment between individual CI problems and developer edits, ReCAP performs bidirectional change attribution by reconciling a backward endpoint view (from the final patch) and a forward development view (across commit trajectories). The extracted records are abstracted into three hierarchical levels (L1 concrete edits, L2 contextual strategies, and L3 transferable patterns) and dynamically retrieved to guide repair agents sequentially. Evaluations on CI-REPAIR-BENCH across multiple backbones and agents show consistent Pass@1 improvements over baselines.

Critical strengths
  1. Well-written paper and easy to follow. The problem formulation of learning from unaligned, noisy PR histories is practical and addresses a realistic gap in current coding agent memory systems.

  2. The complementary design of combining backward endpoint patches and forward commit histories is intuitive and well-supported by instance-level overlap analyses.

  3. Compelling empirical evidence showing that naive reuse of unaligned historical repair plans degrades performance below the no-experience baseline, validating the need for problem-level attribution.

Critical weaknesses
  1. While the dynamic reuse stage introduces dependency expansion and recurring-problem clustering to predict hidden downstream failures, there are no ablation studies for these components. The current ablations in Tables 3 and 4 only test the two views and abstraction levels, leaving it unclear whether guessing unobserved problems actually helps or introduces false-positive edits.

  2. The paper reports inference-time token costs in detail, but completely omits the offline overhead of building the memory bank. Extracting experience from historical PRs relies on a heavy, multi-stage prompting pipeline across intermediate commits, and there is also no direct evaluation of the intermediate attribution accuracy beyond end-to-end Pass@1.

  3. Because the dataset split puts the earliest 30% of multi-instance repos into the history bank while leaving single-instance repos entirely in the test set, the evaluation mixes in-repository reuse with cold-start repositories. Reporting only aggregate metrics makes it unclear whether the highest-level abstractions truly generalize to unseen repositories without prior repo-specific history.

Questions for authors
  1. How does the method perform if the hidden-problem prediction modules (dependency expansion and recurring problems) are removed so the agent only repairs directly observed failures? Also, what is the total token and monetary cost of running the offline extraction pipeline on the historical set?

  2. Could the authors break down the test results between repositories that have prior instances in the history bank versus cold-start repositories that rely purely on cross-repository retrieval?

3 comments

We thank the reviewer for the constructive feedback. We address the questions below.

Q1. Hidden-problem prediction ablation and offline extraction cost.

Excluding hidden-problem prediction (dependency expansion and recurring-problem recall) drops Pass@1 by 7.2 pp (53.6% vs. 46.4%) on 110 evaluation instances.

Hidden-problem prediction. We select 110 evaluation instances and disable dependency expansion and recurring-problem recall, restricting the agent to directly observed CI problems. All other components (experience retrieval, applicability filtering, repair ordering) remain unchanged. Both configurations use mini-SWE-agent with DeepSeek-V4-Flash under identical execution budgets.

ConfigurationPass@1
Full ReCAP53.6% (59/110)
ReCAP without Hidden-Problem Prediction46.4% (51/110)

The 7.2 pp gap shows that hidden-problem prediction helps in cases with multiple CI problems, where dependent or recurring failures are not visible in CI logs until earlier problems are resolved, and historical experience from similar repairs guides their identification in advance.

Offline extraction cost. The total one-time preprocessing cost across the full 565-instance benchmark is $15.96, covering CI log analysis for all 565 instances and experience reconstruction for the 157 historical instances. This cost does not scale with the number of evaluation instances at inference time.

StageInstancesTotal tokensCost (USD)
CI log and workflow analysis56539.3M$6.01
Experience reconstruction and three-level abstraction15780.0M$9.95

Q2. Repository-history availability breakdown.

ReCAP consistently outperforms No-Experience across both repository groups and both backbone models, confirming that problem-level repair experience transfers effectively even without target-repository history.

We partition the 408 evaluation instances into History-Available (375 instances, at least one historical repair in memory) and Cold-Start (33 instances, no target-repository history). Cold-Start instances rely exclusively on cross-repository retrieval. We report existing evaluation outcomes for both models without re-running experiments.

GroupModelInstancesNo-Experience (%)ReCAP (%)Gain (pp)
History-AvailableMiniMax-M2.537519.731.7+12.0
DeepSeek-V4-Flash37523.533.1+9.6
Cold-StartMiniMax-M2.53318.233.3+15.1
DeepSeek-V4-Flash3321.230.3+9.1

Gains persist in the Cold-Start setting (+15.1/+9.1 pp) even without any target-repository history, but the smaller Cold-Start subset (33 instances) makes it difficult to draw strong conclusions. The consistent gains across both models show that Transferable Repair Patterns can generalize to unseen repositories.

Dear Reviewer, The author rebuttal period closes soon. We have submitted a detailed response addressing all your concerns and hope you have had a chance to review it.

If you have any remaining questions, we would be glad to clarify them.

Thank you for your time and consideration.

Thank you for your reply. My concern has been resolved and I have decided to increase my score to 3

Reviewer_xm6zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes ReCAP, an LLM-based pipeline that decomposes completed CI repair histories into problem-level repair records using endpoint patches and commit histories. It stores each repair at three abstraction levels and retrieves relevant records to guide future CI repairs. End-to-end experiments report sizable Pass@1 improvements on CI-Repair-Bench.

Critical strengths

The paper identifies a practically meaningful problem: real PR histories contain multiple failures, unrelated edits, and intermediate attempts, making direct experience reuse difficult. The end-to-end improvements are sizable and are evaluated using complete CI workflows across multiple models and two agent scaffolds.

Critical weaknesses
  1. The paper does not directly validate its central contribution: problem–repair attribution. There is no annotated attribution test set, precision/recall measurement, human assessment, or commit-level execution validation. End-to-end repair success cannot establish that the reconstructed problem–change mappings are correct or that the proposed multi-view attribution is responsible for the gains.

  2. The baseline comparison is substantially under-controlled. ReCAP retrieves up to five records from each of three levels, performs additional LLM filtering and problem expansion, and uses nearly twice the input tokens of the no-experience baseline. The sole experience baseline retrieves two demonstrations and performs worse than no experience. Strong memory methods and matched-context/compute baselines are missing.

  3. Most historical and evaluation repairs come from the same repositories, and L1/L2 explicitly exploit repository- and workflow-specific recurrence. Consequently, the results primarily demonstrate temporal reuse within repositories, while evidence for genuinely transferable cross-repository experience is limited.

Questions for authors
  1. What is the accuracy of the reconstructed problem–repair associations against independently annotated ground truth?
  2. Does ReCAP still outperform strong experience baselines when retrieval count, context length, LLM calls, and inference budget are matched?
  3. What is performance when all historical experience from the target repository is excluded?
4 comments

Q1. Accuracy of reconstructed problem–repair associations against independently annotated ground truth.

Full ReCAP achieves 97.2% recall and 66.2% precision across 30 randomly selected historical repair episodes from the 157-instance historical partition.

An author independently identifies distinct CI problems and their corresponding repair changes from CI logs, commit histories, and workflow configurations, yielding 141 verified associations. We compare ReCAP’s 207 reconstructed associations against this reference using semantic matching with failure-type normalization, reporting micro-averaged precision, recall, and F1.

Attribution MethodPrecision (%)Recall (%)F1 (%)
Full ReCAP66.297.278.7

Full ReCAP correctly recovers 137 of 141 author-verified associations. The lower precision is partly attributable to redundant reconstructions of the same underlying problem across different files, as well as associations that do not match the author reference. These results confirm strong attribution coverage while highlighting opportunities to improve consolidation precision.

Q2. Performance against matched-budget experience baselines.

ReCAP (56% Pass@1) outperforms Whole-Repair-History (40%) while using fewer LLM calls, tokens, and lower inference cost on the same 50 instances.

We introduce a Whole-Repair-History baseline that mirrors ReCAP’s retrieval and inference setup but skips problem-level experience reconstruction, to isolate whether the gains come from attribution rather than additional computation. Both methods use the same 157 historical PRs and CI artifacts. Whole-Repair-History organizes each PR’s changes into dependency-aware chunks, summarizes them, and aggregates the results into the same three memory representations (L1/L2/L3) as ReCAP, without reconstructing individual problem–repair associations.

MethodPass@1 (%)LLM CallsAvg. Tokens (M)Avg. Cost ($)
Whole-Repair-History4058.51.8330.07
Full ReCAP5630.91.2420.04

ReCAP's improvement is not due to a larger inference budget but to more effective problem-level experience reconstruction.

Q3. Performance when all historical experience from the target repository is excluded.

Cross-repository ReCAP achieves 42% Pass@1 (+18 pp over No-Experience); L3-only achieves 47% (+23 pp), only 2 pp below within-repository ReCAP (49%).

We evaluate on 100 instances from 30 target repositories, excluding their histories from memory and retaining 110 historical repairs from 45 disjoint source repositories. Both configurations use mini-SWE-agent with DeepSeek-V4-Flash under identical execution budgets.

ConfigurationPass@1 (%)Δ vs. No-Experience (pp)
No-Experience24—
ReCAP (Cross-Repo, L1/L2/L3)42+18
ReCAP (Cross-Repo, L3-only)47+23
Full ReCAP (within-repository)49+25

L3-only outperforms L1+L2+L3 cross-repository because concrete and contextual experiences provide repository-specific guidance when available, but abstract patterns generalize better when such history is absent.

Thank you for the detailed response. My concerns regarding Q1 and Q2 have been addressed.

For Q3, I have two remaining questions:

  • In the original method, L1/L2 retrieval is restricted to the target repository. How were L1/L2 experiences retrieved in the cross-repository experiment?

  • On 100 instances, 47% versus 42% corresponds to only five additional successful repairs. This difference alone is insufficient to conclude that including L1/L2 causes negative transfer. Please either evaluate this comparison on a larger test set or report a statistical significance test.

Q3.1. In the original method, L1 and L2 retrieval is restricted to the target repository. In the cross-repository setting, no target-repository history exists in memory, so L1 and L2 return empty results by design. The cross-repository L1/L2/L3 and L3-only configurations therefore rely on the same L3 retrieval, which explains why their performance is close.

Q3.2. We expanded the evaluation to 126 instances, the complete population from the 30 target repositories under strict repository-level disjointness, and reran all four configurations. The remaining 282 instances belong to the 45 source repositories and cannot be used without violating cross-repository separation.

ConfigurationPass@1 (%)Δ vs. No-Experience (pp)McNemar p vs. No-Experience
No-Experience24.6 (31/126)——
ReCAP (Cross-Repository, L1/L2/L3)38.9 (49/126)+14.30.0009
ReCAP (Cross-Repository, L3-only)37.3 (47/126)+12.70.0037
Full ReCAP (within-repository)42.1 (53/126)+17.5<0.0001

Both cross-repository configurations significantly outperform No-Experience (p = 0.0009 and p = 0.0037), confirming that L3 abstract patterns transfer across repositories. The 1.6 pp difference between L1/L2/L3 and L3-only is not statistically significant (exact McNemar p = 0.774). Under repository-level disjointness, L1/L2 return empty results and both configurations rely solely on L3 retrieval. We therefore do not claim negative transfer from including L1/L2 for cross-repository evaluation.

Thank you, I've updated my score.

Comments on the paper

6 comments

We thank the reviewer for the constructive feedback. We address the four questions across two comments.

Q1. Pass@1 with whole-PR baseline, and attribution precision and recall.

(i) Whole-Repair-History baseline.

Full ReCAP achieves 56% Pass@1, outperforming Whole-Repair-History (40%) by 16 pp and No-Experience (44%) by 12 pp on a 50-instance subset.

The baseline organizes each PR's changes into dependency-aware, token-bounded chunks, summarizes them individually, and aggregates them with CI and workflow information into a single experience with the same three representations as ReCAP. Unlike ReCAP, it does not reconstruct problem–repair associations. Both methods use the same historical data, retrieval mechanism, and downstream repair agent (mini-SWE-agent with DeepSeek-V4-Flash).

MethodPass@1 (%)
No-Experience44 (22/50)
Historical-Plan30 (15/50)
Whole-Repair-History40 (20/50)
Full ReCAP56 (28/50)

Whole-Repair-History also performs worse than No-Experience (40% vs. 44%), showing that providing more historical information without problem-level attribution does not improve agent performance and can degrade it.

(ii) Human-verified attribution accuracy.

Full ReCAP achieves 97.2% recall and 66.2% precision across 30 randomly selected historical repair episodes from the 157-instance historical partition.

For each episode, an author independently identifies distinct CI problems and their corresponding repair changes from CI logs, commit histories, and workflow configurations, yielding 141 verified associations. We compare ReCAP's 207 reconstructed associations against this reference using semantic matching with failure-type normalization and a minimum 65% repair-file overlap criterion, reporting micro-averaged precision, recall, and F1.

ConfigurationPrecision (%)Recall (%)F1 (%)
Full ReCAP66.297.278.7

Full ReCAP correctly recovers 137 of 141 author-verified associations. The lower precision (66.2%) is partly attributable to redundant reconstructions of the same underlying problem across different files.

Q2. Individual ablation of each dynamic reuse component.

Removing repair ordering causes the largest Pass@1 drop (−18 pp), followed by applicability filtering and recurring-problem recall (−15 pp each), and dependency expansion (−10 pp).

We conduct a leave-one-component-out ablation on 100 CI repair instances, keeping the historical experience pool, retrieval settings, model (mini-SWE-agent with DeepSeek-V4-Flash), and execution budget fixed.

ConfigurationPass@1 (%)Δ (pp)
Full ReCAP480
Without Repair Ordering30−18
Without Applicability Filtering33−15
Without Recurring-Problem Recall33−15
Without Dependency Expansion38−10

These results suggest that all four components contribute to repair performance, with ordering having the largest observed impact. Applicability filtering and recurring-problem recall help select relevant experiences and recover additional problems, respectively, while dependency expansion helps anticipate downstream failures.

Q3. Results per failure type, and gains excluding linting and formatting.

(i) Performance across failure types.

ReCAP improves Pass@1 in 11 of 12 categories with both backbone models. Beyond linting and formatting, notable gains occur in Test Failure (+10.4/+8.1 pp), Dependency Issues (+9.1/+7.6 pp), Type Checking (+6.2/+4.0 pp), and Runtime Error (+6.0/+3.7 pp). Since instances may have multiple failure labels, the categories are not mutually exclusive.

Failure TypeNMiniMax No-ExpMiniMax ReCAPΔDeepSeek No-ExpDeepSeek ReCAPΔ
Test Failure2228.118.5+10.410.418.5+8.1
Code Formatting21521.932.1+10.225.634.4+8.8
Linting21017.126.7+9.619.529.0+9.5
Dependency Issues1974.613.7+9.17.114.7+7.6
Type Checking1769.115.3+6.210.814.8+4.0
Runtime Error1337.513.5+6.09.813.5+3.7
Configuration Error12710.211.8+1.67.911.0+3.1
Doc/Docstring5610.717.9+7.210.714.3+3.6
Assertion Error333.09.1+6.16.112.1+6.0
Package Install Error287.110.7+3.610.714.3+3.6
Environment Error110.09.1+9.10.09.1+9.1
Syntax Error1030.020.0−10.030.030.00.0
Overall40819.631.9+12.323.332.8+9.5

(ii) Excluding linting and formatting.

Excluding all instances involving linting or formatting (including overlapping labels) leaves 129 of 408 instances. ReCAP still achieves +13.2 pp with MiniMax-M2.5 and +9.3 pp with DeepSeek-V4-Flash, comparable to the overall gains of +12.3 pp and +9.5 pp.

ModelNNo-Experience (%)Full ReCAP (%)Gain (pp)
MiniMax-M2.512916.329.5+13.2
DeepSeek-V4-Flash12918.627.9+9.3

Q4. Cross-repository and cold-start performance, and training-data contamination.

Cross-repository ReCAP achieves 38.9% Pass@1 (+14.3 pp over No-Experience), and L3-only achieves 37.3% (+12.7 pp), only 4.8 pp below within-repository ReCAP (42.1%), demonstrating that abstract repair patterns transfer effectively even without any target-repository history.

We evaluate on 126 instances from 30 target repositories, the complete population under strict repository-level disjointness, excluding all target-repository histories from memory and retaining 110 historical repairs from 45 disjoint source repositories. We compare four configurations using mini-SWE-agent with DeepSeek-V4-Flash: (1) No-Experience, (2) cross-repository ReCAP (L1/L2/L3), (3) cross-repository L3-only, and (4) Full ReCAP with within-repository history as reference. To assess whether the difference between L1/L2/L3 and L3-only is meaningful, we apply a two-sided exact McNemar test on the 126 paired per-instance outcomes.

ConfigurationPass@1 (%)Δ vs. No-Experience (pp)McNemar p vs. No-Experience
No-Experience24.6 (31/126)——
ReCAP (Cross-Repository, L1/L2/L3)38.9 (49/126)+14.30.0009
ReCAP (Cross-Repository, L3-only)37.3 (47/126)+12.70.0037
Full ReCAP (within-repository)42.1 (53/126)+17.5<0.0001

Both cross-repository configurations significantly outperform No-Experience (p = 0.0009 and p = 0.0037), confirming that L3 abstract patterns transfer across repositories. The 2 pp difference between L1/L2/L3 and L3-only is not statistically significant (exact McNemar p = 0.774, likely due to runtime variation), as L1/L2 return empty results in the cross-repository setting and both configurations rely solely on L3 retrieval. In the original method, L1 and L2 retrieval is restricted to the target repository. In the cross-repository setting, no target-repository history exists in memory, so L1 and L2 return empty results by design. The cross-repository L1/L2/L3 and L3-only configurations therefore rely on the same L3 retrieval, which explains why their performance is close. We therefore do not claim negative transfer from including L1/L2.

Training-data contamination. Our evaluation data were collected after May 2025, the reported training-data date for the LLMs. We also preserve the historical/evaluation split, exclude target-repository histories from memory, and withhold ground-truth repair patches from the agent. ReCAP does not fine-tune the backbone model. These safeguards reduce the risk of training-data contamination and evaluation leakage.

We have updated our Q4 response with results from the expanded 126-instance evaluation and statistical significance tests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.