acceptodds
Under review as a conference paper at ICLR 2027

ContextForge: Learning What to Retrieve From Diverse Validated Patches

Abstract

Search subagents can reduce the expensive repository performed by coding agents, yet they are commonly trained and evaluated using overlap with locations modified in a single solution patch. This proxy omits useful context in unmodified callers, interfaces, tests, and invariants. It also overlooks that there can be many alternative test-valid repairs that depend on different repository evidence. To address these limitations, we introduce a novel reward function for context retrieval that scores edit locations, supporting code, and explanations using patch-conditioned information gains averaged over multiple test-validated repairs. We implement this reward in ContextForge, a training framework that constructs diverse repair sets through offline generation and test execution. Using GRPO, we train a Qwen3.5-4B search agent and evaluate it with OpenHands and Mini-SWE-Agent on SWE-bench Pro, Verified, and Multilingual. Across all six benchmark-scaffold settings, ContextForge achieves the lowest inference cost and best or tied Pass@1 among evaluated context retrieval agents, and reduces end-to-end cost by up to 29.1% compared to the no-subagent baseline.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Software Engineering, Coding Agent, Reinforcement Learning, Code Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jcqzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ContextForge addresses the limitations of training code retrieval agents against a single reference patch, which may omit useful supporting code and overlook alternative valid repairs. It generates diverse patches, filters them by test results, and uses patch-conditioned information gain to separately reward retrieved edit locations, supporting context, and explanations according to how much they help predict multiple validated repairs. The authors train a Qwen3.5-4B retrieval agent with GRPO and evaluate it with GPT-5.4-high on SWE-bench Pro, Verified, and Multilingual using Mini-SWE-Agent and OpenHands. Across the six benchmark–scaffold settings, ContextForge matches or improves Pass@1 over the no-subagent baseline while reducing end-to-end inference cost by up to 29.1%. Its supervision depends on the quality and coverage of available tests, however, and the authors note that familiarity with benchmark repositories may limit how well the results generalize to unfamiliar codebases.

Critical strengths
  1. The authors propose a subagent-centric method, which is an underexplored field.
  2. This paper presents a novel framework with a patch-conditioned information-gain (PCIG) reward, combining edit locations, context and explanations.
  3. This paper conducts comprehensive evaluation on three SWE benchmarks, covering multiple languages and scenarios.
Critical weaknesses
  1. Although CONTEXTFORGE consistently reports lower end-to-end inference costs, the paper does not adequately explain the source of these savings. The reported reductions in main-agent token usage and illustrative case studies are suggestive, but the paper does not provide an aggregate analysis that attributes the savings to reduced repository exploration or separates the method’s effect from variation in agent trajectories. This makes it difficult to determine whether the cost gains are a robust consequence of the approach.
  2. In addition, the cost analysis focuses on inference and omits the cost of training the retrieval policy. In particular, computing PCIG rewards requires the frozen Qwen3-32B scorer to evaluate candidate-patch log probabilities under multiple conditioning contexts, with these evaluations repeated across repair candidates, rollout groups, and reward components. This repeated scoring may be computationally expensive, yet the paper reports no scorer-token counts, accelerator-hours, or training-cost estimate, nor an amortized analysis comparing this upfront cost with the inference savings.
  3. Finally, the paper states that LLMs were used in patch generation, reward scoring, retrieval, downstream editing, and parts of evaluation, but provides limited detail on independent human auditing of these outputs and experimental results. More information about oversight and verification is needed to assess the reliability and reproducibility of the findings.
Questions for authors

See the above weakness.

4 comments

We are grateful for your effort and time in leaving a thorough review!

W1. Sources and robustness of inference-cost savings.

As suggested by the reviewer, we provide a breakdown of the 17.2% reduction in agent cost. As shown in the table, the cost of initial exploration decreases by 7.1%, while later exploration decreases by 22.2%. The costs associated with editing, review, and completion decrease by 29.2%, whereas testing and environment costs remain unchanged.

Metric/taskBaselineContextForge
Initial exploration calls19.4718.71
Initial exploration cost$0.14$0.13
Later exploration cost$0.09$0.07
Editing, review & completion$0.24$0.17
Testing & environment$0.11$0.11
Agent cost$0.58$0.48

W2. Training cost.

Our candidate generation cost is approximately $1.3K in GPT-5.4 API usage. Our full training required approximately 84 hours on a 4 x H100 node. The training pipeline assigned four GPUs: one for policy optimization, one for rollout generation, and two for reward scoring. We note that these measurements reflect our current implementation rather than an optimized minimum. For a fair comparison across variants, we use the same training data and number of training steps.

W3. Verification and human oversight.

Thank you for raising this. While LLMs are used in patch generation, retrieval, and downstream editing, the effectiveness of our methodology is validated empirically on public benchmarks using Pass@1 and cost (Section 4.1), which are deterministic metrics that cannot be cheated. Thus, our main result does not rely on subjective LLM judgements or human evaluation. Our reward scoring does not use an LLM as a judge. Instead, it deterministically computes the output likelihood of a given patch under different retrieved contexts via teacher-forced inference, without eliciting any subjective judgment from the model. In addition, training patches undergo separate automated validation against the full provided test suite, with invalid patches and test-file modifications rejected (Section 3.5 and Appendix A.1). To support reproducibility and independent verification, we include our training and evaluation code in the anonymous repo and are working toward a public release of the code and models.

We conducted an additional evaluation:

Matched training compute. We compared ContextForge with adapted CodeScout on Qwen3.5-4B using the same training GPU-hour budget. At this budget, ContextForge completed half of its full training steps and matched CodeScout’s 65.3% Pass@1, while lowering inference cost from \0.54to0.54 to \\0.49 per task.

Thank you for your response. I will maintain my initial score.

Thank you for the authors' response. I will maintain my initial score.

Reviewer_kAwYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes "ContextForge", a framework that trains a small LLM code-retrieval agent using GRPO to identify useful code context for fixing issues in repositories. It proposes a reward that quantifies how much a proposed retrieved segment increase the likelihood of a frozen 32B model. Across the proposed benchmarks it achieves modest improvements in pass rates.

Critical strengths

o. Inference efficiency: the model consistently achieves the lowest cost in all the tasks, including substantial reduction in some of them. These gains do not sacrifice the pass rate.

o. Efficient test-time deployment: Once the model is trained, only the final lightweight retrieval model is required. This makes it attractive for deployment.

Critical weaknesses

Missing ablations

There are missing ablations that could support the contribution of the proposed method:

o. Base models: CodeScout is based on Qwen3 while this paper uses 3.5. Therefore, this comparison does not isolate the improvements due to the proposed training method. A comparison with the same backbone would strengthen the claims.

o. Pretrained baseline: the performance of Qwen3.5-4B before GRPO training is not reported. This makes it unclear how much the training improves upon the base model with the same harness, prompts, etc.

o. No report of the retrieval quality: the results quantify the pass rate at the cost of a given task. GPT 5.4 could be doing a lot of the heavy lifting. A standalone evaluation of the retrieval quality could help evaluate how useful if the information retrieved by the agent.

o. Diverse patches: removing the diversity in patches drops the scores minimally. Repeated runs to get uncertainty metrics would be useful to asses if this is an important feature.

o. perhaps a weaker downstream model (vs 5.4) could see a lot higher benefits from this approach.

PCIG

o. Likelihood: an increased likelihood does not necessarily correspond to retrieve utility. The authors should investigate whether PCIG correlates with downstream repair success.

o. Averaging: information is averaged across all valid patches. This penalizes patches that support one correct repair but not to alternative. It would be useful to compare other aggregation strategies.

o. sigmoid: It is not obvious to me why this is the default choice, since the function saturates.

Overall

The improvements in repair success are marginal and the training costs are not accounted for so a practical trade-off between this approach and other can be established.

Questions for authors

o. Why was the 32B chosen as the scoring model? This choice seems arbitrary.

o. Are the scorer's preferences aligned with the downstream model?

4 comments

We really appreciate your time and effort in leaving constructive comments!

W1. Additional ablations and retrieval evidence.

Same backbone and untrained controls. Following the reviewer’s suggestion, we compare the untrained Qwen3.5-4B model, CodeScout adapted to Qwen3.5-4B, and ContextForge. The trained variants use the same initialization, training tasks, prompts, tools, update budget, downstream editor, and output format. ContextForge achieves the best Pass@1 and Cost/task across the three settings.

Pass@1Cost/task
Qwen3.5-4B64.0$0.52
Adapted CodeScout65.3$0.54
Full ContextForge68.2$0.48

Retrieval quality. We argue that pass rate and inference cost are the most relevant measures of context retrieval quality: providing high-quality context will allow an agent to implement a correct solution with less unnecessary work. Any metric that is applied directly to retrieved context, such as F1 on patch location prediction, can only be an indirect proxy for retrieval quality, and we show these metrics do not correlate with pass@1 or cost (see W2). In our repeated evaluation, ContextForge maintains Pass@1 (68.2% v.s. 68.0%) while reducing cost by 17.2% (from $0.58 to $0.48 per task). To assess how ContextForge’s retrieval reduces the work required of GPT-5.4, we further investigate the costs of initial exploration, follow-up exploration, and editing. As shown in the table, ContextForge reduces initial and later exploration costs by 7.1% and 22.2%, respectively, and editing cost by 29.2%. Notably, ContextForge rollouts also modify 13% fewer lines of code on average, suggesting that the retrieved context helps the downstream agent make more targeted edits.

Metric/taskBaselineContextForge
Initial exploration calls19.4718.71
Initial exploration cost$0.14$0.13
Later exploration cost$0.09$0.07
Editing, review & completion$0.24$0.17
Testing & environment$0.11$0.11
Agent cost$0.58$0.48

Uncertainty for diverse patches. Across three evaluation runs on the same 150 tasks, the diverse-reference supervision achieves 68.22% Pass@1, compared with 67.33% for single-reference supervision: a gain of 0.89% with a paired sign test shows that diverse supervision wins on significantly more tasks than it loses on (p = 0.043).

To further investigate its effect on retrieval, we analyze retrieved files and the main agent’s final submitted patch within the same execution in the following table. These results support broader retrieval coverage aligned with downstream edit locations: diverse supervision returns 0.86 more files and increases coverage of final-patch files by 14.87%.

MetricDiverseSingleDifference (95% CI)
Coverage of final-patch files65.13%50.26%+14.87 [10.36, 19.57]
First-call coverage of final-patch files61.43%47.88%+13.55 [9.04, 18.28]
Mean retrieved files per execution2.851.99+0.86 [0.67, 1.06]

Other downstream model. We further evaluate ContextForge with GPT-6.1-terra and compare it with no-subagent and Qwen3.5-4B baselines on the same 150 tasks we used in the repeated evaluation. Relative to no-subagent and Qwen3.5-4B, ContextForge achieves approximately 4.8% and 6.6% lower cost, respectively.

Pass@1Cost
no-subagent70.6%$4.024
Qwen3.5-4B73.3%$4.102
Full ContextForge73.3%$3.832

W2/Q2. PCIG and its alignment with repair utility.

Likelihood and downstream utility. We agree that a likelihood increase is a proxy for useful context, and its relationship to repair success should be examined empirically. On the evaluated SWE-bench Pro tasks with OpenHands, PCIG ranks the three retrieval models in the same order as downstream Pass@1. For example, GPT-5.4 has the highest file, module, and entity F1, while ContextForge has higher repair success (68.2% v.s. 66.3%) and higher PCIG (0.18 v.s. 0.09).

Subagent modelPass@1File F1Module F1Entity F1PCIG
CodeScout-4b67.600.570.460.390.09
GPT-5.466.300.610.700.700.06
ContextForge68.200.540.530.430.18

Averaging across valid repairs. Mean aggregation reflects expected gain across the validated repair set, giving credit to context that supports one repair while favoring evidence useful across several repairs. We agree that repair-specific evidence can receive a smaller score under this objective. Max aggregation is a reasonable alternative when the objective is to support any one repair, although it is more sensitive to a single unusually high estimated gain.
To assess this choice, we replace mean with max for the stored supporting-context gains in 300 randomly selected complete, valid multi-reference training groups, with the other reward components fixed. The top-ranked rollout remains the same in 86.4% of the 300 groups, and the mean within-group Kendall τ-b of 0.881indicates strong agreement across the full rankings. Thus, max aggregation changes some individual selections but largely preserves the rollout ordering in this sample. This suggests that the ranking signal used during training is relatively insensitive to the aggregation choice.

Sigmoid and saturation. We use the sigmoid to bound each unweighted PCIG component between 0 and 1 and control its scale in the weighted reward. For a valid scored field, zero gain contributes λ/2, so evidence that decreases patch likelihood receives a lower reward than neutral evidence. In the analyzed training run (α = 500), none of the valid edit-context, supporting-context, or explanation component scores exceeded 0.99.

W3. Repair success and practical cost.

As shown in Table 2a, although the improvement in repair success is modest (68.2% ± 2.0 vs. 68.0% ± 1.2 for Pass@1), we would like to highlight that ContextForge reduces inference cost from \0.58±0.012to0.58 ± 0.012 to \\0.48 ± 0.020 per task. We do not include training cost in the per-task inference cost, as the trained model can be reused across subsequent tasks. Our full training required approximately 84 hours on a 4 x H100 node. The training pipeline assigned four GPUs: one for policy optimization, one for rollout generation, and two for reward scoring. We note that these measurements reflect our current implementation rather than an optimized minimum. For a fair comparison across variants, we use the same training data and number of training steps.

Q1. Choice of the scoring model.

We chose Qwen3-32B as a practical balance between code-modeling capability and the cost of repeated likelihood evaluations during training. Its open weights allow teacher-forced computation of candidate-patch log probabilities under different retrieved contexts. The scorer is used only during training. Deployment requires only the trained retrieval policy. Thus, Qwen3-32B is an implementation choice rather than a requirement of PCIG or a claim that Qwen3-32B is optimal. We further run GLM-4-32B-0414, a non-Qwen 32B model, on 150 randomly sampled tasks and four fixed contexts per task, as Qwen3-32B. GLM selects the same highest-scoring context as Qwen3-32B on 82.7% of the tasks, with a mean within-task Kendall’s τ-b of 0.731. These results suggest that PCIG scores are not specific to Qwen3-32B and exhibit substantial agreement across comparably sized models, although scorer choice can still affect individual rankings.

We conducted two additional evaluations:

  1. Matched training compute. We compared ContextForge with adapted CodeScout on Qwen3.5-4B using the same training GPU-hour budget. At this budget, ContextForge completed half of its full training steps and matched CodeScout’s 65.3% Pass@1, while lowering inference cost from \0.54to0.54 to \\0.49 per task.

  2. Additional downstream model. With GPT-6.1-Sol as the downstream model, full ContextForge achieved 77.3% Pass@1 at \10.2pertask,comparedwith76.710.2 per task, compared with 76.7% at \\10.9 without a subagent and 75.3% at \$10.9 with a Qwen3.5-4B subagent. ContextForge reduced cost by approximately 6% relative to either baseline, with higher Pass@1 in both comparisons.

Thank you for your detailed response.

Reviewer_B6QdOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ContextForge trains a code-retrieval agent using multiple test-validated repairs and a reward based on their conditional likelihood gains. Across two agent frameworks and three repair benchmarks, it reduces inference costs while maintaining or improving repair success.

Critical strengths
  • Well-motivated distinction between reference-patch locations and useful repair context.
  • Consistent end-to-end cost reductions across evaluation settings.
  • Repeated evaluations and component ablations provide useful supporting evidence.
Critical weaknesses
  • Reproducibility is insufficient. The paper lacks essential details about training-data size and provenance, evaluation overlap checks, candidate-generation budgets, and RL hyperparameters.
  • The ablations do not isolate the reward design. Removing supporting context or explanations changes both training supervision and downstream inputs. Matched-interface comparisons against simpler rewards are needed.
  • Evidence for diverse-repair supervision is limited. Its reported gain is only 0.89 percentage points, without uncertainty estimates. The repeated-run results more clearly support efficiency improvements than accuracy gains.
Questions for authors

Please read the weakness.

2 comments

We sincerely thank the reviewer for their time and valuable feedback!

W1. Reproducibility details.

Thank you for highlighting these omissions. We use 1,185 preprocessed SWE-Smith training tasks and a separate, non-overlapping set of 100 validation tasks. GPT-5.4 generates candidate repairs with up to 5 attempts per task, stopping once 3 alternative patches are accepted; each attempt allows up to 30 agent iterations. Our records contain 4,004 generation attempts and 3,212 accepted alternative patches for the final training set. Candidates are accepted only after passing the provided SWE-Smith test suite.

For RL, we use GRPO advantages with the GSPO policy loss, a learning rate of 1e-6, a batch size of 6 tasks, and 8 rollouts per task. Each trajectory allows 6 agent turns and up to 2,048 generated tokens/call. The context window is 32,768 tokens, with an initial-prompt limit of 16,384 tokens. ContextForge and its trained ablations use one epoch with these same training-budget settings.

Our SWE-smith training data have no repository overlap with the SWE-bench Verified / Pro / Multilingual evaluation sets used in our experiments. Please see Section 3.5 and Appendix A.1 for the data construction and preprocessing procedure.

W2. Reward design with a matched output format.

As suggested by the reviewer, we further evaluate the reward design with a matched output format using the same Qwen3.5-4B initialization, training tasks, prompts, tools, training budget, and output format.

Pass@1Cost/task
Qwen3.5-4B64.0$0.52
w/o supporting context reward69.2$0.51
w/o explaination reward65.4$0.45
Full ContextForge68.2$0.48

The full model improves observed Pass@1 over the untrained Qwen3.5-4B baseline by 6.6% while reducing cost by 7.7%. The component results reveal a trade-off: removing the explanation reward reduces Pass@1 to 65.4%, whereas removing the supporting-context reward reaches 69.2% at a higher cost of $0.51. Thus, the full reward provides a favorable accuracy–cost balance, without each component necessarily maximizing Pass@1 individually.

W3. Evidence for diverse-repair supervision.

We would like to clarify that Table 3b already averaged three evaluation runs on the same 150 tasks. Diverse-reference supervision achieves 68.22% Pass@1, compared with 67.33% for single-reference supervision: a gain of 0.89% with a paired sign test shows that diverse supervision wins on significantly more tasks than it loses on (p = 0.043).

To further investigate its effect on retrieval, we analyze retrieved files and the main agent’s final submitted patch within the same execution in the following table. These results support broader retrieval coverage aligned with downstream edit locations: diverse supervision returns 0.86 more files and increases coverage of final-patch files by 14.87%

MetricDiverseSingleDifference (95% CI)
Coverage of final-patch files65.13%50.26%+14.87 [10.36, 19.57]
First-call coverage of final-patch files61.43%47.88%+13.55 [9.04, 18.28]
Mean retrieved files per execution2.851.99+0.86 [0.67, 1.06]
Reviewer_B6QdOpenReview ↗

Response to authors

Thanks for your response. It addressed some of my concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.