- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ContextForge addresses the limitations of training code retrieval agents against a single reference patch, which may omit useful supporting code and overlook alternative valid repairs. It generates diverse patches, filters them by test results, and uses patch-conditioned information gain to separately reward retrieved edit locations, supporting context, and explanations according to how much they help predict multiple validated repairs. The authors train a Qwen3.5-4B retrieval agent with GRPO and evaluate it with GPT-5.4-high on SWE-bench Pro, Verified, and Multilingual using Mini-SWE-Agent and OpenHands. Across the six benchmark–scaffold settings, ContextForge matches or improves Pass@1 over the no-subagent baseline while reducing end-to-end inference cost by up to 29.1%. Its supervision depends on the quality and coverage of available tests, however, and the authors note that familiarity with benchmark repositories may limit how well the results generalize to unfamiliar codebases.
Critical strengths
- The authors propose a subagent-centric method, which is an underexplored field.
- This paper presents a novel framework with a patch-conditioned information-gain (PCIG) reward, combining edit locations, context and explanations.
- This paper conducts comprehensive evaluation on three SWE benchmarks, covering multiple languages and scenarios.
Critical weaknesses
- Although CONTEXTFORGE consistently reports lower end-to-end inference costs, the paper does not adequately explain the source of these savings. The reported reductions in main-agent token usage and illustrative case studies are suggestive, but the paper does not provide an aggregate analysis that attributes the savings to reduced repository exploration or separates the method’s effect from variation in agent trajectories. This makes it difficult to determine whether the cost gains are a robust consequence of the approach.
- In addition, the cost analysis focuses on inference and omits the cost of training the retrieval policy. In particular, computing PCIG rewards requires the frozen Qwen3-32B scorer to evaluate candidate-patch log probabilities under multiple conditioning contexts, with these evaluations repeated across repair candidates, rollout groups, and reward components. This repeated scoring may be computationally expensive, yet the paper reports no scorer-token counts, accelerator-hours, or training-cost estimate, nor an amortized analysis comparing this upfront cost with the inference savings.
- Finally, the paper states that LLMs were used in patch generation, reward scoring, retrieval, downstream editing, and parts of evaluation, but provides limited detail on independent human auditing of these outputs and experimental results. More information about oversight and verification is needed to assess the reliability and reproducibility of the findings.
Questions for authors
See the above weakness.
4 comments
We conducted an additional evaluation:
Matched training compute. We compared ContextForge with adapted CodeScout on Qwen3.5-4B using the same training GPU-hour budget. At this budget, ContextForge completed half of its full training steps and matched CodeScout’s 65.3% Pass@1, while lowering inference cost from \0.49 per task.
Thank you for your response. I will maintain my initial score.
Thank you for the authors' response. I will maintain my initial score.
We are grateful for your effort and time in leaving a thorough review!
W1. Sources and robustness of inference-cost savings.
As suggested by the reviewer, we provide a breakdown of the 17.2% reduction in agent cost. As shown in the table, the cost of initial exploration decreases by 7.1%, while later exploration decreases by 22.2%. The costs associated with editing, review, and completion decrease by 29.2%, whereas testing and environment costs remain unchanged.
W2. Training cost.
Our candidate generation cost is approximately $1.3K in GPT-5.4 API usage. Our full training required approximately 84 hours on a 4 x H100 node. The training pipeline assigned four GPUs: one for policy optimization, one for rollout generation, and two for reward scoring. We note that these measurements reflect our current implementation rather than an optimized minimum. For a fair comparison across variants, we use the same training data and number of training steps.
W3. Verification and human oversight.
Thank you for raising this. While LLMs are used in patch generation, retrieval, and downstream editing, the effectiveness of our methodology is validated empirically on public benchmarks using Pass@1 and cost (Section 4.1), which are deterministic metrics that cannot be cheated. Thus, our main result does not rely on subjective LLM judgements or human evaluation. Our reward scoring does not use an LLM as a judge. Instead, it deterministically computes the output likelihood of a given patch under different retrieved contexts via teacher-forced inference, without eliciting any subjective judgment from the model. In addition, training patches undergo separate automated validation against the full provided test suite, with invalid patches and test-file modifications rejected (Section 3.5 and Appendix A.1). To support reproducibility and independent verification, we include our training and evaluation code in the anonymous repo and are working toward a public release of the code and models.