- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces an RL recipe to train coding agents for code-loaclisation using only a bash Unix terminal and OpenHands SDK tool (LocalizationFinish), called CodeScout. It is trained on SWE-Smith using ground truth files, modules and functions extracted from the gold patches of GitHub issue PRs. The proposed recipe uses GSPO (Group Sequence Policy Optimisation) without KL or std normalisation, where the reward is a sum of F1 scores at various granularity levels. For certain models like 14B, it uses a bonus term upon finishing k turns. The paper demonstrates the recipe across open-source models like Qwen3-1.7B, Qwen3-4B-Instruct-2507 and Qwen3-14B and shows that the performance of the base models improves across SWE-Bench Verified, Lite and the python subset of Pro using the same scaffolding. The performance is also at par with specialised scaffolding based methods like RepoNavigator. The paper further reports downstream issue-resolution gains, cross-language generalisation at file-level. The analysis shows that tool use narrows to a small set of utilities during RL (only rg and sed for 14B; mainly rg, sed, cat and xargs for 4B). Code, data or model are not included in the supplementary material.
Critical strengths
- The key results are evident and the gains over base models are notable. Specifically, using an identical scaffold, the proposed RL mechanism lifts the Qwen3 bases by 14–54 F1 points across file/module/function levels, and the gains over Qwen3-32B pass a paired bootstrap test.
- The simplicity of the scaffold is a strength and makes it flexible and portable for downstream applications. It does not require pre-indexing or language server.
- Training is not compute intensive, i.e. doable in one 8×H100 node, 9 h for 4B. This has a real practical value to the community.
- Several analyses are useful: Return@K against top-K methods (App. K), performance by number of edited locations (App. H), and the tool-use convergence plot.
Critical weaknesses
- The key contribution is proposed as a recipe covering data, scaffold, reward and algorithm, but certain critical ablations are missing to attribute gains to various components of the methodology. The choice of GSPO over GRPO, is not clearly motivated. GSPO updates GRPO’s token-level importance ratios to a sequence-level ratio, to primarily stabilise RL on MoE models and long responses. While CodeScout's models have long multi-turn trajectories, this alone cannot completely justify the choice of GSPO. Either a concrete discussion or comparison against GRPO as an ablation would help understand whether the gains come from the choice of GSPO or any related RL training paradigm. Similarly, the effect of dropping KL divergence/std normalisation, the structured finish tool vs the string format abandoned in section 3.2 would need further evidence.
- Comparison to prior works are not fully fair and may be confounded by the choice of the base model. E.g., In table 3, the specialised scaffolding baselines (like CoSIL, AgentLess, LocAgent, OrcaLoca, RepoSearcher, RepoNavigator) use Qwen2.5, while CodeScout uses Qwen 3. For example, the untrained Qwen3-4B-Instruct already reaches 49.7 file-F1 on Verified, which is close to the RL-trained RepoNavigator-7B (51.6). Running the proposed method on Qwen2.5-7B/14B-Instruct, or RepoNavigator’s scaffold with a Qwen3 base, would resolve this. While the authors acknowledge that these baseline numbers are copied from prior works, in this paper, they use enhanced patch-parsing scripts (App. B.1). Could the authors clarify that the evaluation ground truth or matching rules are same as the previous works? If they differ, the numbers aren’t comparable.
- The SoTA results are not conclusive. On SWE-Bench Pro, RepoNavigator-32B has higher file-F1 (57.6 vs 53.6), SWE-Bench Verified has a tie on File F1, and SWE-Bench Lite results do not include comparison to specialised scaffolding open-source baselines.
- No comparison to embedding (or embedding+graph)-based localization. The abstract and introduction frame agentic search against embedding retrieval, yet do not discuss or compare against recent dense-retrieval based methods like CoRET, SweRank, SpIDER, CoRNStack/CodeRankEmbed (See refs below). Given the pitch of the paper towards cheap and simplicity, it would benefit from discussion/comparison to such methods.
- Beyond three dollar figures in App. B.3, the token cost comparisons are missing. For the localiser itself, an elaborate account of cumulative input tokens across turns, output tokens and number of turns and tool calls would help solidify the efficiency claims. These numbers should already exist in the logs for every OpenHands-Bash run so that a complete re-run is not needed to report these.
- Cross-language generalization. Besides a sentence reporting file-level F1 in section 6.2, there is no supporting table, per-language breakdown, and reference model performance justifying cross language transfer. The base-vs-CodeScout comparison isolates the effect of RL, but not what transfers. Specifically, CodeScout-4B’s non-Python gain (+15.2) matches its Python Pro gain (+14.8) but CodeScout-14B’s gain falls from +23.6 to +3.6. The gains may mostly reflect language-agnostic protocol learning (e.g. submitting within the turn limit, formatting paths, predicting fewer files) on top of the base model’s existing multilingual knowledge. That would not be transferable localization skill.
- No variance is reported across RL seeds or evaluation samples.
Overall, the paper is well motivated and of real practical value. The same-scaffold gains over the base models are convincing. However, the main concerns are w.r.t. cleaner ablation on design choices, fairer comparisons to minimise confounders and more elaborate reporting of token costs.
References:
- Fehr et al., ACL 2025 (arXiv:2505.24715)
- Reddy et al., ICLR 2026 (arXiv:2505.07849)
- Chaudhari et al., EMNLP 2026 arXiv:2512.16956
- Suresh et al., ICLR 2025 (arXiv:2412.01007)
Questions for authors
Q1. Is evaluation ground truth extracted with the same enhanced scripts as used in training? Is the evaluation setup (including function/class matching and F1 computation) identical to RepoNavigator and other prior works that the proposed method is compared against? How are instances with empty class- or function-level ground truth scored?
Q2. Is it possible that the agent can access git history beyond the base commit, in training (SWE-Smith) or evaluation? SWE-bench’s issue tracker records agents using git log --all to find future commits containing the fix. Please confirm this is blocked, including for the GPT-5 and Claude runs.
Q3. As suggested in section 6.3, have you tested restricting the tool to rg and sed at inference time?
Q4. Do we see cross-language transfer at other granularities (module and function-levels) as well?
2 comments
Token cost comparisons are missing for CodeScout
We agree with this concern and report these statistics from our evaluation logs in the below table (averages per instance on SWE-Bench Verified using OpenHands-Bash).
| Model | # Turns | # Tool Calls | Cumulative Input Tokens | Output Tokens |
|---|---|---|---|---|
| CodeScout-1.7B | 4.10 | 3.10 | 37.8K | 0.18K |
| CodeScout-4B | 4.85 | 9.34 | 57.8K | 0.46K |
| CodeScout-14B | 4.00 | 3.00 | 34.4K | 0.19K |
No variance is reported across RL seeds or evaluation samples.
While reporting variance over RL seeds is prohibitively expensive, we will include variance over multiple localization evaluation runs in the next revision. However, Appendix B.4 does include a discussion on the statistical significance of the reported performance improvements
Cross-language generalization. Besides a sentence reporting file-level F1 in section 6.2, there is no supporting table, per-language breakdown, and reference model performance justifying cross language transfer.
We would like to clarify that our training data and primary evaluations are Python-only, and Section 6.2 is intended as an additional analysis of how CodeScout behaves on programming languages unseen during RL, enabled by our language-agnostic scaffold. Since the patch-processing scripts from LocAgent only support Python, we cannot compute module- or function-level ground truth for these languages. We agree that a per-language breakdown with reference models would strengthen this analysis, and we will include it in our next revision.
Is the evaluation setup (including function/class matching and F1 computation) identical to RepoNavigator and other prior works that the proposed method is compared against? How are instances with empty class- or function-level ground truth scored?
We borrow our evaluation setup from LocAgent’s codebase. We cannot verify the exact implementation of RepoNavigator as their codebase is not public, but their paper’s written description aligns with our approach. For instances with empty module-level or function-level ground truth, we assign an F1 score of 0 at these granularities.
Is it possible that the agent can access git history beyond the base commit, in training (SWE-Smith) or evaluation?
Thank you for raising this point! Our current training code had this bug but we do not observe any instances of reward hacking in our training experiments. We also analyze our evaluation trajectories from CodeScout and do not find such cheating issues for CodeScout models.
As suggested in section 6.3, have you tested restricting the tool to rg and sed at inference time
We would like to clarify that our terminal tool remains fully general, and the convergence to rg and sed is a behavior that emerges during RL rather than something we explicitly enforce or reward. While the models do use rg and sed quite frequently, restricting the model to just these commands can cause a mismatch between the train and test-time harness settings, and potentially lower our reported performance especially for our 4B model that uses other utilities as well.
Thank you for your detailed feedback! We are glad to learn that you appreciate the notable gains of CodeScout over its base LLMs, the simplicity and portability of our scaffold, the low training cost of our recipe, and the usefulness of our analyses. We address your concerns below:
Regarding our choice of GSPO over vanilla GRPO, we train Qwen3-4B-Instruct for 200 steps with different RL algorithms under identical settings and evaluate on SWE-Bench Pro. We find that GSPO is a better alternative over vanilla GRPO and SAPO:
Regarding removing the KL divergence penalty, this is also done by recent work on RL for coding agents such as DeepSWE (TogetherAI) and SkyRL-Agent (Cao et al., 2025 (arXiv:2511.16108)). Our initial experiments trained models without using the structured finish tool but we encountered issues with reward signal fidelity that resulted in unstable RL runs due to brittle parsing of raw text responses (Section 3.2), and use this custom finish tool to mitigate this issue.
To address the base model mismatch, we train Qwen2.5-7B-Instruct (the base model of RepoNavigator-7B) with our CodeScout-4B recipe. We already match RepoNavigator’s training data source (SWE-Smith), and use an identical training budget of 32K rollouts for this experiment. Our recipe outperforms RepoNavigator-7B on both benchmarks:
Regarding the ground truth, we would like to clarify that the evaluation ground truth is extracted using the original patch-processing scripts from LocAgent, and our enhanced scripts (Appendix B.1) are only used for curating training data.
We believe that the above 7B training experiment can help alleviate this concern. Furthermore, CodeScout-14B is 2.3x smaller than RepoNavigator-32B - yet achieves higher function-level F1 on both SWE-Bench Verified (40.32 vs 34.09) and Pro (28.74 vs 20.72), with a comparable file-level F1 on Verified (68.57 vs 67.75). Regarding SWE-Bench Lite, the specialized scaffolds do not report F1 scores on this benchmark, and in some cases like RepoNavigator their code is not open-source which limits us from adding their metrics for this benchmark.
Thanks for the suggestion! We will include a discussion of these methods in the related work and a comparison with them in our next revision. We would also like to clarify that some of our baselines do include methods that rely on embedding-based retrieval or code graphs: Agentless combines embedding-based retrieval with LLM-based ranking, and LocAgent, OrcaLoca, and CoSIL construct code graphs for localization.