- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work explores training small <2B param models for agentic code localization via distillation with SFT and then RL. The RL recipe follows the setting and recipe closely of past methods like CodeScout [1]. But the main difference in this work is using a semantic search tool like colgrep and seeing how that compares with lexical tools like grep.
[1] Sutawika et al., CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents, arXiv 2026.
Critical strengths
- The work tackles an important practical question about code localization which to my knowledge past works do not explore in depth (does using a semantic search tool help beyond just lexical search tools like grep?)
- The results show that training with the colgrep tool helps performance over grep only approaches, helps with efficiency and generalization to other languages.
- The constructed training data could be a valuable resource for the community.
Critical weaknesses
- The paper lacks evaluation of downstream performance to understand whether the localization performed actually helps a SWE agent perform its task (e.g., issue resolution better)
- Methodologically, the paper largely follows similar past papers like CodeScout [1], but with just the small tweak of using colgrep on top of just bash/grep.
- The latency discussion seems to ignore initial indexing cost required for colgrep?
Questions for authors
Please see weaknesses above.
6 comments
Reminder for Reviewer G22a
Dear Reviewer,
We appreciate your time, engagement, and valuable review. As the discussion period concludes, we hope that our responses have addressed your concerns and clarified the points raised. Please let us know if there are any remaining questions or issues we can further address.
Best,
The Authors
Dear Reviewer,
Please read the authors’ rebuttal and join the discussion. Kindly clarify whether your main concerns have been addressed and raise any follow-up questions promptly so the authors have time to respond.
Thank you for your time and effort!
Rebuttal to Reviewer G22a
We thank the reviewer for their time and constructive feedback. Below, we address each of the raised concerns in turn.
W1: We refer the reviewer to our General Rebuttal, where we report additional end-to-end experiments on SWE-bench Lite showing improved issue-resolution success when delegating localization to our smaller agent.
W2: We acknowledge that our training recipe follows standard practice, similar to CodeScout’s setup: SFT distills a stronger teacher’s search behavior and teaches tool-use formatting, while RL improves localization through direct interaction. However, our study goes beyond adding colgrep to an existing agent. We separately train semantic and lexical policies using demonstrations generated with their respective tools, showing that the semantic advantage persists after both receive SFT and RL at this model size. Our agent also outperforms direct semantic retrieval even with oracle selection among the top-10 files (Table 2). These comparisons establish benefits from both semantic access and the learned search policy. We further show reduced dependence on multilingual policy training and lower CPU/GPU trajectory latency. Starting from the same Qwen3-1.7B base model, our agent substantially outperforms CodeScout’s reported 1.7B result on SWE-bench Lite in file F1 (67.29 versus 56.57), while matching its 4B result (67.03) and supporting multilingual localization.
W3: In section 3.1, we state that repository indexes are built once before training and evaluation, and Appendix A.5 explicitly excludes indexing from latency measurements. In practice, the index is reused across searches for the same repository snapshot, amortizing its cost over subsequent queries. Our results therefore measure search latency over prepared indexes.
We once again thank the reviewer for their constructive feedback.