acceptodds
Under review as a conference paper at ICLR 2027

CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents

Abstract

A prerequisite for coding agents to perform tasks on large repositories is code localization - the identification of relevant files, classes, and functions to work on. While repository-level code localization has been performed using embedding-based retrieval approaches such as vector search, recent work has focused on developing agents to localize relevant code either as a standalone precursor to or interleaved with performing actual work. Most prior methods on agentic code search equip the agent with complex, specialized tools, such as repository graphs derived using static analysis. In this paper, we demonstrate that, with an effective reinforcement learning recipe, a coding agent equipped with nothing more than a standard Unix terminal can be trained to achieve strong results. Our experiments on three benchmarks (SWE-Bench Verified, Pro, and Lite) reveal that our models consistently achieve superior or competitive performance over **2-18x** larger base and post-trained LLMs and sometimes approach performance provided by closed models like Claude Sonnet, even when using specialized scaffolds. Furthermore, our models improve over their corresponding base models on programming languages not seen during RL, demonstrating cross-language generalization with a language-agnostic bash-only scaffold. Our work particularly focuses on techniques for re-purposing existing coding agent environments for code search, reward design, and RL optimization. We release the resulting model family, **CodeScout**, along with all our code and data for the community to build upon.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agents, RL, Code search, Coding agents, Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FEpfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an RL recipe to train coding agents for code-loaclisation using only a bash Unix terminal and OpenHands SDK tool (LocalizationFinish), called CodeScout. It is trained on SWE-Smith using ground truth files, modules and functions extracted from the gold patches of GitHub issue PRs. The proposed recipe uses GSPO (Group Sequence Policy Optimisation) without KL or std normalisation, where the reward is a sum of F1 scores at various granularity levels. For certain models like 14B, it uses a bonus term upon finishing k turns. The paper demonstrates the recipe across open-source models like Qwen3-1.7B, Qwen3-4B-Instruct-2507 and Qwen3-14B and shows that the performance of the base models improves across SWE-Bench Verified, Lite and the python subset of Pro using the same scaffolding. The performance is also at par with specialised scaffolding based methods like RepoNavigator. The paper further reports downstream issue-resolution gains, cross-language generalisation at file-level. The analysis shows that tool use narrows to a small set of utilities during RL (only rg and sed for 14B; mainly rg, sed, cat and xargs for 4B). Code, data or model are not included in the supplementary material.

Critical strengths
  • The key results are evident and the gains over base models are notable. Specifically, using an identical scaffold, the proposed RL mechanism lifts the Qwen3 bases by 14–54 F1 points across file/module/function levels, and the gains over Qwen3-32B pass a paired bootstrap test.
  • The simplicity of the scaffold is a strength and makes it flexible and portable for downstream applications. It does not require pre-indexing or language server.
  • Training is not compute intensive, i.e. doable in one 8×H100 node, 9 h for 4B. This has a real practical value to the community.
  • Several analyses are useful: Return@K against top-K methods (App. K), performance by number of edited locations (App. H), and the tool-use convergence plot.
Critical weaknesses
  • The key contribution is proposed as a recipe covering data, scaffold, reward and algorithm, but certain critical ablations are missing to attribute gains to various components of the methodology. The choice of GSPO over GRPO, is not clearly motivated. GSPO updates GRPO’s token-level importance ratios to a sequence-level ratio, to primarily stabilise RL on MoE models and long responses. While CodeScout's models have long multi-turn trajectories, this alone cannot completely justify the choice of GSPO. Either a concrete discussion or comparison against GRPO as an ablation would help understand whether the gains come from the choice of GSPO or any related RL training paradigm. Similarly, the effect of dropping KL divergence/std normalisation, the structured finish tool vs the string format abandoned in section 3.2 would need further evidence.
  • Comparison to prior works are not fully fair and may be confounded by the choice of the base model. E.g., In table 3, the specialised scaffolding baselines (like CoSIL, AgentLess, LocAgent, OrcaLoca, RepoSearcher, RepoNavigator) use Qwen2.5, while CodeScout uses Qwen 3. For example, the untrained Qwen3-4B-Instruct already reaches 49.7 file-F1 on Verified, which is close to the RL-trained RepoNavigator-7B (51.6). Running the proposed method on Qwen2.5-7B/14B-Instruct, or RepoNavigator’s scaffold with a Qwen3 base, would resolve this. While the authors acknowledge that these baseline numbers are copied from prior works, in this paper, they use enhanced patch-parsing scripts (App. B.1). Could the authors clarify that the evaluation ground truth or matching rules are same as the previous works? If they differ, the numbers aren’t comparable.
  • The SoTA results are not conclusive. On SWE-Bench Pro, RepoNavigator-32B has higher file-F1 (57.6 vs 53.6), SWE-Bench Verified has a tie on File F1, and SWE-Bench Lite results do not include comparison to specialised scaffolding open-source baselines.
  • No comparison to embedding (or embedding+graph)-based localization. The abstract and introduction frame agentic search against embedding retrieval, yet do not discuss or compare against recent dense-retrieval based methods like CoRET, SweRank, SpIDER, CoRNStack/CodeRankEmbed (See refs below). Given the pitch of the paper towards cheap and simplicity, it would benefit from discussion/comparison to such methods.
  • Beyond three dollar figures in App. B.3, the token cost comparisons are missing. For the localiser itself, an elaborate account of cumulative input tokens across turns, output tokens and number of turns and tool calls would help solidify the efficiency claims. These numbers should already exist in the logs for every OpenHands-Bash run so that a complete re-run is not needed to report these.
  • Cross-language generalization. Besides a sentence reporting file-level F1 in section 6.2, there is no supporting table, per-language breakdown, and reference model performance justifying cross language transfer. The base-vs-CodeScout comparison isolates the effect of RL, but not what transfers. Specifically, CodeScout-4B’s non-Python gain (+15.2) matches its Python Pro gain (+14.8) but CodeScout-14B’s gain falls from +23.6 to +3.6. The gains may mostly reflect language-agnostic protocol learning (e.g. submitting within the turn limit, formatting paths, predicting fewer files) on top of the base model’s existing multilingual knowledge. That would not be transferable localization skill.
  • No variance is reported across RL seeds or evaluation samples.

Overall, the paper is well motivated and of real practical value. The same-scaffold gains over the base models are convincing. However, the main concerns are w.r.t. cleaner ablation on design choices, fairer comparisons to minimise confounders and more elaborate reporting of token costs.

References:

  • Fehr et al., ACL 2025 (arXiv:2505.24715)
  • Reddy et al., ICLR 2026 (arXiv:2505.07849)
  • Chaudhari et al., EMNLP 2026 arXiv:2512.16956
  • Suresh et al., ICLR 2025 (arXiv:2412.01007)
Questions for authors

Q1. Is evaluation ground truth extracted with the same enhanced scripts as used in training? Is the evaluation setup (including function/class matching and F1 computation) identical to RepoNavigator and other prior works that the proposed method is compared against? How are instances with empty class- or function-level ground truth scored?

Q2. Is it possible that the agent can access git history beyond the base commit, in training (SWE-Smith) or evaluation? SWE-bench’s issue tracker records agents using git log --all to find future commits containing the fix. Please confirm this is blocked, including for the GPT-5 and Claude runs.

Q3. As suggested in section 6.3, have you tested restricting the tool to rg and sed at inference time?

Q4. Do we see cross-language transfer at other granularities (module and function-levels) as well?

2 comments

Thank you for your detailed feedback! We are glad to learn that you appreciate the notable gains of CodeScout over its base LLMs, the simplicity and portability of our scaffold, the low training cost of our recipe, and the usefulness of our analyses. We address your concerns below:

Critical ablations are missing to attribute gains to individual components: the choice of GSPO over GRPO is not clearly motivated, and the effects of dropping KL divergence/std normalization and of the structured finish tool need further evidence

Regarding our choice of GSPO over vanilla GRPO, we train Qwen3-4B-Instruct for 200 steps with different RL algorithms under identical settings and evaluate on SWE-Bench Pro. We find that GSPO is a better alternative over vanilla GRPO and SAPO:

RL AlgorithmFile-level F1Module-level F1Function-level F1
GSPO (ours)54.8331.4323.29
SAPO49.0529.8222.00
GRPO47.5930.8922.20

Regarding removing the KL divergence penalty, this is also done by recent work on RL for coding agents such as DeepSWE (TogetherAI) and SkyRL-Agent (Cao et al., 2025 (arXiv:2511.16108)). Our initial experiments trained models without using the structured finish tool but we encountered issues with reward signal fidelity that resulted in unstable RL runs due to brittle parsing of raw text responses (Section 3.2), and use this custom finish tool to mitigate this issue.

Comparisons to prior works may be confounded by the choice of base model (Qwen2.5 vs. Qwen3)

To address the base model mismatch, we train Qwen2.5-7B-Instruct (the base model of RepoNavigator-7B) with our CodeScout-4B recipe. We already match RepoNavigator’s training data source (SWE-Smith), and use an identical training budget of 32K rollouts for this experiment. Our recipe outperforms RepoNavigator-7B on both benchmarks:

MethodFile-level F1Function-level F1
SWE-Bench Verified
RepoNavigator-7B51.6327.49
CodeScout-7B (Ours)65.6029.90
SWE-Bench Pro
RepoNavigator-7B39.7414.29
CodeScout-7B (Ours)47.8120.79

Could the authors clarify that the evaluation ground truth or matching rules are same as the previous works?

Regarding the ground truth, we would like to clarify that the evaluation ground truth is extracted using the original patch-processing scripts from LocAgent, and our enhanced scripts (Appendix B.1) are only used for curating training data.

The SoTA results are not conclusive. On SWE-Bench Pro, RepoNavigator-32B has higher file-F1 (57.6 vs 53.6), SWE-Bench Verified has a tie on File F1, and SWE-Bench Lite results do not include comparison to specialised scaffolding open-source baselines.

We believe that the above 7B training experiment can help alleviate this concern. Furthermore, CodeScout-14B is 2.3x smaller than RepoNavigator-32B - yet achieves higher function-level F1 on both SWE-Bench Verified (40.32 vs 34.09) and Pro (28.74 vs 20.72), with a comparable file-level F1 on Verified (68.57 vs 67.75). Regarding SWE-Bench Lite, the specialized scaffolds do not report F1 scores on this benchmark, and in some cases like RepoNavigator their code is not open-source which limits us from adding their metrics for this benchmark.

No comparison to embedding (or embedding+graph)-based localization.

Thanks for the suggestion! We will include a discussion of these methods in the related work and a comparison with them in our next revision. We would also like to clarify that some of our baselines do include methods that rely on embedding-based retrieval or code graphs: Agentless combines embedding-based retrieval with LLM-based ranking, and LocAgent, OrcaLoca, and CoSIL construct code graphs for localization.

Token cost comparisons are missing for CodeScout

We agree with this concern and report these statistics from our evaluation logs in the below table (averages per instance on SWE-Bench Verified using OpenHands-Bash).

Model# Turns# Tool CallsCumulative Input TokensOutput Tokens
CodeScout-1.7B4.103.1037.8K0.18K
CodeScout-4B4.859.3457.8K0.46K
CodeScout-14B4.003.0034.4K0.19K

No variance is reported across RL seeds or evaluation samples.

While reporting variance over RL seeds is prohibitively expensive, we will include variance over multiple localization evaluation runs in the next revision. However, Appendix B.4 does include a discussion on the statistical significance of the reported performance improvements

Cross-language generalization. Besides a sentence reporting file-level F1 in section 6.2, there is no supporting table, per-language breakdown, and reference model performance justifying cross language transfer.

We would like to clarify that our training data and primary evaluations are Python-only, and Section 6.2 is intended as an additional analysis of how CodeScout behaves on programming languages unseen during RL, enabled by our language-agnostic scaffold. Since the patch-processing scripts from LocAgent only support Python, we cannot compute module- or function-level ground truth for these languages. We agree that a per-language breakdown with reference models would strengthen this analysis, and we will include it in our next revision.

Is the evaluation setup (including function/class matching and F1 computation) identical to RepoNavigator and other prior works that the proposed method is compared against? How are instances with empty class- or function-level ground truth scored?

We borrow our evaluation setup from LocAgent’s codebase. We cannot verify the exact implementation of RepoNavigator as their codebase is not public, but their paper’s written description aligns with our approach. For instances with empty module-level or function-level ground truth, we assign an F1 score of 0 at these granularities.

Is it possible that the agent can access git history beyond the base commit, in training (SWE-Smith) or evaluation?

Thank you for raising this point! Our current training code had this bug but we do not observe any instances of reward hacking in our training experiments. We also analyze our evaluation trajectories from CodeScout and do not find such cheating issues for CodeScout models.

As suggested in section 6.3, have you tested restricting the tool to rg and sed at inference time

We would like to clarify that our terminal tool remains fully general, and the convergence to rg and sed is a behavior that emerges during RL rather than something we explicitly enforce or reward. While the models do use rg and sed quite frequently, restricting the model to just these commands can cause a mismatch between the train and test-time harness settings, and potentially lower our reported performance especially for our 4B model that uses other utilities as well.

Reviewer_ANeYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a repository-localization agent that uses only a Unix terminal (ripgrep, file reads) instead of a code graph or specialized navigation tools. Training data comes from SWE-Smith issue/patch pairs, converted into file/module/function ground truth (39K instances, 128 repositories, disjoint from evaluation repositories); the reward is the sum of three F1 terms, with an additional termination-step bonus for the 14B model. Training uses the OpenHands-Bash scaffold with a structured Localization Finish action, SkyRL async training, and GSPO; the 1.7B model is first warm-started with 14B trajectories. On SWE-bench Verified, 4B/14B reach file F1 68.52/68.57 and function F1 36.78/40.32, and localization assistance raises downstream solve rates from 13.40% to 17.20% (4B) and 45.20% to 46.00% (30B-A3B).

Critical strengths
  1. Well-posed question with a directly useful answer: whether a simple terminal scaffold plus a good RL recipe suffices for localization. The same-scaffold base-model comparison makes this claim credible.
  2. Appendix quality is high: paired bootstrap over 18 comparisons against Qwen3-32B (Appendix B.4), cost estimates, top-K sensitivity, error analysis, and an explicit limitations section.
  3. Practical value: open models and data, deployable at 1.7B/4B, with reported step and token reductions.
  4. Honest reporting: the Pro-benchmark trade-off (higher function F1, lower file F1 than RepoNavigator-32B) is stated rather than hidden.
Critical weaknesses
  1. The contribution is a recipe, and its components are not attributed. Multi-granularity reward, termination bonus, structured finish action, GSPO, and async training are introduced together; the authors acknowledge that a full ablation was not performed. The generalizable knowledge is therefore limited, and the paper reads as an engineering combination rather than a new method.
  2. End-to-end net benefit is not accounted for. The two-stage pipeline (localize with CodeScout-14B, then solve) adds a full agentic search phase; only component-level cost estimates are given. Whether total tokens, latency, and cost still beat "solve directly" or "complex scaffold in one pass" is the practically relevant question.
  3. Some cross-system comparisons are confounded. Several baselines are Qwen2.5-era, while this work post-trains Qwen3; turn budgets also differ (6 for the 4B comparison, 4 for most others, Appendix A). "Simple scaffold is sufficient" is supported by the same-scaffold control, but "simple scaffold is better" is not.
  4. Scaling is non-monotonic (on SWE-bench Pro, 14B function F1 28.74 < 4B 29.03), indicating the training configuration is not tuned per scale, which weakens the model-family narrative.
  5. Ground truth is the gold-patch location, which misses correct-but-unmodified code and equivalent fix sites (acknowledged, and affecting baselines equally).
Questions for authors
  1. Can you ablate the two or three core recipe elements (e.g., multi-granularity reward vs single F1, with/without termination bonus) at fixed backbone, data, and budget?
  2. What is the total token/latency/cost of the full two-stage pipeline, compared with solving directly and with a complex-scaffold agent at equal budget?
  3. Can you report a comparison at matched turn and token budgets across model generations, to separate scaffold simplicity from backbone vintage?
  4. Why is 14B function F1 below 4B on SWE-bench Pro, and does per-scale tuning remove the inversion?
  5. Is the 30B downstream gain (+0.8pp) accompanied by a paired confidence interval?
1 comment

Thank you for your detailed feedback! We are glad to learn that you appreciate the practical value of our research question, the credibility of our same-scaffold comparisons, the quality of our appendix, our open models and data. We address your concerns below:

The contribution is a recipe, and its components are not attributed. Can you ablate the two or three core recipe elements (e.g., multi-granularity reward vs single F1, with/without termination bonus) at fixed backbone, data, and budget?

We ablate two core elements of our recipe at fixed backbone, data, and budget: the RL algorithm and the turn-limit reward. RL algorithm. We train Qwen3-4B-Instruct for 200 steps with three different RL algorithms under identical settings and evaluate on SWE-Bench Pro. We find that GSPO achieves better or similar performance over vanilla GRPO and SAPO (soft-adaptive policy optimization).

RL AlgorithmFile-level F1Module-level F1Function-level F1
GSPO (ours)54.8331.4323.29
SAPO49.0529.8222.00
GRPO47.5930.8922.20

Turn-limit reward Without the termination bonus, training of the 14B model collapses to zero rewards as the agent frequently exhausts the turn limit without submitting (Section 3.3). Appendix I describes an alternative bonus variant that flexibly rewards terminating within the turn limit, for the 14B model trained for 300 steps (Appendix I). The flexible variant consistently lowers module-level and function-level F1 by up to 6% and 5% respectively, implying that rewarding the model for strictly using 4 turns is a better work-around.

BenchmarkTurn-limit RewardFile-level F1Module-level F1Function-level F1
SWE-Bench VerifiedExact (ours)68.5750.8840.32
Flexible70.3746.4337.30
SWE-Bench ProExact (ours)53.6337.1328.74
Flexible54.2735.7627.60
SWE-Bench LiteExact (ours)71.8459.2344.43
Flexible70.9253.2239.29

We will include these findings in the next revision.

Some cross-system comparisons are confounded. Several baselines are Qwen2.5-era, while this work post-trains Qwen3; turn budgets also differ (6 for the 4B comparison, 4 for most others, Appendix A).

To separate scaffold simplicity from choice of backbone model, we train Qwen2.5-7B-Instruct (the base model of RepoNavigator-7B) with our recipe and an identical training budget (of 32K rollouts) and SWE-smith training environments as done by RepoNavigator-7B. Our recipe outperforms RepoNavigator-7B on both benchmarks with the same base model:

MethodFile-level F1Function-level F1
SWE-Bench Verified
RepoNavigator-7B51.6327.49
CodeScout-7B (Ours)65.6029.90
SWE-Bench Pro
RepoNavigator-7B39.7414.29
CodeScout-7B (Ours)47.8120.79

While training all CodeScout models with an identical turn budget can simplify our experimental setup, this experiment does not change the central findings and insights from our research. We will try to include this in the final revision.

Scaling is non-monotonic, indicating the training configuration is not tuned per scale, which weakens the model-family narrative. Why is 14B function F1 below 4B on SWE-bench Pro, and does per-scale tuning remove the inversion?

We would like to clarify that CodeScout-14B outperforms CodeScout-4B on 8 out of 9 metrics (three granularities across three benchmarks), with gains of up to 6.13% in module-level F1 and 4.56% in function-level F1 on SWE-Bench Lite. The only exception is function-level F1 on SWE-Bench Pro, where the difference is quite small −0.29%. We agree that tuning the training configuration separately for each model scale may further improve performance but this requires several additional RL training runs per scale, which is beyond the scope and computational budget of our work.

Is the 30B downstream gain (+0.8pp) accompanied by a paired confidence interval?

We would like to clarify that our downstream experiments are an analysis of how CodeScout's localized context affects an off-the-shelf issue resolution agent, and repeated SWE-Bench Verified evaluations is prohibitively expensive. For the 30B model, we agree that the 0.8% difference in resolution rate may not be significant but we do observe efficiency gains at a comparable resolution rate. This particular model is strong on this task and already localizes much of the relevant code during its own exploration, implying that locations retrieved by CodeScout-14B do not add much benefit to the final resolve rate, but still makes it more efficient. This is also supported when we use Qwen3-4B-Instr. as the repair agent where both resolution rate and efficiency improved with additional hints as this model can benefit from the hints given by CodeScout-14B.

Reviewer_ViaBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CodeScout trains small code localization agents using standard terminal commands. It applies GSPO with rewards based on file-, class-, and function-level localization F1. The 4B and 14B models undergo direct RL, while the 1.7B model uses distillation followed by RL. Experiments on SWE-bench Verified, Lite, and Pro show improved localization, with additional cross-language and downstream repair evaluations.

Critical strengths
  1. Competitive results with a simple tool interface. CodeScout achieves clear gains over its original models using ordinary terminal commands, providing useful evidence that effective localization does not necessarily require specialized navigation tools.

  2. Downstream repair evaluation. The paper tests whether localization helps actual issue resolution. For example, the Qwen3-4B repair agent improves from 13.4% to 17.2%, supporting the practical value of the retrieved locations.

Critical weaknesses
  1. The contribution beyond closely related work remains unclear. RepoNavigator already trains code localization models directly with RL. CodeScout mainly changes the tool interface and training recipe. However, CodeScout uses Qwen3, while key baselines use Qwen2.5 and results reported in prior papers. Differences in backbones, data, and training settings make it difficult to attribute the gains to the proposed design. Improvements over the original model establish that training helps, but do not establish an advantage over existing methods.

  2. The downstream experiments do not establish an advantage over existing localization approaches. They compare no localization, CodeScout, and oracle locations, without integrating other localization methods into the same repair agent. The results therefore show that localization can help, but not how much additional value CodeScout provides. For the stronger repair model, success increases only from 45.2% to 46.0%, with uncertainty around this difference left unclear.

Questions for authors

1.How does CodeScout compare with RepoNavigator using the same backbone, training data, and budget? Which design choices account for the gains?

2.Does CodeScout improve downstream repair more than existing localization methods under comparable total inference budgets? How does it compare with allocating the additional budget directly to the repair agent?

3.Does Table 6 include CodeScout’s own inference cost? Is the 0.8 percentage-point repair improvement supported by confidence intervals or repeated runs?

1 comment

We thank the reviewer for their feedback and we are glad they appreciate the strong performance of our models, the simple bash-only interface, and how effective localization can potentially benefit downstream issue resolution tasks. We address the concerns and questions below:

The contribution beyond closely related work remains unclear. RepoNavigator already trains code localization models directly with RL. CodeScout mainly changes the tool interface and training recipe. CodeScout uses Qwen3, while key baselines use Qwen2.5 making comparisons difficult.

We respectfully disagree. Our work trains LLM agents for localization using a language-agnostic, bash-only scaffold, whereas RepoNavigator’s jump tool requires implementing a Python language server built with static analysis tools implying that their approach needs additional engineering efforts to be usable for other programming languages. Our scaffold and the post-trained models generalize to other languages like Go, JavaScript, and TypeScript (Section 6.2) and all our baseline scaffolds, including RepoNavigator, cannot be used on other programming languages. To address the base model mismatch, we train Qwen2.5-7B-Instruct (the base model of RepoNavigator-7B) with an identical RL recipe as that of CodeScout-4B and an identical training budget as that of RepoNavigator (4K instances x 8 rollouts) and report evaluation metrics on SWE-bench Verified and SWE-bench Pro in the below table. Our method significantly outperforms its closest baseline by a huge margin despite both approaches using the same model backbone, SWE-smith training data, and identical training budget.

BenchmarkMethodFile-level F1Function-level F1
SWE-Bench VerifiedRepoNavigator-7B51.6327.49
SWE-Bench VerifiedCodeScout-7B (Ours)65.6029.90
SWE-Bench ProRepoNavigator-7B39.7414.29
SWE-Bench ProCodeScout-7B (Ours)47.8120.79

The downstream experiments do not establish an advantage over existing localization approaches and do not integrate other localization methods into the same repair agent. The results therefore show that localization can help, but not how much additional value CodeScout provides.

We would like to clarify that the goal of Section 6.1 is to analyze how the locations retrieved by CodeScout can be used to potentially improve the performance and efficiency of downstream issue resolution agents. Section 5 already compares CodeScout’s localization capabilities over our baselines extensively. Section 6.1 also includes oracle locations in the input prompt to determine an upper bound on potential downstream improvements. Performing this analysis with our baselines is expensive and requires their evaluation outputs, trained checkpoints and harness, which is not publicly available for several baselines like RepoNavigator. Appendix E shows that downstream agents achieve higher resolution rates on instances with higher quality of localization inputs with a more detailed breakdown. Thus, we leave additional analyses to future work.

How does CodeScout compare with RepoNavigator using the same backbone, training data, and budget? Which design choices account for the gains?

We have reported the performance gains in the table above. It is difficult to attribute performance gains to specific differences in design choices as we use a different reward function and agent harness. While understanding this can be interesting, performing exhaustive ablations is prohibitively expensive and beyond the scope of our work.

How does providing localization outputs from CodeScout to downstream repair agents compare with allocating the additional budget directly to the repair agent? Does Table 6 include CodeScout’s own inference cost? Is the 0.8 percentage-point repair improvement supported by confidence intervals or repeated runs?

Table 6 does not include inference cost from CodeScout because this makes it difficult to compare it with the “oracle locations as additional context” where computing inference costs is not feasible. For other proposed experiments, we agree that these are interesting and can further strengthen this analysis. However, we argue that this is not central to the primary research contributions made by our work. Furthermore, additional experiments on SWE bench verified require significant resources and engineering effort. We leave additional analysis to future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.