acceptodds
Under review as a conference paper at ICLR 2027

SWELSP: Symbol-Indexed Navigation for Efficient Coding Agents

Abstract

Coding agents spend substantial token budgets exploring repositories through text searches and code reads. These operations return line-based fragments, often prompting additional queries to identify complete definitions and follow relationships between symbols. Existing approaches reduce this overhead through context pruning or structured navigation, but may introduce additional dependencies, such as auxiliary models or dedicated analysis pipelines. Moreover, providing navigation tools alone does not ensure their effective use within existing agent workflows. We present SWELSP, a symbol-indexed navigation interface that connects Language Server Protocol to agents' established workflows. It augments bash outputs with symbol identifiers that agents can directly use to retrieve definitions and references. Our evaluation shows that SWELSP can reduce API cost by 10.1-21.3% on SWE-bench Multilingual, and 19.8-29.6% on SWE-QA while maintaining task performance on both benchmarks by reducing intermediate searches and windowed code reads of text-based exploration. It also achieves the lowest cost per resolved task against popular open-source tools such as Serena and AiderMap in openhands.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LSP, Coding Agent, Neuro-Symbolic, Code Navigation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vETjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SWELSP, a symbol-indexed navigation interface that connects LSP to workflows. This is a simple yet effective method. SWELSP reduces API cost by a large margin while maintaining task performance on both benchmarks by reducing intermediate searches and windowed code reads.

Critical strengths

SWELSP is a simple yet effective method, reducing API cost by a large margin while maintaining task performance.

Critical weaknesses
  1. The contribution appears to be essentially an interface integration of existing components, with no new methodological insight or mechanism. Without a clear differentiation from prior work, the paper's novelty is insufficient for publication.

  2. The ablation compares the full method against Nav-only but cannot disentangle which factor drives the gain—whether symbol addresses improve tool usage, whether auto-attaching correct code saves subsequent reads, or whether both interact. This leaves the proposed mechanism unverified. Additionally, the reported 19.8% cost reduction is presented as if it reflects "processing 20% less content," but the effect actually depends on caching; this framing overstates the method's intrinsic efficiency gain and misrepresents the underlying mechanism.

Questions for authors
  1. The paper lacks novelty—it does not appear to introduce anything genuinely new; my understanding is that it is essentially just interface integration.
  2. What about the Pass@N metric? Would the full method yield further improvement there?
  3. Experiments show that the full method outperforms Nav-only, but the ablation cannot identify where the actual gain originates:
  • Symbol addresses make it easier for the agent to use navigation tools;
  • Automatically attaching the correct code saves subsequent reading steps;
  • Or is it the combination of both?
  1. The cost-saving effect clearly depends on caching; the mechanism explanation should be more accurate. The 19.8% figure should not be directly interpreted as "processing about 20% less content."

4 comments
AuthorsOpenReview ↗

Responses to Questions and Weakness [1/2]

Thank you for your insightful review. We have addressed all your questions as follows.

(Question 1) The paper lacks novelty—it does not appear to introduce anything genuinely new; my understanding is that it is essentially just interface integration.

Thank you for raising this concern. SWELSP addresses a key limitation of existing symbolic tools (e.g., Serena and CodeStruct) : making precise navigation available does not ensure that agents actually use it to replace redundant text-based exploration. Rather than simply adding another retrieval interface, SWELSP introduces symbol-based output enrichment, which automatically exposes directly usable symbol addresses in existing Bash/grep outputs, connecting agents' familiar search workflows to targeted semantic navigation.

Our experiments demonstrate why this distinction matters. In Appendix B, merely adding symbolic navigation tools to Mini-SWE-Agent increases API cost by 17.5% and decreases Pass@1 by 5.0 percentage points on V4-Flash. In contrast, SWELSP reduces cost by 21.3% while preserving task performance. We further evaluated CodeStruct and Serena+ (Serena augmented with comparable output annotations and tool instructions) under the same OpenHands-GPT5.5 setting:

MetricCodeStructSerena+SWELSP
Resolve rate ↑66.0%67.0%73.3%
Tokens/task ↓666K1,039K563K
Cost/task ↓$0.6400.9290.548
Conventional file-view calls ↓2698443
Search/read output tokens ↓4.291M4.836M3.489M

Neither baseline reproduces SWELSP's efficiency gains. Compared with Serena+ and CodeStruct, SWELSP reduces total tokens by 45.8% and 15.5%, respectively, while improving resolve rates by 6.3 and 7.3 percentage points. The sharp reduction in conventional file-view calls further supports a shift toward targeted semantic retrieval.

Thus, SWELSP's novelty lies not in introducing symbolic retrieval itself, but in making it effective within agents' existing workflows. These results highlight a broader methodological insight: how retrieval capabilities are exposed to agents can fundamentally affect their tool-use behavior and end-to-end efficiency.

(Question 2) What about the Pass@N metric? Would the full method yield further improvement there?

Thanks. Following your suggestion, we conducted three independent GPT-5.5 runs on SWE-bench Multilingual to evaluate Pass@3. The results are summarized below.

ConfigurationMean Pass@1Pass@3Tokens/task
Mini-SWE-Agent71.44%79.33%379K
+ SWE-Pruner71.11%80.33%385K
+ SWELSP72.56%81.67%330K
OpenHands71.22%81.00%653K
+ Serena65.56%79.67%876K
+ AiderMap67.00%79.67%860K
+ SWELSP71.22%79.67%499K

SWELSP improves Pass@3 from 79.33% to 81.67% (+2.34 percentage points) on Mini-SWE-Agent, outperforming SWE-Pruner while reducing average tokens by 12.8%. On OpenHands, SWELSP achieves 79.67% Pass@3, slightly below vanilla (81.00%) but matching Serena and AiderMap, while reducing tokens by 23.7% relative to vanilla.

Overall, SWELSP improves Pass@3 in Mini-SWE-Agent and maintains competitive Pass@3 in OpenHands, with consistent token savings across both frameworks.

AuthorsOpenReview ↗

Responses to Questions and Weakness [2/2]

(Question 3) Experiments show that the full method outperforms Nav-only, but the ablation cannot identify where the actual gain originates: Symbol addresses make it easier for the agent to use navigation tools; Automatically attaching the correct code saves subsequent reading steps; Or is it the combination of both?

Thanks. We conducted additional component-level ablations to distinguish the contributions of the two mechanisms. We compare Nav-only (symbolic navigation on demand), Nav+Grep (additionally annotating grep outputs with symbol addresses), and Full SWELSP (further supplying function bodies automatically after failures).

ModelConfigurationPass@1Tokens/taskCost/resolved
V4-FlashNav-only65.7%12.464M$0.166
Nav+Grep67.7%10.005M$0.135
Full SWELSP71.0%8.329M$0.103
GPT-5.5Nav-only67.0%0.393M$0.710
Nav+Grep68.33%0.360M$0.677
Full SWELSP72.7%0.393M$0.650

Both mechanisms contribute, but in different ways. Grep annotations reduce tokens by 19.7% and 8.4%, respectively, while improving Pass@1 by 2.0 and 1.3 percentage points. Automatically supplying failure-related function bodies further improves Pass@1 by 3.3 and 4.4 percentage points. For V4-Flash, this also reduces tokens by another 16.8%; for GPT-5.5, tokens increase, but cost per resolved task still decreases ($0.677 → $0.650).

Together, the two mechanisms achieve the highest Pass@1 and lowest cost per resolved task in both models, with grep annotations improving retrieval efficiency and automatic failure-context delivery providing additional resolution gains.

(Question 4) The cost-saving effect clearly depends on caching; the mechanism explanation should be more accurate. The 19.8% figure should not be directly interpreted as "processing about 20% less content.

Thank you for pointing this out. We agree that API cost reduction should not be conflated with total-token reduction. Specifically, for GPT-5.5 on SWE-QA, SWELSP reduces total tokens by only 0.8%, while the 19.8% cost reduction reflects cache-hit, cache-miss, and output-token pricing. We will clarify this distinction in the paper.

To isolate the impact of caching discounts, we repriced all cache hits as misses on the same GPT-5.5 trajectories:

Cost savings over vanillaOriginal pricingAll hits priced as misses
SWE-bench Multilingual10.12%6.49%
SWE-QA19.82%2.11%

Caching substantially amplifies GPT-5.5's cost savings, particularly on SWE-QA, but positive savings remain without caching discounts. Moreover, the two DeepSeek models achieve substantial total-token reductions of 18.2–21.5% on SWE-bench Multilingual and 32.5–33.4% on SWE-QA, while maintaining or improving task performance.

Thanks for the additional experiments. The new ablations help clarify where the gains come from, though the interaction between the two components is still unclear. The caching analysis addresses my main concern about the cost claim; please distinguish cost savings from token savings in the paper. I will keep my weak-accept rating.

Thank you for your feedback. We will clearly distinguish API cost savings from total-token savings in the paper, and add the analysis below to clarify how the two components interact.

To clarify how the two mechanisms interact with agent, we analyzed agents' tool-use patterns across the ablation settings:

ModelPost-grep symbolic lookup rate (Nav-only → Nav+Grep)Avg. post-failure search/read calls (Nav+Grep → Full SWELSP)
GPT-5.542.9% → 68.5% (+59.7%)1.36 → 1.28 (-5.6%)
V4-Flash5.5% → 19.7% (+255.5%)2.47 → 1.81 (-26.5%)

The two mechanisms operate at complementary stages of the agent workflow: Grep annotations guide initial search toward target symbols, while automatic failure-context delivery decreases follow-up searches after test failures. Exposing symbol addresses bridges text search and symbolic navigation, boosting post-grep symbolic look up rate by 59.7% (GPT-5.5) and 255.5% (V4-Flash). Once a test fails, supplying failure-related function bodies decreases redundant exploratory reads, reducing follow-up search/read calls by 5.6% (GPT-5.5) and 26.5% (V4-Flash).Together, they achieve the highest Pass@1 and lowest cost per resolved task across both models.

Reviewer_pzxsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SWELSP, which adds enclosing function or class names to shell command search results to help coding agents retrieve definitions without repeatedly searching for locations and line ranges. It implements this via existing language servers. Experiments on SWE-bench Multilingual and SWE-QA show the API costs reduce 10.1–21.3% and 19.8–29.6%, respectively.

Critical strengths
  • The method provides a helpful way to connect existing shell commands with language-server navigation without expensive additional model training.
  • The evidence from comparisons with Serena and AiderMap, together with the ablation studies based on navigation-only, is useful to show how navigation tools enter the agent’s workflow affects their usefulness.
Critical weaknesses
  • The contribution seems a bit incremental. There are several prior works that propose to enrich the retrieval in coding agents. For example, language-server assistance [1], complete-function retrieval through named selectors [2], etc. The paper needs to show more clearly what its automatic output annotations additionally introduced beyond these existing capabilities.
  • It would be helpful to add evaluations of adding names to grep results and automatically inserting function bodies after failures to ablate which component saves calls and whether those savings actually exceed the additional context.

[1] Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. “Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair.” ESEC/FSE, 2023.

[2] Myeongsoo Kim, Chao-Chun Hsu, Dingmin Wang, Shweta Garg, Varun Kumar, and Murali Krishna Ramanathan. “CODESTRUCT: Code Agents over Structured Action Spaces.” ACL, 2026, pp. 13290–13306.

Presentation and Reporting

  • Sections 4.1 and 4.3 could describe standard LSP operations more briefly, while the main text can add ablations (in Appendix B) back.
  • Figure 2 does not look very clear
Questions for authors
  1. How much benefit comes from grep annotations vs. automatically supplying function bodies after failures? How do these compare with retrieving the same information on demand?
  2. Would Serena, with similar annotations and tool instructions, achieve similar savings? How does SWELSP compare with CodeStruct’s named-function retrieval?
1 comment
AuthorsOpenReview ↗

Responses to Questions and Weakness

Thank you for your insightful review. We have addressed all your questions as follows.

(Question 1) How much benefit comes from grep annotations versus automatically supplying function bodies after failures, and how do these compare with retrieving the same information on demand?

Thanks. We conducted additional component-level ablations to isolate the contributions of the two hooks. We compare three configurations: Nav-only (on-demand symbolic retrieval), Nav+Grep (adding symbol annotations to grep outputs), and Full SWELSP (further adding automatic function-body retrieval after failures).

ModelConfigurationPass@1Tokens/taskCost/resolved
V4-FlashNav-only65.7%12.464M$0.166
Nav+Grep67.7%10.005M$0.135
Full SWELSP71.0%8.329M$0.103
GPT-5.5Nav-only67.0%0.393M$0.710
Nav+Grep68.3%0.360M$0.677
Full SWELSP72.7%0.393M$0.650

Both hooks provide complementary benefits beyond on-demand retrieval. Grep annotations reduce tokens by 19.7% and 8.4%, respectively, while improving Pass@1. Adding the traceback hook further improves Pass@1 by 3.3 and 4.4 percentage points. For V4-Flash, it also reduces tokens by another 16.8%; for GPT-5.5, tokens increase, but cost per resolved task still decreases ($0.677 → $0.650).

Compared with Nav-only, Full SWELSP improves Pass@1 by 5.3 and 5.7 percentage points, while reducing tokens by 33.2% on V4-Flash and maintaining comparable token usage on GPT-5.5. These results show that automatically exposing relevant symbolic information can improve agent effectiveness beyond merely making the same retrieval capabilities available on demand.

(Question 2) Would Serena with comparable annotations and tool instructions achieve similar savings, and how does SWELSP compare with CodeStruct’s named-function retrieval?

Thanks. We additionally evaluated Serena+ (Serena augmented with comparable annotations and tool instructions) and CodeStruct under the same OpenHands-GPT5.5 setting.

MetricCodeStructSerena+SWELSP
Resolve rate ↑66.0%67.0%73.3%
Tokens/task ↓666K1,039K563K
Cost/task ↓$0.6400.9290.548
File-view calls ↓2698443
Search/read output tokens ↓4.291M4.836M3.489M

Neither baseline reproduces SWELSP's efficiency gains. Compared with Serena+ and CodeStruct, SWELSP reduces total tokens by 45.8% and 15.5%, respectively, while improving resolve rates by 6.3 and 7.3 percentage points.

The key distinction is not the ability to retrieve named functions, but how agents discover and use that capability. While CodeStruct provides named-function retrieval through readCode, SWELSP automatically enriches existing Bash/grep outputs with directly usable symbol addresses, connecting familiar text searches to targeted semantic retrieval. The substantial efficiency gains above, particularly the sharp reduction in conventional file-view calls, demonstrate the practical value of this workflow integration.

Reviewer_b4JEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SWELSP hooks a Language Server Protocol backend into the agent's existing bash loop: grep and traceback output is annotated with the enclosing symbol name, and three tools are added (outline, read_def, find_refs). No extra model and no per-language pipeline are required. The paper reports API cost reductions of 10.1 to 21.3% on SWE-bench Multilingual (300 instances, mini-swe-agent) and 19.8 to 29.6% on SWE-QA (144 questions), with pass@1 "preserved or improved" across three models, and the lowest cost per resolved task against Serena and AiderMap in OpenHands.

Critical strengths
  1. The ablation in Appendix B, Table 3, is the most interesting finding in the paper: navigation tools without the hooks actually hurt pass@1 (-5.0 and -4.7 points) and raise tokens by 17.5% for V4-Flash. The hooks are the whole story.

  2. The paper compares against tools people actually use (Serena, AiderMap, Figure 5) rather than only research baselines, and measures the hook overhead (Appendix D, Figure 11). The case studies in Figures 7 and 8 use step-matched token accounting, so it is easy to follow where the savings come from; code is released and the cost formula and prices are spelled out (Appendix F.1, Table 4).

Critical weaknesses
  1. One run per configuration, no seeds, no confidence intervals anywhere. The pass@1 deltas are +0.3, +1.7 and +1.0 on 300 instances (Table 1) and -1.3 for GPT-5.5 in Figure 5. "Preserving or improving" is not supportable at this sample size; the honest statement is "no detectable change". Figure 12 shows heavy-tailed per-instance token counts (36 to 53 tail instances per panel), so the percentage cost means are fragile as well.

  2. The GPT-5.5 result is mostly a caching artifact. On SWE-QA, tokens change by -0.8% and rounds rise by 1.9, yet cost falls 19.8%, with the whole saving coming from cache-miss tokens (Table 2). On SWE-bench the GPT-5.5 token reduction is only 5.7%. Cache behaviour is provider-side and non-deterministic, so the headline 19.8 to 29.6% range leans on its weakest leg.

Questions for authors

Clarification requests :

  1. Can you run at least three seeds for one model and report confidence intervals on pass@1 and cost? Is any pass@1 delta distinguishable from zero?
  2. For GPT-5.5, how much of the saving survives if cache hits are priced as misses?
  3. Why not Serena plus your hooks as a baseline? That would isolate the hook contribution from the tool set.
  4. Is there any human agreement check on the SWE-QA judge, even on a small sample?
1 comment
AuthorsOpenReview ↗

Responses to Questions and Weakness

Thank you for your insightful review. We have addressed all your questions as follows.

(Question 1) Can you run at least three seeds for one model and report confidence intervals on pass@1 and cost? Is any pass@1 delta distinguishable from zero?

Thanks. Following your suggestion, we evaluated GPT-5.5 across three seeds and reported pass@1 and cost reductions with 95% confidence intervals below.

Pass@1 across three runs

ConfigurationRun 1Run 2Run 3Mean [95% CI]
MSE (Vanilla)71.67%70.67%72.00%71.44% [66.78, 76.00]
MSE (SWELSP)72.67%72.67%72.33%72.56% [68.00, 76.89]
OpenHands (Vanilla)74.67%74.67%64.33%71.22% [66.78, 75.56]
OpenHands (SWELSP)73.33%71.67%68.67%71.22% [66.67, 75.78]

Cost reductions [95% CI]

AgentRun 1Run 2Run 3
MSE10.12% [4.80, 15.35]9.96% [3.06, 15.85]18.27% [11.53, 24.94]
OpenHands18.49% [11.54, 24.81]27.84% [21.18, 34.09]14.71% [5.23, 23.25]

Across all three seeds, SWELSP consistently reduces cost without statistically significant changes in pass@1. All six cost-reduction confidence intervals exclude zero. The paired pass@1 differences (SWELSP − Vanilla) are +1.11 percentage points for MSE (95% CI [−1.44, +3.67], p=0.449) and 0.00 points for OpenHands (95% CI [−2.11, +2.11], p=1.000). Neither difference is statistically distinguishable from zero.

(Question 2) For GPT-5.5, how much of the saving survives if cache hits are priced as misses?

Thank you. We have repriced all cache hits as misses on the same GPT-5.5 trajectories to isolate the effect of caching discounts:

Cost savings over vanillaOriginal pricingAll hits priced as misses
SWE-bench Multilingual10.12%6.49%
SWE-QA19.82%2.11%

SWELSP retains positive cost savings without caching discounts, although caching substantially amplifies the GPT-5.5 savings, particularly on SWE-QA. Importantly, the gains are not universally cache-dependent: the two DeepSeek models achieve 18.2–21.5% total-token reductions on SWE-bench Multilingual and 32.5–33.4% on SWE-QA, while maintaining or improving task performance.

(Question 3) Why not Serena plus your hooks as a baseline? That would isolate the hook contribution from the tool set.

Thanks. We additionally evaluated Serena + our hooks under the same OpenHands-GPT5.5 setting to assess whether hooks alone can reproduce SWELSP's gains.

ConfigurationResolve rateTokens/taskFile-view callsSearch/read tokens
Serena + Hooks67.0%1.039M8444.836M
SWELSP73.3%0.563M33.489M

Simply adding our hooks to Serena does not reproduce SWELSP's gains. Compared with Serena + Hooks, SWELSP reduces tokens by 45.8% while improving resolve rate by 6.3 percentage points. Serena + Hooks still makes 844 file-view calls, compared with only 3 for SWELSP, while SWELSP makes more targeted semantic calls (1,491 vs. 787).

These results highlight that the gains depend not on hooks alone, but on tightly integrating output enrichment with symbol-indexed navigation in agents' existing workflows.

(Question 4) Is there any human agreement check on the SWE-QA judge, even on a small sample?

Thanks. We conducted a human evaluation on 60 randomly sampled SWE-QA examples. Two annotators with expertise in coding agents independently evaluated the responses using the benchmark's scoring rubric, without access to the LLM judge's scores.

CriterionHuman meanLLM judge meanMean absolute error
Correctness9.429.32+0.15
Completeness9.218,82+0.39
Relevance9.969.93+0.03
Overall9.539.36+0.17

Human assessments closely align with the LLM judge's scores, with an overall mean absolute error of only 0.17 points on a 10-point scale.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

Thank you for your constructive feedback. We have carefully addressed all the reviewers' concerns in our rebuttal, including providing additional experimental evidence and analyses.

As the discussion period approaches its end, we would greatly appreciate it if the reviewers could review our responses and share any further feedback. We hope this additional evidence can be taken into account in your final evaluation of the paper.

Thank you again for your time and consideration.

Best, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.