- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents SWELSP, a symbol-indexed navigation interface that connects LSP to workflows. This is a simple yet effective method. SWELSP reduces API cost by a large margin while maintaining task performance on both benchmarks by reducing intermediate searches and windowed code reads.
Critical strengths
SWELSP is a simple yet effective method, reducing API cost by a large margin while maintaining task performance.
Critical weaknesses
-
The contribution appears to be essentially an interface integration of existing components, with no new methodological insight or mechanism. Without a clear differentiation from prior work, the paper's novelty is insufficient for publication.
-
The ablation compares the full method against Nav-only but cannot disentangle which factor drives the gain—whether symbol addresses improve tool usage, whether auto-attaching correct code saves subsequent reads, or whether both interact. This leaves the proposed mechanism unverified. Additionally, the reported 19.8% cost reduction is presented as if it reflects "processing 20% less content," but the effect actually depends on caching; this framing overstates the method's intrinsic efficiency gain and misrepresents the underlying mechanism.
Questions for authors
- The paper lacks novelty—it does not appear to introduce anything genuinely new; my understanding is that it is essentially just interface integration.
- What about the Pass@N metric? Would the full method yield further improvement there?
- Experiments show that the full method outperforms Nav-only, but the ablation cannot identify where the actual gain originates:
- Symbol addresses make it easier for the agent to use navigation tools;
- Automatically attaching the correct code saves subsequent reading steps;
- Or is it the combination of both?
- The cost-saving effect clearly depends on caching; the mechanism explanation should be more accurate. The 19.8% figure should not be directly interpreted as "processing about 20% less content."
4 comments
Responses to Questions and Weakness [2/2]
(Question 3) Experiments show that the full method outperforms Nav-only, but the ablation cannot identify where the actual gain originates: Symbol addresses make it easier for the agent to use navigation tools; Automatically attaching the correct code saves subsequent reading steps; Or is it the combination of both?
Thanks. We conducted additional component-level ablations to distinguish the contributions of the two mechanisms. We compare Nav-only (symbolic navigation on demand), Nav+Grep (additionally annotating grep outputs with symbol addresses), and Full SWELSP (further supplying function bodies automatically after failures).
| Model | Configuration | Pass@1 | Tokens/task | Cost/resolved |
|---|---|---|---|---|
| V4-Flash | Nav-only | 65.7% | 12.464M | $0.166 |
| Nav+Grep | 67.7% | 10.005M | $0.135 | |
| Full SWELSP | 71.0% | 8.329M | $0.103 | |
| GPT-5.5 | Nav-only | 67.0% | 0.393M | $0.710 |
| Nav+Grep | 68.33% | 0.360M | $0.677 | |
| Full SWELSP | 72.7% | 0.393M | $0.650 |
Both mechanisms contribute, but in different ways. Grep annotations reduce tokens by 19.7% and 8.4%, respectively, while improving Pass@1 by 2.0 and 1.3 percentage points. Automatically supplying failure-related function bodies further improves Pass@1 by 3.3 and 4.4 percentage points. For V4-Flash, this also reduces tokens by another 16.8%; for GPT-5.5, tokens increase, but cost per resolved task still decreases ($0.677 → $0.650).
Together, the two mechanisms achieve the highest Pass@1 and lowest cost per resolved task in both models, with grep annotations improving retrieval efficiency and automatic failure-context delivery providing additional resolution gains.
(Question 4) The cost-saving effect clearly depends on caching; the mechanism explanation should be more accurate. The 19.8% figure should not be directly interpreted as "processing about 20% less content.
Thank you for pointing this out. We agree that API cost reduction should not be conflated with total-token reduction. Specifically, for GPT-5.5 on SWE-QA, SWELSP reduces total tokens by only 0.8%, while the 19.8% cost reduction reflects cache-hit, cache-miss, and output-token pricing. We will clarify this distinction in the paper.
To isolate the impact of caching discounts, we repriced all cache hits as misses on the same GPT-5.5 trajectories:
| Cost savings over vanilla | Original pricing | All hits priced as misses |
|---|---|---|
| SWE-bench Multilingual | 10.12% | 6.49% |
| SWE-QA | 19.82% | 2.11% |
Caching substantially amplifies GPT-5.5's cost savings, particularly on SWE-QA, but positive savings remain without caching discounts. Moreover, the two DeepSeek models achieve substantial total-token reductions of 18.2–21.5% on SWE-bench Multilingual and 32.5–33.4% on SWE-QA, while maintaining or improving task performance.
Thanks for the additional experiments. The new ablations help clarify where the gains come from, though the interaction between the two components is still unclear. The caching analysis addresses my main concern about the cost claim; please distinguish cost savings from token savings in the paper. I will keep my weak-accept rating.
Thank you for your feedback. We will clearly distinguish API cost savings from total-token savings in the paper, and add the analysis below to clarify how the two components interact.
To clarify how the two mechanisms interact with agent, we analyzed agents' tool-use patterns across the ablation settings:
| Model | Post-grep symbolic lookup rate (Nav-only → Nav+Grep) | Avg. post-failure search/read calls (Nav+Grep → Full SWELSP) |
|---|---|---|
| GPT-5.5 | 42.9% → 68.5% (+59.7%) | 1.36 → 1.28 (-5.6%) |
| V4-Flash | 5.5% → 19.7% (+255.5%) | 2.47 → 1.81 (-26.5%) |
The two mechanisms operate at complementary stages of the agent workflow: Grep annotations guide initial search toward target symbols, while automatic failure-context delivery decreases follow-up searches after test failures. Exposing symbol addresses bridges text search and symbolic navigation, boosting post-grep symbolic look up rate by 59.7% (GPT-5.5) and 255.5% (V4-Flash). Once a test fails, supplying failure-related function bodies decreases redundant exploratory reads, reducing follow-up search/read calls by 5.6% (GPT-5.5) and 26.5% (V4-Flash).Together, they achieve the highest Pass@1 and lowest cost per resolved task across both models.
Responses to Questions and Weakness [1/2]
Thank you for your insightful review. We have addressed all your questions as follows.
Thank you for raising this concern. SWELSP addresses a key limitation of existing symbolic tools (e.g., Serena and CodeStruct) : making precise navigation available does not ensure that agents actually use it to replace redundant text-based exploration. Rather than simply adding another retrieval interface, SWELSP introduces symbol-based output enrichment, which automatically exposes directly usable symbol addresses in existing Bash/grep outputs, connecting agents' familiar search workflows to targeted semantic navigation.
Our experiments demonstrate why this distinction matters. In Appendix B, merely adding symbolic navigation tools to Mini-SWE-Agent increases API cost by 17.5% and decreases Pass@1 by 5.0 percentage points on V4-Flash. In contrast, SWELSP reduces cost by 21.3% while preserving task performance. We further evaluated CodeStruct and Serena+ (Serena augmented with comparable output annotations and tool instructions) under the same OpenHands-GPT5.5 setting:
Neither baseline reproduces SWELSP's efficiency gains. Compared with Serena+ and CodeStruct, SWELSP reduces total tokens by 45.8% and 15.5%, respectively, while improving resolve rates by 6.3 and 7.3 percentage points. The sharp reduction in conventional file-view calls further supports a shift toward targeted semantic retrieval.
Thus, SWELSP's novelty lies not in introducing symbolic retrieval itself, but in making it effective within agents' existing workflows. These results highlight a broader methodological insight: how retrieval capabilities are exposed to agents can fundamentally affect their tool-use behavior and end-to-end efficiency.
Thanks. Following your suggestion, we conducted three independent GPT-5.5 runs on SWE-bench Multilingual to evaluate Pass@3. The results are summarized below.
SWELSP improves Pass@3 from 79.33% to 81.67% (+2.34 percentage points) on Mini-SWE-Agent, outperforming SWE-Pruner while reducing average tokens by 12.8%. On OpenHands, SWELSP achieves 79.67% Pass@3, slightly below vanilla (81.00%) but matching Serena and AiderMap, while reducing tokens by 23.7% relative to vanilla.
Overall, SWELSP improves Pass@3 in Mini-SWE-Agent and maintains competitive Pass@3 in OpenHands, with consistent token savings across both frameworks.