- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CorpusWalker separates graph-based candidate discovery from final ranking. Starting from BM25 seeds, an LLM agent reads cached, query-independent document summaries and decides which document and which lexical or semantic neighborhood to expand. A Meta Reranker then ranks the resulting candidate pool using summaries, expansion rationales, and provenance from the traversal. The design aims to find useful evidence beyond an initial retrieval pool and reuse information already obtained during exploration. Experiments on BRIGHT and FreshStack compare candidate discovery, final ranking, and online computational cost under a 100-document candidate budget.
Critical strengths
-
Evidence for both candidate discovery and reuse of traversal information (PDF pp. 6–9, Figure 2 and Tables 1–3). On BRIGHT, candidate Recall@100 increases from RGS's 50.1 to 54.8, while final nDCG@10 increases from 33.0 to 36.1. Reporting both measures is valuable: it distinguishes whether useful documents enter the pool from whether the final ranker places them near the top. The strongest component evidence is Table 3, which holds the candidate pool fixed and improves nDCG@10 from 32.8 with summaries and keywords to 36.1 with the complete traversal-information input. Because candidate discovery is held constant in this comparison, the ranking gain cannot be explained simply by supplying a better pool. This supports the practical value of reusing the traversal information package, including its provenance, ordering, and reasons for expansion. I regard that package as a supported contribution; isolating the prose rationale from every other trace element is not a prerequisite for recognizing this result.
-
Complementary tasks and useful evidence about practical operation (PDF pp. 8–9 and Appendices A–G). FreshStack tests a different aspect of retrieval by rewarding coverage of distinct answer nuggets. CorpusWalker obtains alpha-nDCG@10 of 41.3 versus RGS's 39.3 and Coverage@20 of 67.8 versus 65.0. These results broaden the empirical case beyond BRIGHT's document-relevance measurements. The paper also provides the traversal and ranking prompts, illustrative walks, edge-use and traversal statistics, query-reasoning comparisons, and an agent-size study with the Meta Reranker fixed. Online token and call accounting includes invalid-action retries. Together, these analyses make the system's operation more inspectable and show how compressed document representations can reduce repeated reading during search and ranking. The cost results concern online operation; the summaries themselves are constructed offline. This is a useful deployment distinction, and the reported online savings deserve credit within that scope.
Critical weaknesses
-
The exploration gain is not separated from the change in available representations and graph resources (PDF pp. 1–2, 6, 9 and 15–18, Figures 1–3/5–6 and Appendix D). The central hypothesis concerns reasoning about where to expand, rather than expanding according to current relevance. However, the comparison with RGS changes several ingredients at once: CorpusWalker uses compressed summaries instead of raw documents and can traverse lexical as well as semantic neighborhoods, whereas the evaluated RGS uses its semantic graph. The common BM25 seeding method, listwise LLM, and 100-document budget are useful controls. They make the comparison between the implemented systems informative, but they do not isolate the additional value of the proposed expansion decisions.
The resource difference is consequential rather than merely hypothetical. Figure 1 and the example walks in Figure 6 show relevant documents reached through lexical edges that RGS cannot use. These examples explain a plausible advantage of the complete design, while leaving open whether a simpler policy with the same dual-graph access and summaries would recover much of the gain. The agent-size study and traversal statistics characterize the proposed agent; they do not provide that alternative-policy comparison. Likewise, Table 3 establishes the value of the trace package for ranking a fixed pool, but cannot determine why the exploration procedure found that pool.
A focused comparison with a competitive relevance-guided or simple hybrid-graph policy, supplied with the same summaries and both graphs, would directly address this uncertainty. Candidate-pool recall or nugget coverage is the most informative first outcome, because it measures discovery before a different ranking input can affect the result. Reporting token and call costs alongside it would clarify whether the extra deliberation buys useful evidence. This concern limits the evidence for the paper's central exploration claim; it does not imply that the reported system improvements are incorrect or that the implemented baselines are illegitimate.
I lean toward weak rejection, close to the acceptance boundary. The improvements on two benchmarks, the useful online-cost analysis, and especially the same-pool trace-reuse result give the paper clear positive value. My reservation is concentrated on the exploration contribution: the current comparisons do not sufficiently distinguish better expansion decisions from better representations and additional graph access. A meaningful gain in pool quality over a competitive policy using the same resources would substantially strengthen the case for acceptance. If that policy instead recovered most of the gain at comparable or lower online cost, the contribution would be better supported as a system integration and ranking result, with a narrower claim about reasoning-guided discovery.
Questions for authors
For W1 — main priority: How does CorpusWalker compare with a competitive relevance-guided or clearly specified simple hybrid-graph policy using the same BM25 seeds, summaries, lexical/semantic graphs, LLM, and 100-document budget? Please prioritize candidate-pool Recall@100 or nugget coverage and token/call costs, which assess discovery before reranking. For the simple policy, state how it chooses both the expansion document and edge type. If final ranking is also compared, use a shared ranker with matched inputs so that trace availability does not introduce a second difference. This focused comparison would clarify whether the expansion policy itself is responsible for a meaningful portion of the observed gain.
6 comments
Response to Reviewer 1sNG (2/2)
Q2. If final ranking is also compared, use a shared ranker with matched inputs so that trace availability does not introduce a second difference. This focused comparison would clarify whether the expansion policy itself is responsible for a meaningful portion of the observed gain.
Response. We ran the shared listwise reranker (RankGPT or MetaReranker) over the matched inputs (raw texts or summaries). We only change the pool retrieved by RGS and CorpusWalker. We found that the CorpuWalker’s (ours) pool helps in improving the nDCG@10 by 1.4 points and Recall@10 by 2.1 points over the RGS baseline. We report per-subset performance in the updated version on Table 3 in the revised pdf. Next, we use the one-shot MetaReranker over a fixed pool and we compare the impact of availability of traces (similar to Table 3 in paper). The traces add 1.7 points gain in nDCG@10 and 2.1 in Recall@10. Hence these gains in the pool and ranking are separable and the Corpuswalker agent provides the gains as observed from our experiments.
| Pool | Ranker | doc representation | traces (yes/no) | nDCG@10 | R@10 |
|---|---|---|---|---|---|
| CorpusWalker | RankGPT | raw text | no | 34.4 | 37.2 |
| RGS as published | RankGPT | raw text | no | 33.0 | 35.1 |
| CorpusWalker | MetaReranker | summaries | no | 32.8 | 36.1 |
| CorpusWalker | MetaReranker | summaries | yes | 36.1 | 39.3 |
Q3. A meaningful gain in pool quality over a competitive policy using the same resources would substantially strengthen the case for acceptance. If that policy instead recovered most of the gain at comparable or lower online cost, the contribution would be better supported as a system integration and ranking result, with a narrower claim about reasoning-guided discovery.
Response In comparison to the simple and relevance guided strategies, the ranker guided graph expansion strategy provides better performance, Recall@100 of 50.5% Vs Recall@100 of 40.3% by Breadth-first selection which comes at no cost (it is independent of document representations, e.g, raw or summary). On the other hand, RGS applies listwise ranking over the pool at each iteration resulting in more expensive LLM calls. In vanilla setup, RGS uses raw text of the documents exacerbating token costs and LLM calls in comparison to the CorpusWalker. However, our proposed document summaries help RGS to maintain the pool quality and reduce the token cost from 122k tokens/query to 50k tokens/query which makes the proposed component suitable for system integration to existing graph expansion strategies. But, since the vanilla RGS does not disentangle the exploration and ranking, these summaries degrade the ranking quality, for example with summaries nDCG@10 is 30.3 in comparison to the vanilla RGS over raw document text nDCG@10 is 33.0. However, this still falls behind the CorpusWalker’s pool quality which uses 107k tokens per query. But, the extra tokens leads to gains of points in Recall@100 and points in nDCg@10 in fewer LLM calls (10.1 Vs 14.7-20).
Thanks again for the feedback. We have tried to address all the concerns. We have also uploaded the revised pdf. We keep all changes in blue fonts. We look forward to answer any further queries.
A gentle reminder
Greetings Reviewer 1sNG,
Thank you for your time and valuable feedback. We have tried to address your queries regarding the contribution of the expansion strategy and comparison with relevance-guided strategies.. We have also uploaded the updated version of the paper. We would like to request if we have addressed your concerns. We are happy to answer any further queries. If we have addressed your concerns, we politely and respectfully request a reassessment of the rating in light of our response. Thank you for your time and consideration.
Response to Reviewer 1sNG (1/2)
Thank you for your valuable feedback. We would like to clarify the queries raised in the review as follows. We would like to clarify that we call the agent-based expansion mechanisms as
strategyand notpolicy.Response. We thank the reviewer for the feedback. We ran the experiment as requested: the same BM25 seeds, the same 100-document budget, the same two graphs, the same summaries, the same LLM and the same ranker, with the only difference being the expansion strategy. We report pool quality by Recall@100, LLM calls and token cost per query.
Relevance Guided Expansion Strategies: Relevance-greedy with perfect labels expands the earliest-pooled unexpanded document that the qrels mark relevant, along a fixed or alternating edge type; it knows the true relevance of every document already in the pool, so it is an upper bound on any reranker-guided strategy. We agree that this is not practically possible but we want to see how much we gain from this expansion baseline. If there is no (more) gold document in the pool at any turn t, the first document in the unexpanded pool is expanded. RGS relies on re-ranker signal to guide exploration over document summaries and both the graphs. RGS performs this iteratively using either of the graphs in an alternating manner or union of 2 neighborhoods from both the graphs during each expansion.
The simple hybrid graph strategies barely beat the non-expanding baseline (BM25). The Breadth-first expansion reaches Recall@100 of 40.3% and random strategy degrades the BM25 baseline by 4.9 points which concludes that the availability of the graphs does not help if the expansion strategy is not good.
We then quantify the impact of relevance-guided expansion strategy, mainly relevance greedy that uses the gold labels and RGS that uses the ranker’s relevance. The relevance greedy achieves the Recall@100 of 49.6% lexical edges, 49.4% on semantic and 50.0% using alternating edges. Our CorpusWalker expansion outperforms this by 4.8 points and on 11 out of 12 subsets (we report per subset performance in Table 5 in revised pdf) except pony where the majority of the gold documents are 1-hop away from seed documents. In comparison to RGS, where it uses its own expansion strategy, when given the same resources as CorpusWalker, the pool quality does not change (50.0 Vs 50.5 and 50.1) when compared with given full document texts and single graphs. It even degrades the ranking performance when document summaries are used for reranking, nDCG@10 drops from 33.0 to 30.1, which shows that the documents summaries are not enough to get better ranking. The same observation we made in Table 3 when the same pool with document summaries was reranked using MetaReranker.