- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes CorpusMap, a new entity-based indexing system for agentic retrieval. CorpusMap recognizes and resolves the named entities in documents to create entity pages, and uses the entity pages as input to LLM agents for navigation. On EnterpriseRAG-Bench, WixQA and HERB, the method achieve better results than baselines in most cases. Practical aspects e.g. cost to build the index, and reusing the map over different LLMs and incremental updates are also discussed.
Critical strengths
- The paper is clearly defined, easy to understand, and concise in construction. I think this is an important contribution to advancing enterprise agentic search, and a cheaper yet potentially more effective solution than the well-used GraphRAG.
- Experiments are generally well conducted, with cost reporting, discussion of practical aspects, and quite decent backbone coverage. I agree the method would work in many cases, with some limitations on the dataset coverage; please see weaknesses below.
Critical weaknesses
I am not very satisfied with the scale of the experiments in terms of the number of queries, documents, and diversity of datasets. Did you try to cover diverse types of datasets, i.e., weakly linked documents, entity-dense documents, table-heavy ones, etc. And a scalability test for more documents would also strengthen the efficiency experiments. I am worried that as the number of documents scales up, the accuracy/noise in the NER step would degrade the downstream performance. Besides, according to the results in Table 8, having the entity-linked docs is almost the same performance as the full method, so some discussion on in which cases we need to walk on the map vs. when we only need to retrieve based on entities would be helpful.
Questions for authors
Please see weaknesses. Additionally, please also report the performance of name entity resolution if that is feasible.
6 comments
Response to Reviewer iZj1 (2/2)
Weaknesses 2.1) A scalability test for more documents would also strengthen the efficiency experiments.
We also consider scalability important, which is why we examined CorpusMap with GLinker as the corpus grows in Figure 4, where it outperforms Raw Corpus on every metric from 2.8k to 20k documents. Nevertheless, following your suggestion, we further scale the corpus to 50k and 100k documents. As shown in Table C.2, CorpusMap with GLinker still outperforms Raw Corpus on every metric at both sizes while using fewer tokens. Thank you again for this helpful suggestion!
Table C.2: Performance on larger corpora of EnterpriseRAG-Bench
| Documents | Method | Doc. Rec. | Correct. | Complete. | Tokens |
|---|---|---|---|---|---|
| 50k | Raw Corpus | 43.70 | 41.25 | 63.55 | 110.7k |
| 50k | CorpusMap with GLinker (Ours) | 60.72 | 65.00 | 74.50 | 70.7k |
| 100k | Raw Corpus | 42.54 | 45.00 | 57.94 | 134.4k |
| 100k | CorpusMap with GLinker (Ours) | 62.59 | 57.50 | 70.18 | 71.7k |
Weaknesses 2.2) I am worried that as the number of documents scales up, the accuracy/noise in the NER step would degrade the downstream performance.
We thank you for raising this important concern. Entity recognition (NER) in CorpusMap does not depend on the number of documents, as it recognizes the mentions of each document from that document alone, without referring to other documents. For instance, since our larger corpora are built by adding documents to the 2,819 documents of EnterpriseRAG-Bench, we compare the mentions recognized in these 2,819 documents across corpus sizes, and they are the same from 2.8k to 100k documents. However, we agree that linking these mentions to the same entity across documents (entity resolution) can become more challenging as the corpus grows, since more entities with similar names appear, which may introduce noise into the map. For example, measured on the employees of EnterpriseRAG-Bench, whose ground-truth identities are available, the precision of linking their mentions is 0.989 at 2.8k and 0.987 at 100k documents, while its recall among the linked mentions decreases from 0.957 to 0.912. Nevertheless, entity recognition and linking are intermediate steps for building the map that the agent uses, and CorpusMap continues to outperform Raw Corpus on every metric at 50k and 100k documents (Table C.2), as discussed in our response to Weaknesses 2.1. We will include the evaluation of linking on larger corpora in the updated paper.
Weaknesses 3) According to the results in Table 8, having the entity-linked docs is almost the same performance as the full method, so some discussion on in which cases we need to walk on the map vs. when we only need to retrieve based on entities would be helpful.
We would like to clarify that the setting with the entity-linked documents that you mentioned is exactly our full method, in which the agent receives these documents as its initial candidates while having access to the full map (Section 3.4 and Appendix B.3). Table 8 compares variants of CorpusMap that differ only in the candidates given to the agent, namely none, Entity Pages, entity-linked documents, or both, and giving both is one of these variants rather than the full method. We chose the entity-linked documents as the default for their efficiency, as they achieve comparable quality with the fewest tokens (Table 8). Specifically, with Entity Pages as candidates, the agent reads these pages in addition to the documents they link, whereas with the linked documents, it opens these documents right away and can still explore the map from them when needed. For instance, Table 10 shows a case where the agent needs to walk on the map, in which it reads a draft of a manifest and then reaches its v1 specification, stored in a different application, through the Entity Page of Serving Runtime.
Question 1) Additionally, please also report the performance of name entity resolution if that is feasible.
We sincerely thank you for this suggestion. Accordingly, we evaluate the entity resolution of CorpusMap on the employees of HERB, whose ground-truth identities are available. Specifically, we check whether the map links every document that mentions an employee to the Entity Page of that employee, without mixing different employees in one page. CorpusMap achieves a B³ F1 of 0.976, a standard metric of cross-document coreference [C4], with precision 0.999 and recall 0.954. In addition, we discuss how entity resolution changes as the corpus grows in our response to Weaknesses 2.2. We will include this evaluation in the updated paper.
[C4] Algorithms for Scoring Coreference Chains
Thank you for the additional experiments and clarifications. I now understand that the entity-linked document setting in Table 8 retains full map access.
My remaining question is whether online map navigation provides additional benefits beyond entity-based candidate retrieval. Could you compare the same entity-derived candidate list with versus without subsequent access to the map, keeping the agent, raw-document search tools, and budget fixed? This would help isolate the contribution of navigation and clarify when it is useful.
After considering the other reviewer’s concerns about novelty, I still see a meaningful contribution in the proposed design and empirical results, and I maintain my weak acceptance rating. However, I encourage the authors to narrow the novelty claims and more precisely position the work relative to GraphReader and HippoRAG 2, acknowledging the existing ideas while emphasizing the specific design choices and benefits demonstrated here.
Overall, I remain supportive of this paper.
Response to Reviewer iZj1 (1/2)
We thank you very much for your careful and constructive review. We are also grateful that you recognized CorpusMap as a new entity-based approach to agentic retrieval and an important contribution to advancing enterprise agentic search with a more effective solution at a lower cost, and that you found our method clear and concise and our experiments well conducted, including the cost analysis, practical considerations such as map reuse and incremental updates, and the coverage of multiple LLMs. Below, we respond to each of your comments, including new results on additional benchmarks and larger corpora.
We thank you for this insightful comment, and we fully agree that the diversity of datasets is important, which is why we chose three benchmarks that include both entity-dense and weakly linked documents, while covering diverse types of documents, from issue tickets, wiki pages, and pull requests across nine workplace applications (EnterpriseRAG-Bench) and help-center articles (WixQA) to chat messages and meeting transcripts (HERB). For instance, EnterpriseRAG-Bench is entity-dense, with 24.7 Entity Pages linked to each document on average, whereas WixQA is weakly linked, as the articles needed for the same question typically share only broad entities, such as a product mentioned in many other articles. As shown in Table 1 and summarized in Table C.1, CorpusMap improves over Raw Corpus on all three benchmarks.
In addition, to further address your concern, we evaluate CorpusMap on three more benchmarks of different types, namely entity-dense news articles (MultiHop-RAG [C1]), table-heavy financial reports (ViDoRe V3 Finance-EN [C2]), and documents with renamed entities (SynthWorlds-SM [C3]). As shown in Table C.1, CorpusMap outperforms Raw Corpus on every metric of all three additional benchmarks while using fewer tokens. Regarding scale, these benchmarks include up to 2,255 questions, and we further scale the corpus up to 100k documents, as detailed in our response to Weaknesses 2.1. We thank you again for this valuable comment. We will include these results in the updated paper.
Table C.1: Performance on our three benchmarks and three additional benchmarks
[C1] MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
[C2] ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios
[C3] SynthWorlds: Controlled Parallel Worlds for Disentangling Reasoning and Knowledge in Language Models