- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SELF-INDEX, a framework that improves retrieval by iteratively updating document representations rather than changing the retriever. It keeps each original document and adds LLM-generated retrieval keys. Using simulated queries and retrieval feedback, the framework diagnoses weaknesses, revises keys, and checks them for faithfulness, specificity, and separation from competing representations.
Critical strengths
-
Improves retrieval without retraining the retriever, while keeping original documents as the evidence returned to users.
-
Tests the framework across multiple retrieval backbones and several retrieval and downstream tasks.
-
Reports both offline optimization costs and downstream online costs, providing a useful view of the efficiency trade-offs.
Critical weaknesses
-
The paper presents not relying on predefined revision strategies as an important feature of the method, but the paper’s code uses multiple templates that explicitly specify the forms of expression and combines candidates from multiple calls.
-
The paper requires candidate keys to retrieve their source documents based only on the original document texts, but the public code allows other candidate keys in the same proposal to increase the source document’s score. This means keys that do not meet the paper’s conditions may still pass validation. Therefore, the current ablation cannot directly support the Specificity mechanism as defined in the paper.
-
The paper already includes cost comparisons and component ablations, but it has not sufficiently separated the contributions of diagnosis-guided revision, manually designed template diversity, the candidate generation budget, and expanded document coverage.
-
Experiments show the results of adapting the method separately to three retrievers, but it remains unclear whether keys optimized for one retriever stay effective on others.If changing the retriever requires re-optimizing the index, the authors should clarify the additional cost and how this limits the method’s practical use.
Questions for authors
-
Could the authors report experimental results again that are consistent with the code implementation?
-
Improvements over iterations alone cannot prove that repeated revision is effective. Could the authors add two key experiments:
-
With the same document selection, generation templates, and a comparable compute budget, compare the full method with a direct generation-and-filtering baseline that does not use diagnosis guidance.
-
Fix the set of documents being expanded, control the number of keys and compute budget, and compare settings that allow repeated revision with settings that do not.
-
Given the paper’s generalization claims, could the authors test the method on additional full-corpus retrieval settings, such as NQ, HotpotQA, or 2WikiMultiHopQA, and report both retrieval and end-to-end question-answering results?
-
My main concern is that “the method the authors describe, the method actually executed, and the contributions supported by the experiments” are inconsistent. It seems that the paper is a repackaged presentation of the code. These differences make it difficult for readers to judge whether the reported improvements come from the mechanism proposed in the paper or from additional strategies that were not disclosed. They also weaken the interpretability of the current ablation experiments and the reproducibility of the results.
11 comments
Initial Response (2/3)
About validation implementation (W2, Q1, Q2, Q4)
We would like to clarify that all results reported in our paper were obtained using the logic described in the paper. After reviewing the code, we found that a version used for an internal ablation had been uploaded by mistake, and we corrected it immediately. We apologize for the confusion this caused in reproducing and interpreting our method.
In addition, to show the performance of the two implementations more concretely, we present their results side by side in the table below. Setting 1 is the Specificity implementation based on the original document text that we used in the paper’s experiments, and Setting 2 is the implementation that was uploaded by mistake.
| NL | Code | Math | |
|---|---|---|---|
| Setting 1 | 26.6 | 22.1 | 17.2 |
| Setting 2 | 26.0 (-0.6) | 21.1 (-1.0) | 16.8 (-0.4) |
The mistakenly uploaded implementation (Setting 2) performed slightly worse but remained close to Setting 1 overall. One possible explanation for this small difference is the Faithfulness check that precedes Specificity. Because Faithfulness first selects keys that express information needs supported by the original document, the other candidate keys for the same document may still have served as supplementary representations of its content in Setting 2. This in turn may have largely preserved Specificity’s filtering function and limited the impact of the implementation difference on final performance.
Revise vs Add (Q2)
We conducted additional experiments to assess the importance of repeatedly revising existing keys. Specifically, using GPT-5.4-nano with BM25 on BrowseComp-Plus, we compared performance after 10 iterations under the following three settings.
- Setting 1: uses the configuration reported in the paper.
- Setting 2: add only new keys without revising previously generated keys.
- Setting 3: raises the optimization top-k to 60 and, as in Setting 2, generates only new keys without revising previously generated keys.
The results are shown below.
| Setting | Acc | Recall | Coverage (%) |
|---|---|---|---|
| Baseline | 36.51 | 39.49 | 0 |
| Setting 1 | 54.34 | 59.70 | 34.2 |
| Setting 2 | 51.08 | 54.82 | 34.2 |
| Setting 3 | 53.86 | 57.40 | 62.4 |
First, Setting 2 outperformed the baseline, showing that adding keys to additional documents alone improves performance. However, it performed worse than Setting 1, which allows repeated revision of the existing key sets. This supports that repeatedly revising existing key sets contributes substantially to further performance gains.
Next, to see whether adding keys to more documents could close this performance gap, we evaluated Setting 3, which raises the optimization top-k in Setting 2 to 60. Setting 3 achieved higher coverage than Setting 1, but its performance was still lower. These results show that repeatedly refining existing key sets plays a more important role in improving performance than simply adding keys to more documents.
Initial Response (3/3)
Cross-Retriever generalization (W4)
We conducted additional experiments with the three retrievers used in our evaluation (BM25, BGE, and Qwen3) to test whether an index optimized with one retriever remains effective when applied to another. On BRIGHT, we evaluated each set of optimized keys with all three retrievers. For readability, we show results for one representative dataset per domain (Biology, Robotics, and TheoremQA-Theorem). The full results follow the same trend, and we can share the complete table in a follow-up. The results are shown below.
| Domain | Optimized by | BGE | BM25 | Qwen3 |
|---|---|---|---|---|
| NL | None (Baseline) | 12.4 | 18.8 | 16.8 |
| BGE | 32.0 | 36.4 | 30.5 | |
| BM25 | 30.9 | 34.4 | 31.7 | |
| Qwen3 | 30.0 | 33.9 | 29.7 | |
| Code | None (Baseline) | 12.2 | 13.6 | 14.4 |
| BGE | 22.4 | 21.5 | 20.1 | |
| BM25 | 16.5 | 18.8 | 17.1 | |
| Qwen3 | 20.0 | 21.2 | 22.9 | |
| Math | None (Baseline) | 5.3 | 4.9 | 22.1 |
| BGE | 15.0 | 8.3 | 28.8 | |
| BM25 | 14.7 | 10.8 | 27.8 | |
| Qwen3 | 13.6 | 8.9 | 31.4 |
The experimental results show that an index optimized with one retriever remains effective when applied to another. We would therefore like to emphasize that Self-Index can effectively accommodate a change of retriever at no additional cost.
About generalization (Q3)
In addition to evaluating retrieval performance across diverse domains using BRIGHT and additional benchmarks, we evaluated both the recall of evidence documents during search and final QA accuracy on the full BrowseComp-Plus corpus. This corpus contains diverse types of web content, including Wikipedia documents. We also expanded the corpus to 100K, 200K, and 400K documents to evaluate performance as the search corpus grew. We would like to emphasize that these benchmark characteristics and our experimental design provide sufficient support for Self-Index’s generalizability across domains, corpus compositions, and corpus sizes.
We did not include benchmarks such as NQ and HotpotQA in this evaluation because we wanted to assess more clearly how much improvements in retrieval contribute to final QA performance. Prior work has reported that models can correctly answer a substantial number of HotpotQA and 2WikiMultiHopQA questions without external context [1]. Studies of search agents using benchmarks such as NQ have also raised the concern that what models already know can obscure the contribution of retrieval [2]. For this reason, we chose BrowseComp-Plus, where answering questions correctly requires effective retrieval.
[1] LastingBench: Defend Benchmarks Against Knowledge Leakage; Fang et al
[2] DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments; Zheng et al
Initial Response (1/3)
Thank you for your thoughtful and constructive review. We hope our responses fully address your concerns. If there are any further questions or clarifications needed, we would be happy to address them.
Predefined Strategies and Diagnosis-based Revision (W1, W2, W3, Q1, Q2, Q4)
The key distinction we wanted to emphasize is the autonomy of the loop that uses retrieval results to decide what to improve in the index and how to revise it. As explained in lines 47–51 of our paper, when a predefined index optimization strategy does not adequately address information needs, existing methods require human intervention to diagnose the causes of retrieval failures and adjust and redefine the strategy. By contrast, in Self-Index, our Optimizer uses the current index keys and retrieval feedback to autonomously diagnose what is missing from each document’s representation and decide how to revise its keys accordingly.
In this context, the templates in our implementation serve only as “guidelines” for the form of candidate keys and the scope of their generation. The Optimizer determines revisions primarily through its own diagnosis based on the current keys and retrieval feedback. In practice, the Optimizer conditions its revisions on this diagnosis, and the templates have little effect on how effective those revisions are.
To support this claim, we examined how performance changed when we removed the diagnosis step entirely. The “w/o self-diagnose” row in the table below shows the results.
The results show that performance drops substantially when the diagnosis step is removed and revisions rely on the templates alone. If our templates really acted as a “significant predefined strategy,” performance should have been maintained in the “w/o self-diagnose” setting, where revisions rely solely on those templates. This shows that the templates themselves have little impact in our current implementation and that the main performance gains of Self-Index come from “autonomous diagnosis based on the current retrieval results.”
Moreover, to address your concern more directly, we also ran experiments with a version that removes those templates and replaces the multiple LLM calls with a single call. In this setting, the Optimizer uses its diagnosis to autonomously generate multiple keys in that single LLM call. We have also uploaded this version to the anonymous GitHub repository and would appreciate it if you could take a look. The results are shown in the “w/o template” row of the table above.
The performance difference between the version without templates and the existing implementation was small. In some cases, this version even outperformed the existing implementation. These results support the conclusion that, in the settings we evaluated, Self-Index’s performance does not depend heavily on this set of templates and that diagnosis-based revision can work effectively. If you would like to see additional experimental results for this setting, we would be happy to upload them.