acceptodds
Under review as a conference paper at ICLR 2027

Self-Evolving Search Index

26 views

Abstract

Information retrieval is increasingly important as LLM agents tackle complex tasks involving diverse information needs. Because retrieval relies on an index that represents each document through index keys, retrieval quality depends heavily on how effectively these keys expose the knowledge contained in each document. However, effective index representations vary across retrieval environments, making it difficult for any fixed optimization strategy to perform consistently. Yet evolving an index to its retrieval environment remains largely human-driven, requiring humans to diagnose retrieval failures, refine the optimization strategy, and reprocess the index accordingly. We propose SELF-INDEX, a framework that enables an index to self-evolve without human intervention. Its Optimizer autonomously diagnoses retrieval shortfalls, selectively revises the responsible index keys, and validates each revision before updating the index. Beyond reacting to observed retrieval demands, SELF-INDEX proactively explores additional demands through a Query Simulator, allowing the index to evolve beyond the queries already available for optimization. Across diverse corpora and retrievers, SELF-INDEX consistently improves retrieval performance while outperforming existing index optimization methods. We further show that these benefits extend to downstream applications, improving the effectiveness and efficiency of search agents and helping agent memory systems retrieve useful past interactions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Information retrieval, Index optimization, Self evolving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7iUkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SELF-INDEX, a framework that improves retrieval by iteratively updating document representations rather than changing the retriever. It keeps each original document and adds LLM-generated retrieval keys. Using simulated queries and retrieval feedback, the framework diagnoses weaknesses, revises keys, and checks them for faithfulness, specificity, and separation from competing representations.

Critical strengths
  • Improves retrieval without retraining the retriever, while keeping original documents as the evidence returned to users.

  • Tests the framework across multiple retrieval backbones and several retrieval and downstream tasks.

  • Reports both offline optimization costs and downstream online costs, providing a useful view of the efficiency trade-offs.

Critical weaknesses
  • The paper presents not relying on predefined revision strategies as an important feature of the method, but the paper’s code uses multiple templates that explicitly specify the forms of expression and combines candidates from multiple calls.

  • The paper requires candidate keys to retrieve their source documents based only on the original document texts, but the public code allows other candidate keys in the same proposal to increase the source document’s score. This means keys that do not meet the paper’s conditions may still pass validation. Therefore, the current ablation cannot directly support the Specificity mechanism as defined in the paper.

  • The paper already includes cost comparisons and component ablations, but it has not sufficiently separated the contributions of diagnosis-guided revision, manually designed template diversity, the candidate generation budget, and expanded document coverage.

  • Experiments show the results of adapting the method separately to three retrievers, but it remains unclear whether keys optimized for one retriever stay effective on others.If changing the retriever requires re-optimizing the index, the authors should clarify the additional cost and how this limits the method’s practical use.

Questions for authors
  1. Could the authors report experimental results again that are consistent with the code implementation?

  2. Improvements over iterations alone cannot prove that repeated revision is effective. Could the authors add two key experiments:

  • With the same document selection, generation templates, and a comparable compute budget, compare the full method with a direct generation-and-filtering baseline that does not use diagnosis guidance.

  • Fix the set of documents being expanded, control the number of keys and compute budget, and compare settings that allow repeated revision with settings that do not.

  1. Given the paper’s generalization claims, could the authors test the method on additional full-corpus retrieval settings, such as NQ, HotpotQA, or 2WikiMultiHopQA, and report both retrieval and end-to-end question-answering results?

  2. My main concern is that “the method the authors describe, the method actually executed, and the contributions supported by the experiments” are inconsistent. It seems that the paper is a repackaged presentation of the code. These differences make it difficult for readers to judge whether the reported improvements come from the mechanism proposed in the paper or from additional strategies that were not disclosed. They also weaken the interpretability of the current ablation experiments and the reproducibility of the results.

11 comments
AuthorsOpenReview ↗

Initial Response (1/3)

Thank you for your thoughtful and constructive review. We hope our responses fully address your concerns. If there are any further questions or clarifications needed, we would be happy to address them.

Predefined Strategies and Diagnosis-based Revision (W1, W2, W3, Q1, Q2, Q4)

The key distinction we wanted to emphasize is the autonomy of the loop that uses retrieval results to decide what to improve in the index and how to revise it. As explained in lines 47–51 of our paper, when a predefined index optimization strategy does not adequately address information needs, existing methods require human intervention to diagnose the causes of retrieval failures and adjust and redefine the strategy. By contrast, in Self-Index, our Optimizer uses the current index keys and retrieval feedback to autonomously diagnose what is missing from each document’s representation and decide how to revise its keys accordingly.

In this context, the templates in our implementation serve only as “guidelines” for the form of candidate keys and the scope of their generation. The Optimizer determines revisions primarily through its own diagnosis based on the current keys and retrieval feedback. In practice, the Optimizer conditions its revisions on this diagnosis, and the templates have little effect on how effective those revisions are.

To support this claim, we examined how performance changed when we removed the diagnosis step entirely. The “w/o self-diagnose” row in the table below shows the results.

MethodBio.Ear.Econ.Psy.Sus.Rob.Sta.Leet.Pony.AopsTheoQTheoTAvg.
BM25existing implementation34.438.317.618.418.518.821.622.417.67.718.810.820.4
w/o self-diagnose11.219.09.39.912.312.611.920.99.65.89.25.911.5
w/o template37.740.917.117.116.918.020.121.419.67.117.38.620.1
BGEexisting implementation32.040.119.821.822.322.418.524.719.57.817.315.021.8
w/o self-diagnose11.325.314.617.617.411.512.124.514.75.013.013.515.0
w/o template34.642.820.622.022.821.216.225.818.98.019.214.622.2
Qwen3existing implementation29.740.019.725.621.022.923.131.622.410.635.831.426.1
w/o self-diagnose15.225.813.921.115.712.315.526.112.65.223.126.517.7
w/o template32.842.919.527.622.421.122.328.725.010.332.930.526.3

The results show that performance drops substantially when the diagnosis step is removed and revisions rely on the templates alone. If our templates really acted as a “significant predefined strategy,” performance should have been maintained in the “w/o self-diagnose” setting, where revisions rely solely on those templates. This shows that the templates themselves have little impact in our current implementation and that the main performance gains of Self-Index come from “autonomous diagnosis based on the current retrieval results.”

Moreover, to address your concern more directly, we also ran experiments with a version that removes those templates and replaces the multiple LLM calls with a single call. In this setting, the Optimizer uses its diagnosis to autonomously generate multiple keys in that single LLM call. We have also uploaded this version to the anonymous GitHub repository and would appreciate it if you could take a look. The results are shown in the “w/o template” row of the table above.

The performance difference between the version without templates and the existing implementation was small. In some cases, this version even outperformed the existing implementation. These results support the conclusion that, in the settings we evaluated, Self-Index’s performance does not depend heavily on this set of templates and that diagnosis-based revision can work effectively. If you would like to see additional experimental results for this setting, we would be happy to upload them.

AuthorsOpenReview ↗

Initial Response (2/3)

About validation implementation (W2, Q1, Q2, Q4)

We would like to clarify that all results reported in our paper were obtained using the logic described in the paper. After reviewing the code, we found that a version used for an internal ablation had been uploaded by mistake, and we corrected it immediately. We apologize for the confusion this caused in reproducing and interpreting our method.

In addition, to show the performance of the two implementations more concretely, we present their results side by side in the table below. Setting 1 is the Specificity implementation based on the original document text that we used in the paper’s experiments, and Setting 2 is the implementation that was uploaded by mistake.

NLCodeMath
Setting 126.622.117.2
Setting 226.0 (-0.6)21.1 (-1.0)16.8 (-0.4)

The mistakenly uploaded implementation (Setting 2) performed slightly worse but remained close to Setting 1 overall. One possible explanation for this small difference is the Faithfulness check that precedes Specificity. Because Faithfulness first selects keys that express information needs supported by the original document, the other candidate keys for the same document may still have served as supplementary representations of its content in Setting 2. This in turn may have largely preserved Specificity’s filtering function and limited the impact of the implementation difference on final performance.

Revise vs Add (Q2)

We conducted additional experiments to assess the importance of repeatedly revising existing keys. Specifically, using GPT-5.4-nano with BM25 on BrowseComp-Plus, we compared performance after 10 iterations under the following three settings.

  • Setting 1: uses the configuration reported in the paper.
  • Setting 2: add only new keys without revising previously generated keys.
  • Setting 3: raises the optimization top-k to 60 and, as in Setting 2, generates only new keys without revising previously generated keys.

The results are shown below.

SettingAccRecallCoverage (%)
Baseline36.5139.490
Setting 154.3459.7034.2
Setting 251.0854.8234.2
Setting 353.8657.4062.4

First, Setting 2 outperformed the baseline, showing that adding keys to additional documents alone improves performance. However, it performed worse than Setting 1, which allows repeated revision of the existing key sets. This supports that repeatedly revising existing key sets contributes substantially to further performance gains.

Next, to see whether adding keys to more documents could close this performance gap, we evaluated Setting 3, which raises the optimization top-k in Setting 2 to 60. Setting 3 achieved higher coverage than Setting 1, but its performance was still lower. These results show that repeatedly refining existing key sets plays a more important role in improving performance than simply adding keys to more documents.

AuthorsOpenReview ↗

Initial Response (3/3)

Cross-Retriever generalization (W4)

We conducted additional experiments with the three retrievers used in our evaluation (BM25, BGE, and Qwen3) to test whether an index optimized with one retriever remains effective when applied to another. On BRIGHT, we evaluated each set of optimized keys with all three retrievers. For readability, we show results for one representative dataset per domain (Biology, Robotics, and TheoremQA-Theorem). The full results follow the same trend, and we can share the complete table in a follow-up. The results are shown below.

DomainOptimized byBGEBM25Qwen3
NLNone (Baseline)12.418.816.8
BGE32.036.430.5
BM2530.934.431.7
Qwen330.033.929.7
CodeNone (Baseline)12.213.614.4
BGE22.421.520.1
BM2516.518.817.1
Qwen320.021.222.9
MathNone (Baseline)5.34.922.1
BGE15.08.328.8
BM2514.710.827.8
Qwen313.68.931.4

The experimental results show that an index optimized with one retriever remains effective when applied to another. We would therefore like to emphasize that Self-Index can effectively accommodate a change of retriever at no additional cost.

About generalization (Q3)

In addition to evaluating retrieval performance across diverse domains using BRIGHT and additional benchmarks, we evaluated both the recall of evidence documents during search and final QA accuracy on the full BrowseComp-Plus corpus. This corpus contains diverse types of web content, including Wikipedia documents. We also expanded the corpus to 100K, 200K, and 400K documents to evaluate performance as the search corpus grew. We would like to emphasize that these benchmark characteristics and our experimental design provide sufficient support for Self-Index’s generalizability across domains, corpus compositions, and corpus sizes.

We did not include benchmarks such as NQ and HotpotQA in this evaluation because we wanted to assess more clearly how much improvements in retrieval contribute to final QA performance. Prior work has reported that models can correctly answer a substantial number of HotpotQA and 2WikiMultiHopQA questions without external context [1]. Studies of search agents using benchmarks such as NQ have also raised the concern that what models already know can obscure the contribution of retrieval [2]. For this reason, we chose BrowseComp-Plus, where answering questions correctly requires effective retrieval.

[1] LastingBench: Defend Benchmarks Against Knowledge Leakage; Fang et al
[2] DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments; Zheng et al

Reviewer_j6BEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SELF-INDEX, a method that uses an LLM to improve document index keys through repeated diagnosis, revision, and validation. It generates queries to guide these updates while keeping the documents and retriever fixed. Experiments report improvements in retrieval quality, search-agent performance, and agent memory use.

Critical strengths

It is an interesting topic to keep documents and retrievers fixed while optimizing the index dynamically. Traditional keyword-based indexing builds indexes directly from words in the documents. Later indexing approaches began to link query meaning with the indexing process. With advances in these approaches, evolving indexes is becoming more feasible and increasingly necessary.

Critical weaknesses
  1. The approach is not truly "Self-Evolving". The experiments use a fixed number of optimization iterations, but the paper does not explain when further updates are needed or what triggers them in deployment. This leaves a gap between the demonstrated offline optimization procedure and the claimed ability of the index to evolve autonomously.

  2. The full computational and storage costs are not evaluated. Although the paper reports selected offline LLM costs and online agent costs, it does not fully account for repeated retrieval, embedding and index updates, additional storage, or retrieval latency. Without these measurements, it is difficult to judge whether the performance gains justify the overhead at scale.

Questions for authors
  • How would the system determine when the index needs another update in deployment? Can you clarify how the fixed 20-iteration procedure would extend to a continuously changing corpus?

  • Can you report the total indexing time, storage requirements, and retrieval latency, including the costs of embedding new keys and updating the index? How do these compare with the baselines as corpus size increases?

  • How does the method improve documents that never appear in the optimization retrieval results, even when queries are generated from those documents?

  • How much of the improvement comes from repeatedly revising existing keys, rather than adding keys to more documents?

5 comments
AuthorsOpenReview ↗

Initial Response (1/2)

Thank you for your thoughtful feedback and the opportunity to improve the paper.

About self-evolving (W1, Q1)

In our work, “self-evolving” refers to the autonomous improvement of an index in a given search environment using retrieval feedback, without repeated human intervention. This use of the term follows prior work on self-evolving models and agents [1,2], where the system improves autonomously without human intervention or annotations.

Specifically, people had to apply an indexing strategy, inspect the search results, diagnose retrieval results, and revise the strategy accordingly, repeating this process manually. Self-Index automates this process so that the index can improve for its search environment without a person having to analyze each problem or specify how to revise the index. This autonomy in the index improvement process is what we mean by “self-evolving.”

We chose 20 iterations as a common evaluation point so that the different experimental settings would use the same iteration count and query budget. Prior work on self-evolving [2] similarly presented a method that can continue improving autonomously over successive iterations, while reporting performance at a specific iteration in its experiments. Self-Index can also continue optimizing the existing index beyond 20 iterations. Indeed, as the table below shows, when we extended optimization on Browsecomp-plus from 20 to 30 iterations, downstream performance continued to improve.
(We use GPT-5.4-nano + bm25 for this experiment)

IterAccRecall
2064.9469.58
2565.9069.80
3066.3970.85

Moreover, this process of improving the index can continue in an online environment after deployment. For example, whenever enough user queries have accumulated online to fill a batch of the specified size, we can use that batch to incrementally improve the existing index. This allows us to continue improving the index to better match users’ information needs without requiring additional human intervention or annotations for those queries.

The same optimization procedure can be applied when new documents are added to corpus. Specifically, we can retain the existing index keys and index each new document using its original text as its initial key. We can then either use the Query Simulator to generate queries from the corpus that now includes the new documents or provide newly collected user queries to the Optimizer.

About computational and storage cost (W2, Q2)

The table below compares SPIKE and Self-Index in terms of index size and total optimization time including time spent on embedding and retrieval, and the total index size. We obtained these measurements on the BrowseComp-Plus corpus using the same settings as in the comparison reported in the paper.

RetrieverMethodIndex size (GB)Optimization time
Bm25SPIKE4.1281.90m
Self-index3.98104.68m
BGESPIKE1.3178.12m
Self-index1.1899.68m

We found that Self-Index produced a slightly smaller index than SPIKE, but took somewhat longer to optimize because it also performs query simulation and multiple stages of validation. However, we would like to emphasize that, unlike existing methods, Self-Index can adjust the number of iterations to fit the available budget and outperforms those methods even with a small number of iterations.

In addition, a larger corpus does not inherently reduce Self-Index’s efficiency. Across methods, index size is influenced by the maximum number of keys per document. Therefore, we would like to emphasize that Self-Index has no inherent disadvantage in storage efficiency relative to existing methods as the corpus grows. As for optimization time and token cost, existing methods optimize every document, while Self-Index can optimize a subset of documents to fit the available budget. We would also like to highlight that Self-Index incurs lower token costs than existing methods even when it optimizes a comparable share of the corpus.

[1] R-Zero: Self-Evolving Reasoning LLM from Zero Data; Huang et al; ICLR 2026
[2] Dr. Zero: Self-Evolving Search Agents without Training Data; Yue et al; COLM 2026

AuthorsOpenReview ↗

Initial Response (2/2)

About blind spot and coverage (Q3)

To examine how often blind spots occur in practice, we analyzed how frequently the document used to ground a simulated query was missing from that query’s retrieval results during Self-Index optimization. Specifically, on BrowseComp-Plus, we recorded the grounding document for each of the 2,560 queries generated by the Query Simulator. For each retriever, we computed Recall@30 for which the corresponding grounding document appeared among the Optimizer’s top 30 results.

RetrieverR@30
Bm2575.04
Qwen396.37
Avg85.71

Averaged across the two retrievers, the grounding document appeared in the top 30 results for 85.71% of query–document pairs. This made the grounding document available for consideration as a revision target in most cases.

Moreover, beyond the high recall of grounding documents reported above, Self-Index covers a growing share of the corpus over successive iterations, further limiting the scope for blind spots. Our Query Simulator aims to generate increasingly diverse queries, bringing a wider range of documents into consideration for revision. For domains with a large corpus, the number of queries per iteration can also be raised above the default, which widens the set of documents covered in each iteration and reduces blind spots further. As the table below shows, validated revisions cover a growing share of the corpus over successive iterations. Furthermore, when we extended optimization to 30 iterations, we found that the vast majority of documents in the Browsecomp-plus corpus had been revised.

IterationCorpus Coverage (%)
517.6
1034.2
1549.7
2064.3
2577.8
3090.4

In addition, we also tested whether Self-Index still had an advantage over existing method in terms of cost and performance after extending optimization to cover most of the corpus. Specifically, we evaluated the resulting index on the downstream task using GPT-5.4-nano with BM25 and measured the token cost of optimizing the index.

MethodToken usage (Input/Output)Offline costAccRecall
Self-Index (30 iter)515M / 46.5M$68.6066.3970.85
SPIKE379.3M / 87.4M$87.7347.5953.43

Even when we extend optimization until most documents are covered, Self-Index achieves further performance gains at a cost comparable to that of the existing method.

Revise vs Add (Q4)

We conducted additional experiments to assess the importance of repeatedly revising existing keys. Specifically, using GPT-5.4-nano with BM25 on BrowseComp-Plus, we compared performance after 10 iterations under the following three settings.

  • Setting 1: uses the configuration reported in the paper.
  • Setting 2: add only new keys without revising previously generated keys.
  • Setting 3: raises the optimization top-k to 60 and, as in Setting 2, generates only new keys without revising previously generated keys.

The results are shown below.

SettingAccRecallCoverage (%)
Baseline36.5139.490
Setting 154.3459.7034.2
Setting 251.0854.8234.2
Setting 353.8657.4062.4

First, Setting 2 outperformed the baseline, showing that adding keys to additional documents alone improves performance. However, it performed worse than Setting 1, which allows repeated revision of the existing key sets. This supports that repeatedly revising existing key sets contributes substantially to further performance gains.

Next, to see whether adding keys to more documents could close this performance gap, we evaluated Setting 3, which raises the optimization top-k in Setting 2 to 60. Setting 3 achieved higher coverage than Setting 1, but its performance was still lower. These results show that repeatedly refining existing key sets plays a more important role in improving performance than simply adding keys to more documents.

AuthorsOpenReview ↗

We have incorporated your feedback into the paper

We have incorporated your suggestions as well as our experimental results and the corresponding responses into our paper (Additions and revisions are indicated in blue text). Thank you very much for carefully reviewing our work and providing such valuable feedback. We are also delighted to have improved our work through your feedback. Please let us know if any concerns remain unaddressed.

Reviewer_RybyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work presents Self-Evolving Search Index, a framework that simulates retrieval queries, analyzes the retrieved results of the current index, selectively revises retrieval keys for documents, and updates the index after validation. The proposed approach requires no retraining of the retrieval model and leaves the original documents unchanged. Experiments span natural language, code, mathematical, and tabular retrieval tasks. The authors further evaluate downstream performance on search agents and agent memory benchmarks.

Critical strengths
  • The problem carries practical value. Different corpora and retrievers favor different index representations. Allowing the index to adapt gradually based on retrieval feedback offers greater flexibility than applying a fixed augmentation template uniformly across all documents.
  • The method is well-structured. The joint retrieval profile supplies context for diagnosis. Revising key sets at the whole-document level reduces redundant expressions, while three validation criteria prevent low-quality keys from being added to the index. Ablation studies confirm the contribution of each component.
  • The evaluation covers a broad range of settings and metrics. Beyond nDCG@10, the paper reports final answer accuracy, evidence recall, and memory-based QA performance. Results on BRIGHT include means and standard deviations over three runs, and the authors examine the impact of different score aggregation strategies.
Critical weaknesses
  • Blind spots exist in the diagnostic signal. The optimizer only examines the top-30 keys returned for a given query. If truly relevant documents fall outside this range, the system receives no direct failure signal for those documents. The paper should more clearly analyze whether such completely unretrieved documents can be rescued via subsequent exploration.
  • Scaling cost and coverage remain undercharacterized. The main experiments use only 2560 optimization queries per index, so the observed low cost does not straightforwardly generalize to corpora with millions of documents. I recommend reporting the number of documents diagnosed, proposed for revision, and passing validation in each iteration, along with cumulative unique document coverage. Costs from simulated query generation, rejected revision proposals, embeddings, index storage, and retrieval serving should also be accounted for.
  • Validation criteria do not guarantee monotonic retrieval improvements. Fidelity is judged by an LLM, while specificity and discriminability use retrieval scores as proxies. Neither directly checks relevance on real queries or performance regression on old queries. Minor performance drops are observed across iterative runs.
  • Gains from the proposed method need to be disentangled from resource allocation effects. Self-Index only processes a selected subset of documents, whereas some baselines generate representations for the full corpus. This is a legitimate system advantage, yet it creates mismatched processing scope and compute budgets across comparisons. Comparisons under equal LLM token budgets and equal numbers of revised documents would be helpful.
  • Downstream findings should explicitly report variance. The search agent results are promising, but confidence intervals or repeated runs for final answer accuracy are needed. For the memory experiments, the Gotchas category shows inconsistent gains, including a drop in one setting. The main text should more carefully distinguish two separate issues: whether existing information becomes easier to retrieve, versus whether the memory content itself is insufficient.
Questions for authors
  1. After 20 rounds, what fraction of documents in each dataset have been successfully revised at least once? Do performance gains mainly come from covering more documents, or from repeatedly refining the same set of documents?
  2. What are the token consumption and retrieval call overheads broken down by simulated query generation, filtering, diagnosis, and validation?
  3. For relevant documents that lie outside the top-30 at test time and are never retrieved during optimization, what is the improvement rate achieved by the method?
  4. Under a fixed offline budget, how much additional gain does the proposed method deliver compared with simpler baselines that prioritize documents by retrieval failure frequency and augment them using Doc2Query or SPIKE?
10 comments
AuthorsOpenReview ↗

Initial Response (1/3)

We sincerely appreciate your thoughtful and constructive feedback!

About blind spot and coverage (W1 W2, Q1, Q2, Q3)

To examine how often blind spots occur in practice, we analyzed how frequently the document used to ground a simulated query was missing from that query’s retrieval results during Self-Index optimization. Specifically, on BrowseComp-Plus, we recorded the grounding document for each of the 2,560 queries generated by the Query Simulator. For each retriever, we computed Recall@30 for which the corresponding grounding document appeared among the Optimizer’s top 30 results.

RetrieverR@30
Bm2575.04
Qwen396.37
Avg85.71

Averaged across the two retrievers, the grounding document appeared in the top 30 results for 85.71% of query–document pairs. This made the grounding document available for consideration as a revision target in most cases.

Moreover, beyond the high recall of grounding documents reported above, Self-Index covers a growing share of the corpus over successive iterations, further limiting the scope for blind spots. Our Query Simulator aims to generate increasingly diverse queries, bringing a wider range of documents into consideration for revision. For domains with a large corpus, the number of queries per iteration can also be raised above the default, which widens the set of documents covered in each iteration and reduces blind spots further. As the table below shows, validated revisions cover a growing share of the corpus over successive iterations. Furthermore, when we extended optimization to 30 iterations, we found that the vast majority of documents in the Browsecomp-plus corpus had been revised.

IterationCorpus Coverage (%)Cumulative documents with proposed revisionsCumulative documents with validated revisions
517.618,77117,606
1034.236,47234,220
1549.753,10049,818
2064.368,65864,451
2577.883,14377,995
3090.496,55890,603

In addition, we also tested whether Self-Index still had an advantage over existing method in terms of cost and performance after extending optimization to cover most of the corpus. Specifically, we evaluated the resulting index on the downstream task using GPT-5.4-nano with BM25 and measured the token cost of optimizing the index.

MethodToken usage (Input/Output)Offline costAccRecall
Self-Index (30 iter)515M / 46.5M$68.6066.3970.85
SPIKE379.3M / 87.4M$87.7347.5953.43

Even when we extend optimization until most documents are covered, Self-Index achieves further performance gains at a cost comparable to that of the existing method.

Revise vs Add (Q1)

We conducted additional experiments to assess the importance of repeatedly revising existing keys. Specifically, using GPT-5.4-nano with BM25 on BrowseComp-Plus, we compared performance after 10 iterations under the following three settings.

  • Setting 1: uses the configuration reported in the paper.
  • Setting 2: add only new keys without revising previously generated keys.
  • Setting 3: raises the optimization top-k to 60 and, as in Setting 2, generates only new keys without revising previously generated keys.

The results are shown below.

SettingAccRecallCoverage (%)
Baseline36.5139.490
Setting 154.3459.7034.2
Setting 251.0854.8234.2
Setting 353.8657.4062.4

First, Setting 2 outperformed the baseline, showing that adding keys to additional documents alone improves performance. However, it performed worse than Setting 1, which allows repeated revision of the existing key sets. This supports that repeatedly revising existing key sets contributes substantially to further performance gains.

Next, to see whether adding keys to more documents could close this performance gap, we evaluated Setting 3, which raises the optimization top-k in Setting 2 to 60. Setting 3 achieved higher coverage than Setting 1, but its performance was still lower. These results show that repeatedly refining existing key sets plays a more important role in improving performance than simply adding keys to more documents.

AuthorsOpenReview ↗

Initial Response (2/3)

Cost breakdown (W2, Q2)

We would first like to report the token usage and corresponding cost for each stage of the Self-Index, the total optimization time including time spent on embedding and retrieval, and the total index size. We also report the totals across stages and compare the results with SPIKE. The table below shows results on BrowseComp-Plus with BGE as the retriever, using 20 iterations for Self-Index.

Phasetoken (Input/output)Offline costOptimization timeIndex size (GB)
Self-Exploration32.84M / 0.26M$2.489.7m-
Self-Diagnose85.21M / 2.92M$8.0120.8m-
Self-Revision165.67M / 33.83M$35.2849.2m-
Self-validation228.98M / 3.73M$18.6419.9m-
Self-index (Overall)512.70M / 40.74M$64.4199.68m1.18
SPIKE379.3M / 87.4M$87.7378.12m1.31

We found that Self-Index produced a slightly smaller index than SPIKE, but took somewhat longer to optimize because it also performs query simulation and multiple stages of validation. However, we would like to emphasize that, unlike existing methods, Self-Index can adjust the number of iterations to fit the available budget and outperforms those methods even with a small number of iterations.

In addition, a larger corpus does not inherently reduce Self-Index’s efficiency. Across methods, index size is influenced by the maximum number of keys per document. Therefore, we would like to emphasize that Self-Index has no inherent disadvantage in storage efficiency relative to existing methods as the corpus grows. As for optimization time and token cost, existing methods optimize every document, while Self-Index can optimize a subset of documents to fit the available budget. We would also like to highlight that Self-Index incurs lower token costs than existing methods even when it optimizes a comparable share of the corpus.

About advanced baseline (W4, Q4)

We conducted additional experiments to compare Self-Index more rigorously with the baselines you mentioned. Specifically, we applied SPIKE and Doc2Query only to documents whose revisions passed validation and were incorporated into the index during Self-Index optimization. We then compared their performance, cost, and token usage with that of Self-Index.

(We ran the experiments on BrowseComp-Plus using GPT-5.4-nano with BM25 and calculated offline costs based on Qwen3.6 token prices.)

MethodAccuracyRecallOnline costOffline cost
Baseline36.5139.49$30.86 / 583.2M / 5.76M (516.49M cached)0
+ Doc2query39.0443.97$32.58 / 645.1M / 5.97M (577.22M cached)$10.30 / 56.31M / 9.08M
+ SPIKE48.6752.30$32.17 / 604.4M / 5.70M (532.43M cached)$81.05 / 351.24M / 80.66M
+ Self-Index64.9469.58$28.08 / 463.8M / 4.96M (393.77M cached)$62.13 / 454.39M / 43.32M

Even when all methods revised the same set of documents, Self-Index achieved the best performance at a cost comparable to SPIKE’s. These results show that identifying shortfall specific to each document and optimizing the index accordingly can be more effective than applying a single strategy across corpus, as existing methods do.

About our validation criteria (W3)

As we noted in the Introduction, the main goal of Self-Index is to improve the index autonomously using retrieval feedback, without requiring human intervention in the revision process, such as providing relevance annotations. As you suggested, directly measuring retrieval performance is an intuitive and effective way to validate revisions. However, this approach requires relevance annotations for the input queries, and obtaining those annotations would require human involvement.

We therefore needed a way to validate revisions without relevance annotations. To this end, we designed our validation criteria based on prior work [1][2][3]. Our ablation study (Table 5) shows that each criterion makes a meaningful contribution to retrieval performance. We would like to emphasize that Self-Index’s validation criteria serve as an effective proxy for assessing whether revisions improve retrieval, without relying on relevance annotations.

[1] A vector space model for automatic indexing; Salton et al
[2] A theory of term importance in automatic text analysis.; Salton et al
[3] Contextual document embeddings; Morris & Rush

AuthorsOpenReview ↗

Initial Response (3/3)

About results variance and “Gotchas” (W5)

We also ran the downstream evaluations with Self-Index three times and report the means and standard deviations below. The baseline scores are those reported in the manuscript.

The following table shows the three-run means and standard deviations for GPT-5.4-nano on BrowseComp-Plus.

RetrieverAccuracyRecall
Bm2536.5139.49
+Self-index64.98 (0.18)69.62 (0.46)
Qwen3-Emb-8B52.1757.30
+Self-index58.92 (0.37)63.89 (0.49)

The three-run means and standard deviations for the overall scores on LongMemEval-V2 are as follows.

MethodOverall
Query -> Slice0.415
+Self-index0.474 (0.016)
Query -> Slice + Notes0.448
+Self-index0.513 (0.015)
AgentRunbook-R0.532
+Self-index0.570 (0.01)

In both tables, the three-run means for Self-Index are close to the results reported in the manuscript, and the standard deviations are substantially smaller than the gains over the corresponding baselines. We will continue running these experiments and make the three-run means and standard deviations available for all results reported in the manuscript.

Turning to Gotchas, as we noted in line 392 of our manuscript and as discussed in the LongMemEval-V2 paper, “Gotchas” performance depends heavily on the memory’s “value content” supplied to the reader as context. [4] In LongMemEval-V2, Gotchas performance was much less sensitive to improvements in memory retrieval achieved by revising the memory’s “key value” with Self-Index [4]. We therefore believe that the lack of improvement on Gotchas offers little basis for concluding that Self-Index is less effective. We would like to emphasize that this result reflects a limitation of the memory’s “value content” produced by the baseline memory systems provided with LongMemEval-V2.

[4] LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues; Wu et al

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your valuable feedback and the follow-up discussions, which have helped us further improve our work. To help clarify the points raised and address any remaining concerns, we have prepared additional responses to your latest comments. We would greatly appreciate it if you could take a look.

Thank you again for your time and thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.