acceptodds
Under review as a conference paper at ICLR 2027

Evolving Semantic Indexes for Generative Retrieval

Abstract

Generative retrieval (GR) casts search and recommendation as autoregressive generation of discrete semantic identifiers (SID). SIDs play one of the most important roles to determine the quality of GR results. For example, different SID models, collision handling mechanisms, decoding frameworks, and training recipes, can significantly vary the retrieval performance. However, these discrete and interdependent design choices cannot be optimized end-to-end using gradients from downstream retrieval and therefore remain largely designed by hand. In this paper, we introduce a framework for automatically improving these components through LLM-driven evolutionary search on the SID framework, which to the best of our knowledge is the first application of LLM-driven evolutionary program search to GR. A locally servable, open-weight 31B language model is used to mutate executable GR components, so no proprietary LLM API is used at any point, and a 1M-parameter proxy decoder provides fast downstream feedback in place of repeatedly training the 1B target generator, which would otherwise make evolutionary search computationally infeasible. Across 21 SID programs, proxy and target Recall@100 exhibit strong rank correlation (). Guided by this proxy, evolution consistently improves the seed GR on NQ320K, MSMARCO300K, and a multilingual e-commerce dataset, under a training recipe held fixed across conditions. On NQ320K, jointly evolving SID and collision handling improves Recall@1 from 55.8% to 59.3% and Recall@100 from 85.8% to 90.6%. Our component study reveals that proxy reliability depends on what component is evolved. Improvements transfer reliably when evolving the semantic index, but transfer less reliably when evolving the generator’s training objective or the full GR pipeline. These results suggest that LLM-driven evolutionary search is a practical method for improving GR design and show that carefully assessing proxy reliability is essential for successful GR pipeline optimization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Generative Retrieval, Evolutionary Search, Semantic Index

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_on4WOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the use of LLM-driven evolutionary program search to automatically improve semantic identifier (SID) construction for generative retrieval (GR). Building on AdaEvolve, the authors employ an open-weight 31B LLM to iteratively modify executable implementations of SID construction, collision handling, and other GR components. A lightweight generative retriever with approximately 1M non-embedding parameters serves as a proxy evaluator to reduce the computational cost of evaluating candidate programs. Experiments on NQ320K, MSMARCO300K, and an e-commerce dataset demonstrate that the evolved SID construction programs can improve retrieval performance over the initial GR system. The authors further investigate proxy-to-target transferability across different GR components, observing that proxy evaluations are relatively reliable for SID optimization but may fail to transfer when optimizing training objectives or the entire GR pipeline.

Critical strengths
  1. The paper addresses an important and practically meaningful research problem. I find the research direction highly interesting and worthwhile. Generative retrieval involves numerous interdependent algorithmic design choices, including SID construction, collision handling, model training, and decoding. Optimizing these components often requires substantial human expertise and extensive experimentation. Exploring whether AutoResearch can automatically discover and improve such algorithmic designs is an important question with considerable potential impact. The authors conduct a meaningful initial investigation of this problem, providing a valuable starting point for further research.

  2. The experimental results are encouraging and reveal potentially important phenomena. The authors demonstrate retrieval improvements through evolutionary optimization across three datasets, with particularly noticeable gains on NQ320K. More importantly, the paper goes beyond reporting successful optimization results and investigates proxy-to-target transferability across different GR components. The finding that proxy evaluations can be relatively reliable for SID optimization but may fail when optimizing training objectives or the full pipeline is particularly interesting. I believe this observation has potential implications for understanding and improving AutoResearch systems more broadly.

Critical weaknesses
  1. The paper lacks a sufficiently clear and substantial research contribution. While I find the research problem important and the authors' exploration valuable, it is unclear what the central contribution is. As a methodological paper, it largely applies the existing AdaEvolve framework without introducing significant innovations in AutoResearch. As an algorithm discovery paper, the evolved SID method mainly combines established techniques rather than introducing a fundamentally new design. As an empirical study, the analysis remains relatively preliminary. I do not expect contributions along all three dimensions, but I believe the paper should establish a more substantial contribution in at least one of them.

  2. The empirical analysis does not provide sufficiently deep or generalizable insights into AutoResearch. The paper demonstrates that evolutionary search can improve an initial GR system and that proxy-to-target transferability varies across components, but these observations remain largely descriptive. More fundamental questions are left unanswered: How does AutoResearch compare with human-designed methods or alternative automated search techniques? What are its key capability limitations? How much does domain-specific prior knowledge influence its success? Why does proxy transfer fail for certain components, and what makes GR distinct from other AutoResearch applications? I believe investigating these questions would provide much stronger insights into when, why, and how AutoResearch can effectively improve GR systems.

Questions for authors
  1. How do the authors define the paper's primary research contribution?
  2. What advantages does AutoResearch actually provide for SID algorithm design?
  3. Can the authors provide a deeper analysis of the factors governing the success and failure of AutoResearch in this setting?
2 comments
AuthorsOpenReview ↗

Response to Reviewer on4W

Thank you for the careful review and appreciation of our work. We clarify our claims and address your concerns as follows:

W1, Q1: Our contribution is empirical. We identify and propose solutions to the bottlenecks that prevent LLM-driven evolutionary search from autonomously optimizing GR. This includes studying proxy evaluation and search space boundaries, all conducted with a locally served 31B mutator. Evaluating each candidate on the 1B target model is too expensive. A 1M-param proxy is reliable for searching the semantic index, but is anti-correlated or not well correlated when the search changes the generator’s (and the proxy’s) objective. This is expected, since changing the objective is asking the search to chase a moving target. We also observed that the small 31B mutator rarely writes runnable code in one shot. Even when it does, the SID index usually has >50% collision rate. An error-feedback inner loop fixes this within about 5 retries. Without it the mutator could retry tens of times without making any progress.

W2, Q3: The generalizable insight we provide are that for evolving GR, a cheap proxy reliably guides the search when the search changes what the generator learns from, such as the SIDs, but it fails once the search space includes the generator’s training objective, because then each candidate changes how the proxy itself is trained, so the search can raise the proxy score by exploiting the 1M model’s training dynamics, which do not carry over to the much larger target model. Also that for evolving GR to be scalable, a small 31B mutator model can be used only if its errors are fed back to itself, to give additional information for it to self-correct, otherwise blind sampling rarely produces runnable/usable programs.

What’s distinct about GR is that SID quality is hard to measure. You can’t score a SID program from its output alone, you would have to train a generator on it. Every candidate therefore costs a training run, so evolutionary search is only affordable with a proxy model as the evaluator. Many AutoResearch successes have exact, cheap evaluators. GR has neither, which is why proxy validity is important.

Q2: Besides making GR optimization much more scalable, our main experiments and the added experiments (see general comment) show that the evolved program’s gains are genuine. (1) +4.75 on Recall@100 over the seed. (2) tuning the seed’s key hyper-parameters gains at most +0.73. The search found use of the relevance labels and query embeddings which the seed ignores, and on many other good candidates evolved on all 3 datasets, the search departed from the RQVAE paradigm entirely (Section 6.1)

We hope our response address your concerns and would appreciate your reassessment.

Dear reviewer on4W, thanks for your questions and suggestions. As the discussion period ends today, we would be grateful if you could take a look at our updates and let us know if there is something missing we could still help to clarify. Thanks a lot!

Reviewer_RfL7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper applies LLM-driven evolutionary program search to semantic index design for generative retrieval. Starting from an RQVAE index with fixed collision handling, a locally served 31B model proposes code edits, while a small autoregressive retrieval generator evaluates candidates using Recall@100. After search, the program selected by proxy performance constructs the index used to train a 1B target generator. Experiments on NQ320K, MSMARCO300K, and proprietary e-commerce data report improvements over the seed pipeline. Index changes transfer from the small proxy to the larger model more successfully than changes to the generator's training objective. The selected NQ program uses cosine-based quantization and larger codebooks, with query supervision and geometry-based collision suffixes.

Critical strengths

The paper uses a small retriever to evaluate candidate semantic indexes and shows that proxy-selected indexes can improve the target generator's retrieval performance. With the input encoder and target-generator training recipe held fixed, results across several datasets support the feasibility of this transfer. The main text and appendix explain the final index changes, making the programs produced by search easier to understand. The authors also report transfer failures when evolving the training objective or the full pipeline. These cases show that better proxy scores need not improve the target model and provide useful observations about the scope of proxy guidance.

Critical weaknesses

[W1] - The weak baseline does not establish that the performance gains support replacing manual index design with automated search

The paper motivates LLM-driven evolutionary search by the continued reliance on manual design for SID construction and collision handling. However, its main baseline is an RQVAE trained only for document reconstruction, with smaller codebooks and collision suffixes ordered by document ID. It is not compared against a carefully designed SID method. The selected NQ program changes quantization and suffix construction, adds query-relevance supervision, and expands the codebooks. Improvements over this seed show that the resulting index is better suited to retrieval, but do not establish how much additional value evolutionary search provides beyond known design choices absent from the starting point.

The use of query data needs to be distinguished from the contribution of automated search. The seed index uses only document reconstruction, whereas the evolved index uses relevant queries to learn identifiers. Even with identical input data, the two methods use supervision differently. Search excludes synthetic queries, but final index construction and generator training on NQ and MSMARCO include them. The results do not separate the effects of supervision from original queries and synthetic queries, or their interactions with the index changes. Query supervision or increased capacity could therefore explain a substantial part of the improvement, but targeted ablations are missing.

The related work already discusses relevance-supervised identifier learning in MERGE and RIPOR. A direct control would strengthen the seed SID with relevance supervision under the same query data, match its codebook capacity, and improve its collision suffixes before measuring the remaining benefit of evolutionary search. Ablating these changes in the final program would also help avoid attributing their combined gains directly to the search framework.

[W2] - Practical cost of complete proxy-guided searches remain unclear

The proxy–target correlation experiment supports the potential of this evaluator, but covers only NQ joint SID–collision search without synthetic queries. It does not cover the final setting with synthetic queries or the other datasets. Table 1's error bars come from repeated target-model training rather than independent complete searches. They therefore do not establish how often rerunning the automated design process produces a transferable improvement. The transfer failures in training-objective and full-pipeline search also show that proxy gains cannot generally be treated as reliable evidence of target gains.

The approximately 15-fold cost reduction compares proxy training with target training for one candidate, excluding code mutation, index construction, and failed retries. Although candidates can have long execution times and undergo repeated repairs, the paper does not consolidate the total costs like completed candidate counts or total GPU time for each search. Whether automation is preferable to manual design or simple candidate selection depends on its complete cost and success rate. Independent search repetitions and full cost records would directly affect the assessment of practical value.

Questions for authors
  1. Under the same query data and generator training settings, how much improvement remains over a SID baseline with relevance supervision, matched codebook capacity, and improved collision suffixes?

  2. Are component ablations available for the final program that separate the effects of query supervision from increased codebook capacity and collision handling? Does the gap between the seed and evolved indexes remain similar with and without synthetic queries?

  3. At the same budget, what existing evidence supports the additional value of iterative evolution over independent LLM proposals selected by the same proxy?

  4. Are the results stable across independent complete searches? Please report completed candidate counts and rejection/retry counts, together with total GPU time including code mutation and index construction.

2 comments
AuthorsOpenReview ↗

Response to Reviewer RfL7

Thank you for the detailed review. Please take a look at the general comment. We address the concerns as follows:

W1, Q1, Q2: We do not claim that search replaces manual SID design or that the evolved index beats the best hand-designed ones. Our claim is that starting from a standard GR setup as seed and with everything else fixed, proxy-guided search finds an index building method that improves the target generator. And the focus of our paper is mainly on identifying and overcoming the bottlenecks that prevent evolutionary search from autonomously optimizing GR.

The seed and evolved programs receive identical inputs, including the training relevance pairs. The seed ignores them, and the search found a use for them, with a formulation different from RIPOR and MERGE (Section 6). So the gain from query supervision is a gain the search produced, not something we added. We agree the effect of the larger codebook should be measured separately.

For synthetic queries, they are used identically in the training data for Seed GR and SID evolution results in Table 1, so they do not explain the gap. Also, e-commerce does not use synthetic queries and any stage and still improves over baseline.

Q3: Without the inner gatekeeping loop, the gemma-4-31b-it mutator LLM is not strong enough to always propose diff that produces a runnable solution. In fact, in our earlier attempts, repeated sampling without appending the error to the context shows no signs of improvement. It would retry tens of times and not produce a runnable program. The code errors are often syntax errors related to missing/extra tabs, or memory object misplacement such as when an object should be put on CPU instead of on the GPU device, etc. Even when the program runs, the SID index produced from it often suffers from codebook collapse, where the majority of the documents are mapped to the same SID sequence.

It is only when we have the inner feedback loop, the mutator LLM finally iteratively writes satisfactory diffs. The inner loop rarely hits the 20 retry ceiling, usually around 5 iterations would produce a bug-free program that produces less than 50% collision rate SIDs.

W2, Q4: The error bars in Table 1 do come from different training seeds, not from independent searches. Regarding cost, we did not record the exact retries per step or token counts in and out of the gemma-4-31b-it mutator model throughout each evolution, but we can confidently report that GPU time is dominated by evaluation (build SID index + retriever training) instead of candidate proposal + retries. This is true even when our evaluator is already a proxy, which is why our paper argues that a cheap and reliable evaluator is essential for GR evolution.

We hope our response address your concerns and would appreciate your reassessment.

Thank you for the additional experiments and clarifications. The capacity-matched results and query-alignment ablation strengthen the evidence for the evolved index. However, the added value of evolutionary search over stronger baselines remains unclear, and concerns about search reproducibility and total computational cost remain unresolved. After also considering the other reviewers’ assessments, I maintain my score of 2.

Reviewer_GohqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. This paper examines LLM-powered evolutionary search as an approach to improve generative retrieval. It chooses a residual quantization version of generative retrieval, characterizes the design space for it, and uses an open source, moderately sized LLM to generate and execute edits.
  2. The authors design proxies whose performances correlate well with those of the target systems/models. The results indicate that this proxy works well for portions of the design space (SID design) and poorly for others (generation objective).
Critical strengths
  1. I believe this approach is likely to scale well with compute and data, and seems to trend in the direction of how generative retrieval systems will be built in the future. The authors also create a reusable recipe to develop and improve an end-to-end generative retrieval system using limited resources.
  2. The design space of algorithms for (this version of) generative retrieval is well characterized at a high level, and the results that show how well proxies can be used to explore it could be of use in future generative retrieval papers.
Critical weaknesses
  1. The results hinge on one very specific prior / Seed GR. This is good enough for a proof of concept, but given the novelty seems to be in the application to generative retrieval and not the approach itself, the strength of the result depends on how comprehensive the exploration is. It would also help to have a hyper-parameter tuned baseline (within the design space the evolutionary search algorithm explores), so that any gains from the evolutionary search are above and beyond what a hand-crafted approach could yield.
  2. The paper does not offer comparisons with other evolutionary search approaches or auto research baselines. Suggestion: Identifying the bottlenecks preventing evolutionary search or auto research algorithms from autonomously optimizing generative retrieval would be the most valuable contribution of this paper.
Questions for authors
  1. How did you identify this design/search space? It seems to be index building, decoding, training objective, and collision handling at a high-level, and within index building, input representation, quantization, codebook initialization, etc. How much of this needs to be specified in advance, and how strongly does the algorithm depend on it? A lot of this is well defined by your seed, but generative retrieval methods have become very diverse, and even if the broader search space remains the same, the set of options available within each part will become very different. How would this evolutionary search algorithm change for other variants of generative retrieval (for example, lexical identifiers)?
  2. The chosen proxy is very defensible, but the results indicate that, for various possible reasons, it doesn’t work well for certain parts of the search space such as the objective. There seems to be a tradeoff here: you could use a cheap proxy and have faster experiments, or an expensive proxy and more reliable results. Do you have any results that explain how to make this tradeoff, or allow the system to adjust the tradeoff automatically?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Gohq

Thank you for the careful review and appreciation of our work. The contribution of this work is to identify and propose solutions to the bottlenecks that prevent evolutionary search from autonomously optimizing GR. We will rewrite the appropriate parts of the introduction to say this directly.

We address the reviewer’s concerns as follows:

W1: We tuned the seed's codebook size and latent dim (see general comment). The best tuned seed reaches 86.55 Recall@100. The evolved program reaches 90.57 with the same codebook size and latent dim. The search also makes changes that no seed constant can reach. The best NQ program adds a query-alignment loss, direction-based code assignment, and centroid-ranked collision suffixes (Section 6). Some other programs departed from the RQVAE paradigm entirely, using the relevance label and query embeddings in very different formulations (Section 6.1). A sweep over the seed’s constants cannot reach any of these.

W2: The proxy and gatekeeping loop sit on the evaluation side and can work with any search controller. Adding different search controllers would not change our two findings on (1) when the proxy can be trusted, and (2) what a small locally-servable mutator must do to be usable.

Q1: The high-level search space is identified by human, and the interface is fixed in advance. For example for SID index building, it is just a function and the inputs are doc/query embeddings and training relevance pairs, the output is one integer sequence per document. The seed is only a starting point and the evolved program often departs dramatically from it. The mutation prompt suggests algorithm families (Appendix B) but does not restrict the search to them. Lexical identifiers would need a different interface such as raw text in, token sequences out, and a new seed program. The outer loop, inner loop, and proxy stay as they are, and the proxy must be re-tested in order to be trusted.

Q2: Regarding the proxy, we mainly present a calibration protocol. Before trusting a proxy on a to-be-searched component, evaluate about 20 sampled programs on both proxy and target. This costs about 100 GPU-hours in our case, versus the 500 GPU-hour run if we were to not use the proxy. The cost of LLM-driven evolutionary search is often bottlenecked by evaluation, and search quality is heavily influenced by evaluation precision, thus motivating our study for the proxy.

We hope our response address your concerns and would appreciate your reassessment.

Dear reviewer Gohq, thanks for your questions and suggestions. As the discussion period ends today, we would be grateful if you could take a look at our updates and let us know if there is something missing we could still help to clarify. Thanks a lot!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment: new NQ320K results

We thank the reviewers for their time and valuable feedback. During this discussion phase, we mainly want to clarify our contribution and claims (commented under individual reviews). The following are additional experiments on NQ320K to clarify the genuineness of the evolved program’s gains. Specifically, it answers the following question:

  1. Can a simple hyper-parameter search achieve the performance of the evolved program?
  2. How much of the evolved program gains come from the mutated logic?

K = codebook size, d = latent dimension. Scores reported on the 1B target model (gemma-3-1b-pt)

#SID programKdR@1R@10R@100
1Seed12812852.9671.6682.75
2Seed12851247.5462.8071.98
3Seed (in paper)25625655.8275.2785.82
4Seed51212857.1576.5486.41
5Seed51225657.3977.2886.55
6Seed51251247.4862.5071.97
7Evolved25625656.8578.7590.23
8Evolved (in paper)51225659.2881.0590.57
9Evolved w/o query-alignment loss51225651.0565.3071.12
  1. K = 512 and d = 256 is the best setting for seed and that is also what Evolved found. (row 3, 5 vs. 8)
  2. Tuning key hyper-parameters on the seed does not outperform Evolved (row 1,2,3,4,5,6 vs. 7,8)
  3. When Evolved use the same hyper-parameters as seed, the gains are still large. (row 3 vs. 7)
  4. Removing the query-alignment loss from the evolved program breaks it (row 9 vs. 8). The index collapses, with very high collision rate, and Recall@100 falls to 71.12, below the seed. The other evolved components likely depend on this loss, so the evolved program's changes are co-adapted, not independently additive.

These results demonstrate that the codebook capacities are not the source of the gain. The mutated logic, is what contributes to the gain. Also, a simple hyper-parameter tuning on the seed does not come close to what evolve achieved.

AuthorsOpenReview ↗

Follow-up Response

Thank you for the follow-up. The revised introduction states the two bottlenecks, and the new Section 5.3 covers proxy calibration, its cost, and what to do when the proxy is unreliable.

The best NQ program differs from the seed in about a dozen places, while each prompt asks for one or a few edits per proposal. The changes are accumulated across generations. We updated the general comment with the evolved program without the query-alignment loss, and it degrades heavily. This suggests that through evolution the changes grow to collectively maximize the evaluation score and to depend on each other, and the gains from each change are not always independently additive.

The prompts do list plenty of options, but under the same prompts the search produced vastly different programs on each dataset (Section 6.1). Some choices, such as direction-based code assignment, are not in the prompts. We agree that a run with the suggestion lists removed would isolate the prompt's role further, and we leave it to future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.