- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the use of LLM-driven evolutionary program search to automatically improve semantic identifier (SID) construction for generative retrieval (GR). Building on AdaEvolve, the authors employ an open-weight 31B LLM to iteratively modify executable implementations of SID construction, collision handling, and other GR components. A lightweight generative retriever with approximately 1M non-embedding parameters serves as a proxy evaluator to reduce the computational cost of evaluating candidate programs. Experiments on NQ320K, MSMARCO300K, and an e-commerce dataset demonstrate that the evolved SID construction programs can improve retrieval performance over the initial GR system. The authors further investigate proxy-to-target transferability across different GR components, observing that proxy evaluations are relatively reliable for SID optimization but may fail to transfer when optimizing training objectives or the entire GR pipeline.
Critical strengths
-
The paper addresses an important and practically meaningful research problem. I find the research direction highly interesting and worthwhile. Generative retrieval involves numerous interdependent algorithmic design choices, including SID construction, collision handling, model training, and decoding. Optimizing these components often requires substantial human expertise and extensive experimentation. Exploring whether AutoResearch can automatically discover and improve such algorithmic designs is an important question with considerable potential impact. The authors conduct a meaningful initial investigation of this problem, providing a valuable starting point for further research.
-
The experimental results are encouraging and reveal potentially important phenomena. The authors demonstrate retrieval improvements through evolutionary optimization across three datasets, with particularly noticeable gains on NQ320K. More importantly, the paper goes beyond reporting successful optimization results and investigates proxy-to-target transferability across different GR components. The finding that proxy evaluations can be relatively reliable for SID optimization but may fail when optimizing training objectives or the full pipeline is particularly interesting. I believe this observation has potential implications for understanding and improving AutoResearch systems more broadly.
Critical weaknesses
-
The paper lacks a sufficiently clear and substantial research contribution. While I find the research problem important and the authors' exploration valuable, it is unclear what the central contribution is. As a methodological paper, it largely applies the existing AdaEvolve framework without introducing significant innovations in AutoResearch. As an algorithm discovery paper, the evolved SID method mainly combines established techniques rather than introducing a fundamentally new design. As an empirical study, the analysis remains relatively preliminary. I do not expect contributions along all three dimensions, but I believe the paper should establish a more substantial contribution in at least one of them.
-
The empirical analysis does not provide sufficiently deep or generalizable insights into AutoResearch. The paper demonstrates that evolutionary search can improve an initial GR system and that proxy-to-target transferability varies across components, but these observations remain largely descriptive. More fundamental questions are left unanswered: How does AutoResearch compare with human-designed methods or alternative automated search techniques? What are its key capability limitations? How much does domain-specific prior knowledge influence its success? Why does proxy transfer fail for certain components, and what makes GR distinct from other AutoResearch applications? I believe investigating these questions would provide much stronger insights into when, why, and how AutoResearch can effectively improve GR systems.
Questions for authors
- How do the authors define the paper's primary research contribution?
- What advantages does AutoResearch actually provide for SID algorithm design?
- Can the authors provide a deeper analysis of the factors governing the success and failure of AutoResearch in this setting?
2 comments
Dear reviewer on4W, thanks for your questions and suggestions. As the discussion period ends today, we would be grateful if you could take a look at our updates and let us know if there is something missing we could still help to clarify. Thanks a lot!
Response to Reviewer on4W
Thank you for the careful review and appreciation of our work. We clarify our claims and address your concerns as follows:
W1, Q1: Our contribution is empirical. We identify and propose solutions to the bottlenecks that prevent LLM-driven evolutionary search from autonomously optimizing GR. This includes studying proxy evaluation and search space boundaries, all conducted with a locally served 31B mutator. Evaluating each candidate on the 1B target model is too expensive. A 1M-param proxy is reliable for searching the semantic index, but is anti-correlated or not well correlated when the search changes the generator’s (and the proxy’s) objective. This is expected, since changing the objective is asking the search to chase a moving target. We also observed that the small 31B mutator rarely writes runnable code in one shot. Even when it does, the SID index usually has >50% collision rate. An error-feedback inner loop fixes this within about 5 retries. Without it the mutator could retry tens of times without making any progress.
W2, Q3: The generalizable insight we provide are that for evolving GR, a cheap proxy reliably guides the search when the search changes what the generator learns from, such as the SIDs, but it fails once the search space includes the generator’s training objective, because then each candidate changes how the proxy itself is trained, so the search can raise the proxy score by exploiting the 1M model’s training dynamics, which do not carry over to the much larger target model. Also that for evolving GR to be scalable, a small 31B mutator model can be used only if its errors are fed back to itself, to give additional information for it to self-correct, otherwise blind sampling rarely produces runnable/usable programs.
What’s distinct about GR is that SID quality is hard to measure. You can’t score a SID program from its output alone, you would have to train a generator on it. Every candidate therefore costs a training run, so evolutionary search is only affordable with a proxy model as the evaluator. Many AutoResearch successes have exact, cheap evaluators. GR has neither, which is why proxy validity is important.
Q2: Besides making GR optimization much more scalable, our main experiments and the added experiments (see general comment) show that the evolved program’s gains are genuine. (1) +4.75 on Recall@100 over the seed. (2) tuning the seed’s key hyper-parameters gains at most +0.73. The search found use of the relevance labels and query embeddings which the seed ignores, and on many other good candidates evolved on all 3 datasets, the search departed from the RQVAE paradigm entirely (Section 6.1)
We hope our response address your concerns and would appreciate your reassessment.