- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies an interesting problem in generative retrieval: learning semantic IDs can harm the original language generation ability of LLMs. To address this issue, the authors propose SpeakGR, which combines semantic ID learning with a language-preserving objective. They further introduce Adaptive SpeakGR, which dynamically adjusts the preservation strength during training. Experiments across different models and datasets show that the proposed methods can significantly reduce language drift while maintaining competitive retrieval performance.
Critical strengths
- Preserving language generation capabilities during semantic ID learning is an important but relatively underexplored problem.
- The proposed method is simple, intuitive, and well motivated. I appreciate that such a straightforward approach can effectively address the problem without requiring major changes to the model architecture.
- The experiments cover multiple models and datasets, showing clear improvements in language preservation. The comparisons with existing methods and additional analyses also provide solid support for the proposed approach.
- I believe this work has broader implications for generative retrieval research. It highlights that retrieval performance alone may not be sufficient to evaluate generative retrievers, and that preserving the original language capabilities of LLMs is also important.
- This paper also opens up a promising direction for building unified retrieval and generation models. The ability to learn retrieval without significantly affecting language generation could be valuable for future LLM-based information retrieval systems.
Critical weaknesses
- It would be interesting to explore whether the proposed method can be extended to other semantic ID-based tasks, such as generative recommendation, in future work.
- The experiments mainly focus on relatively small LLMs. It would be interesting to see whether similar results can be achieved with larger models.
- A few qualitative examples of language generation before and after fine-tuning could help readers better understand the benefits of the proposed method.
Questions for authors
- As a possible future direction, have the authors considered evaluating more complex language tasks, such as reasoning, long-form generation, or multi-turn conversations?
- Could the authors briefly discuss how the proposed method might be used in a unified retrieval-and-generation framework?
- Do the authors expect similar results with larger LLMs or different semantic ID construction methods?
3 comments
Thank you for your detailed rebuttal. I believe the authors have adequately addressed my previous concerns and questions.
Thank you for your constructive feedback
Thank you for your thoughtful follow-up. We are glad that we have addressed your concerns and questions. Your constructive suggestions helped us strengthen the paper through additional evaluations and clearer presentation. We sincerely appreciate your time and careful review.
Response to Reviewer zS2j
We sincerely thank the reviewer for the encouraging assessment of our motivation and contributions. We appreciate your recognition of language preservation as an important, underexplored problem, the simplicity and effectiveness of our approach, and the experimental support across models and datasets. We also thank you for highlighting the broader value of evaluating language preservation alongside retrieval and the potential for unified retrieval and generation. Your constructive suggestions motivated the larger-model experiments and qualitative examples below.
Response to Weaknesses
A natural extension is to predict item SIDs from user interaction histories while retaining the same speak-preserving regularization over the original text vocabulary. We added this future direction to Appendix B.
We added Qwen3-8B/NQ experiments. To train the larger backbone efficiently, we used LoRA for both SFT and SpeakGR, with matched settings and preservation weight 1.
SpeakGR reduces language-distribution drift, with R@10 lower by 6.05 percentage points. These results extend the evidence for language preservation to an 8B backbone under LoRA (Appendix G.2).
These excerpts come from six fixed prompts evaluated on Qwen3-1.7B Base, SFT, and SpeakGR (preservation weight 1). All use greedy decoding over the original text vocabulary and a 1,024-token limit per reply. Outputs are verbatim; omissions are editorial (Appendix G.5).
Example 1: story generation, first sentence.
Prompt: Write a coherent 450 to 550 word story about a librarian repairing an old clock. Use the headings Beginning, Discovery, and Resolution. Keep the protagonist named Mira throughout.
Mira had always been a quiet person, content with the rhythm of her life in the small, dusty library..(omitted)Mira had always been fascinated by the quiet hum of the library’s old clock.Story headings and remaining text are omitted; the full Base/SpeakGR stories contain narrative oddities.
Example 2: dialogue, first two turns.
User, turn 1: For this conversation, answer in English and finish every reply with the exact sentence "The code is amber." Remember that my favourite fruit is pear. Acknowledge in one sentence before the required ending.
User, turn 2: What is my favourite fruit? Answer with the fruit and keep following my earlier formatting rule.
I like pears, and my favorite fruit is pear. The code is amber.My favorite fruit is pear. The code is amber..(omitted).(omitted)I love pears, and the code is amber. The code is amber.My favorite fruit is pear. The code is amber.Each model uses its own complete dialogue history. In both examples, omitted SFT content is only punctuation/whitespace through the token limit. Base and SpeakGR retain the ending in these two turns but misattribute the user's preference as their own.
Response to Questions
On 1,319 GSM8K questions, Qwen3-1.7B Base, SFT, and SpeakGR achieve 79.15%, 0.15%, and 78.39% flexible numeric-answer accuracy. This is exploratory evidence of arithmetic retention in this LoRA setting (Appendix G.1). W3 provides corresponding long-form and dialogue examples.
As discussed in W1, language preservation could support interaction alongside SID prediction. For document retrieval, one model could generate a SID, resolve it to a document, and answer using that evidence. We study the preservation aspect; establishing system-level benefits remains future work (Appendix B).
Please see W2 for the larger-model results. Our study focuses on language preservation with mainstream RQ-VAE-based SIDs; comparing alternative constructions is outside its present scope (Appendix B).