- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes UNREAL, a framework that unifies corpus retrieval and long-context evidence selection using a single frozen LLM. They encodes document chunks using the LLM’s hidden states and trains retrieval tokens and layer mixing weights through contrastive learning. Given a question and retrieved context by BM25, it constructs query representations to select relevant chunks, whose original text is then fed to the same LLM for answer generation. Experiments across multiple model architectures show improvements over baselines on corpus retrieval and long-context benchmarks.
Critical strengths
Conventional retrieval pipelines often rely on separate retriever and reranker models to identify query-relevant chunks from large corpus. In contrast, UNREAL develops a unified framework that enables both effective retrieval and answer generation using a single frozen decoder-only LLM backbone, without requiring explicit neural retrieval or reranking models.
Critical weaknesses
-
The paper does not provide a clear procedure for selecting hyperparameters that substantially affect the algorithm’s behavior and performance, such as the number of query vectors and the number of pooled vectors per chunk. The one that considers adopting the method is likely to use different datasets and different LLMs from those used in the paper, making reliance on ablation experiments for hyperparameter selection less appealing. Given the positive performance demonstrated in the experiments, an appropriate hyperparameter selection strategy could help make the method more scalable and generalizable.
-
The process of constructing the retrieval token suggests that their hidden states incorporate substantial information from the 5 chunks retrieved by BM25. Using these representations to score every chunk in the corpus appears conceptually similar to prior approaches that perform BM25-based retrieval followed by semantic search [1]. Eliminating a separate retriever is a distinguishing feature and a potential novelty, but the benefits of this design choice that sharing ranking evidence in the representation space of the LLM that will use it are not clearly proved. Furthermore, although the method avoids separate retriever and reranker models, it introduces an additional LLM forward pass to construct multiple vectors. A comparison of computational costs (including GPU-hours and memory usage) against standard RAG and full-context approaches would be helpful for showing strength of UNREAL.
[1] Shen, Tao, et al. "Large language models are strong zero-shot retriever." arXiv preprint arXiv:2304.14233 (2023).
Questions for authors
-
How are the token vectors within each chunk partitioned into contiguous groups? If the groups are formed by dividing the token sequence into equal-length segments based on position, could this overlook semantic boundaries within the chunk? Furthermore, a strategy that encourages each of the seven vectors to capture information that is as semantically distinct as possible might yield more robust chunk representations. Could the authors comment on this alternative?
-
Although using a single frozen LLM for both retrieval and generation is an interesting idea, the overall pipeline appears similar to first retrieving keyword-matched chunks with BM25 and then using a separate retriever to search for chunks that are semantically relevant to both the question and the initially retrieved chunks. An experimental comparison between these approaches would be helpful. Also, is there evidence that UNREAL’s use of a single frozen LLM offers an advantage over this alternative? For example, could the authors provide a controlled experiment to test whether UNREAL’s internal representations selection actually produces evidence that is particularly useful to the corresponding generator, rather than simply improving retrieval quality in general?
-
Following up on the previous question, could the authors compare UNREAL with standard RAG and full-context approaches in terms of computational cost, such as GPU-hours, and memory usage?
4 comments
Thank you for the additional experiments and clarifications. The results address several of my earlier questions, but I still have concerns about the practical efficiency gains.
The reported recall uses exhaustive search, while the strongest latency gains rely on approximate search. Could the authors report complete-evidence R@10 under the approximate-search configuration, rather than top-10 overlap with exhaustive search? I understand that synthetic vectors can be useful for timing fixed-size computations. However, as the authors acknowledge, PLAID’s candidate pruning can depend on the vector values. It would therefore be helpful to compare recall, latency, and cost using actual corpus embeddings for both search methods and the corresponding baselines.
I also appreciate the clarification that sharing representations has not been shown to produce evidence particularly useful to the corresponding generator. This weakens the original motivation for sharing representations, although the strong retrieval results remain valuable. Using one frozen backbone also simplifies model management, but I am not yet convinced that this translates into a practical advantage over separate retriever and reranker models when accuracy, latency, and memory are considered together.
The rebuttal provides useful additional evidence, but these concerns remain unresolved. I therefore maintain my rating.
Response to Reviewer 1MqB
We appreciate the reviewer's note that the new results address several of the earlier questions. We believe our response resolves those main points: hyperparameter selection (App. E.2, E.4), the BM25-augmented dense baselines (App. E.3), and pooling. On the remaining efficiency concern:
Accuracy, latency and memory are compared in one consistent setting. All recall numbers in the paper use exhaustive search (App. B.3), and Table 11 (App. E.6) reports latency and index size for that same exhaustive search, for UNREAL and the baselines. Our earlier table mixed exhaustive and approximate rows, the comparison below uses exhaustive search throughout:
| Method | Index | Latency, batch 1 / 64 (ms) | Mean R@10 |
|---|---|---|---|
| Qwen3-Emb-4B + reranker | 108 GB | 36.2 / 23.4 | 36.5 |
| UNREAL-Nemotron, L_p=1 | 113 GB | 26.2 / 10.2 | 45.4 |
| LateOn + reranker | 758 GB | 63.7 / 36.4 | 36.0 |
| UNREAL-Nemotron, L_p=7 | 790 GB | 46.5 / 10.8 | 49.0 |
At a comparable index size, UNREAL is both faster and more accurate, and it needs no separate embedder or reranker in memory.
Approximate search is an extra, not a basis for our claims. It was not part of the paper. We added these columns during the rebuttal only to show how far latency can drop with standard compression. We will remove them. As with ColBERT, whose index compression and search engine (ColBERTv2, PLAID) came later, we leave index optimization to future work.
Shared representations. This argument appeared in a single sentence of the original Introduction, which we removed in the revision (Sec. 1), the paper does not rely on it. The case for UNREAL rests on the measured accuracy and efficiency above.
We hope this resolves the two remaining concerns, and we would be grateful if you would reconsider your assessment.
As requested: ANN search confirms UNREAL’s lead (+9.3 R@10)
Recall under approximate search
Following your remaining concern, we ran the requested experiment: approximate nearest-neighbor search (cuVS IVF-PQ) with the real corpus embeddings on all eight datasets:
| Method | Search | Index | Latency, batch 1 / 64 (ms) | Mean R@10 |
|---|---|---|---|---|
| Qwen3-Emb-4B + reranker | Exhaustive | 108 GB | 36.2 / 23.4 | 36.5 |
| Qwen3-Emb-4B + reranker | Approximate | 13.6 GB | 32.8 / 23.5 | 32.9 |
| UNREAL-Nemotron, L_p=1 | Approximate | 14.3 GB | 22.5 / 10.3 | 42.2 |
In a practical setting, UNREAL beats the strongest baseline by a large margin. With approximate search, and memory, latency and recall considered together, UNREAL is 5.7 points more accurate than the baseline with exhaustive search and 9.3 points more accurate than the baseline with approximate search, while being faster at a comparable index size and needing no separate embedder or reranker in memory. We will add these results to App. E.6. Multi-vector (L_p=7, PLAID) results with real embeddings will follow in the final version.
We believe this resolves your last remaining concern, and we would be grateful if you would reconsider your score.
Follow-up
Dear Reviewer 1MqB,
Thank you again for your detailed review. As the discussion period ends soon, we wanted to check whether our response addressed your concerns. In our response and the revised paper, we:
We would be happy to answer any further questions. If your concerns have been addressed, we would be grateful if you would consider raising your score.
Warm regards, The Authors