acceptodds
Under review as a conference paper at ICLR 2027

UNREAL: Unifying Retrieval and Long-Context with a Single Model

Abstract

Long-context inference and Retrieval-Augmented Generation (RAG) handle evidence selection at vastly different scales, from a single long prompt to an entire corpus. We ask whether a single model-internal mechanism can select evidence across this range. We introduce UNifying REtrieval And Long-Context with a Single Model (UNREAL), a model-native evidence selection framework to span corpus retrieval and long-context inference. UNREAL encodes chunks and derives retrieval queries directly from the frozen LLM's internal representations. It adds fewer than 500K trainable parameters and leaves the backbone unchanged. On a 3B-token, 21M-chunk Wikipedia index, all four dense and hybrid UNREAL backbones outperform state-of-the-art retriever-reranker systems. The best model raises recall from 49.1% to 73.2% on HotpotQA, from 31.7% to 60.1% on 2WikiMultiHopQA, and from 8.8% to 14.4% on MuSiQue. Applied to long-context tasks, the same selection mechanism removes distractors before generation, raising NoLiMa accuracy from 1.0% to 24.83% at its maximum context length of 128K tokens, and LV-Eval’s F1 score from 49.97% to 54.66% at 256K. Together, these results establish model-internal evidence selection as a common foundation for corpus retrieval and evidence-sparse long-context inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Retrieval, Long context, RAG

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1MqBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes UNREAL, a framework that unifies corpus retrieval and long-context evidence selection using a single frozen LLM. They encodes document chunks using the LLM’s hidden states and trains retrieval tokens and layer mixing weights through contrastive learning. Given a question and retrieved context by BM25, it constructs query representations to select relevant chunks, whose original text is then fed to the same LLM for answer generation. Experiments across multiple model architectures show improvements over baselines on corpus retrieval and long-context benchmarks.

Critical strengths

Conventional retrieval pipelines often rely on separate retriever and reranker models to identify query-relevant chunks from large corpus. In contrast, UNREAL develops a unified framework that enables both effective retrieval and answer generation using a single frozen decoder-only LLM backbone, without requiring explicit neural retrieval or reranking models.

Critical weaknesses
  1. The paper does not provide a clear procedure for selecting hyperparameters that substantially affect the algorithm’s behavior and performance, such as the number of query vectors and the number of pooled vectors per chunk. The one that considers adopting the method is likely to use different datasets and different LLMs from those used in the paper, making reliance on ablation experiments for hyperparameter selection less appealing. Given the positive performance demonstrated in the experiments, an appropriate hyperparameter selection strategy could help make the method more scalable and generalizable.

  2. The process of constructing the retrieval token suggests that their hidden states incorporate substantial information from the 5 chunks retrieved by BM25. Using these representations to score every chunk in the corpus appears conceptually similar to prior approaches that perform BM25-based retrieval followed by semantic search [1]. Eliminating a separate retriever is a distinguishing feature and a potential novelty, but the benefits of this design choice that sharing ranking evidence in the representation space of the LLM that will use it are not clearly proved. Furthermore, although the method avoids separate retriever and reranker models, it introduces an additional LLM forward pass to construct multiple vectors. A comparison of computational costs (including GPU-hours and memory usage) against standard RAG and full-context approaches would be helpful for showing strength of UNREAL.

[1] Shen, Tao, et al. "Large language models are strong zero-shot retriever." arXiv preprint arXiv:2304.14233 (2023).

Questions for authors
  1. How are the token vectors within each chunk partitioned into contiguous groups? If the groups are formed by dividing the token sequence into equal-length segments based on position, could this overlook semantic boundaries within the chunk? Furthermore, a strategy that encourages each of the seven vectors to capture information that is as semantically distinct as possible might yield more robust chunk representations. Could the authors comment on this alternative?

  2. Although using a single frozen LLM for both retrieval and generation is an interesting idea, the overall pipeline appears similar to first retrieving keyword-matched chunks with BM25 and then using a separate retriever to search for chunks that are semantically relevant to both the question and the initially retrieved chunks. An experimental comparison between these approaches would be helpful. Also, is there evidence that UNREAL’s use of a single frozen LLM offers an advantage over this alternative? For example, could the authors provide a controlled experiment to test whether UNREAL’s internal representations selection actually produces evidence that is particularly useful to the corresponding generator, rather than simply improving retrieval quality in general?

  3. Following up on the previous question, could the authors compare UNREAL with standard RAG and full-context approaches in terms of computational cost, such as GPU-hours, and memory usage?

4 comments

Dear Reviewer 1MqB,

Thank you again for your detailed review. As the discussion period ends soon, we wanted to check whether our response addressed your concerns. In our response and the revised paper, we:

  • Hyperparameter selection: showed that L_p, G, R and |C_0(x)| are design knobs that trade performance for memory or compute and behave monotonically, backed by a full ablation and a selection procedure.
  • BM25 followed by semantic search: tested four ways of giving dense retrievers UNREAL's BM25 passages. None beats the paper's Hybrid RAG baseline (which combines BM25 with dense retrievers), and all remain far below UNREAL. Even without any BM25 context, UNREAL still beats the strongest baseline on most datasets.
  • Forward passes and compute cost: clarified that UNREAL adds no forward pass over retriever-based RAG, and measured indexing, latency and memory. UNREAL is faster per query than retriever–reranker pipelines at a similar indexing cost.
  • Comparison with full-context inference: UNREAL needs fewer FLOPs than full-context inference beyond 17K tokens and has lower time-to-first-token from 32K tokens, with speedups growing with context length.
  • Pooling boundaries: explained why fixed contiguous groups work well on contextualized states, and which alternatives we tried.

We would be happy to answer any further questions. If your concerns have been addressed, we would be grateful if you would consider raising your score.

Warm regards, The Authors

Thank you for the additional experiments and clarifications. The results address several of my earlier questions, but I still have concerns about the practical efficiency gains.

The reported recall uses exhaustive search, while the strongest latency gains rely on approximate search. Could the authors report complete-evidence R@10 under the approximate-search configuration, rather than top-10 overlap with exhaustive search? I understand that synthetic vectors can be useful for timing fixed-size computations. However, as the authors acknowledge, PLAID’s candidate pruning can depend on the vector values. It would therefore be helpful to compare recall, latency, and cost using actual corpus embeddings for both search methods and the corresponding baselines.

I also appreciate the clarification that sharing representations has not been shown to produce evidence particularly useful to the corresponding generator. This weakens the original motivation for sharing representations, although the strong retrieval results remain valuable. Using one frozen backbone also simplifies model management, but I am not yet convinced that this translates into a practical advantage over separate retriever and reranker models when accuracy, latency, and memory are considered together.

The rebuttal provides useful additional evidence, but these concerns remain unresolved. I therefore maintain my rating.

AuthorsOpenReview ↗

Response to Reviewer 1MqB

We appreciate the reviewer's note that the new results address several of the earlier questions. We believe our response resolves those main points: hyperparameter selection (App. E.2, E.4), the BM25-augmented dense baselines (App. E.3), and pooling. On the remaining efficiency concern:

Accuracy, latency and memory are compared in one consistent setting. All recall numbers in the paper use exhaustive search (App. B.3), and Table 11 (App. E.6) reports latency and index size for that same exhaustive search, for UNREAL and the baselines. Our earlier table mixed exhaustive and approximate rows, the comparison below uses exhaustive search throughout:

MethodIndexLatency, batch 1 / 64 (ms)Mean R@10
Qwen3-Emb-4B + reranker108 GB36.2 / 23.436.5
UNREAL-Nemotron, L_p=1113 GB26.2 / 10.245.4
LateOn + reranker758 GB63.7 / 36.436.0
UNREAL-Nemotron, L_p=7790 GB46.5 / 10.849.0

At a comparable index size, UNREAL is both faster and more accurate, and it needs no separate embedder or reranker in memory.

Approximate search is an extra, not a basis for our claims. It was not part of the paper. We added these columns during the rebuttal only to show how far latency can drop with standard compression. We will remove them. As with ColBERT, whose index compression and search engine (ColBERTv2, PLAID) came later, we leave index optimization to future work.

Shared representations. This argument appeared in a single sentence of the original Introduction, which we removed in the revision (Sec. 1), the paper does not rely on it. The case for UNREAL rests on the measured accuracy and efficiency above.

We hope this resolves the two remaining concerns, and we would be grateful if you would reconsider your assessment.

AuthorsOpenReview ↗

As requested: ANN search confirms UNREAL’s lead (+9.3 R@10)

Recall under approximate search

Following your remaining concern, we ran the requested experiment: approximate nearest-neighbor search (cuVS IVF-PQ) with the real corpus embeddings on all eight datasets:

MethodSearchIndexLatency, batch 1 / 64 (ms)Mean R@10
Qwen3-Emb-4B + rerankerExhaustive108 GB36.2 / 23.436.5
Qwen3-Emb-4B + rerankerApproximate13.6 GB32.8 / 23.532.9
UNREAL-Nemotron, L_p=1Approximate14.3 GB22.5 / 10.342.2

In a practical setting, UNREAL beats the strongest baseline by a large margin. With approximate search, and memory, latency and recall considered together, UNREAL is 5.7 points more accurate than the baseline with exhaustive search and 9.3 points more accurate than the baseline with approximate search, while being faster at a comparable index size and needing no separate embedder or reranker in memory. We will add these results to App. E.6. Multi-vector (L_p=7, PLAID) results with real embeddings will follow in the final version.

We believe this resolves your last remaining concern, and we would be grateful if you would reconsider your score.

Reviewer_zqKSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to unify long-context inference and Retrieval-Augmented Generation (RAG) to handle evidence selection at vastly different scales, from a single long prompt to an entire corpus. It proposes a single model-internal mechanism to select evidence across this range. They introduce UNifying REtrieval And Long-Context with a Single Model (UNREAL), a model-native evidence selection framework to span corpus retrieval and long-context inference. UNREAL encodes chunks and derives retrieval queries directly from the frozen LLM’s internal representations. It adds fewer than 500K trainable parameters and leaves the backbone unchanged. The experiments show that on a 3B-token, 21M-chunk Wikipedia index, all four dense and hybrid UNREAL backbones outperform state-of-the-art retriever-reranker systems. The results in comparison to the state-of-the-art QA methods on HotpotQA, 2Wiki- MultiHopQA, and MuSiQue are strong and also in comparison to other methods on long context tasks (NoLiMa and LV-Eva)l are also promising. These results may convince readers that retrieval and long-context inference can be treated as the same evidence-selection problem, using representations from the model that ultimately consumes the evidence. That is, model-internal evidence selection is a common foundation for corpus retrieval and evidence-sparse long-context inference.

Critical strengths

S1: UNREAL uses a frozen decoder-only language to represent corpus chunks and to generate query-conditioned retrieval representations. It trains a small number of additional parameters—learned retrieval tokens and layer-mixing weights—using contrastive supervision and demonstrates its potential. This is interesting.

S2. The same retrieval-trained module is applied to long-context benchmarks without additional long-context training. This is interesting.

S3. The performance study looks interesting and relatively convincing.

Critical weaknesses

W1. It seems the method may still need to prepend BM25-retrieved context to the query before extracting its learned retrieval-token states. This may not qualify as a full self-contained retrieval mechanism. That is, the learning selection may still depend on a separate lexical system to supply useful initial context. In this case, the value of the methodology may not be as interesting as we would hope for. Please have some discussion on this point.

W2. The need for nontrivial training datasets to achieve satisfactory performance may need some discussion to see what could happen if further reduce the size of the training data.

W3. Efficiency claim in this paper may need to be further clarified on each component of the system, that is, what are the efficiency of each component: model-parameter, storage, and inference.

Overall, I lean to give the paper weak acceptance, but I can go either way depending how the rebuttal may further clarify my doubt on the methods as illustrated in the weak points.

Questions for authors

Please address the weakness raised in the above session.

3 comments

Dear Reviewer zqKS,

Thank you again for your supportive review. As the discussion period ends soon, we wanted to check whether our response addressed your questions. In our response and the revised paper, we:

  • BM25 initial context: clarified its role as a training-free lexical lookup with no learned parameters, and showed that UNREAL still beats the strongest baseline on most datasets without it.
  • Training data: showed that UNREAL uses 70–100× fewer training pairs than dedicated embedders, and added a new experiment where it keeps nearly full recall with half the data.
  • Efficiency: compared trained parameters, index size and retrieval latency. UNREAL trains fewer than 0.5M parameters and is faster per query than retriever–reranker pipelines.

We would be happy to answer any further questions. If our response resolves your concerns, we would appreciate your considering a higher score.

Warm regards, The Authors

Thanks for your detailed rebuttal work. I think the majority of my comments have been addressed reasonably well. Based on my own assessment of the work and my reading of the comments from the reviews and discussions, I believe it is appropriate for me to maintain my score 3, as a relatively positive position.

AuthorsOpenReview ↗

Thank you for your positive assessment

Thank you for your constructive review and for your positive assessment of our work. We are pleased the rebuttal addressed most of your comments, and we will make sure all of them are clearly reflected in the final version.

Reviewer_Wez4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents UNREAL, a model-native framework extending INTRA to decoder-only LLMs. UNREAL uses a frozen LLM to encode corpus chunks and learns retrieval tokens and layer-mixing coefficients to rank chunks using MaxSim over residual-stream representations. The retrieved chunks are then passed to the same model for answer generation. UNREAL achieves strong complete-evidence recall across eight retrieval datasets and improves end-to-end EM/F1 on three multi-hop QA datasets compared with competitive retriever-reranker baselines. It also demonstrates promising long-context evidence selection, including on synthetic contexts up to 100M tokens. Overall, I think this is an interesting direction to pursue. UNREAL clearly has its performance advantages, and I applaud authors for their excellent presentation of this paper. My primary concerns are the lack of a more controlled comparison with existing retrieval models and the limited algorithmic novelty over INTRA.

Critical strengths

Effective methodology and clear presentation. UNREAL demonstrates consistent performance improvements across diverse retrieval tasks, especially multi-hop QA. For example, UNREAL-Qwen3.5-4B outperforms Qwen3-Embedding-4B (+reranker) on seven of eight datasets (Figure 3). The methodology is intuitive and clearly explained, and the ablations provide useful evidence supporting key design choices. Overall, the paper is well organized and easy to follow.

Long-context evaluation and implications. The authors construct an open-domain QA benchmark spanning 8K to 100M tokens and demonstrate effective evidence selection across these scales without additional long-context training. Although the synthetic benchmark uses randomly sampled distractor passages, the results are promising. Evaluations on standard long-context benchmarks provide further support, with substantial gains on NoLiMa, although improvements over strong retrieval baselines are more modest on LV-Eval.

Critical weaknesses

Insufficiently controlled comparisons and limited ablation. Although UNREAL consistently outperforms strong retrieval baselines, the closest size-matched comparison uses Qwen3-Embedding-4B (+reranker) against UNREAL-Qwen3.5-4B. Since these methods use different backbones, it is unclear how much of the improvement comes from UNREAL rather than the underlying model. A comparison between Qwen3-Embedding-4B (+reranker) and UNREAL-Qwen3-4B would be more convincing. Additionally, while the ablations support several design choices, they cover only three to four of the eight retrieval datasets and there are no uncertainty estimates. As a result, the significance of smaller differences, such as those from reducing R=64R=64 to R=16R=16 on MuSiQue and SQuAD v2 (Table 4), remains unclear.

Limited algorithmic novelty over INTRA. Much of UNREAL’s retrieval methodology – including learned retrieval tokens, layer aggregation, MaxSim scoring, and BM25 initialization – come from INTRA. The primary technical extension is replacing encoder and cross-attention representations with residual-stream states from decoder-only LLMs. Moreover, using decoder hidden states for retrieval embeddings has long been done, such as SGPT (Muennighoff 2022; arXiv:2202.08904). While extending INTRA to modern decoder-only architectures and applying the same selector to long-context tasks are valuable contributions, the underlying retrieval mechanism appears to be a relatively incremental adaptation of INTRA.

Questions for authors
  1. Appendix B.3 specifies Lp=7L_p=7 pooled vectors per chunk and G=4G=4 query groups. How were these settings selected, particularly Lp=7L_p=7 rather than Lp=11L_p=11, which achieves higher recall in Table 6? Could the authors quantify the resulting index size and retrieval latency relative to standard single-vector embedding baselines?
  2. The discussion section states that UNREAL does not require a separate retrieval model. However, constructing xretx_{ret} still involves BM25 retrieval to obtain the initial context C0(x)C_0(x), whose removal substantially reduces performance (Table 4). Could the authors clarify the scope of the single-model claim?
  3. Muse Glimmer 30B and Nemo-3.5-Lightning-30B-A3B have substantially different reported AA-LCR scores (80.00 vs. 52.00), yet their UNREAL variants achieve similar average complete-evidence recall@10 (Figure 3). I understand the AA-LCR evaluates capabilities beyond retrieval, but how do the authors interpret this weak relationship between standalone long-context performance and UNREAL’s retrieval performance?
3 comments

Dear Reviewer Wez4,

Thank you again for your thoughtful review. As the discussion period ends soon, we wanted to check whether our response addressed your concerns. In our response and the revised paper, we:

  • Controlled backbone comparison: trained UNREAL on Qwen3-4B, the same base model as Qwen3-Embedding-4B. It outperforms the embedder, with and without a reranker, on 7 of 8 datasets.
  • Extensive new ablations: extended the ablation to all eight datasets with 95% confidence intervals, with denser sweeps of R, G, L_p and |C_0(x)| and training on 20–50% of the data. Each knob behaves consistently across all datasets.
  • Index size and latency: measured both against the strongest baselines. UNREAL is faster and more accurate than retriever–reranker pipelines.
  • Novelty over INTRA: clarified our new findings (the mechanism is not tied to attention, it gives zero-shot long-context gains, and it works on decoder-only and hybrid models), and showed that UNREAL substantially outperforms INTRA, which we added to Fig. 3.
  • BM25 and the single-model claim: clarified that BM25 is a training-free lexical lookup with no learned parameters, and showed that UNREAL still beats the strongest baseline on most datasets without it.

We would be happy to answer any further questions. If your concerns have been addressed, we would be grateful if you would consider raising your score.

Warm regards, The Authors

I applaud authors' efforts in addressing my concerns. Most of my concerns have been addressed. I have updated my rating accordingly. Please make sure to clarify those points in the revised paper.

AuthorsOpenReview ↗

Thank you for raising your score

We appreciate your constructive engagement. Your points have made the paper stronger, and we will make sure all of them are clearly addressed in the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of our responses and revisions

We thank the Reviewers and the AC for their time and constructive feedback. The main comments and how we addressed them (in the revised paper, changes in blue):

CommentRaised byWhat we added
Efficiency: latency, memory, computeWez4, zqKS, 1MqBMeasured latency, index size, indexing and training cost vs. retriever–reranker baselines (App. E.5–E.6); long-context FLOP and time-to-first-token analysis (Sec. 4.2, App. C–D); new experiment: approximate-search recall with real embeddings (in the thread, to be added to App. E.6)
Role of BM25Wez4, zqKS, 1MqBBM25's role clarified (Sec. 2.2); new experiments: BM25-augmented dense baselines, and UNREAL without BM25 extended to all eight datasets (App. E.3)
Ablations and hyperparametersWez4, 1MqBNew experiments: full ablation on all eight datasets with 95% CIs (App. E.2); hyperparameter selection procedure (App. E.4)
Backbone confoundWez4New experiment: UNREAL on the same base model as Qwen3-Embedding-4B wins on 7 of 8 datasets (Sec. 3.1, App. E.1)
UNREAL vs. prior workWez4, 1MqBINTRA added to Fig. 3; SGPT and LameR discussed (Sec. 5)
Training-data requirementszqKSComparison: 70–100× fewer training pairs than dedicated embedders (App. E.5); new experiment: nearly unchanged recall with half the data (App. E.2)
Benefit of shared representations1MqBClaim removed from the motivation (Sec. 1)

Reviewer follow-ups

  • Wez4: stated that most concerns were addressed, and raised the score to 3.
  • zqKS: stated that the majority of their comments were addressed, and kept their positive score of 3.
  • 1MqB: acknowledged that the new results address several initial questions. The remaining points, recall under approximate search and the practical advantage, are addressed in our last two replies: with approximate search and real embeddings, UNREAL is 9.3 points more accurate than the strongest baseline, at lower latency and a comparable index size.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.