acceptodds
Under review as a conference paper at ICLR 2027

Policy-Integrated Query Encoding for Fixed-Index Agentic RAG: Feasibility and Limits

Abstract

Dense-retrieval agentic RAG usually runs two neural models at inference time: a policy language model that writes search queries, and a separate dense encoder that embeds every query for the passage index. We ask whether the policy’s own search-token representations can act as the query-side interface to a fixed pas- sage index, so that the external encoder is not needed online. We mean-pool the search-token hidden states, map them into the passage-embedding space with a 6.55M-parameter linear projection, retrieve a top-M candidate set with the pro- jected query, define a stochastic policy over that candidate set, and optimize token and selected-passage log-probabilities jointly under the same final-answer reward. We refer to this architecture as policy-integrated query encoding (PIQE). PIQE removes the external 4B query encoder from the online retrieval path while the corpus index stays fixed. Under a controlled comparison with a matched Search- R1 baseline over three seeds, PIQE reaches 36.11 ± 0.34 macro and 42.36 micro exact match on the frozen reporting partition against 36.77 ± 0.72 and 43.16 for the baseline, and 56.55 against 57.17 micro exact match out of domain. PIQE is within one exact-match point of the baseline on each aggregate, but remains below it on average. Retrieval diagnostics associate the deficit with candidate generation: the hidden-state interface puts an answer-containing passage into the candidate set less often than the text-query interface (Pool@20 45.67 against 47.14), and passage-level credit assignment does not compensate through better selection in- side the candidate set. PIQE is thus architecturally feasible but not lossless, and the results motivate learning signals that improve candidate generation without establishing that better selection could not close the gap.

Primary area:
reinforcement learning
Keywords:
agentic retrieval-augmented generation, query encoding, representation learning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RL49OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PIQE, which replaces an external query encoder with a linear projection of the policy's search-token hidden states. It jointly optimizes token and passage actions against a fixed passage index. A matched Search-R1 comparison shows small average accuracy losses, and retrieval diagnostics examine the remaining gap.

Critical strengths
  1. The architectural question is clear, and a 6.55M-parameter projection retains most of the matched baseline's accuracy without updating the passage index.

  2. The paper reports three seeds, paired tests, and extensive retrieval diagnostics, while clearly stating the limitations of the prototype.

Critical weaknesses
  1. The prototype requires an additional policy forward pass to recover hidden states and demonstrates no actual speedup. Removing a separate encoder therefore does not yet establish a practical efficiency advantage.

  2. The baseline uses a 4B query encoder, whereas the original Search-R1 uses the much smaller e5-base-v2. A smaller-encoder comparison is needed to assess the accuracy-resource trade-off.

  3. The benefit of joint passage-level RL remains unclear. In Table 4, PIQE does not improve ID macro EM over the frozen-projection control (35.73 vs. 35.76), and the controls also differ in sampling and gradient flow.

  4. Experiments cover only one policy family, one embedding space, and one corpus. It is unclear whether the replacement remains effective with different policy-retriever combinations.",

Questions for authors

See the weaknesses above.

1 comment

Thank you for the careful review and for recognizing the clear architectural question and the multi-seed, paired evaluation. All new results below are from post-submission runs on Cluster B (4× H100), kept separate from the submitted Cluster A results.

1. Extra forward pass and practical efficiency

We agree. We timed one greedy Dev1200 pass (1,200 questions) per system on the same node with identical vLLM and batch settings.

Per Dev1200 passSearch-R1 (4B encoder)PIQESearch-R1 (e5-base-v2)
Total646 s1,169 s312 s
vLLM generation300 s291 s237 s
Query side11.5 s encoding53.5 s extra forward + 0.8 s projection3.7 s encoding
Passage-text lookup316 s (5 per search)783 s (20 per search)60 s (5 per search)
Peak memory, encoder GPU83.5 GiB75.9 GiB (encoder removed)58.5 GiB

The extra forward costs about 4.7 times the encoding it replaces, and the prototype fetches text for all 20 candidates although only the 5 selected enter the context. Excluding text lookup, a pass takes ≈331 s for Search-R1 and ≈386 s for PIQE. PIQE times use a lossless binary transfer (identical outputs; 1,326 s with the JSON transfer of the submitted runs), and a training step takes 1.46× longer (JSON transfer, as in Table 14). Removing the online encoder saves 7.5 GiB on the GPU hosting it. The current prototype does not reduce latency; reusing hidden states from generation and fetching only the selected passages are implementation improvements that remain untested.

2. Smaller query encoder

We trained Search-R1 with e5-base-v2, as in the original Search-R1, with its own index of the same corpus on all three seeds. On the 11,262-question reporting partition (averaging steps 180/190/200 rather than the submitted selection rule), micro EM is 42.97 ± 0.73 for the 4B baseline, 42.23 ± 0.42 for PIQE, and 41.08 ± 0.83 for e5 (mean ± s.d.). Paired micro EM differences (McNemar):

Seede5 − 4BPIQE − 4BPIQE − e5
13−1.46 (p<0.001)−1.14 (p=0.004)+0.32 (p=0.45)
21−1.46 (p<0.001)+0.44 (p=0.25)+1.90 (p<0.001)
42−2.78 (p<0.001)−1.54 (p<0.001)+1.24 (p=0.002)

PIQE has higher micro EM than e5 on all three seeds, significantly on two. Both remove the 4B encoder from the online path: PIQE keeps the index built offline by that encoder, whereas e5 replaces both the encoder and the index. e5 is the fastest and lightest option (≈58 vs. ≈76 GiB peak on every GPU, mainly from its smaller 768-d index) but the least accurate. We will add this trade-off to the paper.

3. Benefit of joint passage-level RL

We agree that Table 4 does not show a benefit and that its controls differ in sampling and gradient flow. As a more direct test, we kept a trained actor fixed and swapped only the projection. At the step-100 checkpoints of seeds 13 and 21, replacing the RL-adapted projection with its warm-started initialization changes micro EM on Dev1200 + reserve (11,136 questions) by +0.31 [−0.15, +0.77] and +0.23 [−0.25, +0.71] (paired bootstrap 95% CI). The projection changes during RL (≈1.0% relative weight change by step 200), but we find no evidence that this improves accuracy. This test isolates projection adaptation under a fixed actor, not the passage objective's effect on actor learning.

We will not claim an accuracy gain from joint passage-level training. Instead, we will emphasize the hidden-state query interface, which queries the fixed external index without an online encoder, together with the measured accuracy–memory trade-off and the remaining latency overhead.

4. One policy family, embedding space, and corpus

We agree that this limits generality, as our Limitations section states. We have not evaluated other policy-retriever pairs or corpora and will keep the claims scoped to the tested configuration.

Reviewer_PNwVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate whether an agentic RAG policy can substitute the heavy‑weight online query encoder using internal model hidden‑states. They propose PIQE that mean‑pools hidden states of generated search tokens and employs a lightweight 6.55M linear projection to map them into a fixed passage embedding space. The projection is warm‑started via knowledge distillation from an external encoder. Built upon the Search‑R1 setup, PIQE applies GRPO reinforcement learning with final‑answer exact‑match reward, where passage selection from top‑20 candidates is treated as an explicit action and jointly optimized alongside token generation. In controlled multi‑seed evaluation against matched Search‑R1 baseline, PIQE underperforms across all main metrics. In addition, diagnostic analysis attributes most performance deficit to candidate generation stage instead of within‑candidate passage selection.

Critical strengths
  1. PIQE replaces an online 4B query encoder with a tiny 6.55M linear projection while keeping the original fixed passage index unchanged. It retains nearly all baseline accuracy (only 0.6‑0.8 exact‑match point drop) under controlled multi‑seed experimental setup.
  2. This work systematically pinpoints performance gap originates mainly from candidate generation, rather than the passage‑selection policy inside retrieved candidate pool. It honestly distinguishes architectural possibility from realized empirical speed‑up, without over‑claiming practical efficiency improvements.
Critical weaknesses
  1. Limited methodological novelty and lack of demonstrated practical benefit:
    • Conceptually related prior work such as Search‑R3 already constructs retrieval embeddings from LLM hidden states; training only query‑side parameters against fixed index is established practice in dense retrieval. PIQE’s additions are incremental: mean‑pooling + linear projection plus passage‑level GRPO objective.
    • No end‑to‑end accuracy improvement over baseline. The PIQE+RRF variant which recovers comparable performance re‑introduces the external online encoder, so it is not a valid instantiation of the target architecture. Moreover, RL‑updating the projection brings no tangible gain compared with frozen distillation‑initialized projection on the held‑out reporting partition.
    • Efficiency benefit remains purely architectural. The prototype requires an extra forward pass of the main policy LLM to obtain hidden states, and no measured inference latency/throughput gain is shown. Meanwhile training cost increases to ~1.5× compared with Search‑R1 baseline. There is no empirical evidence showing concrete real‑world advantage for adopting PIQE.
  2. The experimental scope is restricted and baseline comparisons are insufficient:
    • The primary evaluation only uses Search‑R1 as baseline. Important closely‑related RL‑agent‑RAG systems cited in the paper including Search‑R3, Agentic‑R, ReSearch, R‑Search are missing from empirical comparison.
    • Experiments are limited to one policy‑encoder pair; evaluations across diverse policy models / embedding backbones are absent. Comparisons against small‑sized lightweight online query encoders are also not provided. This restricts how general the feasibility conclusion can be.
Questions for authors
  1. Could you add empirical comparisons against closely‑related published methods (Search‑R3, Agentic‑R etc.) as well as small‑size online query encoders?
  2. Please provide end‑to‑end analysis or measurement to illustrate concrete practical benefits brought by removing the online query encoder, beyond pure architectural feasibility.
  3. Could you extend experiments over multiple different policy models and embedding spaces to strengthen the generalisability of your findings?
1 comment

Thank you for the detailed review and for recognizing the controlled multi-seed setup and the candidate-generation diagnosis. All new results below are from post-submission runs on Cluster B (4× H100), kept separate from the submitted Cluster A results.

1. Related methods and small online encoders

We trained Search-R1 with e5-base-v2 (110M parameters) and its own index of the same corpus on all three seeds. On the 11,262-question reporting partition (averaging steps 180/190/200 rather than the submitted selection rule), micro EM is 42.97 ± 0.73 for the 4B baseline, 42.23 ± 0.42 for PIQE, and 41.08 ± 0.83 for e5 (mean ± s.d.). Paired micro EM differences (McNemar):

Seede5 − 4BPIQE − 4BPIQE − e5
13−1.46 (p<0.001)−1.14 (p=0.004)+0.32 (p=0.45)
21−1.46 (p<0.001)+0.44 (p=0.25)+1.90 (p<0.001)
42−2.78 (p<0.001)−1.54 (p<0.001)+1.24 (p=0.002)

PIQE achieves higher micro EM than the e5-base-v2 system on all three seeds, significantly on two. This comparison captures our target use case: PIQE retains an index built offline by the strong encoder without running that encoder online, whereas e5 uses its own index of the same corpus. e5 remains the fastest and lightest option.

Our study asks whether an agentic policy can query an existing, fixed passage index without an online query encoder. This differs from approaches that train the retrieval model and update the corpus representation, such as Search-R3 and Agentic-R. Search-R3 trains the LLM as an embedding model, updates document embeddings, and is evaluated with ranking metrics on retrieval benchmarks. Agentic-R fine-tunes the retriever and re-encodes the corpus, with a different agent (Qwen2.5-7B-Base), training data, and search budget. A matched comparison requires retraining under one protocol, which we could not complete within the discussion period. We will discuss both in the paper as differently constrained alternatives. ReSearch and R-Search change the trajectory formulation and rewards while using a standard external retriever, so they are complementary to the query interface; Search-R1 is the matched representative of this family.

2. Practical benefit of removing the online encoder

We timed one greedy Dev1200 pass (1,200 questions) per system in one job on the same node, with identical vLLM and batch settings.

Per Dev1200 passSearch-R1 (4B encoder)PIQESearch-R1 (e5-base-v2)
Total646 s1,169 s312 s
vLLM generation300 s291 s237 s
Query side11.5 s encoding53.5 s extra forward + 0.8 s projection3.7 s encoding
Passage-text lookup316 s (5 per search)783 s (20 per search)60 s (5 per search)
Peak memory, encoder GPU83.5 GiB75.9 GiB (encoder removed)58.5 GiB

The measured benefit is limited to removing the encoder and its memory (7.5 GiB on that GPU; the other three GPUs stay at 75.7–76.8 GiB in both systems). The prototype is slower: the extra policy forward costs about 4.7 times the encoding it replaces, and it fetches text for all 20 candidates although only the 5 selected enter the context. PIQE times use a lossless binary transfer (identical outputs; 1,326 s with the JSON transfer of the submitted runs), and a training step takes 1.46× longer (JSON transfer, as in Table 14). We agree that this does not establish a practical advantage and will state so. A latency benefit would require reusing hidden states from generation and fetching only the selected passages, which we have not implemented. We also agree that PIQE + RRF is not an instance of the target architecture and will keep it only as a diagnostic.

**3. Other policies and embedding spaces **

We agree that this limits generality; our Limitations section states that the evidence covers one policy family and one passage-embedding space. We have not evaluated other pairs and do not claim generality beyond the tested configuration.

4. Novelty and what the study shows

our contribution is to show that an agentic policy can query a fixed external index through its own hidden states, without an online query encoder, and to characterize the remaining accuracy gap in a controlled multi-seed comparison. Query-side training against a fixed index is established; our focus is this replacement within a multi-step QA policy.

With the trained actor fixed, replacing the RL-adapted projection by its warm-started initialization changes micro EM by +0.31 and +0.23 points on Dev1200 + reserve (seeds 13 and 21; 95% CIs include 0), so we will not claim a gain from joint training. In a seed-13 Dev1200 analysis, 78% of search events with no gold supporting title in the top 20 also have none in the top 100, so a larger pool would not resolve most misses. We will revise the contribution statement accordingly.

Reviewer_6n2MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether an agentic RAG policy can replace an external query encoder while keeping the passage index fixed. PIQE pools the hidden states of generated search tokens and maps them into the passage embedding space with a 6.55M-parameter linear projection. The projection is first trained by distilling the external encoder. The policy and projection are then trained with a joint token and passage policy objective using final-answer rewards. Across three seeds, PIQE removes the external online query encoder and stays within one EM point of the matched Search-R1 baseline on the three main aggregates, but its average accuracy is lower. The paper also analyzes the retrieval gap and the limits of training on a truncated candidate set.

Critical strengths
  1. Keeping an existing passage index fixed is a useful practical constraint. The paper shows that a small projection can connect the policy to a large external passage space while retaining most of the baseline accuracy. This is a useful feasibility result.

  2. The main comparison uses a matched baseline and three random seeds. The authors clearly report the accuracy loss, the extra forward pass in the implementation, and the limits of the retrieval diagnostics.

Critical weaknesses
  1. The added contribution and the benefit of joint training are limited. PIQE differs from Search-R1 by using a fixed external passage space and final-answer rewards in a multi-step QA setting. These are meaningful differences, but the paper does not show a clear benefit from its added joint training stage over the distilled projection. In Table 4, the frozen-projection control reaches 35.76 in-domain macro EM, compared with 35.73 for full PIQE. Out-of-domain micro EM changes only from 56.45 to 56.64. They do not establish a clear gain from the added training complexity. This leaves the contribution beyond a narrow feasibility demonstration relatively small.

  2. The practical accuracy-cost trade-off is not established. In Table 1, PIQE is below the matched Search-R1 baseline on all three mean aggregates: 36.11 vs. 36.77 in-domain macro EM, 42.36 vs. 43.16 in-domain micro EM, and 56.55 vs. 57.17 out-of-domain micro EM. A small accuracy loss could be reasonable if it brought a clear deployment benefit. However, Section 3.5 states that the current implementation needs an additional policy forward pass to recover hidden states and demonstrates no wall-clock speedup. Fewer query-side parameters therefore do not establish lower inference cost. The reported 200-step training runs also take 27.2–27.7 hours for PIQE, compared with 18.6–18.9 hours for Search-R1 (Table 14). The paper does not compare the trade-off with a smaller external encoder. PIQE + RRF improves accuracy, but it uses the external encoder again and therefore does not establish the value of removing it. With lower average accuracy and no measured inference benefit, the practical reason to adopt PIQE remains unclear.

Questions for authors
  1. Table 4 shows almost no difference between the frozen-projection control and full PIQE on the reporting sets. Is this result consistent across seeds under matched training conditions? What evidence shows that the joint passage objective adds value beyond the warm-started projection?

  2. Does the current implementation reduce end-to-end inference latency or peak GPU memory once the additional policy forward pass is included? Please provide any available measurements under matched hardware and batch settings, and clarify how this trade-off compares with using a smaller external query encoder. These measurements would help determine whether the accuracy loss is worthwhile.

1 comment

Thank you for the careful review. Our additional experiments show that PIQE achieves higher micro EM than e5 on all three seeds, significantly on two, while retaining the strong encoder’s fixed index without running it online. Its three-seed mean micro EM is 0.74 points below the 4B baseline. All new results are from post-submission runs on Cluster B (4× H100), reported separately from the submitted Cluster A results.

1. Does joint training add value beyond the warm-started projection? We agree that Table 4 does not show a benefit of full PIQE over the frozen-projection control. Table 4 is a single-seed diagnostic, and its conditions also differ in training-time passage selection (stochastic vs. deterministic), so it is not a matched multi-seed comparison.

We therefore ran a more direct test: we kept a trained actor fixed and swapped only the projection. At the step-100 checkpoints of seeds 13 and 21, replacing the RL-adapted projection with its warm-started initialization changes micro EM on Dev1200 + reserve (11,136 questions) by +0.31 [−0.15, +0.77] and +0.23 [−0.25, +0.71] (paired bootstrap 95% CI). The projection does change during RL (≈1.0% relative weight change by step 200 on all three seeds), but we find no evidence that this change improves accuracy.

Accordingly, we will not claim a gain from joint training. The revision will present PIQE as a feasibility result for querying a fixed index from policy hidden states, together with its accuracy cost and the candidate-generation deficit that RL did not resolve. Our matched Search-R1 baseline shares the passage space, reward, and QA setting; the differences are the query interface, the passage-action objective, and training-time passage selection.

2. End-to-end latency and peak GPU memory (Q2)

We timed one greedy Dev1200 pass (1,200 questions) with identical vLLM and batch settings and the passage index sharded over the four GPUs.

Per Dev1200 passSearch-R1 (4B encoder)PIQESearch-R1 (e5-base-v2)
Total646 s1,169 s312 s
vLLM generation300 s291 s237 s
Query side11.5 s encoding53.5 s extra forward + 0.8 s projection3.7 s encoding
Passage-text lookup316 s (5 per search)783 s (20 per search)60 s (5 per search)
Peak memory, encoder GPU83.5 GiB75.9 GiB (encoder removed)58.5 GiB

The current implementation does not reduce latency. The extra policy forward costs about 4.7 times as much as the 4B query encoding it replaces; reusing the hidden states already computed during generation could remove it, but we have not implemented or measured this. Most of the remaining gap is passage-text lookup: the prototype fetches text for all 20 candidates, although selection needs only their embeddings and only the 5 selected passages enter the context, so this is avoidable overhead. Excluding this disk-bound, run-dependent cost, a pass takes ≈331 s for Search-R1 and ≈386 s for PIQE.

PIQE timings use a lossless binary transfer of candidate embeddings without the unused encoder (identical outputs; 1,326 s with the JSON transfer of the submitted runs). The training times in Table 14 also used JSON transfer; we have not remeasured training with binary transfer.

On the encoder GPU, PIQE peaks at 75.9 GiB vs. 83.5 GiB for Search-R1; within PIQE, removing the encoder saves 7.50 GiB, consistent with its weights. The other three GPUs stay at 75.7–76.8 GiB in both systems, set by vLLM preallocation and index shards, so these are observed usage, not minimum requirements.

3. Smaller external encoder

We trained Search-R1 with e5-base-v2 and its own index of the same corpus on all three seeds. On the 11,262-question reporting partition (averaging steps 180/190/200 rather than the submitted selection rule), micro EM is 42.97 ± 0.73 for the 4B baseline, 42.23 ± 0.42 for PIQE, and 41.08 ± 0.83 for e5 (mean ± s.d.). Paired micro EM differences (McNemar):

Seede5 − 4BPIQE − 4BPIQE − e5
13−1.46 (p<0.001)−1.14 (p=0.004)+0.32 (p=0.45)
21−1.46 (p<0.001)+0.44 (p=0.25)+1.90 (p<0.001)
42−2.78 (p<0.001)−1.54 (p<0.001)+1.24 (p=0.002)

The small encoder is the fastest and lightest option (≈58 vs. ≈76 GiB peak on every GPU), but it is less accurate than the 4B baseline on every seed. PIQE has higher micro EM than e5 on all three seeds, significantly on seeds 21 and 42. The PIQE–4B gap (0.74 micro points). The PIQE–4B gap (0.74 micro points) is close to the submitted one (0.80).

We will add these measurements and their conditions to the paper, state that a benefit of joint training is not established, and keep PIQE + RRF only as a diagnostic that reintroduces the encoder.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.