acceptodds
Under review as a conference paper at ICLR 2027

Avoiding Search Behavior Homogenization in Agentic RAG with Query-and-Document Trajectory Steering

Abstract

Agentic retrieval-augmented generation (RAG) integrates dynamic search tool calling with reinforcement learning with verifiable rewards (RLVR) to overcome the static parametric knowledge of LLMs. However, training a single policy across diverse search patterns induces behavioral conflicts, causing the model to collapse onto an average search pattern, which we call **behavioral homogenization**. Since existing approaches leverage extra compute or memory for explicit planning and self-correction, they incur significant computational inefficiency and fail to foster positive transfer among similar search patterns while disentangling conflicting search patterns. To overcome this challenge, we propose **query-and-document trajectory steering** (**QDTS**), a framework designed to maximize cooperative learning across complementary search patterns while suppressing conflicting signals with minimal cost, by steering generation with query and document contexts. Specifically, **query-side pre-rollout steering** categorizes queries into structural patterns via a rule-based parser, supplying corresponding class contexts to guide targeted reasoning pathways during both training and inference. Furthermore, **document-side mid-rollout steering** evaluates online retrieval statistics and deploys dynamic mid-execution contexts across multi-turn rollouts to identify search failures and steer subsequent actions. By integrating this two-stage mechanism to generate and optimize agentic search trajectories, QDTS effectively learns an adaptive policy across diverse query patterns, outperforming the state-of-the-art method by 9.2% on average across comprehensive agentic RAG benchmarks.

Primary area:
reinforcement learning
Keywords:
Agentic RAG, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rHKqOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies behavioral homogenization in agentic RAG trained with RLVR, query-side search-depth homogenization and document-side integration-selectivity homogenization. It proposes Query-and-Document Trajectory Steering (QDTS), which injects a pre-rollout context based on query depth and a mid-rollout context based on retrieval state, optimized before RL with GEPA. Across six QA benchmarks, QDTS reports an average EM gain of 3.7 points over the strongest baseline.

Critical strengths
  1. The author gives a clear formalization of two homogenization modes and their metrics, depth slope, over-/under-search, and selectivity.
  2. Prospective context steering is lightweight and does not change the reward or rollout budget.
  3. The author conducts extensive experiments, including 6 QA benchmarks, multiple backbones, and ablations.
Critical weaknesses
  1. The theoretical bound assumes independent per-hop retrieval success p<1p<1 and does not empirically estimate pp.
  2. GEPA context-optimization cost is not reported in the efficiency table. It looks like there is no direct evidence found in the manuscript for its total training overhead.
  3. Mis-triggered context cost is not quantified, Fig. 5 shows the agent can ignore context, but no aggregate failure rate is given.
Questions for authors
  1. How is the per-hop success probability pp in Eqn. 4 & 13 estimated or validated?
  2. What is the wall-clock and token cost of GEPA context optimization, and is it included in Tab. 3 training time?
  3. Suggestions for future work: Report context mis-trigger rates and the over-search/under-search trade-off across more retrievers and larger backbones.
4 comments
AuthorsOpenReview ↗

Response to Reviewer rHKq (1/2)

We sincerely appreciate the positive feedback on our work and the suggestions for strengthening the empirical side of the analysis. We hope that our rebuttal addresses all major concerns and clarifies any remaining questions.

W1 / Q1. The bound assumes independent per-hop retrieval success and does not empirically estimate p. How is the per-hop success probability in Equations 4 and 13 estimated or validated?

We thank the reviewer for this careful observation; it is a good catch. Independence is not essential to the argument. We assumed it in the submission only for simplicity of exposition, so that the product over hops could be written with a single pp. The submission did not estimate pp, because the argument uses only p<1p < 1. Below, we show that the bound holds without independence, and we estimate pp empirically.

The bound holds without independence. Let EhE_h denote the event that the search at hop hh retrieves the passage required for that hop. A trajectory completes a query of depth H∗(x)H^\ast(x) only if all of E1,…,EH∗E_1, \dots, E_{H^\ast} occur, and by the chain rule

P(⋂h=1H∗Eh)=∏h=1H∗P(Eh∣E1,…,Eh−1)=∏h=1H∗ph≤pH∗,p=max⁡hph,P\Big(\bigcap_{h=1}^{H^\ast} E_h\Big) = \prod_{h=1}^{H^\ast} P\big(E_h \mid E_1, \dots, E_{h-1}\big) = \prod_{h=1}^{H^\ast} p_h \le p^{H^\ast}, \qquad p = \max_h p_h ,

where php_h is the success probability of hop hh conditional on the earlier hops having succeeded. This is an identity followed by a maximum, so it holds for any dependence between hops; the bound itself needs only p<1p < 1. We have updated Equations 4 and 13 in this conditional form and define pp as max⁡hph\max_h p_h in the revision.

Empirical estimate of hop success probability. We also estimated hop success probability (php_h) directly from rollouts of the base policy θ0\theta_0 on the 3-hop+ questions of MuSiQue, the deepest bucket in our evaluation (Table R16): p^h\hat p_h is the fraction of search steps at hop hh whose top-3 passages contain the gold passage of that hop, computed on trajectories whose earlier hops succeeded. Two observations follow. First, every p^h\hat p_h is far below 1, so the condition p<1p < 1 required by the bound holds with a wide margin. Second, php_h decreases as the hops proceed, from 0.244 at the first hop to 0.157 at the second and 0.089 at the third. As a result, the mass of full-chain trajectories ∏hp^h=0.0034\prod_h \hat p_h = 0.0034 decays faster than the constant-pp bound pH∗=0.0145p^{H^*} = 0.0145 of Equation 4, by a factor of 4.3. The bound of Equation 4 is therefore conservative, and the depth-dependent decay that motivates QDTS is stronger in practice than the bound states.

Table R16: Estimated per-hop retrieval success of the base policy θ0\theta_0 on MuSiQue 3-hop+ questions (Qwen2.5-7B-Instruct, E5, top-3; 400 questions, 8 rollouts each). p^h\hat p_h is conditional on the earlier hops having succeeded. "Bound" is the constant-pp bound pH∗p^{H^\ast} with p=max⁡hp^h=p^1p = \max_h \hat p_h = \hat p_1, and "Estimated" is the product ∏hp^h\prod_h \hat p_h.

p^1\hat p_1p^2\hat p_2p^3\hat p_3BoundEstimated
0.2440.1570.0890.01450.0034

W2 / Q2. What is the wall-clock and token cost of GEPA context optimization, and is it included in Table 3?

It was not included in Table 3, and we agree that it should be reported.

Wall-clock time. On the same 4×H100 used for training, context optimization takes only 38.8 minutes in total (17.7 min for CpreC_{\text{pre}} and 21.1 min for CmidC_{\text{mid}}), that is, 2.9% of the total training cost, with 1,731 rollouts and 121 reflection calls (Table R17). The dominant cost of the method is the RL rollouts, as for the baselines. We have added this time to the training time of QDTS in Table 3, which becomes 22.1 h and still remains below that of GRPO + MGS (24.7 h) and AutoSearch (22.4 h).

Token cost. In tokens, the optimization processes 4.1M input tokens and 256k output tokens, against 35M and 2.8M for the rollouts of RL training. Its share of the tokens (about 10%) is larger than its share of the time, because RL training also spends time on the gradient updates, which the token counts do not include. All tokens are processed by θ0\theta_0 on the same GPUs, and no external API is called.

Table R17: Cost of context optimization and RL training (4×H100).

StageWall-clockRollouts / reflectionsToken cost (input/output)Share of total (time)
CpreC_{\text{pre}} optimization (Eq. 9)17.7 min849 / 571.64M / 124k1.3%
CmidC_{\text{mid}} optimization (Eq. 12)21.1 min882 / 642.46M / 132k1.6%
RL training (150 steps)21.5 h19,200 / —35M / 2.8M97.1%
Total22.1 h20,931 / 12139.1M / 3.1M100%
AuthorsOpenReview ↗

Response to Reviewer rHKq (2/2)

W3 / Q3. Mis-triggered context cost is not quantified; Figure 5 shows the agent can ignore context, but no aggregate rate is given; report context mis-trigger rates.

We now measure two new quantities, which are the rate of mis-triggered contexts and the aggregate failure rate of the behavior in Figure 5, where the policy ignores a context. We call this behavior a policy override, because the policy receives a context and does not follow it, and Table R20 shows that it does not lower accuracy. The aggregate failure rate that the reviewer asks for is the share of test questions with at least one override. We first quantify the mis-triggers of each stage and then the overrides. The three analyses use the per-question outputs of a single seed.

Mis-triggered pre-rollout context (Table R18). A pre-rollout context (CpreC_{\text{pre}}) is mis-triggered when the parser assigns the wrong class, which happens for only 13.5% of test questions. QDTS still outperforms GRPO on these questions, so a wrong class reduces the gain but does not turn it into a loss. GRPO, which never sees the class, also drops on this subset, which shows that the parser fails mostly on questions that are difficult in themselves.

Table R18: Mis-triggered CpreC_{\text{pre}}: test EM stratified by parser correctness (Qwen2.5-7B-Instruct, E5; average over four multi-hop benchmarks; single seed).

SubsetShare of test setGRPO EMQDTS EMΔ
Parser correct86.5%37.544.8+7.3
Parser wrong (CpreC_{\text{pre}} mis-triggered)13.5%26.430.6+4.2
All100%36.042.9+6.9

Mis-triggered mid-rollout context (Table R19). A mid-rollout context (CmidC_{\text{mid}}) is mis-triggered when it is emitted although the gold passage is already among the retrieved passages. This is rare, and QDTS still outperforms GRPO on the affected questions, because the context describes a state and leaves the decision to the policy.

Table R19: Mis-triggered CmidC_{\text{mid}} (QDTS, Qwen2.5-7B-Instruct, E5; average over four multi-hop benchmarks; single seed). EM is computed on the questions that received a mis-triggered context of the given state; GRPO is evaluated on the same questions.

StateMis-trigger share of test setGRPO EMQDTS EMΔ
Distant Query-Doc1.6%37.440.7+3.3
Dense Docs4.6%34.538.6+4.1

Policy overrides of mid-rollout contexts (Table R20). An override, which is the behavior shown in Figure 5, happens in only 7.8% of test questions. These questions are relatively easy, and GRPO is also more accurate on them than on the rest. The retrieved passages already suffice there, and the policy answers instead of searching again. An override is therefore not always harmful, and QDTS remains ahead of GRPO on these questions. For CpreC_{\text{pre}}, not following the context means falling back to a query-independent depth, which is the homogenization that Table 2 measures.

Table R20: Policy overrides of CmidC_{\text{mid}} (Qwen2.5-7B-Instruct, E5; average over four multi-hop benchmarks; single seed). An override is a mid-rollout context that the policy receives and does not follow; GRPO is evaluated on the same questions.

SubsetShare of test setGRPO EMQDTS EMΔ
At least one override of CmidC_{\text{mid}}7.8%40.642.2+1.6
No override92.2%35.643.0+7.4

Suggestion. Report the over-/under-search trade-off across more retrievers and larger backbones.

We thank the reviewer for this valuable suggestion. We will add the query-side metrics (depth slope, over-search, under-search) for the BGE-M3 retriever and the Qwen3-8B and Llama-3.1-8B backbones to Appendix M in the final version, and extend to larger backbones and additional retrievers in future work, where the over-search term in particular deserves study since part of it reflects legitimate re-searches after a missed retrieval rather than homogenization.


Overall, we have incorporated these responses and additional results into the revised paper. Please refer to the uploaded PDF, where the changes are highlighted in red. We hope that most of your concerns have been addressed and would be glad to engage in further discussion if needed.

I thank the authors for their detailed rebuttal, which addresses most review comments. Particularly, the derivations relaxing the independence assumption, empirical success rates for step-by-step retrieval, and quantified GEPA optimization overhead resolve most of my concerns. Consequently, I maintain my positive stance on the paper and have raised my confidence.

We are very glad that our rebuttal strengthened your confidence in your positive evaluation. Thank you again for your positive assessment of our work. Your comments were of great help in improving our work further!

Reviewer_V1czOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies behavioral homogenization in RL-trained agentic RAG, where one policy adopts similar search depths and document-integration decisions across heterogeneous queries. It proposes QDTS, combining rule-based pre-rollout query classification with embedding-statistics-based mid-rollout steering and offline prompt optimization. Experiments on six QA benchmarks report improved answer accuracy, search-depth adaptivity, document selectivity, and competitive efficiency, with additional analyses across model backbones, retrievers, sampling strategies, qualitative trajectories, and training-sample construction choices.

Critical strengths
  1. The paper formulates a concrete failure mode through complementary query-side and document-side views, supported by explicit behavioral metrics rather than answer accuracy alone.
  2. The exposition is unusually coherent: the theoretical motivation, two-stage design, behavioral metrics, and trajectory visualizations align, making the intended intervention points easy to understand.
Critical weaknesses
  1. The component ablation does not isolate the claimed document-side contribution: adding Cmid after Cpre provides essentially no gain, while performance improves only after GEPA. Cmid-only, GEPA-only, and separately optimized component controls are needed for causal attribution.
  2. The strongest baselines are partly copied from their original papers rather than reproduced under the same backbone, retriever, corpus, coreset, and budget. This weakens the stated 9.2% comparison.
  3. QDTS relies on hand-designed query classes, surface-rule hop prediction, and thresholds fitted on held-out retrievals, yet evaluation remains within closely related Wikipedia QA settings. Cross-dataset (i.e. out-of-domain data) transfer without refitting is needed to establish generality.
Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer V1cz (1/2)

We sincerely appreciate the positive feedback on our work and the careful reading of the ablations. Below, we address each of the concerns.

W1. The component ablation does not isolate the document-side contribution: adding Cmid after Cpre gives no gain, while performance improves only after GEPA. Cmid-only, GEPA-only, and separately optimized controls are needed.

We first clarify the role of GEPA (i.e., context optimization). QDTS steers through injected contexts, and a context steers only as well as it is written, so optimizing the contexts before RL is a step of the method (Equations 9 and 12). GEPA is applied only to the strings that QDTS adds; the base agent prompt, which QDTS shares with the Search-R1 baselines, is unchanged, and StepSearch, ReasonRAG, and AutoSearch keep the agent prompts of their own papers. We trained the controls requested by the reviewer and now ablate one factor at a time against the full method; §4.3 of the revision is organized in the same way.

Document-side contribution and separately optimized components (Table R12). With optimized contexts, each stage alone improves accuracy, and the full method is better than either. The document-side contribution is now isolated: CmidC_{\text{mid}} alone restores selectivity (SS), which CpreC_{\text{pre}} alone leaves low. "+ CpreC_{\text{pre}}" and "+ CpreC_{\text{pre}} + CmidC_{\text{mid}}" share an identical pre-rollout context, so their difference is the net contribution of mid-rollout steering.

Table R12: Component ablation (Qwen2.5-7B-Instruct, E5, average over four multi-hop benchmarks). All contexts are optimized. The Search-R1 (GRPO) and QDTS rows are three-seed means and differ slightly from the submission.

SettingEM / F1m(H,H∗)↑m(H,H^*)\uparrowP(int∣g)↑P(\text{int}\mid g)\uparrowP(int∣¬g)↓P(\text{int}\mid\neg g)\downarrowS↑S\uparrow
Search-R1 (GRPO)35.5 / 43.80.180.850.770.09
+ CpreC_{\text{pre}}40.5 / 50.00.430.860.670.22
+ CmidC_{\text{mid}}40.1 / 48.90.390.720.280.61
+ CpreC_{\text{pre}} + CmidC_{\text{mid}} (QDTS)43.3 / 50.60.650.880.320.64

Context optimization by GEPA (Table R13). The seed contexts without GEPA lower P(int∣¬g)P(\text{int}\mid\neg g) as intended, but they also lower P(int∣g)P(\text{int}\mid g) below that of GRPO: the policy searches again even when the gold passage is already retrieved. This is why adding seed CmidC_{\text{mid}} gave no gain in the submitted Table 4 (Table 5 in the revision). (Table 4 in the original submission was intended to show that unoptimized contexts may not fully realize the benefits of steering.) Optimization did not reword the retained strings (Table 12); it removed the states and reminders that caused this over-search, and both selectivity and accuracy improve as a result.

Table R13: Context-optimization ablation (Qwen2.5-7B-Instruct, E5, average over four multi-hop benchmarks). Both QDTS rows use CpreC_{\text{pre}} + CmidC_{\text{mid}}.

SettingEM / F1m(H,H∗)↑m(H,H^*)\uparrowP(int∣g)↑P(\text{int}\mid g)\uparrowP(int∣¬g)↓P(\text{int}\mid\neg g)\downarrowS↑S\uparrow
Search-R1 (GRPO)35.5 / 43.80.180.850.770.09
+ QDTS w/o optimization39.4 / 47.80.450.720.410.43
+ QDTS43.3 / 50.60.650.880.320.64

Class structure (Table R14). This control keeps both optimized stages and merges the classes into one, so every query receives the same optimized context. It improves accuracy over GRPO but leaves the depth slope and the selectivity far below those of QDTS, because a query-independent context cannot separate queries with opposite demands. The gain of QDTS is therefore not explained by optimization alone.

Table R14: Class-conditioning ablation (Qwen2.5-7B-Instruct, E5, average over four multi-hop benchmarks). Both rows use optimized CpreC_{\text{pre}} + CmidC_{\text{mid}}.

Class structureEM / F1m(H,H∗)↑m(H,H^*)\uparrowP(int∣g)↑P(\text{int}\mid g)\uparrowP(int∣¬g)↓P(\text{int}\mid\neg g)\downarrowS↑S\uparrow
Single class40.5 / 48.00.360.830.550.34
Mid-grained classes (QDTS)43.3 / 50.60.650.880.320.64
AuthorsOpenReview ↗

Response to Reviewer V1cz (2/2)

W2. The strongest baselines are partly copied from their original papers rather than reproduced under the same backbone, retriever, corpus, coreset, and budget. This weakens the stated 9.2% comparison.

For every baseline, Table 1 reports the more favorable of two numbers: one from its original paper under its own setting, and our rerun under exactly our setting (same backbone, retriever, corpus, coreset, and budget). Table R15 lists both, and QDTS outperforms every baseline in both columns.

The 9.2% figure is a matched comparison. It compares QDTS with Search-R1 (GRPO + MGS), the strongest baseline in the submission, which we trained ourselves under the identical setting. Over three training seeds the gap on the six-benchmark average is 44.3 vs. 40.2 EM, which is now 10.2%. With the three-seed means, AutoSearch (40.3 EM) becomes the strongest baseline in Table 1 by a small margin. Its numbers are taken from its original paper, which is its more favorable setting (Table R15). The revision therefore reports the gain over AutoSearch, which is 9.9% (44.3 vs. 40.3 EM), and the comparison holds in both settings.

Table R15: Baselines under their reported setting and under our identical rerun (average EM / F1 over four multi-hop benchmarks). The Search-R1 paper reports EM only. The more favorable number of each baseline, which is the one used in Table 1, is underlined.

MethodReported Avg. EM / F1Our rerun Avg. EM / F1
Search-R1 (GRPO)32.4 / —3̲5̲.̲5̲ / 4̲3̲.̲8̲
StepSearch3̲4̲.̲5̲ / 4̲4̲.̲5̲32.1 / 41.1
ReasonRAG3̲2̲.̲7̲ / 4̲1̲.̲4̲31.0 / 39.6
AutoSearch3̲7̲.̲0̲ / 4̲6̲.̲0̲34.1 / 44.1
QDTS (ours)—43.3 / 50.6

W3. QDTS relies on hand-designed classes, surface-rule hop prediction, and fitted thresholds, yet evaluation remains within Wikipedia QA. Cross-dataset transfer without refitting is needed.

We believe that QDTS supports cross-dataset transfer without refitting for the following two reasons.

(1) Bamboogle is an out-of-distribution test. It is hand-written rather than crowd-sourced [1] and contains no question from any of our training sources. The parser rules, the class contexts, and the thresholds are applied to it unchanged, and QDTS keeps its gain over GRPO there (36.5 → 43.4 EM).

(2) The hand-designed parts are deliberately small. Six patterns, three depth buckets, and two retrieval states describe the structure of search rather than the content of a corpus.


Overall, we have incorporated these responses and additional results into the revised paper. Please refer to the uploaded PDF, where the changes are highlighted in red. We hope that most of your concerns have been addressed and would be glad to engage in further discussion if needed.


References:

[1] "Measuring and Narrowing the Compositionality Gap in Language Models", In Findings of EMNLP, 2023

Thanks for the response and revision. I would like to maintian my positive score. Good luck.

Thank you for reading our response and the revision, and for maintaining your positive score. We are grateful for your careful review, and your comments were of great help in improving our work.

Reviewer_8AE7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how RL-trained search agents can adopt similar search behavior across queries with different needs. QDTS adds query-specific guidance before search and retrieval-dependent guidance during search. It improves answer accuracy and the authors' measures of search adaptivity. On six QA benchmarks, QDTS gets 44.1 average EM, compared with 40.4 for the best baseline.

Critical strengths
  • The paper evaluates search behavior directly, rather than relying only on answer accuracy.
  • QDTS improves results across multiple backbones and retrievers without adding inference-time LLM calls.
Critical weaknesses
  • Contribution of each component is not fully isolated. In Table 5, a single GEPA-optimized context already provides a sizeable improvement (36.0 to 40.5 EM), while class-specific contexts reach 42.9. So most of the gain comes from the optimized prompt, which the baselines don't get. I'm not convinced yet that the class-specific part is what matters. The paper should separate the effects of prompt optimization, class conditioning, and mid-rollout steering more clearly.
  • Evaluation details limit confidence. No variance across seeds is reported. Some baseline results come from previous papers rather than identical reruns, and the gains depend in part on a curated training coreset. Table 2's ReasonRAG selectivity value also appears inconsistent with the stated formula.
Questions for authors
  1. Can you compare the full method against a single GEPA-optimized context under the same training setup, and isolate the gain from mid-rollout steering?
  2. Can you report results over multiple seeds and clarify GEPA's data, model, and optimization cost?
  3. Please clarify the ReasonRAG value in Table 2 and how sensitive QDTS is to query-classification errors.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 8AE7 (1/4)

We sincerely appreciate the reviewer's insightful comments, which have significantly contributed to improving the quality of our paper.

Before addressing each point in detail, we clarify two facts that underlie the first weakness.

Context optimization is an integral part of QDTS, rather than a generic prompt improvement applied only to our method. QDTS steers the policy by injecting query- and retrieval-dependent contexts (CpreC_{\text{pre}}, CmidC_{\text{mid}}). Simply injecting these contexts does not guarantee effective steering; their content must be optimized to elicit the desired search behaviors. Accordingly, QDTS explicitly optimizes these contexts before RL (Equations 9 and 12), using GEPA as the optimization algorithm. Importantly, the base agent prompt remains unchanged: QDTS and the Search-R1 baselines share the same base agent prompt, which follows the Search-R1 and StepSearch multi-turn templates. Thus, the improvement comes from the proposed context-steering framework, including its context optimization, rather than modifications to the underlying agent prompt.

The "Single" rows of class-coarseness ablation (Table 5 in the submission, Table 7 in the revision) run our full pipeline (CpreC_{\text{pre}} + CmidC_{\text{mid}}) with only the class set merged to one class. They are variants of QDTS with one class for every query, not a prompt added on top of a baseline.

Below, we first separate the three effects as the reviewer requests (W1-1), and then answer the remaining points (W1-2, W1-3, and W2-1 to W2-3). Unless noted otherwise, the EM and F1 of Search-R1 (GRPO) and QDTS are means over three training seeds (Table R6); the ablation variants and the behavioral metrics are from single runs.

W1-1 / Q1 (second part). Separate the effects of prompt optimization, class conditioning, and mid-rollout steering more clearly; isolate the gain from mid-rollout steering.

We now ablate one factor at a time against the full method, and we have organized §4.3 of the revision accordingly.

Context optimization (Table R1). With both stages fixed, optimizing the contexts improves accuracy, depth adaptivity, and selectivity together. The seed contexts make the policy search again even when the gold passage is already retrieved (P(int∣g)P(\text{int}\mid g) falls below that of GRPO), and optimization removes this side effect.

Table R1: Context-optimization ablation (Qwen2.5-7B-Instruct, E5, average over four multi-hop benchmarks). Both QDTS rows use CpreC_{\text{pre}} + CmidC_{\text{mid}}.

SettingEM / F1m(H,H∗)↑m(H,H^*)\uparrowP(int∣g)↑P(\text{int}\mid g)\uparrowP(int∣¬g)↓P(\text{int}\mid\neg g)\downarrowS↑S\uparrow
Search-R1 (GRPO)35.5 / 43.80.180.850.770.09
+ QDTS w/o optimization39.4 / 47.80.450.720.410.43
+ QDTS43.3 / 50.60.650.880.320.64

Class conditioning (Table R2). With both optimized stages fixed, the mid-grained classes of QDTS are the best on every benchmark. A single class for all queries cannot separate queries with opposite demands, and overly fine classes leave too few queries per class.

Table R2: Class-coarseness ablation (Qwen2.5-7B-Instruct, E5; EM / F1 on the four multi-hop benchmarks). All rows use optimized CpreC_{\text{pre}} + CmidC_{\text{mid}}.

Class structureHotpotQA2WikiMuSiQueBamboogleAvg (EM / F1)
Single (1 class)45.3 / 52.248.7 / 54.527.0 / 31.340.8 / 54.140.5 / 48.0
Fine-grained42.9 / 46.146.8 / 50.122.2 / 26.736.8 / 48.637.2 / 42.9
Mid-grained (QDTS)50.3 / 54.752.2 / 57.027.1 / 33.143.4 / 57.543.3 / 50.6

Mid-rollout steering (Table R3). With optimized contexts, each stage alone improves accuracy, but they act on different quantities. CmidC_{\text{mid}} is what restores selectivity, which remains low under CpreC_{\text{pre}} alone, and depth adaptivity needs both stages. The gain from mid-rollout steering is isolated by comparing "+ CpreC_{\text{pre}}" with "+ CpreC_{\text{pre}} + CmidC_{\text{mid}}", which share an identical pre-rollout context.

Table R3: Component ablation (Qwen2.5-7B-Instruct, E5, average over four multi-hop benchmarks). All contexts are optimized.

SettingEM / F1m(H,H∗)↑m(H,H^*)\uparrowP(int∣g)↑P(\text{int}\mid g)\uparrowP(int∣¬g)↓P(\text{int}\mid\neg g)\downarrowS↑S\uparrow
Search-R1 (GRPO)35.5 / 43.80.180.850.770.09
+ CpreC_{\text{pre}}40.5 / 50.00.430.860.670.22
+ CmidC_{\text{mid}}40.1 / 48.90.390.720.280.61
+ CpreC_{\text{pre}} + CmidC_{\text{mid}} (QDTS)43.3 / 50.60.650.880.320.64
AuthorsOpenReview ↗

Response to Reviewer 8AE7 (2/4)

W1-2. A single GEPA-optimized context already provides a sizeable improvement (36.0 → 40.5 EM). Not convinced that the class-specific part is what matters.

The "Single + GEPA" row is QDTS with one class, so its improvement is a gain of our steering. The class-specific part is what a unified context cannot provide, in two respects.

(1) Accuracy. Class-specific contexts are better than the single context on every benchmark (Table R2).

(2) Cost. A single context must carry the guidance for every search pattern and depth in one string, so every rollout pays for all of it at every turn while only one part applies. It is more than twice as long as a class context and slower at both inference and training (Table R4).

Table R4: Cost of a single query-independent context versus mid-grained class contexts ("Ctx tokens": length of CpreC_{\text{pre}}; latency per question on one H100; training time for 150 GRPO steps on 4×H100).

Context settingCtx tokensLatency (s)Train time (h)
Single (1 class)28723.224.5
Mid-grained (18 classes, QDTS)12618.921.5

W1-3 / Q1 (first part). Most of the gain comes from the optimized prompt, which the baselines don't get; compare the full method against a single GEPA-optimized context under the same training setup.

The baselines do not "miss" the optimized prompt; they have no steering context to optimize. To test the reviewer's hypothesis directly, we optimized the base agent prompt itself, with the same optimizer, data, and budget as Equation 9 and no steering context, and trained it with GRPO (Table R5). This "baseline with an optimized prompt", which the reviewer asks for, gains small accuracy and its depth slope stays at the level of GRPO. Prompt optimization alone therefore does not explain the gain; conditioning the context on the query and on the retrieval does.

Table R5: Prompt optimization without steering versus QDTS, under the same training setup (Qwen2.5-7B-Instruct, E5, average over four multi-hop benchmarks).

MethodWhat is optimizedEM / F1m(H,H∗)↑m(H,H^*)\uparrowUnder-search ↓\downarrowS↑S\uparrow
Search-R1 (GRPO)nothing (base agent prompt)35.5 / 43.80.180.280.09
Search-R1 (GRPO + optimized base prompt)agent prompt, no steering context38.1 / 46.80.220.310.35
QDTSCpreC_{\text{pre}} + CmidC_{\text{mid}}43.3 / 50.60.650.070.64
AuthorsOpenReview ↗

Response to Reviewer 8AE7 (3/4)

W2-1 / Q2. No variance across seeds is reported; report results over multiple seeds and clarify GEPA's data, model, and optimization cost.

Seeds. We retrained Search-R1 (GRPO), its strongest variant (GRPO + MGS), and QDTS with three seeds each (Table R6). The gain of QDTS over both exceeds the seed variation of every method. For the final revision, we will run all experiments with three seeds.

Table R6: Answer accuracy in EM / F1 (mean ± standard deviation over three training seeds; Qwen2.5-7B-Instruct, E5).

MethodNQPopQAHotpotQA2WikiMuSiQueBamboogleAvg.
Search-R1 (GRPO)42.2 ± 2.1 / 51.6 ± 3.542.0 ± 1.2 / 46.9 ± 1.738.7 ± 0.5 / 47.3 ± 0.844.8 ± 1.1 / 51.3 ± 1.321.8 ± 0.7 / 26.9 ± 0.536.5 ± 1.0 / 49.5 ± 1.737.7 ± 1.0 / 45.6 ± 1.5
Search-R1 (GRPO + MGS)44.3 ± 0.8 / 49.5 ± 1.442.1 ± 1.4 / 48.9 ± 2.646.5 ± 0.2 / 53.0 ± 0.346.6 ± 1.0 / 53.8 ± 1.323.3 ± 0.4 / 30.1 ± 0.838.6 ± 0.8 / 55.7 ± 1.340.2 ± 0.7 / 48.5 ± 1.1
Search-R1 (GRPO + QDTS)46.2 ± 1.0 / 52.4 ± 1.246.5 ± 1.3 / 50.8 ± 1.650.3 ± 0.5 / 54.7 ± 0.852.2 ± 0.9 / 57.0 ± 1.127.1 ± 0.9 / 33.1 ± 1.443.4 ± 1.2 / 57.5 ± 1.744.3 ± 0.5 / 50.9 ± 0.9

Context optimization. The optimization uses questions sampled from the RL training set itself, so it needs no additional data and no test question. Candidates are scored with the base policy θ0\theta_0 (Qwen2.5-7B-Instruct), and the same θ0\theta_0 serves as the reflection model, so no external or stronger model is involved at any point. It runs before RL and takes only 38.8 minutes on 4×H100, which is 2.9% of the total training cost (Table R7). The resulting strings are frozen and reused by every seed, backbone, and retriever run.

Table R7: Cost of context optimization and RL training (4×H100).

StageWall-clockRollouts / reflectionsShare of total
CpreC_{\text{pre}} optimization (Eq. 9)17.7 min849 / 571.3%
CmidC_{\text{mid}} optimization (Eq. 12)21.1 min882 / 641.6%
RL training (150 steps)21.5 h19,200 / —97.1%
Total22.1 h20,931 / 121100%

W2-2. Some baselines come from previous papers rather than identical reruns, and the gains depend in part on a curated coreset.

Baselines. For every baseline, Table 1 reports the more favorable of two numbers: the one from its original paper under its own setting, and our rerun under the identical backbone, retriever, corpus, coreset, and budget as QDTS. Table R8 lists both. The comparison in the paper is therefore against the strongest available version of each baseline, and QDTS outperforms every baseline in both columns.

Table R8: Baselines in their own reported setting and under our identical rerun (average EM / F1 over four multi-hop benchmarks). The Search-R1 paper reports EM only. The more favorable number of each baseline, which is the one used in Table 1, is underlined.

MethodReported EM / F1Our rerun EM / F1 (Qwen2.5-7B-Instruct, E5, wiki-18, same coreset and budget)
Search-R1 (GRPO)32.4 / —3̲5̲.̲5̲ / 4̲3̲.̲8̲
StepSearch3̲4̲.̲5̲ / 4̲4̲.̲5̲32.1 / 41.1
ReasonRAG3̲2̲.̲7̲ / 4̲1̲.̲4̲31.0 / 39.6
AutoSearch3̲7̲.̲0̲ / 4̲6̲.̲0̲34.1 / 44.1
QDTS (ours)—43.3 / 50.6

Coreset. The coreset is designed to cover diverse search demands, rather than to favor QDTS specifically. Deeper questions are particularly challenging for RL-based search agents, as discussed in our analysis of successful-trajectory probability mass in Equation 4. Importantly, all methods are trained on the same coreset in our controlled comparisons. Without QDTS, GRPO achieves nearly identical EM with random sampling and diversity-based coreset sampling (Table 20), suggesting that increasing training diversity alone is insufficient to improve its performance in this setting. In contrast, QDTS benefits substantially from diversity-based sampling, consistent with its goal of helping the policy learn from heterogeneous search demands. Crucially, QDTS improves over GRPO under both sampling strategies: by 2.6 EM points with random sampling (38.8 vs. 36.2) and by 6.9 points with diversity-based sampling (42.9 vs. 36.0).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response Summary to All Reviewers

Dear Reviewers,

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are grateful for acknowledging the direct evaluation of search behavior beyond answer accuracy (8AE7, V1cz, rHKq), the clear formalization of the two homogenization modes and their metrics (V1cz, rHKq), the lightweight design that adds no inference-time LLM calls and changes neither the reward nor the RL rollout budget (8AE7, rHKq), and the extensive evaluation across benchmarks, backbones, and retrievers (8AE7, rHKq).

Following the reviewers' comments, we have made our best effort to address the concerns raised.

  • Component attribution: We clarify that context optimization is part of our method rather than an ingredient that baselines lack, and then separate the effects of steering-context optimization, class conditioning, and mid-rollout steering, each with its own one-factor ablation (8AE7, V1cz).
  • Evaluation reliability: We report three training seeds for the main comparison, list every baseline both in its own reported setting and in an identical rerun, and correct three selectivity entries of Table 2 (8AE7, V1cz).
  • Robustness and scope: We report the sensitivity to parser errors and a quantified analysis of mis-triggered and overridden contexts, and we clarify the out-of-distribution evaluation and the scope of the method (8AE7, V1cz, rHKq).
  • Theory and cost: We show that the bound of Equation 4 holds without the independence assumption, estimate the per-hop success probability empirically, and report the cost of context optimization next to the training cost (8AE7, rHKq).

We hope that our rebuttal successfully addresses these concerns and contributes to a favorable reassessment of our work.

Best regards, Authors

AuthorsOpenReview ↗

Author Final Remarks

Dear ACs, SACs, and PCs:

We summarize the outcome of the rebuttal for each reviewer. The initial ratings were 3, 3, and 2.

Reviewer rHKq (3): Reconfirmed acceptance and raised his/her confidence. The reviewer stated that the relaxed independence assumption, the empirical per-hop success rates, and the quantified cost of context optimization resolve most of his/her concerns.

Reviewer V1cz (3): Reconfirmed acceptance after reading our response and revision. We answered his/her concerns on component attribution, baseline comparison, and cross-dataset transfer with new one-factor ablations, identical reruns of all baselines, and three training seeds.

Reviewer 8AE7 (2): Has not yet responded to our rebuttal. His/her main concerns were the attribution of the gain to prompt optimization and the reliability of the evaluation. We answered both with new experiments, namely one-factor ablations, a control that optimizes the agent prompt without steering, three training seeds, identical reruns of all baselines, and a sensitivity analysis of parser errors. These concerns overlap with those of Reviewer V1cz, who reconfirmed acceptance after our response.

Conclusion: Two reviewers reconfirmed acceptance after the rebuttal, one with raised confidence, and we believe that the remaining concerns of Reviewer 8AE7 are fully addressed in our response and revision. We hope for a positive final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.