acceptodds
Under review as a conference paper at ICLR 2027

Document Optimization for Black-Box Retrieval via Reinforcement Learning

Abstract

Generative large language models (LLMs) are increasingly used as inference-time components in retrieval pipelines, for tasks such as query rewriting and document reranking. However, these online approaches place costly autoregressive computation directly on the latency-critical retrieval path. We explore an alternative axis: using LLMs to improve documents instead, rewriting them into better representations and shifting computation offline. Yet producing a useful document rewrite is not straightforward: retrieval is inherently discriminative, so an effective rewrite must make a document more similar to relevant queries than competing candidates under the retriever’s notion of similarity. We therefore formulate document transformation as an optimization problem, directly training an LLM or VLM to produce rewrites that improve retrieval. Our approach, DocOpt, uses GRPO with retriever ranking improvements as rewards, requires only black-box access to retrieval ranks, and applies across single-vector, multi-vector, and lexical retrievers. We evaluate zero-shot LLM rewriting and DocOpt on code and visual retrieval tasks, finding that document rewriting can improve retrieval and that optimizing rewrites yields further gains. For example, OpenAI text-embedding-3-small achieves 58.35 nDCG@5 on average with direct retrieval; zero-shot rewriting improves this to 60.83 with GPT-5.4-mini, 63.75 with Claude Haiku 4.5, and 64.23 with Qwen3. DocOpt further improves performance to 67.94, surpassing the 6.5 more expensive text-embedding-3-large retriever at 66.15.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Embedding Models, RAG, Information Retrieval, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_z3i9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates improving retrieval performance by optimizing document representations rather than modifying queries or retriever parameters. The authors propose DocOpt, a framework that uses reinforcement learning to train LLMs or VLMs to rewrite documents into representations better suited to a target retriever. Specifically, the method introduces a counterfactual retrieval reward based on changes in nDCG@5, considering both positive and negative queries, and optimizes the rewriting policy using GRPO. Since document transformations are performed offline, the approach avoids additional LLM inference during online retrieval and supports black-box retrievers. Experiments on code retrieval and visual document retrieval demonstrate consistent improvements over direct retrieval and zero-shot rewriting across lexical, single-vector, and multi-vector retrievers. Additional experiments investigate retriever fine-tuning, cross-retriever transfer, and generalization to natural language retrieval.

Critical strengths
  1. The paper explores an interesting and practical direction for improving retrieval by shifting LLM-based optimization from the query side to the document side. Performing document rewriting offline is particularly appealing for black-box retrieval systems where modifying the retriever is not possible.

  2. The proposed counterfactual retrieval reward is intuitive and directly aligned with the retrieval objective. Considering both positive and negative queries allows the model to optimize document representations relative to competing candidates rather than simply maximizing query-document similarity.

  3. The experiments cover diverse retriever architectures and retrieval tasks, including code and visual document retrieval. The consistent improvements over zero-shot rewriting are promising, and the comparisons with retriever fine-tuning further demonstrate the potential of document-side optimization.

Critical weaknesses
  1. My main concern is about the faithfulness of the rewritten documents. The RL reward focuses on retrieval rankings without explicitly ensuring that the transformations preserve the original information. This is particularly important for code retrieval, where rewriting may alter program functionality, and visual document retrieval, where generated descriptions may introduce incorrect information. A systematic evaluation of this would make the method more convincing eg 通过semantic preservation

  2. The baseline comparisons could be more comprehensive. Existing document expansion approaches, such as doc2query, as well as closely related reinforcement learning methods for document rewriting, are not directly evaluated. Including these baselines would help clarify the benefits of the proposed ranking-based optimization over existing document-side adaptation techniques.

  3. The requirement to rewrite the entire document corpus may introduce substantial computational overhead, especially for large-scale or frequently updated collections. Moreover, the rewritten documents are considerably longer than the originals (346 vs. 183 tokens on average), potentially increasing embedding computation, indexing time, and storage costs. Although rewriting is performed offline, these additional costs are not fully accounted for in the efficiency analysis, raising concerns about the scalability and practicality of the proposed approach.

Questions for authors

see weakness

5 comments

We thank the reviewer for their time and positive assessment of the paper’s strengths. We argue that the first and main concern (regarding the faithfulness of rewritten documents) is misplaced and explain why below. We address the concerns about baselines and computational costs with additional results and analysis.

My main concern is about the faithfulness of the rewritten documents.

The intention behind document rewriting is to replace only the document’s representation in the index, while keeping the original document for downstream use, as in prior work on document expansion (Nogueira et al., 2019). The rewrite does not need to preserve factual correctness, since the original document is used for processing. We will make this distinction clearer in the revised manuscript.

The baseline comparisons could be more comprehensive.

We added comparisons to two non-RL baselines, while matching the training data between those.

  • Supervised doc2query. Following prior work (Nogueira et al., 2019), we train the LM to generate relevant queries for the documents and append those to the documents. We experiment with 5, 10 and 15 queries variants.
  • Query-conditioned rewriting. We evaluate two ways of incorporating training queries into document rewriting. (a) Prompt-conditioned expansion: we select five training queries per dataset and include the same examples in the rewriting prompt for every document, without fine-tuning. (b) Query-conditioned Rewrite SFT: a teacher LM receives each training document and its relevant training queries and generates a target rewrite. We then fine-tune Qwen on these documents–rewrite pairs using supervised learning. The Qwen LM receives only the document when generating rewrites at inference time.

The results can be viewed here - https://bashify.io/i/GMgz8U

As shown in the table, DocOpt achieves the best aggregate performance across all six retrievers. Doc2Query performs slightly better on visual retrieval on two retrievers. We suspect that putting training queries directly into the prompt/document can steer the LM/retriever too strongly toward those particular queries at the expense of other potential queries (a form of overfitting). DocOpt instead uses queries only to compute the retrieval reward, while the LM/retriever sees only the document. The query contributes a scalar reward rather than exposing its full text and this may reduce overfitting to it.

Regarding alternative RL-based optimization methods, we evaluated three modifications inspired by Liu et al.: (i) retaining the original document alongside the rewrite, (ii) centering advantages without standard-deviation normalization, and (iii) removing KL regularization. We use the same configuration as the other ablations in the paper. We find (https://bashify.io/i/HfNEPD) that (i) and (iii) decrease performance, and (ii) has a advantage over the default configuration, though not a large one. As mentioned in the related works section (lines 456-461), we believe the difference between our findings and Liu et al's mainly reflect differences in evaluation and data, and not optimization methodology.

The requirement to rewrite the entire document corpus may introduce substantial computational overhead

We expanded our end-to-end offline cost analysis to cover 2K, 10K, and 100K documents, across short code documents (182 input tokens on average), long code documents (1,715 input tokens on average), and visual inputs. We measure GPU hours for DocOpt across these nine configurations and extrapolate API costs from the 2K samples to 10K and 100K documents (measuring them directly at larger scales is prohibitively expensive). We also break down the costs across the different pipeline components. For these measurements, we run DocOpt for 200 steps with one refresh round. Each refresh round costs approximately as much as rewriting the corpus once, so the number of refresh rounds should balance data staleness against available compute resources.

This plot (https://bashify.io/i/qkDDCt) shows the comparison across data types (also shown here in a tabular format - https://bashify.io/i/Wak3hz). It's clear that DocOpt offers substantial savings in both time and cost from using API models on larger scales.

This plot (https://bashify.io/i/BTVfnd) shows the breakdown of the pipeline into parts across scales. As can be seen, RL training costs are independent from index size, and rewrite and refresh time grow with the corpus size. We show these costs remain manageable and much cheaper and faster than API alternatives.

This table (https://bashify.io/i/tvVgyO) reports embedding time and storage for DocOpt versus direct retrieval. At the evaluated size, the additional embedding time is small in absolute terms. Single-vector index storage remains unchanged, while multi-vector storage varies with rewritten document length. For longer code documents, rewriting produces shorter representations, reducing embedding time and text storage, as well as multi-vector index storage. ANN index construction and synthetic query curation are not measured in our setup, and we assume differences to be negligible between options.

Dear Reviewer,

We thank you again for your time and effort. We have addressed the issues you raised in your review in our reply. Please let us know if you have any remaining questions before the rebuttal period ends.

Reviewer_QctvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes DocOpt to improve retrieval through offline document rewriting, avoiding query-time LLM costs. It trains LLM/VLM rewriters using GRPO with counterfactual nDCG rewards from black-box retriever rankings. Experiments on code, visual-document, and BRIGHT retrieval show gains over direct retrieval and zero-shot rewriting.

Critical strengths
  1. Optimizing documents offline addresses the serving cost of LLM-assisted retrieval with query rewriting, meanwhile accommodating retrievers whose weights are inaccessible. This paradigm may find its value in real retrieval applications.
Critical weaknesses
  1. Although discussed as existing work in the paper, the experiments omit query-only rewriting (a missing citation: [1]), joint query–document optimization, and established document-expansion approaches beyond generic zero-shot rewriting. Consequently, they establish improvements over the selected baselines but do not adequately determine DocOpt’s position relative to prior approaches, in terms of both retrieval performance and computational costs.

[1] "Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning." arXiv preprint arXiv:2503.00223 (2025).

  1. The method primarily combines standard GRPO with a retrieval-metric reward, while the paper itself acknowledges prior RL-based document rewriting. Although the formulation is sensible, the distinctive methodological advance remains modest.
Questions for authors

Following Weakness 1, including experimental comparisons over query-only rewriting, query-document co-augmentation, and document-expansion approaches would strengthen the positioning of the proposed approach.

4 comments

We thank the reviewer for their time and appreciation of our work. We disagree that query-rewriting baselines and lack of RL methodological innovation are critical weaknesses of our work and explain why below. We agree with the point about comparison to other document-side baselines and attempt to satisfy those concerns with additional experiments.

Although discussed as existing work in the paper, the experiments omit query-only rewriting (a missing citation: [1]), joint query–document optimization, and established document-expansion approaches beyond generic zero-shot rewriting.

We disagree that query-only optimization and joint query-document optimization baselines are required to establish the value of our work. The motivation of our paper (as stated in the abstract and introduction) is to improve retrieval by optimizing documents offline, without adding any generative inference-time component. Query rewriting introduces exactly the cost we aim to avoid. Beyond their irrelevance, methods that use online generation extend beyond query rewriting, including LLM-based PRF and reranking. Properly exploring the design space between offline and online modifications is a substantial undertaking, well beyond the scope of our work. We will clarify this in the related work section and add the citation the reviewer mentioned.

We agree that comparisons to other document-side approaches are useful and important. We added comparisons to two non-RL baselines, while matching the training data between those.

  • Supervised doc2query. Following prior work (Nogueira et al., 2019), we train the LM to generate relevant queries for the documents and append those to the documents. We experiment with 5, 10 and 15 queries variants.
  • Query-conditioned rewriting. We evaluate two ways of incorporating training queries into document rewriting. (a) Prompt-conditioned expansion: we select five training queries per dataset and include the same examples in the rewriting prompt for every document, without fine-tuning. (b) Query-conditioned Rewrite SFT: a teacher LM receives each training document and its relevant training queries and generates a target rewrite. We then fine-tune Qwen on these documents–rewrite pairs using supervised learning. The Qwen LM receives only the document when generating rewrites at inference time.

The results can be viewed here - https://bashify.io/i/GMgz8U

As shown in the table, DocOpt achieves the best aggregate performance across all six retrievers. Doc2Query performs slightly better on visual retrieval on two retrievers. We suspect that putting training queries directly into the prompt/document can steer the LM/retriever respectively too strongly toward those particular queries at the expense of other potential queries (a form of overfitting). DocOpt instead uses queries only to compute the retrieval reward, while the LM/retriever sees only the document. The query contributes a scalar reward rather than exposing its full text and this may reduce overfitting to it.

The method primarily combines standard GRPO with a retrieval-metric reward, while the paper itself acknowledges prior RL-based document rewriting. Although the formulation is sensible, the distinctive methodological advance remains modest.

We do not claim to introduce a novel RL algorithm. Both our method and Liu et al.’s are largely direct applications of GRPO to retrieval, and we disagree that this constitutes a critical weakness. The contribution of our work is in supplying the first broad empirical evidence that document-only adaptation can improve retrieval performance while preserving a fast retrieval pipeline with no query-time generation. We believe this can lead to adoption of document re-writing in practical retrieval systems. The use of an optimization algorithm that is already well established in other domains does not diminish that contribution.

Dear Reviewer,

We thank you again for your time and effort. We have addressed the issues you raised in your review in our reply. Please let us know if you have any remaining questions before the rebuttal period ends.

Thanks for the detailed discussions. I appreciate the new results for baseline comparison. On the other hand, I still think that comparison against query-side augmentation approaches are useful, since this would bring useful insights for understanding the difference between paradigms. Furthermore, my concerns on technical novelty remain. Therefore, my evaluation remains unchanged.

We sincerely thank the reviewer for their time and effort. We respectfully disagree that these remaining concerns provide sufficient grounds for rejection.

Our work targets offline document adaptation without query-time generation. The requested query-side methods introduce precisely the online computation we aim to avoid. The relevant comparisons within this setting are with document-side methods, including the additional baselines we evaluated. A broader comparison of online and offline approaches would require studying retrieval quality, latency, and serving cost across query rewriting, reranking, and other inference-time methods. This is a substantial research question beyond our scope.

Regarding novelty, our contribution is to formulate and systematically evaluate document optimization for fixed retrievers, providing broad empirical evidence across architectures and tasks. We believe these findings constitute a substantive contribution, and that introducing a new RL algorithm is not necessary to establish their value.

We further explain these points in our main comment.

The Authors

Reviewer_d4LvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DocOpt, a reinforcement learning framework that optimizes document representations for black-box retrieval. The method trains an LLM or VLM rewriter using GRPO with retrieval ranking improvements as rewards. Experiments demonstrate improvements over direct retrieval and zero-shot rewriting across multiple retrievers and datasets.

Critical strengths

S1. Practical document-side optimization for black-box retrieval.

The proposed approach improves retrieval without modifying retriever parameters or introducing additional query-time inference costs. Its applicability to different retrieval architectures is a practical advantage.

S2. Improvements across multiple retrievers.

The experiments show consistent improvements over zero-shot document rewriting across lexical, single-vector, and multi-vector retrievers, demonstrating the potential of retrieval-oriented document optimization.

Critical weaknesses

W1. Insufficient baselines to justify RL-based optimization.

The paper lacks comparisons with simpler query-conditioned document expansion methods, where relevant training queries are incorporated into documents using an LLM. Since DocOpt also relies on query supervision, it remains unclear whether reinforcement learning is necessary to achieve the reported improvements.

W2. Limited evaluation of reasoning-intensive retrieval and factual reliability.

Although BRIGHT is evaluated, the improvements are modest, particularly for OpenAI text-embedding-3-small (0.1647 → 0.1670). More importantly, the paper does not evaluate whether rewritten documents preserve factual information or introduce hallucinations. Optimizing retrieval rankings alone does not guarantee better document representations, especially for reasoning-intensive or knowledge-sensitive tasks.

W3. Incomplete end-to-end cost analysis.

Although training costs are reported, the analysis relies on a small training setup and extrapolates costs to larger corpora. Query generation and index construction costs are excluded, and the scalability of training requirements is not validated. This weakens the claimed computational efficiency advantages.

Questions for authors

Q1. Is reinforcement learning necessary for the reported improvements?

Could the authors compare DocOpt against query-conditioned document expansion using the same query supervision and computational budget?

Q2. Does DocOpt preserve factual correctness in rewritten documents?

Could the authors evaluate factual consistency, hallucination rates, and downstream QA accuracy, particularly on reasoning-intensive retrieval tasks?

Q3. How does the actual end-to-end cost scale?

Could the authors include query preparation and indexing costs and investigate whether the same training budget remains effective for larger and more diverse corpora?

2 comments

We thank the reviewer for their time and appreciation of our work! We address W1 and W3 below. We provide additional results and analysis in an attempt to satisfy the reviewer concerns. Regarding W2, we disagree that it reflects a critical weakness, and clarify why below.

Q1. Is reinforcement learning necessary for the reported improvements?

We thank the reviewer for raising this point. We added comparisons to two non-RL baselines, while matching the training data between those.

  • Supervised doc2query. Following prior work (Nogueira et al., 2019), we train the LM to generate relevant queries for the documents and append those to the documents. We experiment with 5, 10 and 15 queries variants.
  • Query-conditioned rewriting. We evaluate two ways of incorporating training queries into document rewriting. (a) Prompt-conditioned expansion: we select five training queries per dataset and include the same examples in the rewriting prompt for every document, without fine-tuning. (b) Query-conditioned Rewrite SFT: a teacher LM receives each training document and its relevant training queries and generates a target rewrite. We then fine-tune Qwen on these documents–rewrite pairs using supervised learning. The Qwen LM receives only the document when generating rewrites at inference time.

The results can be viewed here - https://bashify.io/i/GMgz8U

As shown in the table, DocOpt achieves the best aggregate performance across all six retrievers. Doc2Query performs slightly better on visual retrieval on two retrievers. We suspect that putting training queries directly into the prompt/document can steer the LM/retriever respectively too strongly toward those particular queries at the expense of other potential queries (a form of overfitting). DocOpt instead uses queries only to compute the retrieval reward, while the LM/retriever sees only the document. The query contributes a scalar reward rather than exposing its full text and this may reduce overfitting to it.

Q2. Does DocOpt preserve factual correctness in rewritten documents?

The intention behind document rewriting is to replace only the document’s representation in the index, while keeping the original document for downstream use, as in prior work on document expansion (Nogueira et al., 2019). The rewrite should therefore improve retrieval for relevant queries and reduce spurious matches for irrelevant ones. Our training reward directly optimizes this objective. The instruction in the code-rewriting prompt encourages the policy to keep the representation grounded in the original document, but is not meant to be enforced. The rewritten representation serves as a retrieval surrogate, while the original document remains the source of information or executable code used downstream. We will make this distinction clearer in the revised manuscript.

Q3. How does the actual end-to-end cost scale?

We expanded our end-to-end offline cost analysis to cover 2K, 10K, and 100K documents, across short code documents (182 input tokens on average), long code documents (1,715 input tokens on average), and visual inputs. We measure GPU hours for DocOpt across these nine configurations and extrapolate API costs from the 2K samples to 10K and 100K documents (measuring them directly at larger scales is prohibitively expensive). We also break down the costs across the different pipeline components. For these measurements, we run DocOpt for 200 steps with one refresh round. Each refresh round costs approximately as much as rewriting the corpus once, so the number of refresh rounds should balance data staleness against available compute resources.

This plot (https://bashify.io/i/qkDDCt) shows the comparison across data types (also shown here in a tabular format - https://bashify.io/i/Wak3hz). It's clear that DocOpt offers substantial savings in both time and cost from using API models on larger scales.

This plot (https://bashify.io/i/BTVfnd) shows the breakdown of the pipeline into parts across scales. As can be seen, RL training costs are independent from index size, and rewrite and refresh time grow with the corpus size. We show these costs remain manageable and much cheaper and faster than API alternatives.

This table (https://bashify.io/i/tvVgyO) reports embedding time and storage for DocOpt versus direct retrieval. At the evaluated size, the additional embedding time is small in absolute terms. Single-vector index storage remains unchanged, while multi-vector storage varies with rewritten document length. For longer code documents, rewriting produces shorter representations, reducing embedding time and text storage, as well as multi-vector index storage. ANN index construction and synthetic query curation are not measured in our setup, and we assume differences to be negligible between options.

Dear Reviewer,

We thank you again for your time and effort. We have addressed the issues you raised in your review in our reply. Please let us know if you have any remaining questions before the rebuttal period ends.

Reviewer_ZYJ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DocOpt, which trains an LLM or VLM to rewrite documents into representations that work better with a fixed retriever. For each candidate rewrite, the reward measures the change in nDCG when that document is replaced in a temporarily fixed index, using both relevant queries and mined negative queries. The rewriting policy is trained with GRPO, and the document index is refreshed periodically. At deployment, the policy rewrites the corpus offline, so queries do not require an additional generative model call. The experiments cover code retrieval, visual-document retrieval through text surrogates, and seven BRIGHT domains, using lexical, single-vector, multi-vector, and commercial embedding retrievers. The authors also study joint retriever adaptation, cross-retriever transfer, synthetic-query supervision, and preparation costs. DocOpt generally improves on the corresponding zero-shot rewriter, although gains vary across retrievers and domains, and cross-family transfer is less consistent.

Critical strengths

1、Optimizing documents for a fixed retriever is a useful way to improve retrieval when retriever weights cannot be changed. The method moves generation offline and uses ranking feedback rather than gradients through the retriever, which makes the same approach applicable to BM25, commercial embedding models, and open-weight neural retrievers. Including negative queries is also well motivated: a document should become easier to retrieve for relevant requests without becoming an attractive false match for unrelated ones. This gives the method a clear practical use case and a training objective that is more closely connected to retrieval than a generic request to produce a richer description.

2、The experiments show that the improvement is not limited to one retrieval architecture. For text-embedding-3-small, DocOpt raises the reported mean nDCG@5 from 64.23 with the same zero-shot Qwen rewriter to 67.94. There are also gains with BM25 and Jina-ColBERT-V2, and the joint-adaptation experiment shows that document rewriting can complement retriever fine-tuning in the multi-vector setting. The transfer and synthetic-query results are useful qualifications rather than uniformly positive results: they show that the policy is partly retriever-specific and that labeled queries remain valuable. Together, these experiments make a reasonable case that learning the document transformation is worth studying.

Critical weaknesses

1、The comparisons do not yet establish the added value over the closest learned document-side methods. Most rewriting baselines in Table 1 are zero-shot LLMs, while DocOpt receives retrieval supervision and repeated optimization. Retriever fine-tuning is a useful control, but it does not replace a comparison with a trained document rewriter. Section 6 already identifies Liu et al.'s query-document co-augmentation work (2025, arXiv:2506.18670) as a close RL-based approach, yet its document-only setting is not evaluated on the same tasks. The explanation that its earlier result came from a different dataset does not tell us whether DocOpt's reward and training procedure are better under a matched setting. A matched document-only RL baseline, and a simpler supervised rewriter trained on selected high-reward outputs, would help determine whether the contribution is the particular optimization method or mainly the benefit of adding retrieval supervision.

2、The empirical conclusions need stronger evidence of stability, particularly where the gains are small. Appendix A.5 uses one random 20% query split and trains a separate policy for each dataset and retriever. Some settings are very small: HumanEval has 31 training and 127 test queries, while ESG-H has 10 training and 42 test queries. The paper does not report uncertainty or variation across policy-training seeds and query splits for the main results. This matters for claims of being competitive with fine-tuning and for the larger-scale extension: on BRIGHT, the OpenAI-small average moves only from 0.1647 to 0.1670 and improves on three of seven domains. The broad pattern is encouraging, but the current results do not show which smaller improvements are dependable. The reported low overlap of supervised positive documents is helpful, but it does not answer this stability question.

Questions for authors

、How does DocOpt compare with the document-only variant of Liu et al.'s RL co-augmentation approach under the same backbone, training queries, retriever, and optimization budget? Which part of the proposed reward or update procedure explains any difference?

2、Can you report variation across training seeds and query splits, together with paired uncertainty for the main comparisons? The small code and ESG-H settings, the comparisons with retriever fine-tuning, and the OpenAI-small BRIGHT result are the most important cases.

3、For a fixed document and frozen index, the baseline nDCG term in the counterfactual reward is constant across the group's candidate rewrites and therefore cancels in GRPO's centered advantages. Is the main benefit of the reward its positive/negative query construction rather than the subtraction itself? How sensitive are the gains to the index refresh interval?

4、Table 2 extrapolates rewriting time from 1,998 short code documents and reuses the same trained policy at one million documents. How does the cost change for long text or visual pages, and when training or refreshing the index must use a substantially larger collection? Please distinguish measured costs from extrapolated costs.

5、The rewriting prompt asks the policy to preserve code behavior and document facts, but the reward only measures retrieval. Do checks on optimized outputs show that the gains come from faithful representations? Are the original documents, rather than the rewritten surrogates, what the final retrieval system returns to users?

3 comments

We thank the reviewer for their appreciation of our work. We addressed comments below, and added additional analysis and results.

Weaknesses:

The comparisons do not yet establish the added value over the closest learned document-side methods.

We added comparisons to two non-RL baselines, while matching the training data between those.

  • Supervised doc2query. Following prior work (Nogueira et al., 2019), we train the LM to generate relevant queries for the documents and append those to the documents. We experiment with 5, 10 and 15 queries variants.
  • Query-conditioned rewriting. We evaluate two ways of incorporating training queries into document rewriting. (a) Prompt-conditioned expansion: we select five training queries per dataset and include the same examples in the rewriting prompt for every document, without fine-tuning. (b) Query-conditioned Rewrite SFT: a teacher LM receives each training document and its relevant training queries and generates a target rewrite. We then fine-tune Qwen on these documents–rewrite pairs using supervised learning. The Qwen LM receives only the document when generating rewrites at inference time.

The results can be viewed here - https://bashify.io/i/GMgz8U

As shown in the table, DocOpt achieves the best aggregate performance across all six retrievers. Doc2Query performs slightly better on visual retrieval on two retrievers. We suspect that putting training queries directly into the prompt/document can steer the LM/retriever respectively too strongly toward those particular queries at the expense of other potential queries (a form of overfitting). DocOpt instead uses queries only to compute the retrieval reward, while the LM/retriever sees only the document. The query contributes a scalar reward rather than exposing its full text and this may reduce overfitting to it.

The empirical conclusions need stronger evidence of stability, particularly where the gains are small.

We added 95% bootstrap confidence intervals to our main results table (https://bashify.io/i/V4ieUF). Across retrievers, DocOpt’s confidence intervals lie above the direct retrieval and zero-shot rewriting baseline mean scores for all retrievers except Qwen-Emb-4B. For the Qwen retrievers, DocOpt performs roughly as well as retriever fine-tuning, with each method’s interval containing the other method’s mean score. For Jina, the confidence interval for DocOpt–Joint Adaptation lies strictly above all other methods’ mean scores. Questions:

How does DocOpt compare with the document-only variant of Liu et al.'s RL co-augmentation approach under the same backbone, training queries, retriever, and optimization budget? Which part of the proposed reward or update procedure explains any difference?

We evaluated three modifications to the RL algorithm inspired by Liu et al.: (i) retaining the original document alongside the rewrite, (ii) centering advantages without standard-deviation normalization, and (iii) removing KL regularization. We use the same configuration as the other ablations in the paper. We find (https://bashify.io/i/HfNEPD) that (i) and (iii) decrease performance, and (ii) has an advantage over the default configuration, though not a large one.

As mentioned in the related works section (lines 456-461), we believe the difference between our findings and Liu et al's mainly reflect differences in evaluation and data, and not optimization methodology.

Can you report variation across training seeds and query splits

Re-running the experiments with enough training seeds to assess variance would require more compute budget than we have available. We acknowledge this limitation, which is unfortunately common in ML research under stringent compute.

The small code and ESG-H settings, the comparisons with retriever fine-tuning, and the OpenAI-small BRIGHT result are the most important cases.

This table includes the CIs for code and ESG-H: https://bashify.io/i/9CqWba. As shown, they are fairly wide given the small sample sizes. That said, we disagree with the reviewer about the CIs for these specific retriever/dataset pairs being the most important. The main results table shows that DocOpt's CIs are above the baseline means when aggregated. We have not argued in our work that DocOpt improves over baselines on every retriever/dataset pair, which is a much stronger claim.

For a fixed document and frozen index, the baseline nDCG term in the counterfactual reward is constant across the group's candidate rewrites and therefore cancels in GRPO's centered advantages. Is the main benefit of the reward its positive/negative query construction rather than the subtraction itself?

We retain the counterfactual formulation because it makes the reward directly interpretable as the change in retrieval quality caused by replacing a document. This makes it easier to monitor gains across different queries and documents during RL training. We found this useful when running experiments and understanding the effects of different modifications. We will clarify this point in the paper.

Table 2 extrapolates rewriting time from 1,998 short code documents and reuses the same trained policy at one million documents. How does the cost change for long text or visual pages, and when training or refreshing the index must use a substantially larger collection? Please distinguish measured costs from extrapolated costs.

We expanded our end-to-end offline cost analysis to cover 2K, 10K, and 100K documents, across short code documents (182 input tokens on average), long code documents (1,715 input tokens on average), and visual inputs. We measure GPU hours for DocOpt across these nine configurations and extrapolate API costs from the 2K samples to 10K and 100K documents (measuring them directly at larger scales is prohibitively expensive). We also break down the costs across the different pipeline components. For these measurements, we run DocOpt for 200 steps with one refresh round. Each refresh round costs approximately as much as rewriting the corpus once, so the number of refresh rounds should balance data staleness against available compute resources.

This plot (https://bashify.io/i/qkDDCt) shows the comparison across data types (also shown here in a tabular format - https://bashify.io/i/Wak3hz). It's clear that DocOpt offers substantial savings in both time and cost from using API models on larger scales.

This plot (https://bashify.io/i/BTVfnd) shows the breakdown of the pipeline into parts across scales. As can be seen, RL training costs are independent from index size, and rewrite and refresh time grow with the corpus size. We show these costs remain manageable and much cheaper and faster than API alternatives.

This table (https://bashify.io/i/tvVgyO) reports embedding time and storage for DocOpt versus direct retrieval. At the evaluated size, the additional embedding time is small in absolute terms. Single-vector index storage remains unchanged, while multi-vector storage varies with rewritten document length. For longer code documents, rewriting produces shorter representations, reducing embedding time and text storage, as well as multi-vector index storage. ANN index construction and synthetic query curation are not measured in our setup, and we assume differences to be negligible between options.

The rewriting prompt asks the policy to preserve code behavior and document facts, but the reward only measures retrieval. Do checks on optimized outputs show that the gains come from faithful representations? Are the original documents, rather than the rewritten surrogates, what the final retrieval system returns to users?

The intention behind document rewriting is to replace only the document’s representation in the index, while keeping the original document for downstream use, as in prior work on document expansion (Nogueira et al., 2019). The rewrite should therefore improve retrieval for relevant queries and reduce spurious matches for irrelevant ones. Our training reward directly optimizes this objective. The instruction in the code-rewriting prompt encourages the policy to keep the representation grounded in the original document, but semantic preservation is not enforced as a hard constraint by the reward. The rewritten representation serves as a retrieval surrogate, while the original document remains the source of information or executable code used downstream. We will make this distinction clearer in the revised manuscript.

Dear Reviewer,

We thank you again for your time and effort. We have addressed the issues you raised in your review in our reply. Please let us know if you have any remaining questions before the rebuttal period ends.

Comments on the paper

1 comment

To the Reviewers, Area Chair, Senior Area Chair, and all others whom it may concern:

We thank the reviewers for their time and feedback. We first summarize their main concerns and the additional evidence and clarifications we provided. We then state our position on the reviews, the subsequent discussion, and the standing assessments.

  • Additional Document-side baselines (ZYJ5, d4Lv, Qctv, z3i9). The reviewers requested document-rewriting baselines beyond zero-shot rewriting. We added three: supervised doc2query, prompt-conditioned expansion, and query-conditioned rewrite SFT. DocOpt achieves the best aggregate performance among these methods across all six retrievers.
  • Faithfulness of rewrites (ZYJ5, d4Lv, z3i9). The reviewers were concerned that optimizing retrieval rankings could introduce factual errors or alter code behavior. We clarified that rewrites replace only the indexed representation; original documents remain the source of information or executable code used downstream.
  • Computational costs and scalability (ZYJ5, d4Lv, z3i9). The reviewers questioned extrapolations from short documents and the additional costs of rewriting, embedding, and indexing. We expanded the analysis to 2K, 10K, and 100K documents across short code, long code, and visual inputs, with pipeline breakdowns and embedding/storage measurements. We distinguished measured costs from extrapolations and identified excluded components.
  • Statistical uncertainty (ZYJ5, d4Lv). The reviewer questioned the reliability of smaller gains, particularly on datasets with few queries. We added 95% bootstrap confidence intervals to the main results and provided intervals for the small datasets. We acknowledged that evaluating variation across training seeds and query splits exceeds our available compute budget.
  • Comparison with prior RL methods (ZYJ5, z3i9). The reviewers requested comparisons with closely related RL-based document rewriting. We evaluated three modifications inspired by Liu et al.: retaining the original document alongside the rewrite, removing advantage standard-deviation normalization, and removing KL regularization. The first and third reduce performance; the second yields a modest improvement..

Two reviewers (ZYJ5 and d4Lv) did not respond during the discussion period, despite substantial additional experiments addressing their concerns. Reviewers Qctv and z3i9 acknowledged parts of the added evidence but maintained their scores. Their remaining concerns centered on query-side comparisons, methodological novelty, and comparisons with prior RL methods. We respectfully disagree that these concerns provide sufficient grounds for rejection.

Query-side comparisons (Qctv). Our work asks whether retrieval can be improved through offline document adaptation without introducing query-time generation. The requested query-side and joint optimization methods introduce precisely the online computation we aim to avoid. The relevant comparisons are therefore with methods that preserve this constraint, including the additional document-side baselines we evaluated. Comparisons with online methods could characterize broader tradeoffs between retrieval quality, latency, and serving cost, but would address a different research question. A meaningful study of these tradeoffs would extend beyond query rewriting to approaches such as generative reranking and pseudo-relevance feedback, well beyond the scope of this work.

Methodological novelty (Qctv). We disagree that using an established optimization algorithm such as GRPO constitutes a weakness. Our contribution is to formulate and systematically evaluate document optimization for fixed retrievers, providing broad empirical evidence for its effectiveness across retrieval architectures and tasks. These findings demonstrate a practical way to improve retrieval without modifying retriever parameters or adding query-time generation. We believe this constitutes a substantive contribution, and that introducing a new RL algorithm is not necessary to establish its value.

Prior RL methods (z3i9). To our knowledge, Liu et al. is the only directly relevant prior RL work on document optimization. We argue that their conflicting conclusion about its utility reflects the limited scope of their evaluation data, as discussed in our rebuttal. Both methods are standard variants of GRPO, with slightly different design choices and hyperparameters. RL offers a large configuration space, and selecting settings suited to the hardware, data and model is standard practice. In addition to our existing ablations of RL design choices, including the reward formulation, we evaluated several choices used by Liu et al. and found no consistent advantage over our configuration. We do not believe that the absence of a full reproduction of another GRPO variant is sufficient grounds to reject the contribution.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.