- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates improving retrieval performance by optimizing document representations rather than modifying queries or retriever parameters. The authors propose DocOpt, a framework that uses reinforcement learning to train LLMs or VLMs to rewrite documents into representations better suited to a target retriever. Specifically, the method introduces a counterfactual retrieval reward based on changes in nDCG@5, considering both positive and negative queries, and optimizes the rewriting policy using GRPO. Since document transformations are performed offline, the approach avoids additional LLM inference during online retrieval and supports black-box retrievers. Experiments on code retrieval and visual document retrieval demonstrate consistent improvements over direct retrieval and zero-shot rewriting across lexical, single-vector, and multi-vector retrievers. Additional experiments investigate retriever fine-tuning, cross-retriever transfer, and generalization to natural language retrieval.
Critical strengths
-
The paper explores an interesting and practical direction for improving retrieval by shifting LLM-based optimization from the query side to the document side. Performing document rewriting offline is particularly appealing for black-box retrieval systems where modifying the retriever is not possible.
-
The proposed counterfactual retrieval reward is intuitive and directly aligned with the retrieval objective. Considering both positive and negative queries allows the model to optimize document representations relative to competing candidates rather than simply maximizing query-document similarity.
-
The experiments cover diverse retriever architectures and retrieval tasks, including code and visual document retrieval. The consistent improvements over zero-shot rewriting are promising, and the comparisons with retriever fine-tuning further demonstrate the potential of document-side optimization.
Critical weaknesses
-
My main concern is about the faithfulness of the rewritten documents. The RL reward focuses on retrieval rankings without explicitly ensuring that the transformations preserve the original information. This is particularly important for code retrieval, where rewriting may alter program functionality, and visual document retrieval, where generated descriptions may introduce incorrect information. A systematic evaluation of this would make the method more convincing eg 通过semantic preservation
-
The baseline comparisons could be more comprehensive. Existing document expansion approaches, such as doc2query, as well as closely related reinforcement learning methods for document rewriting, are not directly evaluated. Including these baselines would help clarify the benefits of the proposed ranking-based optimization over existing document-side adaptation techniques.
-
The requirement to rewrite the entire document corpus may introduce substantial computational overhead, especially for large-scale or frequently updated collections. Moreover, the rewritten documents are considerably longer than the originals (346 vs. 183 tokens on average), potentially increasing embedding computation, indexing time, and storage costs. Although rewriting is performed offline, these additional costs are not fully accounted for in the efficiency analysis, raising concerns about the scalability and practicality of the proposed approach.
Questions for authors
see weakness
5 comments
This table (https://bashify.io/i/tvVgyO) reports embedding time and storage for DocOpt versus direct retrieval. At the evaluated size, the additional embedding time is small in absolute terms. Single-vector index storage remains unchanged, while multi-vector storage varies with rewritten document length. For longer code documents, rewriting produces shorter representations, reducing embedding time and text storage, as well as multi-vector index storage. ANN index construction and synthetic query curation are not measured in our setup, and we assume differences to be negligible between options.
Dear Reviewer,
We thank you again for your time and effort. We have addressed the issues you raised in your review in our reply. Please let us know if you have any remaining questions before the rebuttal period ends.
We thank the reviewer for their time and positive assessment of the paper’s strengths. We argue that the first and main concern (regarding the faithfulness of rewritten documents) is misplaced and explain why below. We address the concerns about baselines and computational costs with additional results and analysis.
The intention behind document rewriting is to replace only the document’s representation in the index, while keeping the original document for downstream use, as in prior work on document expansion (Nogueira et al., 2019). The rewrite does not need to preserve factual correctness, since the original document is used for processing. We will make this distinction clearer in the revised manuscript.
We added comparisons to two non-RL baselines, while matching the training data between those.
The results can be viewed here - https://bashify.io/i/GMgz8U
As shown in the table, DocOpt achieves the best aggregate performance across all six retrievers. Doc2Query performs slightly better on visual retrieval on two retrievers. We suspect that putting training queries directly into the prompt/document can steer the LM/retriever too strongly toward those particular queries at the expense of other potential queries (a form of overfitting). DocOpt instead uses queries only to compute the retrieval reward, while the LM/retriever sees only the document. The query contributes a scalar reward rather than exposing its full text and this may reduce overfitting to it.
Regarding alternative RL-based optimization methods, we evaluated three modifications inspired by Liu et al.: (i) retaining the original document alongside the rewrite, (ii) centering advantages without standard-deviation normalization, and (iii) removing KL regularization. We use the same configuration as the other ablations in the paper. We find (https://bashify.io/i/HfNEPD) that (i) and (iii) decrease performance, and (ii) has a advantage over the default configuration, though not a large one. As mentioned in the related works section (lines 456-461), we believe the difference between our findings and Liu et al's mainly reflect differences in evaluation and data, and not optimization methodology.
We expanded our end-to-end offline cost analysis to cover 2K, 10K, and 100K documents, across short code documents (182 input tokens on average), long code documents (1,715 input tokens on average), and visual inputs. We measure GPU hours for DocOpt across these nine configurations and extrapolate API costs from the 2K samples to 10K and 100K documents (measuring them directly at larger scales is prohibitively expensive). We also break down the costs across the different pipeline components. For these measurements, we run DocOpt for 200 steps with one refresh round. Each refresh round costs approximately as much as rewriting the corpus once, so the number of refresh rounds should balance data staleness against available compute resources.
This plot (https://bashify.io/i/qkDDCt) shows the comparison across data types (also shown here in a tabular format - https://bashify.io/i/Wak3hz). It's clear that DocOpt offers substantial savings in both time and cost from using API models on larger scales.
This plot (https://bashify.io/i/BTVfnd) shows the breakdown of the pipeline into parts across scales. As can be seen, RL training costs are independent from index size, and rewrite and refresh time grow with the corpus size. We show these costs remain manageable and much cheaper and faster than API alternatives.