- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes CoGR, a framework that jointly trains query-side and item-side keyword generators (LLMs) through alternating RL. The key idea of this paper is to let both sides adapt to each other, using retrieval F1 to train the query generator and rewarding the item generator when its changes (to the item keywords) improve retrieval performance.
Critical strengths
-
I think the motivation is great - jointly optimizing query and item representations is a natural and promising direction, and the proposed approach is simple and clean.
-
The reward design for the item-side RL is good.
-
The results look strong, with clear improvements over strong baselines. The ablations are also great.
Critical weaknesses
-
The evaluation is somehow limited - it only tests two datasets, and WANDS has only 50 validation queries. More datasets can improve this paper.
-
The method performs well on full-set F1, but its advantages on top-k ranking metrics are less consistent. This part worth a discussion.
-
Training two generators through multi-round RL is expensive. A more detailed cost analysis and compute-matched comparison are good to add.
Questions for authors
see weaknesses.
3 comments
Results on new dataset
Thank you again for your positive assessment of our work. Your support means a lot to us.
As promised in our previous response, we are pleased to share that we have now completed additional experiments on NFCorpus, a publicly available retrieval benchmark, including comparisons with CoAugRetriever and ANCE-Qwen4B. The results are summarized below:
| Method | MRR@100 | NDCG@100 | F1@100 | P | R | F1 |
|---|---|---|---|---|---|---|
| CoAugRetriever-4B | 0.5706 | 0.3311 | 0.1055 | 0.2076 | 0.2221 | 0.1470 |
| ANCE-4B | 0.6010 | 0.4770 | 0.1790 | 0.3090 | 0.3230 | 0.2300 |
| CoGR-4B | 0.5710 | 0.4120 | 0.2450 | 0.3280 | 0.3370 | 0.2460 |
CoGR achieves the highest F1@100 and overall precision, recall, and F1. These results extend our evaluation to a third benchmark and provide further evidence of CoGR's effectiveness beyond the original two datasets.
We are extremely grateful for your positive score and your supportive feedback on our work. While we hesitate to ask for more, if these additional results further strengthen your confidence, we would be truly grateful if you would consider raising your score.
Thanks for your response. I will keep my score.
We sincerely thank you for your positive assessment of our work and recommendation for acceptance. We greatly appreciate your encouraging feedback and constructive suggestions.
We are adding experiments on additional public benchmarks and hope to share the results soon.
W2 (Top-k Metrics)
We acknowledge that our current top-k metrics are relatively less consistent. This is primarily because we use BM25 over generated keywords for ranking, which limits the ranking performance. To better illustrate this, we conduct two additional experiments: (1) using a dense retrieval baseline (ANCE-Qwen3-4B) to rerank the candidates retrieved by CoGR, and (2) applying Reciprocal Rank Fusion (RRF) to combine the ranking results of CoGR and ANCE-Qwen3-4B. Both approaches substantially improve the top-k metrics.
W3 (Computation Cost)
We acknowledge that training two generators through multi-round RL incurs substantial computational costs. Our primary focus is on improving retrieval effectiveness rather than minimizing training compute. However, since training is performed infrequently in practice, its cost can be amortized over a large volume of retrieval requests in industrial applications. Below, we provide a detailed breakdown of the training cost.
Training cost. Training is conducted on a node with 8 B200 GPUs. SFT takes 0.13 hours, while each round of query-side and item-side RL takes 3.95 and 4.20 hours, respectively. The five rounds of RL optimization take approximately 40.8 hours in total.
We also provide the indexing, inference, and retrieval costs below to offer a more comprehensive view.
Indexing cost. Indexing the item corpus of 39.6K items takes 3.3 minutes on one B200 GPU. Since item keyword generation can be performed independently, this process is naturally parallelizable and can be accelerated with parallel sampling.
Inference cost. CoGR employs autoregressive generation, with an average query-side inference latency of 128 ms. In industrial applications, frequently occurring queries can be cached, reducing the need for online inference. For new queries, techniques such as speculative decoding or replacing the generator with a diffusion language model could reduce latency.
Retrieval cost. CoGR benefit from reverted index thus achieves low retrieval latency and a compact index compared with the dense baseline as summarized below.