- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies the problem of sparse retrieval rewards and fine-grained credit assignment in reinforcement learning-based product query rewriting. The authors propose Counterfactual Generative Search (CGS), which combines dense semantic retrieval feedback with counterfactual credit assignment for individual query edits. Specifically, a relevance teacher evaluates retrieved products using available evidence about user intent, while each edit is assessed through necessity and sufficiency interventions. The resulting edit-level credit is incorporated into GRPO alongside the original sequence-level ranking reward. Experiments on H&M and Amazon ESCI demonstrate improvements over existing query-rewriting and credit-assignment methods. Additional ablations investigate different utility functions, credit estimators, and relevance teacher configurations.
Critical strengths
-
The paper addresses an important limitation of RL-based query rewriting by identifying both the sparsity of ranking rewards and the difficulty of assigning credit to individual edits. The motivation is clear and well supported by the analysis of zero-reward training examples.
-
The proposed combination of semantic retrieval feedback and counterfactual edit credit is intuitive and interesting. Evaluating both the necessity and sufficiency of individual edits provides a reasonable way to obtain more informative training signals.
-
The experiments include comparisons with several recent credit-assignment methods and comprehensive ablation studies. The consistent improvements across two datasets and different model sizes support the effectiveness of the proposed approach.
Critical weaknesses
-
My main concern is about the potential bias introduced by the semantic relevance teacher. The teacher relies heavily on metadata overlap, particularly the attributes of labeled target products, which may favor certain query edits without accurately reflecting the user's actual intent. Since the resulting semantic feedback is used for edit-level credit assignment, such biases could be propagated into the RL training process.
-
The evaluation is limited to a single MPNet dense retriever. Since the proposed feedback and credit assignment depend on retrieval outcomes, it is unclear whether the improvements generalize to other retriever architectures, such as lexical or alternative dense retrievers.
-
The theoretical analysis relies on submodular or supermodular edit contributions, but it is unclear how often these assumptions hold in practical query rewriting. More discussion or empirical analysis of complex interactions between edits would make the theoretical justification more convincing.
Questions for authors
see weakness
Rebuttal Summary
Dear Area Chairs and Reviewers,
We sincerely thank all reviewers for their constructive feedback and thoughtful discussions throughout the rebuttal process. Below, we summarize our main contributions and the further evidence presented during the rebuttal.
Our work makes three main contributions:
Informative semantic retrieval feedback beyond sparse ranking rewards. We introduce a relevance teacher and semantic list utility to evaluate retrieved products using available evidence about user intent. This provides richer training supervision for learning effective query rewrites.
Counterfactual credit assignment for individual query edits. CGS evaluates each edit through complementary necessity and sufficiency comparisons and assigns fine-grained training credit based on its contribution to semantic retrieval utility, enabling more targeted reinforcement of beneficial edits.
Effective and generalizable query rewriting. Across H&M and Amazon ESCI, CGS consistently outperforms existing query-rewriting and credit-assignment baselines, improving NDCG@100 by 10.3% and 4.9%, respectively. The improvements extend across retrieval metrics and model scales.
We greatly appreciate the reviewers' recognition of the importance of addressing sparse ranking feedback and fine-grained credit assignment, the value of the counterfactual credit mechanism and supporting ablations, and the consistent empirical improvements.
During the rebuttal, we strengthened our evidence through:
Additional baseline comparisons. We extended OSPO-Prop and TEMPO with the same relevance teacher and semantic list utility. CGS consistently outperforms both methods, including their semantic-feedback variants, across all reported metrics on both datasets, providing further evidence for the contribution of counterfactual edit-level credit.
Experimental setup and reporting. We clarified the differences between native and adapted baselines, including their training and evaluation protocols. We additionally reported standard deviations across three seeds for the main experiments and ablations (Appendix B.8, Tables 11–14).
Qualitative examples and user-profile clarification. We provided qualitative examples showing that CGS produces meaningful, intent-preserving rewrites, including cases with zero ranking reward. We also clarified the construction and role of H&M user profiles.
Generalization across retrieval models. We additionally evaluated CGS using two alternative dense retrievers, E5 and BGE. Across both datasets and alternative retrievers, CGS achieves the highest NDCG@100 and Recall@100, supporting the transferability of the learned rewriting strategies.
Theoretical clarification and edit-interaction analysis. We clarified the theoretical assumptions and the scope of the guarantees provided by Proposition 1. We further examined whether positively credited edits remain useful under complex edit combinations and found that they improve target-product NDCG@100 more frequently than nonpositive-credit edits (25.9% versus 12.6% on H&M and 41.5% versus 25.6% on ESCI). These results support the usefulness of counterfactual credit under complex edit interactions.
Additional implementation details. We clarified the edit-alignment procedure, counterfactual query construction, and response-token credit assignment for insertions, replacements, and deletions.
Computational efficiency and hyperparameter sensitivity. We reported measured training times. We also conducted additional sensitivity experiments on the edit-credit coefficient β, providing further evidence on the practical behavior and efficiency of CGS.
Together, these results further support CGS's effectiveness and clarify its theoretical scope and practical applicability. We will incorporate the additional experiments and clarifications into the revised manuscript.
We sincerely thank the Area Chairs and reviewers again for their time, constructive feedback, and thoughtful consideration of our work.
Best regards,
Submission 49148 Authors