acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual Edit Credit with Semantic Retrieval Feedback for Product Query Rewriting

Abstract

Product queries are often short and underspecified, leaving much of the user's preference implicit. Recent work trains large language models (LLMs) as query rewriters with reinforcement learning (RL) using ranking rewards based on the ranks of labeled relevant products. However, these rewards are sparse when relevant products are not retrieved and coarse when different retrieval outcomes leave their ranks unchanged. We propose Counterfactual Generative Search (CGS), an RL framework that uses dense semantic retrieval feedback to assign counterfactual credit to individual query edits, while retaining ranking reward as the objective for the complete rewrite. uses a relevance teacher to score retrieved products against available evidence about user intent and a semantic list utility to distinguish retrieval outcomes beyond labeled-product ranks. It assigns credit to each edit by comparing retrieval quality when the edit is removed from the complete rewrite and when it is applied to the source query on its own. With extensive experiments on H&M and Amazon ESCI, we show that CGS consistently outperforms baselines, improving NDCG@100 by 10.3% and 4.9%, respectively.

Primary area:
reinforcement learning
Keywords:
Information Retrieval, Query Rewriting, Product Search, Reinforcement Learning, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_jqDJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the problem of sparse retrieval rewards and fine-grained credit assignment in reinforcement learning-based product query rewriting. The authors propose Counterfactual Generative Search (CGS), which combines dense semantic retrieval feedback with counterfactual credit assignment for individual query edits. Specifically, a relevance teacher evaluates retrieved products using available evidence about user intent, while each edit is assessed through necessity and sufficiency interventions. The resulting edit-level credit is incorporated into GRPO alongside the original sequence-level ranking reward. Experiments on H&M and Amazon ESCI demonstrate improvements over existing query-rewriting and credit-assignment methods. Additional ablations investigate different utility functions, credit estimators, and relevance teacher configurations.

Critical strengths
  1. The paper addresses an important limitation of RL-based query rewriting by identifying both the sparsity of ranking rewards and the difficulty of assigning credit to individual edits. The motivation is clear and well supported by the analysis of zero-reward training examples.

  2. The proposed combination of semantic retrieval feedback and counterfactual edit credit is intuitive and interesting. Evaluating both the necessity and sufficiency of individual edits provides a reasonable way to obtain more informative training signals.

  3. The experiments include comparisons with several recent credit-assignment methods and comprehensive ablation studies. The consistent improvements across two datasets and different model sizes support the effectiveness of the proposed approach.

Critical weaknesses
  1. My main concern is about the potential bias introduced by the semantic relevance teacher. The teacher relies heavily on metadata overlap, particularly the attributes of labeled target products, which may favor certain query edits without accurately reflecting the user's actual intent. Since the resulting semantic feedback is used for edit-level credit assignment, such biases could be propagated into the RL training process.

  2. The evaluation is limited to a single MPNet dense retriever. Since the proposed feedback and credit assignment depend on retrieval outcomes, it is unclear whether the improvements generalize to other retriever architectures, such as lexical or alternative dense retrievers.

  3. The theoretical analysis relies on submodular or supermodular edit contributions, but it is unclear how often these assumptions hold in practical query rewriting. More discussion or empirical analysis of complex interactions between edits would make the theoretical justification more convincing.

Questions for authors

see weakness

Reviewer_LsMwOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the query rewriting for product research via improved GRPO with (1) dense sequence reward via user-intent aware semantic list utility and (2) solve the credit assignment with PNS-inspired credit esitmator by counterfactual edits (necessity vs suffciency). The experiments are organzied over two datasets(H&M and Amazon ESCI ), and Qwen2.5 model family for RL post-training. The main results are good (Rq1,Rq2). The ablation study covers over credit construction(Rq3), teacher evidence(Rq4), model scaling (1.5B, 3B) (Rq5), and efficiency.

Critical strengths
  1. The two research problem of better sequence reward and credit assignment is well-known important problems in GRPO-based RL study. The proposed methods on enriched user-intent aware feedback and counterfactual edits are conceptually straight-forward and practically useful. The considering marginal discussion of the combinatory edit in 4.4 is a valid conditional results although the submodularity may not hold (acknowledged in limitation).
  2. The ablation studies are thought-provoking.
Critical weaknesses
  1. As mentioned the limitation, the section 4.4 discussion the minimal of two boundary counterfactual edits are only conditionally correct. Although the experiments shows good results, might be interesting to how those assumptions hold in the experiments. Any other factors. For example, useful edits may need each other, rather than add or delete by each one, such as coffee machine.
    1. Did you consider those multi-word expressions? each word may not help, but the combination helps. Maybe the edits here could be some meaning segment. Section 5.7 says edited segments, please clarify more.
    2. For each pair (x, q), we need to extra retrivals to test the edits. It seems quite expensive. Any optimization on this having be done?
  2. The performance results looks wired.
    1. Could you report the performance for the original query as a comparasion?
    2. The model only contains small size 1.5 and 3B, the performance are generally not comparable with original OSPO paper. The reported numbers from OSPO-prop seems underperforman GRPO, but in the OSPO paper it is better than GRPO. The OSPO native is almost the same as zero-shot. The overall performance are worth double check based on this inconsistency (at least for OSPO). Please clarify.
Questions for authors

See the weakness.

Reviewer_s9txOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper's research topic is product query rewriting: short, underspecified user queries are reformulated to improve retrieval of relevant items. The authors argue that rewards what related works use are sparse (identical zero reward when no target product is retrieved) or coarse (unchanged when the retrieved list changes but target ranks stay fixed), so re-allocating a ranking reward cannot give informative per-edit credit. They propose an RL framework with two components: (i) a dense semantic retrieval utility — a relevance teacher scores each retrieved product against intent evidence (source query, target-product metadata, optional user history) and a rank-decayed list utility aggregates the list; and (ii) counterfactual edit credit — each rewrite is decomposed into textual edits, and each edit receives credit which is MAD-normalized, thresholded to be positive only when both effects are positive, and added as a token-level bonus to the GRPO sequence advantage, which still optimizes ranking reward for the complete rewrite.

Experiments on H&M and Amazon ESCI with a fixed MPNet retriever and Qwen2.5-1.5B/3B rewriters report consistent gains over baselines. Ablations cover the credit utility (RQ2), credit construction (RQ3), teacher evidence (RQ4), and model scale (RQ5).

Critical strengths
  1. The central design choice is directly validated by a decisive ablation. Table 2 isolates credit construction while holding everything else fixed and shows that the counterfactual minimum beats necessity-only, sufficiency-only, their average, and adding the semantic utility directly to the sequence reward (R + αU_z), which degrades all H&M metrics. This shows that where the semantic feedback is placed (edit-level credit vs. sequence reward) matters, not merely that it is present. This is the strongest evidence that the proposed mechanism, rather than an unrelated training change, drives the gains.

  2. An honest, internally consistent empirical package. Gains are consistent across two datasets, two model scales, and four metrics; every claimed percentage improvement recomputes correctly against the tables; the credit-signal analysis directly evidences the sparse/coarse-reward motivation; and the paper proactively discloses its limitations (synthetic H&M user profiles, teacher dependence, extra retrieval evaluations).

Critical weaknesses
  1. The empirical case rests on baseline reproductions whose strength is not established, and statistical significance is not reported. The "native" runs of Rec-R1 and OSPO are far weaker than the paper's own in-house GRPO baseline on the same setup (Table 1, H&M NDCG@100: Rec-R1 native 0.0563 and OSPO native 0.0536 vs. GRPO 0.1004), and the strongest baseline differs by metric (VinePPO for most, PPO/GAE for ESCI MRR), which makes the headline "+10.3%/+4.9%" less interpretable. Several headline gains are modest in absolute terms (ESCI Recall@100 +2.9%, MRR@100 +4.4%), and results are averaged over only three seeds with no standard deviations or significance tests. Since the contribution is fundamentally empirical (the theoretical guarantee is conditional on unverified sub-/supermodularity of the edit-level utility, as the paper itself notes in Appendix A.1), the lack of variance/significance reporting materially weakens the case.

  2. The training feedback is dominated by a target-metadata term-overlap teacher whose proxy-misalignment risk is unexamined, and the H&M "user history" is synthetic by construction. The relevance teacher gives weight 0.60 to target-product metadata term overlap (Sec. 4.1.1, App. B.3), so the semantic utility largely measures how textually similar the retrieved list is to the target product's metadata. The paper does not analyze whether learned rewrites degenerate toward emitting target-metadata vocabulary rather than improving generalizable query understanding — no qualitative examples of rewritten queries are provided anywhere. In addition, H&M user profiles are assigned from customers who purchased the target product, not the original query author (App. B.1), so the profile is partly a deterministic function of the target; the RQ4 conclusion that "user history improves precision" is confounded by this construction. The disclosure is honest, but the risk that part of the gain comes from exploiting teacher structure rather than from the credit mechanism is not quantified.

Questions for authors
  1. Edit construction (material for soundness). Section 4.2 defines edits by "aligning" the source query and rewrite, but no alignment algorithm is given. How are inserted and deleted spans extracted and normalized? For a deleted span, what exactly are the two counterfactual queries q^{−e} and x^{+e}? How are token positions mapped to edits (S_t(e)), and is the decomposition deterministic? The entire credit estimator depends on this step.
  2. Statistical significance. Please report mean ± std over the three seeds (and significance tests where feasible) for Tables 1–4. Several improvements are below 5% (e.g., ESCI Recall@100 +2.9%); can you confirm these are not within seed variance? Relatedly, please name the single strongest baseline per metric when reporting "best baseline" improvements.
  3. Teacher/proxy analysis. Given the 0.60 weight on target-metadata overlap, can you show qualitative examples of learned rewrites (especially for zero-ranking-reward sequences) and analyze whether CGS rewrites remain diverse and intent-appropriate rather than collapsing toward target-metadata vocabulary? Also, why does the Y_z-NDCG variant underperform Target-NDCG on H&M NDCG@100 (0.0969 vs. 0.1018, Table 1) while the ordering reverses on ESCI? This asymmetry seems worth explaining for a paper whose core claim is that dense semantic feedback is more informative than target ranks.
  4. H&M user context. Since user profiles come from customers who purchased the target product (not the query author) and splitting is example-level (App. B.1), how should the RQ4 "user history" gain be interpreted? Can the same profile customer appear in both train and test splits? Without a cleaner user-context construction, is this a fair test of user-history utility?
  5. Efficiency accounting. Table 6 counts retrieval evaluations but not compute. For ESCI (1.37M-item catalog), what is the wall-clock training cost of CGS vs. GRPO given the up to 2+2E extra retrievals per response? Also, how sensitive are the results to β (0.02/0.05) and the credit clip bound (3.0)?
Reviewer_khtmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Counterfactual Generative Search (CGS), an RL approach for product-query rewriting. It keeps a ranking-based reward for the complete rewrite and adds edit-level training signals derived from a semantic retrieval utility. A lightweight teacher scores retrieved products using the source query, target-product metadata, and, for H&M, an associated customer's purchase history. For each edit, CGS measures the utility change when removing it from the complete rewrite and when applying it alone to the source query. The smaller of these two changes becomes the edit credit, which is normalized, clipped to positive values, and added to the corresponding token advantages in GRPO. Experiments use Qwen2.5-1.5B and 3B with a fixed MPNet retriever on H&M and Amazon ESCI. The paper reports improvements over ranking-reward and credit-assignment baselines, along with ablations of the teacher evidence, list utility, and edit-credit construction.

Critical strengths

1、The paper identifies a real limitation of ranking-only training feedback. Two rewrites can receive the same zero reward even when one retrieves products that are much closer to the intended target. CGS tries to use that difference at the level of individual edits rather than applying an undifferentiated bonus to the whole sequence. The credit-signal analysis is useful here: among sequences with zero ranking reward, 40.5% on H&M and 55.7% on ESCI still receive positive edit credit. These numbers do not by themselves prove that every credited edit is useful, but they show that the proposed feedback reaches cases where the ranking reward cannot distinguish the retrieved results.

2、The evaluation includes several relevant ablations rather than only comparing the full method with standard GRPO. Table 1 changes the utility while keeping the minimum-credit construction fixed, and Table 2 compares sequence-level semantic feedback with necessity-only, sufficiency-only, averaged, and minimum edit credit. The full method performs best across the reported metrics in those comparisons, and the gains also appear with the 3B rewriter. This supports the combination of semantic feedback and selective edit-level credit. I also appreciate that the theoretical statement is explicitly conditional on submodular or supermodular edit effects, and that the appendix distinguishes this assumption from submodularity of the utility over fixed product-rank pairs.

Critical weaknesses

1、A central part of the method is not specified clearly enough to reproduce or fully assess. Section 4.2 defines edits as inserted and deleted spans, but the paper does not give the alignment procedure, how replacements or overlapping changes are handled, or how a deletion is mapped to response-token positions in Equation 10. A deleted span has no corresponding emitted tokens, so this mapping is not a minor implementation detail. It affects which actions receive the positive bonus and whether the two interventions represent the same edit in comparable contexts. Appendix B.6 also reports 1,186 H&M edits but says both effects are available for only 830, without explaining how the remaining edits are handled. Since the paper's contribution is the credit assigned to individual edits, these choices are needed to understand what was actually trained and measured.

2、The experiments do not fully separate the new credit rule from the additional supervision and training resources it uses. Table 3 shows that target metadata is important: the query-only teacher falls below GRPO on both datasets, while target-informed teachers give the main gains. The existing ranking-based credit baselines do not receive that same semantic signal. Table 2 is a useful partial control, but it does not compare an existing segment-credit method, such as OSPO-Prop, using the same teacher and list utility. The paper also does not give enough baseline training and tuning details to explain the large native-versus-adapted performance differences, or a measured equal-compute comparison despite CGS's extra retrieval calls. Three-seed averages are reported, but not their variability. These gaps make it hard to tell how much of the advantage specifically requires the proposed minimum-credit estimator.

Questions for authors

1、What exact algorithm produces the edit set and the counterfactual queries? Please show one replacement and one deletion example, including the response-token positions that receive each edit's credit in Equation 10.

2、Why are both counterfactual effects available for only 830 of the 1,186 H&M edits in Appendix B.6? Are the remaining edits omitted, assigned zero credit, or handled differently, and does the reported positive-credit rate depend on that choice?

3、How does an existing segment-credit method perform when it uses the same target-informed teacher, semantic list utility, and training budget as CGS? Were the semantic-reward coefficient and edit-bonus coefficients in Table 2 selected with comparable validation effort?

4、What differs between native Rec-R1/OSPO and the adapted ranking-reward baselines? Please report their optimization budgets, checkpoint-selection rules, seed variability, and actual training time or retrieval-call counts so the main comparison can be assessed.

5、Appendix B.1 says H&M profiles come from customers who purchased the target, rather than the original query authors, and that splitting is by example. How much normalized-query or target-product overlap is there across splits? Does the result hold with query-disjoint splits, and how should the user-profile result be interpreted given this constructed association?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chairs and Reviewers,

We sincerely thank all reviewers for their constructive feedback and thoughtful discussions throughout the rebuttal process. Below, we summarize our main contributions and the further evidence presented during the rebuttal.

Our work makes three main contributions:

  1. Informative semantic retrieval feedback beyond sparse ranking rewards. We introduce a relevance teacher and semantic list utility to evaluate retrieved products using available evidence about user intent. This provides richer training supervision for learning effective query rewrites.

  2. Counterfactual credit assignment for individual query edits. CGS evaluates each edit through complementary necessity and sufficiency comparisons and assigns fine-grained training credit based on its contribution to semantic retrieval utility, enabling more targeted reinforcement of beneficial edits.

  3. Effective and generalizable query rewriting. Across H&M and Amazon ESCI, CGS consistently outperforms existing query-rewriting and credit-assignment baselines, improving NDCG@100 by 10.3% and 4.9%, respectively. The improvements extend across retrieval metrics and model scales.

We greatly appreciate the reviewers' recognition of the importance of addressing sparse ranking feedback and fine-grained credit assignment, the value of the counterfactual credit mechanism and supporting ablations, and the consistent empirical improvements.

During the rebuttal, we strengthened our evidence through:

  • Additional baseline comparisons. We extended OSPO-Prop and TEMPO with the same relevance teacher and semantic list utility. CGS consistently outperforms both methods, including their semantic-feedback variants, across all reported metrics on both datasets, providing further evidence for the contribution of counterfactual edit-level credit.

  • Experimental setup and reporting. We clarified the differences between native and adapted baselines, including their training and evaluation protocols. We additionally reported standard deviations across three seeds for the main experiments and ablations (Appendix B.8, Tables 11–14).

  • Qualitative examples and user-profile clarification. We provided qualitative examples showing that CGS produces meaningful, intent-preserving rewrites, including cases with zero ranking reward. We also clarified the construction and role of H&M user profiles.

  • Generalization across retrieval models. We additionally evaluated CGS using two alternative dense retrievers, E5 and BGE. Across both datasets and alternative retrievers, CGS achieves the highest NDCG@100 and Recall@100, supporting the transferability of the learned rewriting strategies.

  • Theoretical clarification and edit-interaction analysis. We clarified the theoretical assumptions and the scope of the guarantees provided by Proposition 1. We further examined whether positively credited edits remain useful under complex edit combinations and found that they improve target-product NDCG@100 more frequently than nonpositive-credit edits (25.9% versus 12.6% on H&M and 41.5% versus 25.6% on ESCI). These results support the usefulness of counterfactual credit under complex edit interactions.

  • Additional implementation details. We clarified the edit-alignment procedure, counterfactual query construction, and response-token credit assignment for insertions, replacements, and deletions.

  • Computational efficiency and hyperparameter sensitivity. We reported measured training times. We also conducted additional sensitivity experiments on the edit-credit coefficient β\beta, providing further evidence on the practical behavior and efficiency of CGS.

Together, these results further support CGS's effectiveness and clarify its theoretical scope and practical applicability. We will incorporate the additional experiments and clarifications into the revised manuscript.

We sincerely thank the Area Chairs and reviewers again for their time, constructive feedback, and thoughtful consideration of our work.

Best regards,

Submission 49148 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.