acceptodds
Under review as a conference paper at ICLR 2027

Personalized Query Enhancement via Reward-Guided Feedback Descent

Abstract

Users in real-world interactive systems often submit vague queries that only partially reflect their latent intent. Query enhancement aims to make such queries more informative by adding context that better captures the user’s underlying needs. However, a key challenge is that enhancement quality is difficult to measure: direct satisfaction feedback is sparse, while abundant indirect signals are only noisy proxies. In this paper, we study personalized query enhancement under sparse feedback. We formulate user satisfaction through a hierarchical model that links sparse direct satisfaction signals with denser indirect behavioral feedback, yielding a reward model for evaluating query enhancements. Using this reward model, we define objectives for generating and refining personalized enhancements. To iteratively improve the enhancements, we propose reward-guided feedback descent, a novel text-feedback-based query refinement algorithm. A key novelty of our approach is that it uses the reward model and the enhancement objective as a grounding mechanism for determining the preferred candidate numerically and synthesizing textual feedback, which in turn guides the LLM enhancer toward generating higher-quality enhancements. We evaluate our method on both public benchmarks and real-world e-commerce data, demonstrating that the proposed algorithm consistently improves user satisfaction through enhanced queries.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
personalized query enhancement, text optimization, behavioral feedback

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rcdXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies personalized query enhancement when satisfaction feedback is sparse and behavioral signals provide indirect supervision. It develops a hierarchical satisfaction model and reward-tilted intent distributions, from which it derives two enhancement objectives, PESO and PUBAO, balancing intent refinement or utility with faithfulness to the original query. The proposed reward-guided feedback descent algorithm selects candidate rewrites numerically using these objectives and uses an LLM to verbalize objective differences into feedback for further refinement. For latency-sensitive serving, it constructs an offline enhancement pool indexed by prototypical personas and interpretation branches, then retrieves and selects candidates using a learned reward model. Evaluation combines a proprietary offline e-commerce study, using LLM-derived reference satisfaction scores and supplementary human assessments, with a held-out LongMemEval-M study. The reported results show improvements in estimated satisfaction, evidence retrieval, and downstream QA accuracy, supported by reward-supervision and refinement-objective ablations; the e-commerce results do not establish online user-satisfaction gains.

Critical strengths
  1. The paper presents a coherent framework that connects sparse satisfaction labels, abundant behavioral signals, and iterative query refinement.

  2. The evaluation is broad and deployment-aware, covering a proprietary e-commerce setting, two public benchmarks, human judgments, statistical tests, and several ablations.

  3. Separating numerical candidate selection from textual feedback generation is a practical and well-motivated design choice.

Critical weaknesses
  1. The main e-commerce evaluation relies heavily on GPT-5.2-generated satisfaction labels, which are used both to train the reward model and to report the primary satisfaction gains. Since GPT-5.2 also generates the rewrites, model-specific preference may inflate the reported improvements. The human studies help, but they evaluate query or branch preference rather than actual user satisfaction.

  2. There is a gap between the theoretical formulation and the implementation. The theoretical branch reward is a log-sum-exp over latent intents, whereas the implementation substitutes a conditional-mean Random Forest prediction. The propositions are valid under the definitions, but the experiments do not directly validate several key assumptions or this substitution.

  3. The experiments do not fully isolate the benefit of iterative textual feedback from reward-based candidate search. A compute-matched baseline that samples the same number of rewrites and selects the best one using PUBAO would clarify whether feedback descent contributes beyond repeated generation and reward selection.

Questions for authors
  1. Can the authors evaluate the main satisfaction gains using independently collected human labels or online outcomes rather than the same LLM-derived target used for reward-model training?

  2. How well does the practical conditional-mean reward proxy preserve candidate rankings induced by the theoretical branch reward?

  3. Under an equal LLM-call budget, how does PUBAO-guided feedback descent compare with sampling multiple independent rewrites and selecting the highest-PUBAO candidate?

2 comments

[W1, Q1] Independent Human Validation and Satisfaction Evaluation

We agree that independent human satisfaction labels or online outcomes would provide more direct validation. As the reviewer correctly noted, our current human studies assess query and interpretation preferences, rather than user satisfaction from actual use. Establishing satisfaction gains in a deployed service would require a prospective user study or an online A/B test, ideally including direct user-reported satisfaction. Such an evaluation was not conducted in this study and remains an important step toward deployment. Nevertheless, the current experiments provide meaningful evidence for the proposed approach: human studies support the quality of its query-refinement and branch-selection decisions (Appendix B), while the additional evaluation with the Qwen3-235B judge shows that the estimated satisfaction improvements are not confined to the original GPT-5.2 judge (Table R.1 of our response to Reviewer Cfoo). Together, these results support the effectiveness of the proposed approach in the offline research setting considered in this study. We thank the reviewer for this constructive and insightful feedback and will prioritize direct validation of user satisfaction as we move toward deployment.

[W2, Q2] Theoretical Branch Reward and Practical Approximation

Thank you for raising this question. As defined in Section 2.2, the theoretical branch reward aggregates latent-intent rewards through log-sum-exp:

\log \mathbb{E}_{h \sim \pi_{u,q}(\cdot\mid b)} \left[\exp(\gamma R_{u,q}(h))\right].$$ Since latent intents are unobserved, our two-stage predictor estimates **conditional-mean satisfaction as a practical proxy**. This choice is **theoretically motivated**: under our probabilistic formulation, Jensen's inequality gives $$\underbrace{\mathbb{E}[z\mid q,D_u,b]}_{\text{Lower bound}} \leq R^{\mathcal B}_{u,q}(b) \leq \underbrace{ \frac{1}{\gamma} \log\mathbb{E}_{z\mid q,D_u,b} \left[\exp(\gamma z)\right] }_{\text{Upper bound}}.$$ Thus, under the assumed probabilistic model, the **theoretical reward lies between the population conditional mean and the log-sum-exp of conditional satisfaction scores**. Our Random Forest estimates the former, while the proximity-weighted satisfaction distribution provides an empirical estimate of the latter. To evaluate this approximation, we estimated both bounds using the same fitted Random Forest. We constructed a **conditional satisfaction distribution** using Random Forest proximity weights, which assign higher weights to training samples that more frequently share terminal leaves with each query–user–branch input. The proximity-weighted mean closely matches the original RF predictions (Pearson 0.991, MAE 0.047), while the estimated distribution achieves a 37% lower ranked probability score (RPS) than an unconditional baseline on held-out data. **Table R.8. Evaluation of the branch-reward approximation ($\gamma=1.5$).** | Metric | Result | |-|-| | Median estimated Jensen bound gap (selected branch) | 0.15 | | Candidate-ranking Kendall's $\tau$ | 0.900 | | Final candidate selection agreement | 89.2% | | Difference in mean reference satisfaction | <0.01 | The median estimated gap between the Jensen bounds is 0.15 points, or 1.7% of the 1–10 reward range, for the selected branches. Across 4,210 evaluation instances, mean-based and log-sum-exp-based scoring produce **highly similar candidate rankings** and select the same query in 89.2% of cases, with a difference below 0.01 in overall mean reference satisfaction. Since latent intents are unobserved, this comparison evaluates an estimated Jensen-bounded alternative rather than the exact theoretical reward. The small estimated bound gap and high agreement between mean-based and log-sum-exp-based scoring support the **practical robustness of the conditional-mean proxy**. We will clarify this theoretical relationship and include the additional analysis in the revised manuscript.

[W3, Q3] Comparison with Best-of-N Sampling

Thank you for this suggestion. To isolate the contribution of iterative textual feedback from repeated generation and reward-based selection, we conducted an additional Best-of-N sampling experiment on all 4,210 e-commerce evaluation instances.

Best-of-N independently generates multiple query enhancements without iterative feedback and selects the highest-scoring candidate using the same PUBAO objective as our method. Each FD step requires two LLM calls, one for query generation and one for feedback generation, resulting in six calls over three steps. We compare three-step Reward-Guided FD against Best-of-6 under an identical budget of six LLM calls, using the same reward model and evaluation protocol.

Table R.9. Comparison under equal LLM-call budgets.

MethodLLM callsR1R_1R2R_2zˉ\bar z
Best-of-668.6548.4137.724
Reward-Guided FD (3-step)68.7018.4997.747

Reward-Guided FD achieves a paired satisfaction improvement of +0.023 [0.005, 0.042] (p=0.017) over Best-of-6, while also achieving higher R1R_1 and R2R_2. These results demonstrate that iterative, objective-grounded textual feedback provides additional benefits beyond independently generating candidates and selecting the best one using the same reward objective. Additional comparisons and implementation details are provided in our response to Reviewer yybD [W1, Q3]. We will include these additional results in the revised manuscript.

Reviewer_yybDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at query enhancement for personalized search when direct satisfaction feedback is rare but behavioral signals (clicks, carts, purchases, etc.) are plentiful. The authors build a probabilistic model linking user intent, product interpretations, behavior and satisfaction, and train a reward model from it. From this they define two objectives for scoring a rewritten query, PESO and PUBAO, and show the two differ by a log-partition term.

The main method is "reward-guided feedback descent". Candidate rewrites are ranked numerically by the reward model, and an LLM only turns the score differences into text feedback that guides the next rewrite. For latency, they precompute an offline pool of rewrites per persona and interpretation branch, and serving is just retrieval plus reward-based selection.

They test on a proprietary e-commerce dataset (satisfaction measured mainly by an LLM judge, plus some human studies) and on LongMemEval-M. They report small but consistent gains over pseudo-relevance feedback baselines and prompt-based variants on e-commerce, and better recall and QA accuracy than direct LLM rewriting on LongMemEval.

Critical strengths
  1. A useful way to use behavioral data in text optimization. Feedback descent normally needs a critic that writes critiques, and behavior logs don't give you that. Here the reward model picks the winner and the LLM only explains the difference. That avoids trusting the LLM's own sense of relevance, and avoids RL over scalar rewards. The PUBAO terms (plausibility, utility, drift) also give the feedback prompt a sensible structure. Proposition 2 is a short algebraic identity, so for me the value is in the design, not the theory.

  2. Some public validation. Many industry papers only report on private data. This one also uses LongMemEval-M, where the method gets the best Recall@5 and QA accuracy and keeps its lead with a shared reranker. It suggests the idea can transfer, though I have doubts about how strong this evidence is (see below).

Critical weaknesses
  1. The e-commerce evaluation is circular, and I'm not convinced the gains come from personalization or the feedback design. The satisfaction labels come from a GPT-5.2 judge whose rubric rewards properties of the rewritten query itself (e.g., "hooking attractiveness", "specificity"). The last stage of the reward model is trained on those labels, R1/R2 are computed with that same reward model, and the main metric is again the judge. So the system is largely scored against its own proxy. The extra evaluators in the appendix (LightGBM, XGBoost, kNN) help a bit, but they use the same labels and features. The oracle Rubric baseline beating PUBAO-FD on the judge score is expected. What bothers me more is that its R2 is lower than both PUBAO-FD and the raw query, so the reward model and the judge clearly disagree.

    The effect is also small (+0.17 SD after three steps). Persona-blind 1-step is almost the same as PUBAO-FD 1-step (7.702 vs 7.709), and Compare is close at three steps (7.712 vs 7.747), yet every significance test in Table 4 is against the original query only. So I don't think the paper shows that personalization or reward-grounded feedback matters. There is also no reward-selected best-of-N baseline on e-commerce, no direct online LLM baseline (although the introduction says the method beats one), and the human study leaves out Rubric, Persona-blind and Compare. In Table 6, humans agree about equally with PESO and PUBAO (nominally more with PESO), which doesn't back up the PUBAO > PESO claim.


  1. The code differs from the paper in ways that could affect the results. In scorer.py, PUBAO is computed only over the top-200 branches and every other branch gets the mean reward of those 200 (line 199). The paper doesn't mention this and doesn't ablate it. In fd_iter2.py and fd_iter3.py, the pair compared for feedback is always the raw query against the new rewrite, not the current best rewrite. The final query is then picked with a max over scores that came from different calls, each with its own top-200 set and its own imputed value, so I don't think those scores are directly comparable. The intent-level theory also isn't used in practice: Eq. 6 swaps the log-sum-exp branch reward for a conditional-mean proxy, and I couldn't find anything that enforces the support assumption.

  1. The LongMemEval result is overstated. There are only 200 test questions and no confidence intervals or seeds. Without the reranker, the QA gap to Direct-rewrite is 0.715 vs 0.705, which is two questions, and Direct-rewrite is better on Recall@1. The reward model is also trained on gold answer-session labels rather than behavioral signals, so this benchmark doesn't test the paper's main premise.
Questions for authors
  1. How sensitive are the results to the top-200 cutoff and the mean imputation in scorer.py, and are scores computed on different branch sets comparable in the final max selection?

  1. Can you report paired significance tests of PUBAO-FD against Persona-blind and Compare, since the difference to Persona-blind is only 0.007?

  1. Can you add a reward-selected best-of-N baseline and a direct online LLM rewrite on e-commerce, to show what the feedback loop adds beyond rejection sampling under the same reward model?

  1. In fd_iter2.py and fd_iter3.py, candidate A is the raw query rather than the incumbent. Do the reported 1-, 2- and 3-step results come from the procedure in Section 3.2 or from this code?
7 comments

[W1] Evaluation Independence and Human Validation

Thank you for raising this concern. Our evaluation extends beyond learned rewards and LLM-generated satisfaction scores through three human studies reported in Appendix B, covering offline query refinement, serving-time branch selection, and end-to-end query preference. Study 1 shows higher human agreement for PESO/PUBAO than RF-only. Study 2 reports a 69.5% non-tied human win rate for reward reranking over posterior-only selection. Study 3 finds PUBAO-FD most preferred (40.3%) in a blind four-way comparison with the original query, RM3, and Rocchio. We also evaluated 1,500 examples with Qwen3-235B, from a different model family. PUBAO-FD scores highest among the tested methods under both judges (Table R.1, Reviewer Cfoo [W2]). Together with human judgments, this supports improvements beyond the learned reward model and reduces concerns about judge-specific bias.

Regarding the Rubric baseline, the difference arises from what each metric evaluates. Rubric selects candidates using the reference satisfaction rubric, which considers query quality and available behavioral outcomes, whereas R2R_2 evaluates the selected interpretation using the learned reward under the original user–query context. Thus, a rewrite preferred by Rubric may select a branch with lower R2R_2, even compared with the original query. Nevertheless, the learned reward remains positively correlated with held-out satisfaction labels (Table 1), indicating that the two signals are related despite differences in how they evaluate individual enhancements.

[W1, Q2] Statistical Comparison with Strong Baselines

Thank you for highlighting the importance of direct comparisons with strong baselines. We conducted additional paired significance tests between PUBAO-FD and competing methods, rather than relying only on comparisons against the original query.

Table R.2. Paired satisfaction comparisons on the e-commerce dataset.

ComparisonΔzˉ\Delta\bar z95% CIPaired pp
PUBAO-FD (1-step) vs. Persona-blind (1-step)+0.007[-0.021, 0.034]0.60
PUBAO-FD (1-step) vs. Static (1-step)+0.081[0.055, 0.107]<1e-8
PUBAO-FD (3-step) vs. Text-feature (3-step)+0.107[0.081, 0.133]<1e-13
PUBAO-FD (3-step) vs. Persona-blind (3-step)+0.038[0.021, 0.055]<1e-5
PUBAO-FD (3-step) vs. Compare (3-step)+0.034[0.005, 0.061]0.020

PUBAO-FD significantly outperforms the closely related three-step Compare baseline (p=0.020p=0.020), supporting the benefit of reward-guided refinement. While the one-step difference from Persona-blind is small and not statistically significant, the three-step comparison shows a clearer advantage. This suggests that the benefit of persona conditioning becomes more apparent over successive refinement steps.

Regarding Table 6, the human study measures agreement on individual pairwise query selections, whereas Table 3 evaluates the quality of queries after iterative refinement. Thus, PESO's slightly higher human agreement does not necessarily imply better final enhancement performance, where PUBAO performs better. We will include the additional paired significance results and clarify this distinction in the revised manuscript.

[W1, Q3] Best-of-N Sampling and Direct LLM Rewriting

Thank you for this suggestion. To isolate the contribution of iterative textual feedback from repeated generation and reward-based candidate selection, we conducted additional comparisons with direct LLM rewriting and Best-of-N sampling. Best-of-N independently generates candidates without feedback, then selects the best using the same PUBAO objective. Each FD step uses two LLM calls (challenger and feedback), matching Best-of-2 at one step and Best-of-6 at three steps.

Table R.3. Comparison with direct LLM rewriting and call-matched Best-of-N sampling.

MethodLLM callsR1R_1R2R_2zˉ\bar z
Direct LLM Rewriting18.4378.0777.019
Best-of-228.6178.3847.684
Reward-Guided FD (1-step)28.6448.4317.709
Compare (3-step)68.5928.3107.712
Best-of-668.6548.4137.724
Reward-Guided FD (3-step)68.7018.4997.747

Reward-Guided FD achieves higher scores than Best-of-2 at one step and a paired satisfaction improvement of +0.023 [0.005, 0.042] (p=0.017p=0.017) over Best-of-6 at three steps, while also achieving higher R1R_1 and R2R_2. It also outperforms Direct LLM Rewriting and the three-step Compare baseline. Since Best-of-N uses the same PUBAO objective for candidate selection, this comparison supports the benefit of iterative, objective-grounded textual feedback beyond independent generation and reward-based selection. We will include these additional comparisons in the revised manuscript.

[W2, Q1] Top-200 Approximation and Score Comparability

Thank you for examining the scoring implementation. The top-200 cutoff limits Random Forest reward predictions, not the interpretation space: remaining rewards are imputed using the mean of scored branches, while the reward-tilted distribution, entropy, and KL divergence are computed over all branches. We recomputed rewards for all 6,348 branches and repeated candidate ranking and selection without imputation.

Table R.4. Sensitivity to the top-200 scoring approximation.

MetricResult
Candidate-ranking Kendall's τ\tau0.9994
Final candidate selection agreement99.93%
Maximum change in reported Table 2 metrics0.009
Mean reward-tilted probability mass outside top 2000.07%

As in Table 4, full-branch scoring yields nearly identical rankings and selections, indicating little effect on candidate comparability. We will clarify this implementation in the revised manuscript.

[W2, Q4] Consistency Between the Reported Procedure and Submitted Code

Thank you for identifying this discrepancy in the supplementary code. The reported results follow the incumbent-based procedure in Section 3.2, which compares each challenger with the current best rewrite. During anonymization, we inadvertently submitted iteration-2/3 scripts that instead compare challengers against the original query. This error did not affect the reported results. We evaluated both procedures on all 4,210 instances.

Table R.5. Manuscript results and submitted-code variant.

StepReported zˉ\bar zSubmitted-code variant zˉ\bar z
17.7097.709
27.7417.735
37.7477.745

The submitted-code variant produces similar aggregate satisfaction scores, although its feedback comparison differs from the procedure used for the reported results. We will replace the incorrect scripts and clarify the discrepancy in the revised supplementary materials.

[W2] Theoretical Formulation and Practical Approximation

We clarify the relationship between our intent-level theory and practical implementation. Since the latent-intent log-sum-exp branch reward is unobserved, our two-stage predictor uses conditional-mean satisfaction as a tractable proxy. By Jensen's inequality, this conditional mean lower-bounds the theoretical branch reward, which is upper-bounded by the log-sum-exp of conditional satisfaction scores. Empirically, the median estimated Jensen bound gap is 0.15 points for selected branches, with candidate-ranking Kendall's τ=0.900\tau=0.900, 89.2% final selection agreement, and a mean reference satisfaction difference below 0.01 (Table R.8, Reviewer rcdX [W2, Q2]).

Regarding semantic faithfulness, the practical KL penalty encourages but does not guarantee theoretical support containment. Compared with λ=0\lambda=0, the reported λ=0.1\lambda=0.1 reduces reward-tilted mass on low-original-probability branches from 0.127 to 0.050 and semantic drift from 18.5% to 13.3% (Reviewer Cfoo [W3]). We will clarify these theoretical–practical distinctions in the revised manuscript.

[W3] LongMemEval Evaluation and Statistical Robustness

Thank you for highlighting the importance of statistical uncertainty and experimental robustness in the LongMemEval evaluation. To examine the variability of our results, we evaluated three seeds under the same experimental settings. We used the same held-out 200-question split.

Table R.6. LongMemEval results.

MethodRecall@1Recall@5NDCG@5Recall@10QA accuracy
Direct-rewrite0.199 ±\pm 0.0180.716 ±\pm 0.0080.761 ±\pm 0.0030.811 ±\pm 0.0160.690 ±\pm 0.030
Reward-Guided FD0.197 ±\pm 0.0140.758 ±\pm 0.0070.780 ±\pm 0.0140.821 ±\pm 0.0230.715 ±\pm 0.005
Direct-rewrite + shared reranking0.228 ±\pm 0.0090.817 ±\pm 0.0120.857 ±\pm 0.0150.878 ±\pm 0.0100.735 ±\pm 0.005
Reward-Guided FD + shared reranking0.234 ±\pm 0.0070.827 ±\pm 0.0130.863 ±\pm 0.0130.882 ±\pm 0.0120.760 ±\pm 0.017

Across seeds, Reward-Guided FD achieves higher mean Recall@5 and QA accuracy with and without shared reranking, while Direct-rewrite remains slightly better on first-stage Recall@1. QA gains are positive in all three seeds. We will report these results and moderate our performance claims.

Regarding supervision, LongMemEval uses answer-session relevance annotations as a benchmark-specific scalar reward target, corresponding to the role of zz in our framework. Unlike e-commerce, it does not provide the hierarchical behavioral signals y1:Ly_{1:L}. Thus, LongMemEval evaluates reward-guided refinement under a different supervision source, while the e-commerce experiments directly examine the contribution of behavioral supervision.

(Part 2 of 2)

Partly addressed (continued)

  1. Judge independence. The Qwen3 cross-check is useful. Does it use the same rubric (Figure 2)? If so, it removes model-family bias but not the circularity I raised, because the rubric still scores properties of the rewritten query itself. The human studies help, but Study 3 compares only against the original query, RM3 and Rocchio. None of them includes Persona-blind, Compare or Best-of-N, which are exactly the baselines where the automatic gaps are smallest.

    I accept the explanations of the Rubric R2 gap and of Table 6. Together, though, they mean the learned reward and the judge measure somewhat different things, and the human data don't separate PESO from PUBAO. So the PUBAO > PESO claim rests on the automatic metrics alone. Both points are worth stating in the paper.

  2. Code mismatch (Q4). Thank you for the explanation and Table R.5. Because the reported results come from scripts that are not in the supplement, I can't verify them. Please make the corrected incumbent-based scripts available, either as an anonymous link now or in the camera-ready release. I also note that the two procedures give almost the same numbers (7.745 vs 7.747). This suggests the final outcome is not very sensitive to which pair the feedback is based on, which is worth a sentence in the paper.

  3. LongMemEval variance. The three-seed results are helpful (the QA gain is positive in every seed), and I appreciate the commitment to moderate the claims. Two points remain. First, the reranked QA mean for Reward-Guided FD is 0.760, while Table 5 reports 0.780, so the reranked QA gap over Direct-rewrite shrinks from 0.050 to 0.025. Please report the means in Table 5. Second, seeds capture generation randomness but not variance over the 200 questions. A paired bootstrap over questions would show whether the QA gap of 0.025 (about 5 questions) is reliable.

Overall

The rebuttal largely resolves my implementation concerns (pending the corrected scripts) and adds the baselines I asked for. My main reservation is unchanged: the gains over the strongest baselines are real but small, and the evidence that personalization matters rests on one small multi-step effect. These were the tests and baselines my review asked for. They show the advantage holds up, but they also show it is small, so I will keep my score of 3 (weak accept) rather than raise it.

Reviewer_CfooOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reward-Guided Feedback Descent for personalized query enhancement under sparse satisfaction feedback. It combines behavioral reward modeling with LLM-based iterative refinement and an offline enhancement pool. Experiments on e-commerce data and public benchmarks demonstrate improvements in query quality, retrieval performance, and estimated user satisfaction.

Critical strengths
  1. The paper addresses personalized query enhancement under sparse user satisfaction feedback, a problem with clear practical significance.

  2. The experiments cover real-world e-commerce data and public benchmarks, with ablation studies and human evaluations supporting the effectiveness of the proposed method.

  3. The method combines behavioral reward modeling with LLM-based textual feedback optimization and employs an offline enhancement pool to reduce online inference costs.

Critical weaknesses
  1. The method largely extends the existing Feedback Descent framework, with limited methodological novelty. Additional comparisons are needed to establish the independent contribution of the proposed feedback mechanism.

2.The satisfaction evaluation relies on LLM-generated labels with criteria overlapping the optimization objectives, raising concerns about evaluation independence and actual improvements in user satisfaction.

  1. The practical implementation deviates from the theoretical formulation in reward estimation and semantic faithfulness, with insufficient analysis of their impact.
Questions for authors

Please refer to the Weaknesses

2 comments

[W1] Methodological Contribution

Thank you for the opportunity to clarify our methodological contribution. Our contribution goes beyond applying Feedback Descent to query rewriting. While Feedback Descent provides a general framework for iterative text optimization, our work addresses a distinct challenge: constructing actionable optimization feedback when direct satisfaction supervision is sparse and textual critiques are unavailable.

Specifically, we leverage hierarchical behavioral signals to estimate personalized utility and derive PESO/PUBAO, which provide numerical candidate preferences and decomposed, objective-grounded feedback through posterior sharpening (PESO), branch plausibility and reward-aligned utility (PUBAO), and their shared semantic-faithfulness term. Rather than relying on the LLM to determine candidate preferences, our objectives determine the preferred candidate numerically, while the LLM translates objective differences into actionable refinement guidance. This also enables offline construction of reusable persona–branch enhancements without serving-time LLM inference.

To further validate this contribution, we conducted additional comparisons against conventional LLM-based refinement and reward-based candidate sampling. As shown in Table R.2 (Reviewer yybD [W1, Q2]), three-step PUBAO-FD outperforms the closely related Compare (3-step) baseline by +0.034 [0.005, 0.061] (p=0.020p=0.020) in paired satisfaction. The three-step comparison with Persona-blind also shows an improvement of +0.038 [0.021, 0.055] (p<10−5p<10^{-5}).

Furthermore, under an equal budget of six LLM calls, PUBAO-FD outperforms Best-of-6 sampling, which independently generates candidates and selects the best one using the same PUBAO objective, by +0.023 [0.005, 0.042] (p=0.017p=0.017) (Table R.3, Reviewer yybD [W1, Q3]; Table R.9, Reviewer rcdX [W3, Q3]). These results support the contribution of objective-grounded textual feedback beyond conventional LLM-based refinement and reward-based candidate selection. We will incorporate these additional comparisons and clarify our methodological contribution in the revised manuscript.

[W2] Evaluation Independence and Human Validation

Thank you for highlighting the importance of evaluation independence. Our evaluation also includes evidence beyond learned rewards and LLM-generated satisfaction scores. As reported in Appendix B, we conducted three human studies covering offline query refinement, serving-time branch selection, and end-to-end query preference, providing complementary evidence from human judgments collected independently of reward-model training and the GPT-5.2 satisfaction judge.

Specifically, Study 1 reports higher agreement with human query preferences for PESO/PUBAO than for RF-only selection. Study 2 demonstrates that reward-based branch selection achieves a 69.5% win rate among non-tied human comparisons against posterior-only selection, while Study 3 shows that PUBAO-FD receives the highest share of preferences (40.3%) in a blind four-way comparison against the original query, RM3, and Rocchio.

To further examine potential judge-specific bias, we additionally evaluated 1,500 examples using Qwen3-235B, a model from a different family than GPT-5.2.

Table R.1. Cross-judge satisfaction scores on 1,500 examples.

MethodGPT-5.2Qwen3-235B
Original query7.6058.383
BM25 + RM37.6648.481
Rocchio-PRF7.6618.391
Persona-aware PRF7.6808.494
PUBAO-FD7.7318.504

PUBAO-FD remains the highest-scoring method among the evaluated alternatives under both judges, improving over the original query under both evaluation models. Together, the human judgments and cross-family evaluation provide complementary evidence for the quality of the selected queries and interpretations, while showing that the satisfaction-score improvements are not specific to the original GPT-5.2 judge. We will highlight these human evaluation results and incorporate the cross-family judge analysis in the revised manuscript.

[W3] Theoretical Formulation and Practical Implementation

Thank you for highlighting the gap between our theoretical formulation and practical implementation. The latent-intent formulation establishes how tractable interpretation-level objectives relate to the underlying intent-level goals. In particular, Proposition 1 decomposes the intent-level PESO into its interpretation-level counterpart and a residual capturing within-branch uncertainty and distributional mismatch. This clarifies both the motivation for branch-level optimization and the information it does not capture.

Reward approximation.

Our two-stage predictor does not directly compute the theoretical log-sum-exp branch reward, which requires the unobserved within-branch intent distribution and intent-specific rewards. Instead, it estimates conditional-mean satisfaction from abundant behavioral signals and sparse satisfaction labels. This proxy is theoretically motivated: Jensen's inequality shows that the conditional mean lower-bounds the theoretical branch reward, which is itself upper-bounded by the log-sum-exp of conditional satisfaction scores. Using the learned proxy consistently also preserves the algebraic relationship between the practical PESO and PUBAO objectives.

We further evaluated this approximation using Random Forest proximity-based estimates of the conditional satisfaction distribution. The estimated Jensen bounds have a median gap of 0.15 points on the 1–10 reward scale for the selected branches. Mean-based and log-sum-exp-based scoring agree on the final query in 89.2% of the 4,210 evaluation instances, with a difference below 0.01 in mean reference satisfaction. These results support the practical use of the conditional-mean proxy without assuming exact recovery of the latent-intent reward. The theoretical bounds, estimation procedure, and detailed results are provided in our response to Reviewer rcdX [W2, Q2, Table R.8].

Semantic faithfulness.

The support-containment condition in Section 2.3 is a theoretical assumption, while the practical KL penalty encourages faithfulness without strictly enforcing it. To assess its effect, we varied λ\lambda over the same candidate queries across all 4,210 e-commerce instances, keeping γ=1.5\gamma=1.5 fixed. Compared with λ=0\lambda=0, the reported setting (λ=0.1\lambda=0.1) reduces the mean reward-tilted probability mass assigned to branches with negligible probability under the original query (<10−6<10^{-6}) from 0.127 to 0.050. A separate semantic evaluation also shows fewer selected rewrites that change product categories or contradict the original query (18.5% to 13.3%), while mean reference satisfaction remains unchanged (7.744 vs. 7.744, p=0.85p=0.85).

Together, these analyses support the conditional-mean reward as a practical approximation and the KL penalty as an effective regularizer for reducing semantic drift. We will clarify their relationship to the theoretical formulation and incorporate the additional results in the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript Revision Update Following the Rebuttal

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their constructive feedback and careful engagement during the rebuttal period. We have uploaded a revised manuscript and supplementary material incorporating the additional experiments, analyses, and clarifications discussed in our responses. All revisions are highlighted in blue.

The main updates include:

  1. Additional Baselines: Direct LLM Rewriting, compute-matched Best-of-N, and paired statistical comparisons with strong baselines.
  2. Evaluation Robustness: Cross-family and rewrite-hidden evaluations, along with expanded human-evaluation analyses.
  3. Theory and Implementation: Clarifications and sensitivity analyses for the reward approximation, top-200 scoring, and semantic faithfulness.
  4. LongMemEval: Three-seed results and question-level bootstrap analysis. We plan to extend the three-seed evaluation to the remaining baselines and update Table 5 accordingly in the final version.

We believe these revisions strengthen the empirical evidence, improve reproducibility and transparency, and more clearly establish the scope and limitations of our contributions.

We sincerely appreciate the reviewers' time and valuable suggestions.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.