- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies personalized query enhancement when satisfaction feedback is sparse and behavioral signals provide indirect supervision. It develops a hierarchical satisfaction model and reward-tilted intent distributions, from which it derives two enhancement objectives, PESO and PUBAO, balancing intent refinement or utility with faithfulness to the original query. The proposed reward-guided feedback descent algorithm selects candidate rewrites numerically using these objectives and uses an LLM to verbalize objective differences into feedback for further refinement. For latency-sensitive serving, it constructs an offline enhancement pool indexed by prototypical personas and interpretation branches, then retrieves and selects candidates using a learned reward model. Evaluation combines a proprietary offline e-commerce study, using LLM-derived reference satisfaction scores and supplementary human assessments, with a held-out LongMemEval-M study. The reported results show improvements in estimated satisfaction, evidence retrieval, and downstream QA accuracy, supported by reward-supervision and refinement-objective ablations; the e-commerce results do not establish online user-satisfaction gains.
Critical strengths
-
The paper presents a coherent framework that connects sparse satisfaction labels, abundant behavioral signals, and iterative query refinement.
-
The evaluation is broad and deployment-aware, covering a proprietary e-commerce setting, two public benchmarks, human judgments, statistical tests, and several ablations.
-
Separating numerical candidate selection from textual feedback generation is a practical and well-motivated design choice.
Critical weaknesses
-
The main e-commerce evaluation relies heavily on GPT-5.2-generated satisfaction labels, which are used both to train the reward model and to report the primary satisfaction gains. Since GPT-5.2 also generates the rewrites, model-specific preference may inflate the reported improvements. The human studies help, but they evaluate query or branch preference rather than actual user satisfaction.
-
There is a gap between the theoretical formulation and the implementation. The theoretical branch reward is a log-sum-exp over latent intents, whereas the implementation substitutes a conditional-mean Random Forest prediction. The propositions are valid under the definitions, but the experiments do not directly validate several key assumptions or this substitution.
-
The experiments do not fully isolate the benefit of iterative textual feedback from reward-based candidate search. A compute-matched baseline that samples the same number of rewrites and selects the best one using PUBAO would clarify whether feedback descent contributes beyond repeated generation and reward selection.
Questions for authors
-
Can the authors evaluate the main satisfaction gains using independently collected human labels or online outcomes rather than the same LLM-derived target used for reward-model training?
-
How well does the practical conditional-mean reward proxy preserve candidate rankings induced by the theoretical branch reward?
-
Under an equal LLM-call budget, how does PUBAO-guided feedback descent compare with sampling multiple independent rewrites and selecting the highest-PUBAO candidate?
2 comments
[W3, Q3] Comparison with Best-of-N Sampling
Thank you for this suggestion. To isolate the contribution of iterative textual feedback from repeated generation and reward-based selection, we conducted an additional Best-of-N sampling experiment on all 4,210 e-commerce evaluation instances.
Best-of-N independently generates multiple query enhancements without iterative feedback and selects the highest-scoring candidate using the same PUBAO objective as our method. Each FD step requires two LLM calls, one for query generation and one for feedback generation, resulting in six calls over three steps. We compare three-step Reward-Guided FD against Best-of-6 under an identical budget of six LLM calls, using the same reward model and evaluation protocol.
Table R.9. Comparison under equal LLM-call budgets.
| Method | LLM calls | |||
|---|---|---|---|---|
| Best-of-6 | 6 | 8.654 | 8.413 | 7.724 |
| Reward-Guided FD (3-step) | 6 | 8.701 | 8.499 | 7.747 |
Reward-Guided FD achieves a paired satisfaction improvement of +0.023 [0.005, 0.042] (p=0.017) over Best-of-6, while also achieving higher and . These results demonstrate that iterative, objective-grounded textual feedback provides additional benefits beyond independently generating candidates and selecting the best one using the same reward objective. Additional comparisons and implementation details are provided in our response to Reviewer yybD [W1, Q3]. We will include these additional results in the revised manuscript.
We agree that independent human satisfaction labels or online outcomes would provide more direct validation. As the reviewer correctly noted, our current human studies assess query and interpretation preferences, rather than user satisfaction from actual use. Establishing satisfaction gains in a deployed service would require a prospective user study or an online A/B test, ideally including direct user-reported satisfaction. Such an evaluation was not conducted in this study and remains an important step toward deployment. Nevertheless, the current experiments provide meaningful evidence for the proposed approach: human studies support the quality of its query-refinement and branch-selection decisions (Appendix B), while the additional evaluation with the Qwen3-235B judge shows that the estimated satisfaction improvements are not confined to the original GPT-5.2 judge (Table R.1 of our response to Reviewer Cfoo). Together, these results support the effectiveness of the proposed approach in the offline research setting considered in this study. We thank the reviewer for this constructive and insightful feedback and will prioritize direct validation of user satisfaction as we move toward deployment.
Thank you for raising this question. As defined in Section 2.2, the theoretical branch reward aggregates latent-intent rewards through log-sum-exp:
\log \mathbb{E}_{h \sim \pi_{u,q}(\cdot\mid b)} \left[\exp(\gamma R_{u,q}(h))\right].$$ Since latent intents are unobserved, our two-stage predictor estimates **conditional-mean satisfaction as a practical proxy**. This choice is **theoretically motivated**: under our probabilistic formulation, Jensen's inequality gives $$\underbrace{\mathbb{E}[z\mid q,D_u,b]}_{\text{Lower bound}} \leq R^{\mathcal B}_{u,q}(b) \leq \underbrace{ \frac{1}{\gamma} \log\mathbb{E}_{z\mid q,D_u,b} \left[\exp(\gamma z)\right] }_{\text{Upper bound}}.$$ Thus, under the assumed probabilistic model, the **theoretical reward lies between the population conditional mean and the log-sum-exp of conditional satisfaction scores**. Our Random Forest estimates the former, while the proximity-weighted satisfaction distribution provides an empirical estimate of the latter. To evaluate this approximation, we estimated both bounds using the same fitted Random Forest. We constructed a **conditional satisfaction distribution** using Random Forest proximity weights, which assign higher weights to training samples that more frequently share terminal leaves with each query–user–branch input. The proximity-weighted mean closely matches the original RF predictions (Pearson 0.991, MAE 0.047), while the estimated distribution achieves a 37% lower ranked probability score (RPS) than an unconditional baseline on held-out data. **Table R.8. Evaluation of the branch-reward approximation ($\gamma=1.5$).** | Metric | Result | |-|-| | Median estimated Jensen bound gap (selected branch) | 0.15 | | Candidate-ranking Kendall's $\tau$ | 0.900 | | Final candidate selection agreement | 89.2% | | Difference in mean reference satisfaction | <0.01 | The median estimated gap between the Jensen bounds is 0.15 points, or 1.7% of the 1–10 reward range, for the selected branches. Across 4,210 evaluation instances, mean-based and log-sum-exp-based scoring produce **highly similar candidate rankings** and select the same query in 89.2% of cases, with a difference below 0.01 in overall mean reference satisfaction. Since latent intents are unobserved, this comparison evaluates an estimated Jensen-bounded alternative rather than the exact theoretical reward. The small estimated bound gap and high agreement between mean-based and log-sum-exp-based scoring support the **practical robustness of the conditional-mean proxy**. We will clarify this theoretical relationship and include the additional analysis in the revised manuscript.