- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper trains a natural-language user-profile generator using a groundedness reward based on cohesion, alignment, and truthfulness, together with cluster-size and coverage regularizers. Groupwise DPO uses only past behavior. Across three datasets and several retrieval, ranking, and personalized-search settings, the resulting profiles improve downstream performance and often match profiles trained with future-interaction supervision.
Critical strengths
-
The problem formulation is useful. Optimizing user representations without requiring future-interaction labels could make profile learning more reusable across downstream tasks. The paper clearly separates the retrospective training signal from the future-facing evaluation.
-
The comparisons control several important confounders. The methods share the backbone, sampled rollouts, and optimization pipeline, allowing a more focused comparison of reward functions. Evaluation on disjoint users and temporal windows, together with checkpoint selection based on groundedness, strengthens the claim that future labels are unnecessary for the proposed training procedure.
-
The evaluation covers multiple consumers and provides some informative decomposition. Gains are tested across different retrievers, rerankers, and two generator families. The observation that truthfulness contributes more to retrieval while alignment contributes more to fine-grained ranking is potentially useful, as is the complementary benefit of combining groundedness with downstream supervision.
Critical weaknesses
-
The central explanation is not sufficiently isolated. Groundedness optimization simultaneously changes profile faithfulness, profile count, evidence grouping, coverage, and total length. Removing reward components demonstrates their contribution to the complete procedure, but does not establish whether faithfulness itself explains the gains over simpler evidence-preserving summarization or improved compression.
-
The groundedness measurement lacks independent validation after optimization. Agreement among three LLM judges on training rollouts is reassuring, but agreement is not equivalent to factual correctness. Since the policy is explicitly optimized against these scores, an independent audit of optimized test profiles is particularly important for assessing whether the reward improves grounding or favors judge-preferred phrasing.
-
The comparison with downstream supervision supports a narrower conclusion than the framing suggests. RLPF uses four-way prediction with random negatives, while LettinGo uses sentiment classification with random negatives. These are specific proxy rewards and may be poorly aligned with large-pool retrieval or hard-negative ranking. Matching them does not establish that groundedness is comparable to stronger task-aligned downstream supervision.
Questions for authors
-
How does the method compare with extractive or evidence-preserving summaries under matched profile-count and token budgets?
-
Do independent human judgments confirm improved groundedness on optimized test profiles?
-
Would the relative advantage persist against a downstream reward using hard negatives or the actual evaluation ranker?
4 comments
Author Response #2
(2) Independent validation on optimized test profiles (W2, Q2)
We evaluate groundedness of the optimized profiles on the MerRec test set in two ways with external judges, and a human audit is currently in progress a human audit is reported in Author Response #3 below.
(2-a) External judge on the same criteria. To separate the effect of training from any bias of the training judges, we score alignment and truthfulness against the behavioral units with GPT-5.4-mini, using the same judge prompts.
(2-b) Faithfulness against the raw log. We also score each profile directly against the raw interaction log. Raw logs are much larger than their behavioral units (aggregation compresses raw events into about 31.5% as many units), so we sample 500 test instances shared across all methods to keep the evaluation cost manageable. Two judges from different model families (GPT-5.4-mini and Claude Sonnet 5.5) score each profile as 1.0 (clearly supported), 0.5 (partially supported or overstated), or 0.0 (unsupported), and we average their scores.
| Method | (2-a) Alignment | (2-a) Truthfulness | (2-b) Raw-log faithfulness |
|---|---|---|---|
| Base | 0.941 | 0.628 | 0.722 |
| RLPF | 0.959 | 0.666 | 0.758 |
| LettinGo | 0.960 | 0.674 | 0.742 |
| Ours | 0.971 | 0.755 | 0.824 |
Under the external judge, Ours scores highest on both criteria, with the largest margin on truthfulness (0.755 vs. 0.628 for Base), while RLPF and LettinGo improve much less (0.666 and 0.674). Against the raw log, our training improves faithfulness over Base by 0.102, compared to +0.036 for RLPF and +0.020 for LettinGo, and Ours is significantly higher than all other methods (paired t-test, p < 0.001). Both judges independently rank Ours first. Since these judges differ from the training judges and the raw-log criterion differs from the training reward, the gains are unlikely to reflect judge-preferred phrasing alone. These results are now in the revised manuscript (Sec. 5, Groundedness of Optimized Profiles, Table 5; judge prompt in App. A.9.3 App. A.10.3).
(2-c) Human audit (in progress). (done; results in Author Response #3 below) Two independent annotators with expertise in search and recommendation are reviewing a 100-instance subset of these 500 instances. Through a web interface we deployed, each annotator sees a user's raw input log alongside the profiles from each method, with method identities masked, and judges whether each profile goes beyond the evidence. We will share the results, along with human-LLM agreement, within the discussion period. We hope the LLM judge results in (2-a, 2-b) partially address your concern in the meantime. The results are now reported in Author Response #3 below.
We will address W3 (+ Q3) in a separate response shortly. W3 (+ Q3) is addressed in Author Response #3 below.
Author Response #3: Human audit results (W2, Q2) and hard-negative downstream reward (W3, Q3)
As promised in (2-c) and at the end of Author Response #2, we report the human audit results and the W3 (+ Q3) response below.
(2-c) Human audit results (W2, Q2)
Setup. Two human annotators with expertise in search and recommendation reviewed a 100-instance subset of the 500 raw-log evaluation instances. Through the web interface (App. A.5, Figure 8), each annotator sees the raw input log and the generated profile set from each method, with method identities masked and presentation order randomized. Each profile is rated 1.0 (clearly supported), 0.5 (partially supported or overstated), or 0.0 (unsupported).
Results. Mean raw-log faithfulness, with the LLM judge from (2-b) for reference:
| Method | LLM judge (500 inst.) | Annotator 1 | Annotator 2 | Human mean |
|---|---|---|---|---|
| Base | 0.722 | 0.607 | 0.430 | 0.518 |
| RLPF | 0.758 | 0.654 | 0.442 | 0.548 |
| LettinGo | 0.742 | 0.677 | 0.442 | 0.560 |
| Ours | 0.824 | 0.765 | 0.522 | 0.643 |
Both annotators rate Ours highest. The paired advantage of Ours is significant against every method for each annotator (paired t-test, p < 0.001):
| Gain of Ours over | Annotator 1 | Annotator 2 |
|---|---|---|
| Base | +0.158 | +0.092 |
| RLPF | +0.111 | +0.079 |
| LettinGo | +0.088 | +0.080 |
Training with the groundedness reward improves faithfulness judged directly against raw logs by human annotators, over Base and over both downstream-supervised controls. The absolute levels differ between annotators (Annotator 2 is stricter; per-annotator rating distributions are in App. A.5, Table 11), but the ordering of methods is the same.
Human-LLM agreement. We measure pairwise preference agreement, i.e., how often two raters prefer the same method in a method pair (random agreement is 0.5).
| Rater | vs. other annotator | vs. Claude Sonnet 5.5 | vs. GPT-5.4-mini |
|---|---|---|---|
| Annotator 1 | 0.70 | 0.70 | 0.72 |
| Annotator 2 | 0.70 | 0.74 | 0.72 |
Human-LLM agreement (0.70-0.74) is comparable to agreement between the two annotators (0.70). At the method level, all four raters rank Ours first and Base last or tied for last. The ordering of RLPF and LettinGo in the middle varies across raters, and their human means differ by only 0.012.
These results are now in the revised manuscript: Sec. 5 Groundedness of Optimized Profiles, App. A.5 Human Evaluation Details.
(3) Stronger downstream supervision (W3, Q3)
To examine whether a harder prediction task makes downstream supervision more effective, we additionally train RLPF on a variant of its four-way task in which the random distractors are replaced by hard negatives constructed in the same way as the L2 ranking slates (App. A.2.1). For each target item in the training set, we sample three negatives from the hard negative pool (BM25 top-50 and BGE-M3 top-50 retrieved with the positive item as the query). All other components are identical. We evaluate on MerRec and report the change relative to Base.
| vs. Base | L1 (E5) | L1 (BGE) | L2 (E5) | L2 (Q3-R) | L2 (LRL) |
|---|---|---|---|---|---|
| RLPF-HN | −0.0026 | −0.0034 | +0.0005 | +0.0027‡ | +0.0044‡ |
(‡: significant improvement over Base, paired t-test, p < 0.001; no mark: not significant.)
Relative to Base, hard-negative RLPF yields no improvement on L1 retrieval, and its gains on L2 ranking are small (not significant with E5). These gains are smaller than those of RLPF with random negatives and of Ours (Table 4). This is consistent with hard negatives emphasizing fine-grained discrimination among similar items, as in L2 ranking. Thus, making the downstream task harder with hard negatives (RLPF-HN) does not yield large gains. The experiment is in App. A.7 (Table 13).
All revisions are reflected in the manuscript and highlighted in blue. We hope these additions address your concerns, and we would be glad to discuss any remaining questions. Thank you again for your time and constructive feedback.
Thanks a lot for the detailed rebuttal, which addresses most of my previous concerns. I'll raise my score to 3.
Author Response #1
Thank you for the thoughtful review. Since each weakness (W#) has a corresponding question (Q#), we address them together.
(1) Isolating groundedness from profile structure with budget-matched comparison (W1, Q1)
To isolate groundedness from profile structure (total length, coverage, and number of profiles), we re-analyze the MerRec results in Table 4. For each test instance, we compute the paired score difference between Ours and each method on the same user, input window, and target items, and apply ANCOVA (analysis of covariance) on these paired differences:
Δscore = β₀ + β₁·Δwords + β₂·Δcoverage + β₃·Δ#profiles + ε
The intercept β₀ is the expected advantage of Ours when the two profiles have the same length, coverage, and number of profiles. Collinearity among the covariates is low (VIF ≤ 2.7), so β₀ is reliably estimated.
(* p < 0.05, ** p < 0.01, *** p < 0.001)
In addition, among these paired instances, we keep only the 25% where the two methods' profiles are already most similar in structure (within this subset, the profiles differ on average by only 13–20 words and 0.5 or fewer profiles, versus 46–65 words and 0.7–1.6 profiles over all instances). The table below reports the mean paired difference (Ours − method) on this subset.
(* p < 0.05, ** p < 0.01, *** p < 0.001)
Both analyses agree and follow the same pattern as the main results in Table 4. On L1, Ours remains ahead of Base and both downstream-supervised methods. On L2, Ours remains ahead of Base under all three rankers and is comparable to RLPF and LettinGo (no significant paired difference), without using any future labels. As expected, Ours+RLPF shows no significant difference from Ours in most settings, and is slightly better on L2 (E5) as it additionally uses future labels. These results suggest that the gains of Ours largely come from how its profiles describe the user, rather than from their structure. These analyses are included in Sec. 5 (Budget-Controlled Comparison,
Table 7Table 8) andApp. A.8 (Table 11)App. A.9 (Table 14).Regarding extractive or evidence-preserving summaries, Base is the closest control. It compresses the same behavioral units into a few profiles using a prompt that requires each profile to cite its supporting units and to avoid exaggerated or overly abstract descriptions (
App. A.9.1App. A.10.1). All methods share this prompt and pipeline, differing only in whether and how they are trained. However, prompting alone does not ensure faithful compression. As shown in App. A.1, the same prompt yields unsupported claims not only from our backbone but also from frontier models such as Claude Sonnet 5 and GPT-5.1. This suggests that directly training for groundedness is what keeps Ours ahead of Base even when matched in structure.