- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether personalized output quality adequately reflects the quality of user representations. It proposes evaluating explicit user states through recoverability, stability, and predictivity. The authors introduce HUMANLM-PROBE and PERSONA-LR, which updates user profiles through behavioral scoring and alternating fine-tuning. Experiments report improvements in persona recovery and several personalization tasks.
Critical strengths
- The paper distinguishes personalized output quality from user representation quality and introduces HUMANLM-PROBE as a concrete framework for evaluating the recoverability, stability, and predictivity of user states.
- PERSONA-LR uses feedback from behavior prediction to update explicit user profiles, connecting persona inference with behavior modeling. It improves persona recovery and performance on several downstream personalization tasks.
Critical weaknesses
-
The evidence for the central claim is not sufficiently direct. Figure 3 compares generation and recovery scores across different datasets, making it difficult to rule out differences in task capabilities. The shuffling experiments in Tables 5 and 7 show that user information is useful, but do not demonstrate reuse of the same state across contexts. Table 6 also lacks baselines under the same conditions. It therefore remains unclear whether the method builds more stable user models beyond improving persona recovery scores.
-
The contribution of repeated refinement remains unclear. Section 5.4 mentions an ablation using a single refinement step without cache updates, but provides no numerical results. It is also unclear whether the SFT-only baseline uses a matched training budget. In Table 3, this baseline has a PANDORA parse rate of 0%, which confounds the comparison with differences in output formatting ability. Comparisons with matched budgets and output constraints are needed to establish whether iterative refinement accounts for the gains.
Questions for authors
- If a user state is inferred once and then kept fixed, does it still outperform SFT-only in other held-out contexts?
- Can the authors provide results for the single-step, no-refresh ablation with matched training budgets and output constraints?
- How are the history views in Table 6 sampled? After controlling for overlap between views, is the full method still more stable than SFT-only?
2 comments
I appreciate the authors' efforts in addressing my concerns through the additional results. After considering the rebuttal, my final rating is 2.
Response to questions on Persona Identifiability
Thank you for your feedback. We conducted additional experiments to address the concerns about predictive validity, independent-history consistency, and missing baselines.
1. Frozen-persona transfer — We agree that the original cross-benchmark association was insufficient to establish predictive utility.
To address this, we infer personas from observed histories, freeze them, and evaluate held-out behavior using the same frozen predictor (LM1). No evaluation targets are used to update personas. CE gain measures the reduction in held-out cross-entropy from conditioning on the correct user's inferred persona.
Paired user-clustered bootstrap CE improvements (95% CI):
All three paired intervals exclude zero. Unlike Figure 3's cross-benchmark association, this experiment fixes the predictor and evaluation contexts. Thus, we now have direct evidence that PERSONA-LR's inferred personas provide greater held-out predictive utility than the principal baselines.
Independent predictor check: With an unadapted HumanLM predictor, PERSONA-LR obtains CE gain +1.103 [0.540, 1.676], compared with untrained +1.008 [0.493, 1.554], SFT +1.014 [0.482, 1.551], and one-shot +1.064 [0.536, 1.630]. The trend is consistent, although statistical superiority is not established. HumanLM's larger CE gains result from overconfident predictions and the 10-12 probability floor, which inflates cross-entropy differences. Since changing this floor also changes model rankings, we report these results for completeness but do not draw conclusions from them.
Disjoint-history consistency — We agree that shared-history agreement alone does not establish user-specific consistency, but we have conducted a new experiment evidence beyond that confound.
We independently infer personas from two non-overlapping history subsets for each user. Separation measures same-user minus different-user cosine similarity.
Recovery MAE also favors PERSONA-LR across all observation budgets: 0.410/0.375/0.357/0.337, compared with SFT's 0.435/0.402/0.383/0.362.
At five items/view, the separation improvement is statistically significant. At smaller budgets, the differences remain inconclusive. Unlike Table 6, the two representations share no observed history items, and SFT follows the identical protocol.
We therefore disagree that the observed consistency can be explained solely by overlapping evidence. The new results support improved user discrimination from independent histories, although they do not establish uniformly higher absolute within-user similarity or formal identifiability.
2. One-shot refinement and compute — We agree that the original manuscript did not report the one-shot baseline clearly, and that compute differences matter for causal attribution.
We therefore added a one-shot baseline under the same evaluation protocol. PERSONA-LR improves recovery MAE and held-out CE gain.
These results establish that the full method outperforms the evaluated one-shot baseline. However, because training budgets differ, we do not claim that this isolates the effect of iterative refinement alone.The comparison establishes a performance advantage under the evaluated training budgets, but does not isolate the effect of repeated refinement from additional training compute.
PANDORA evaluation. We agree that the SFT-only model's 0% parse rate makes its reported correlation unreliable. The evaluation requires five numerical Big Five trait predictions; missing values are imputed to 50. Because SFT-only failed to produce complete valid outputs, its predictions became constant, making Spearman correlation undefined rather than 0.000. We will correct this to N/A.
We evaluate 200 test records (seed 17), using up to 10 comments per record. Spearman correlation is computed separately for each of the five traits and then averaged. We will clarify the parsing procedure and distinguish output-format compliance from trait-recovery accuracy. Consequently, we do not treat SFT-only's formatting failure as evidence of inferior persona recovery.