- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper first mines 13 factors from large-scale data to represent fine-grained explanations of user preferences, and then dynamically injects a subset of these factors as additional context for personalized question answering. Experiments across both open-source and proprietary models show improvements compared with direct RAG methods.
Critical strengths
The proposed framework is evaluated across a range of both open-source and proprietary LLMs, as well as multiple personalization benchmarks.
Critical weaknesses
-
Limited methodological contribution and heuristic mining process. The main contribution of this paper is injecting 13 static factor definitions into the prompt context for personalized question answering. However, the offline factor mining pipeline is heavily heuristic-driven and restricted to specific training datasets, without sufficient discussion on out-of-distribution generalization or why these specific 13 generic factors are representative enough to cover diverse user preferences.
-
Insufficient comparison with prior personalization methods. While the introduction categorizes existing literature into model adaptation and context conditioning, the experimental evaluation completely omits model adaptation methods and even narrows context conditioning down to simple RAG variants, leaving out stronger published profile-generation or memory-construction baselines.
-
Unjustified inference overhead, method inconsistency, and questionable necessity for strong models. The modest gains of the proposed dynamic selection mechanism hardly justify its complexity, specifically in three aspects:
(i) Unreported overhead on open-source models: The score-based selector requires multiple preliminary generations followed by leave-one-factor-out likelihood evaluations across all factors for every single query, introducing substantial inference latency and compute overhead that are not analyzed in the main paper.
(ii) Degradation on proprietary models: Because sequence log-likelihoods are inaccessible in black-box APIs, the method degenerates into a simple prompt-based selector for proprietary models, an inconsistency where the mathematical formulation in Section 3.2 is not actually used, and which the appendix ablation shows can even underperform vanilla RAG on open-source models.
(iii) Questionable necessity as the base policy scales: When the underlying LLM is sufficiently capable, it should already be able to attend to relevant preference cues directly from the retrieved user history, making it unclear whether injecting and selecting from a fixed checklist of 13 generic definitions remains necessary or meaningful at scale.
Questions for authors
Please refer to my weaknesses
8 comments
Q2: Clarification of baseline selection and comparisons(Response to Weakness 2)
Q2: Clarification of baseline selection and comparisons(Response to Weakness 2)
A2: Thank you for suggesting that we include stronger personalization baselines. Existing context-conditioning methods, such as MemoryBank [1] and A-MEM [2], primarily focus on accurately storing, organizing, and retrieving user-specific memories. PersonaFactor instead targets fine-grained aspects of user personality and preferences, including self-concept, cognitive consistency, and schema structure. The two lines of work therefore address complementary objectives. To provide a direct empirical comparison, we additionally evaluate MemoryBank and A-MEM under the same backbone models, user histories, prompts, and evaluation metrics. As reported in the table below, Dynamic PersonaFactor achieves higher scores than both memory-based baselines across all evaluated backbones and metrics. These results support the effectiveness of explicitly modeling fine-grained preference factors beyond factual user-memory retrieval.
| Model | Method | LaMP R-1 | LaMP R-L | LongLaMP R-1 | LongLaMP R-L | LaMP-QA | PersonalLLM |
|---|---|---|---|---|---|---|---|
| DeepSeek-V4-Pro | Direct | 0.2932 | 0.2580 | 0.2125 | 0.1242 | 0.4915 | -0.9013 |
| DeepSeek-V4-Pro | MemoryBank | 0.3375 | 0.2996 | 0.2955 | 0.1655 | 0.5410 | 0.1433 |
| DeepSeek-V4-Pro | A-MEM | 0.3357 | 0.3023 | 0.3039 | 0.1731 | 0.4865 | -0.3117 |
| DeepSeek-V4-Pro | Static PersonaFactor | 0.3495 | 0.3076 | 0.3141 | 0.1581 | 0.7418 | 0.4795 |
| DeepSeek-V4-Pro | Dynamic PersonaFactor | 0.3560 | 0.3140 | 0.3645 | 0.1935 | 0.7575 | 0.4915 |
| Gemini-2.5-Flash | Direct | 0.3286 | 0.2918 | 0.2305 | 0.1309 | 0.4354 | -0.4993 |
| Gemini-2.5-Flash | MemoryBank | 0.3439 | 0.3237 | 0.3011 | 0.1597 | 0.5516 | -0.4573 |
| Gemini-2.5-Flash | A-MEM | 0.3421 | 0.3277 | 0.3186 | 0.1601 | 0.4976 | -0.5388 |
| Gemini-2.5-Flash | Static PersonaFactor | 0.4100 | 0.3696 | 0.3461 | 0.1789 | 0.5914 | 0.2822 |
| Gemini-2.5-Flash | Dynamic PersonaFactor | 0.4225 | 0.3761 | 0.3888 | 0.2051 | 0.6141 | 0.3022 |
| Qwen3-8B | Direct | 0.3145 | 0.2785 | 0.2151 | 0.1287 | 0.3271 | -1.2800 |
| Qwen3-8B | MemoryBank | 0.3109 | 0.2764 | 0.2575 | 0.1536 | 0.2218 | -0.9754 |
| Qwen3-8B | A-MEM | 0.3265 | 0.2916 | 0.2644 | 0.1590 | 0.2116 | -0.9499 |
| Qwen3-8B | Static PersonaFactor | 0.3276 | 0.2892 | 0.2719 | 0.1476 | 0.4328 | -0.1876 |
| Qwen3-8B | Dynamic PersonaFactor | 0.3336 | 0.2956 | 0.3377 | 0.1859 | 0.4417 | -0.1722 |
| Gemma-3-4B | Direct | 0.2522 | 0.2211 | 0.2005 | 0.1194 | 0.3762 | -0.2998 |
| Gemma-3-4B | MemoryBank | 0.2620 | 0.2298 | 0.2286 | 0.1326 | 0.3922 | -1.1280 |
| Gemma-3-4B | A-MEM | 0.2792 | 0.2482 | 0.2327 | 0.1387 | 0.4194 | -0.7723 |
| Gemma-3-4B | Static PersonaFactor | 0.2710 | 0.2353 | 0.2346 | 0.1339 | 0.4944 | -0.1977 |
| Gemma-3-4B | Dynamic PersonaFactor | 0.2818 | 0.2490 | 0.2943 | 0.1651 | 0.5107 | -0.1836 |
References
[1] Zhong, W., Guo, L., Gao, Q., et al. MemoryBank: Enhancing Large Language Models with Long-Term Memory. Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38(17): 19724–19731.
[2] Xu, W., Liang, Z., Mei, K., et al. A-MEM: Agentic Memory for LLM Agents. Advances in Neural Information Processing Systems, 2026, 38: 17577–17604.
Q3: Response to concerns about the inference overhead of dynamic factor selection(Response to Weakness 3(i))
Q3: Response to concerns about the inference overhead of dynamic factor selection(Response to Weakness 3(i))
A3: Thank you for raising this important concern. We additionally measured the inference overhead of dynamic factor selection on DeepSeek-V4-Pro and Qwen3-8B, with the results reported below; we will include these measurements in the camera-ready version. As the reviewer correctly points out, dynamic selection introduces additional computational cost. However, in a session-based personalization setting, factor selection typically needs to be performed only once per session, allowing this cost to be amortized over subsequent interactions. We also provide Static PersonaFactor, which avoids the dynamic selection overhead while still delivering effective personalization. In future work, we will investigate more efficient approaches for constructing and selecting personalization factors. We appreciate the reviewer’s valuable suggestion.
Qwen3-8B
| Dataset | Method | Input tokens | Generated tokens | Average latency (s) |
|---|---|---|---|---|
| LaMP | RAG | 603.17 | 24.82 | 0.950 |
| LaMP | Static | 1,278.17 | 25.33 | 1.026 |
| LaMP | Dynamic | 3,235.73 | 154.48 | 6.417 |
| LongLaMP | RAG | 1,168.00 | 160.87 | 5.789 |
| LongLaMP | Static | 1,843.00 | 175.73 | 6.369 |
| LongLaMP | Dynamic | 5,003.33 | 443.63 | 18.609 |
| LaMP-QA | RAG | 656.80 | 108.12 | 3.167 |
| LaMP-QA | Static | 1,282.80 | 124.84 | 3.692 |
| LaMP-QA | Dynamic | 3,284.52 | 352.44 | 11.869 |
| PersonalLLM | RAG | 2,737.00 | 487.65 | 17.031 |
| PersonalLLM | Static | 3,330.00 | 355.75 | 12.584 |
| PersonalLLM | Dynamic | 9,499.05 | 731.05 | 26.350 |
DeepSeek-V4-Pro
| Dataset | Method | Input tokens | Generated tokens | Average latency (s) |
|---|---|---|---|---|
| LaMP | RAG | 547.28 | 25.52 | 2.880 |
| LaMP | Static | 1,217.28 | 25.00 | 3.025 |
| LaMP | Dynamic | 1,566.93 | 228.00 | 8.579 |
| LongLaMP | RAG | 1,316.42 | 214.77 | 7.086 |
| LongLaMP | Static | 1,986.42 | 200.67 | 6.632 |
| LongLaMP | Dynamic | 3,207.83 | 504.58 | 13.348 |
| LaMP-QA | RAG | 753.10 | 190.72 | 7.021 |
| LaMP-QA | Static | 1,374.10 | 207.05 | 7.260 |
| LaMP-QA | Dynamic | 2,017.57 | 490.37 | 14.184 |
| PersonalLLM | RAG | 2,698.35 | 519.40 | 12.409 |
| PersonalLLM | Static | 3,286.35 | 560.45 | 14.130 |
| PersonalLLM | Dynamic | 5,961.40 | 738.85 | 19.507 |
Q1: Methodological contribution and generality of PersonaFactor(Response to Weakness 1)
Dear Reviewer,
We thank you for your careful and critical assessment of our work. Your comments raise important questions about the methodological contribution, experimental comparisons, and practical implications of our approach. We respond to each concern point by point below.
Q1: Methodological contribution and generality of PersonaFactor(Response to Weakness 1)
A1: Thank you for raising this concern. PersonaFactor is built from a broad collection of training data covering heterogeneous personalization scenarios. Specifically, we use the LaMP and LongLaMP training splits together with CoPA and HiCUPID, totaling 66,018 instances across 14 task-specific subsets. These data cover diverse tasks and domains, including news headline generation, scholarly title generation, tweet paraphrasing, abstract generation, product review generation, topic writing, and personalized question answering. This broad coverage exposes the factor-mining process to different task objectives and user preference patterns, reducing its dependence on any single benchmark or domain. We then construct a reusable factor library through clustering and semantic aggregation, rather than manually defining a task-specific checklist. The resulting library is effective across LaMP, LongLaMP, LaMP-QA, and PersonalLLM. Notably, LaMP-QA and PersonalLLM are not used to construct the factor library, yet PersonaFactor remains effective on both benchmarks, supporting the transferability of the mined factors.