- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a dynamic explore-exploit evaluation framework to measure meta-cultural competence in LLM-based personalization. Unlike existing static evaluations that focus on models’ knowledge representations, the proposed framework assesses whether LLMs can adapt their exploration behavior to available user information and select informative questions during interaction. Specifically, it introduces three metrics, Context Sensitivity, Question Quality, and Trend Alignment, to evaluate these capabilities in multi-turn conversational recommendation. Experiments on MovieLens-1M with 11 LLMs reveal significant differences in interactive behavior across models, demonstrating that dynamic evaluation captures personalization capabilities overlooked by static assessments.
Critical strengths
- The paper shifts from static knowledge assessment to dynamic behavioral evaluation, providing a new perspective on measuring meta-cultural competence in LLM-based personalization.
- The proposed framework introduces three complementary metrics, Context Sensitivity, Question Quality, and Trend Alignment, to assess different aspects of exploration and interaction behavior.
- The experiments cover 11 LLMs from 6 model families under 16 experimental conditions, revealing meaningful differences in interactive personalization capabilities across models.
Critical weaknesses
- The proposed framework primarily evaluates explore-exploit behavior, which overlaps with traditional conversational recommendation. Its connection to meta-cultural competence is not sufficiently justified, especially without explicit cross-cultural evaluation.
- In Section 4 (Question Construction and Tiering), question utility is determined by human experts and GPT-5.5 rather than actual information gain or recommendation improvement. This may introduce subjective bias and fail to capture user-specific question utility.
- The paper lacks a thorough analysis of how the proposed metrics relate to actual recommendation accuracy and interaction costs, making it unclear whether higher evaluation scores translate into better personalization performance.
Questions for authors
Could the authors clarify how the proposed framework distinguishes meta-cultural competence from general explore-exploit capabilities? Additionally, how well do the predefined question utility scores correlate with actual recommendation improvements?
Re: Response to Reviewer z3bA
Dear Reviewer z3bA,
Thank you once again for your review. We have posted a detailed response to your comments and would be grateful if you could take a look. We would like to know whether it addresses your concerns, and we are happy to answer any follow-up questions before the discussion period ends. If our response has resolved your concerns, we would greatly appreciate it if you would consider updating your score. Thank you for your time and feedback.
Best regards,
The authors