acceptodds
Under review as a conference paper at ICLR 2027

Measuring Meta-Cultural Competency in Action: A Behavioral Framework for LLM-Based Personalization

Abstract

Personalization requires a system to know when it has enough information about a user and when it should ask for more. These abilities correspond to two capacities of meta-cultural competence: variational awareness, or sensitivity to the limits of one's knowledge about a user, and explication strategy, or the ability to elicit missing information efficiently. Existing evaluations of meta-cultural competence mostly measure static model-implied distributions. They do not test whether models can use uncertainty to guide interaction. We propose a dynamic explore-exploit framework for evaluating meta-cultural competence behaviorally in conversational recommender systems. The framework measures whether models explore when initial user context is limited, exploit when context is sufficient, and ask informative questions when they explore. We instantiate it as a multi-turn movie recommendation task using simulated users from MovieLens-1M, and evaluate 11 models from 6 families. While static evaluation reports a plateau in meta-cultural competence beyond a certain model size, our dynamic framework detects statistically significant differences across models above that threshold. This separates them into distinct performance tiers, and we find evidence that model size increases meta-cultural competence in action. These findings suggest that measuring meta-cultural competency through distributions and behavior captures different aspects of model capacity.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
meta-culture, large language models, llm personalization, variational awareness, behavioral analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_z3bAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a dynamic explore-exploit evaluation framework to measure meta-cultural competence in LLM-based personalization. Unlike existing static evaluations that focus on models’ knowledge representations, the proposed framework assesses whether LLMs can adapt their exploration behavior to available user information and select informative questions during interaction. Specifically, it introduces three metrics, Context Sensitivity, Question Quality, and Trend Alignment, to evaluate these capabilities in multi-turn conversational recommendation. Experiments on MovieLens-1M with 11 LLMs reveal significant differences in interactive behavior across models, demonstrating that dynamic evaluation captures personalization capabilities overlooked by static assessments.

Critical strengths
  1. The paper shifts from static knowledge assessment to dynamic behavioral evaluation, providing a new perspective on measuring meta-cultural competence in LLM-based personalization.
  2. The proposed framework introduces three complementary metrics, Context Sensitivity, Question Quality, and Trend Alignment, to assess different aspects of exploration and interaction behavior.
  3. The experiments cover 11 LLMs from 6 model families under 16 experimental conditions, revealing meaningful differences in interactive personalization capabilities across models.
Critical weaknesses
  1. The proposed framework primarily evaluates explore-exploit behavior, which overlaps with traditional conversational recommendation. Its connection to meta-cultural competence is not sufficiently justified, especially without explicit cross-cultural evaluation.
  2. In Section 4 (Question Construction and Tiering), question utility is determined by human experts and GPT-5.5 rather than actual information gain or recommendation improvement. This may introduce subjective bias and fail to capture user-specific question utility.
  3. The paper lacks a thorough analysis of how the proposed metrics relate to actual recommendation accuracy and interaction costs, making it unclear whether higher evaluation scores translate into better personalization performance.
Questions for authors

Could the authors clarify how the proposed framework distinguishes meta-cultural competence from general explore-exploit capabilities? Additionally, how well do the predefined question utility scores correlate with actual recommendation improvements?

1 comment
AuthorsOpenReview ↗

Re: Response to Reviewer z3bA

Dear Reviewer z3bA,

Thank you once again for your review. We have posted a detailed response to your comments and would be grateful if you could take a look. We would like to know whether it addresses your concerns, and we are happy to answer any follow-up questions before the discussion period ends. If our response has resolved your concerns, we would greatly appreciate it if you would consider updating your score. Thank you for your time and feedback.

Best regards,

The authors

Reviewer_1hwrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a benchmark for evaluating when an LLM should ask for more information from a user (aka explore) and when it should make recommendations using the information already available (aka exploit). It uses a movie recommendation task and scores how models change their questioning behavior across different amounts of initial context, and whether they select questions considered informative.

Critical strengths

I think studying when an assistant should ask for clarification is worthwhile. Asking unnecessary questions can frustrate users, while making recommendations without enough information can also be unhelpful. Evaluating the interaction process alongside the final answer is a useful direction.

Critical weaknesses
  1. My main concern is that the benchmark removes an important part of the ability it is trying to evaluate. The model chooses from 15 questions written by the authors. In a real conversation, the assistant has to figure out what information is missing and come up with a useful question itself. Recognizing a sensible question from a menu does not establish that it would generate that question when needed. I understand that a fixed question bank makes evaluation easier to control, but I am not convinced that performance in this setting supports the broader claims about conversational personalization or meta-cultural competence.

  2. I found the paper hard to follow and was left unclear about what I should learn from its scores. The abstract introduces several terms without a concrete example showing the behavior being measured. The framework figure and definitions help, but I still wanted a worked example from an actual benchmark interaction: what information the model receives, what it chooses, what answer it gets, and how those decisions produce the reported scores. I also do not find the interpretation of asking less with more context sufficiently justified. Having more information about a user does not necessarily mean that the assistant has the information needed for the current recommendation. The paper needs to explain more clearly how its expected behavioral patterns establish that a model knows when it has enough information.

I think the underlying problem is worth studying, but I am not convinced that this benchmark establishes the practical ability the paper claims to measure. The fixed question menu leaves a central part of real interaction untested, and the presentation makes it difficult to understand what the scores demonstrate beyond behavior within this particular setup.

Questions for authors
  1. What evidence shows that choosing questions from the fixed bank predicts a model’s ability to generate useful questions during a real conversation?

  2. Have you checked whether selecting the highly ranked questions actually improves movie predictions enough to justify asking the user?

  3. Can you provide a worked example from an actual benchmark run, including the resulting scores, and explain why the behavior should be considered competent?

1 comment
AuthorsOpenReview ↗

Re: Response to Reviewer 1hwr

Dear Reviewer 1hwr,

Thank you once again for your review. We have posted a detailed response to your comments and would be grateful if you could take a look. We would like to know whether it addresses your concerns, and we are happy to answer any follow-up questions before the discussion period ends. If our response has resolved your concerns, we would greatly appreciate it if you would consider updating your score. Thank you for your time and feedback.

Best regards,

The authors

Reviewer_NXUrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a behavioral framework for evaluating two aspects of meta-cultural competence in LLM-based personalization,namely the models’ability to recognize insufficient user information (variational awareness) and efficiently elicit missing preferences (explication strategy). The main contribution is a multi-turn explore–exploit evaluation framework with three metrics: Context Sensitivity, Question Quality, and Trend Alignment, measuring how models adapt their exploration decisions to available context and select informative questions.

The framework is instantiated as a 20-round movie recommendation task using 2,038 simulated users from MovieLens-1M. The authors evaluate 11 LLMs across six model families under 16 experimental conditions, varying initial user context, target visibility, and exploration cost. Question quality is evaluated against context-dependent, precomputed utility rankings derived from human experts and GPT-5.5.

The results reveal distinct behavioral profiles across models and significant performance differences. The authors further argue that dynamic behavioral evaluation reveals differences not captured by prior static evaluations, highlighting the complementary nature of representational and behavioral assessments of meta-cultural competence.

Critical strengths

Major Strength: Providing a meaningful problem formulation and a fine-grained evaluation framework.

The paper addresses an important aspect of LLM-based personalization. Moving beyond static evaluations of knowledge distributions toward behavioral evaluation is a meaningful research direction. The proposed framework separates different aspects of interactive personalization through three metrics, providing a more detailed characterization of model behavior than evaluating final recommendation accuracy alone. The experimental results provide preliminary evidence that deciding when to explore and determining what questions to ask are two distinct capabilities, highlighting the value of evaluating them separately.

Critical weaknesses

Major Weakness: The static question-utility oracle may not provide adequate guidance to measuring adaptive exploration.

The proposed Question Quality metric relies heavily on pre-computed question rankings, unable to be updated based on specific users' interaction history, acquired answers or downstream recommendation targets. Table 6 assigns Q1-Q4 to the highest utility tier across all context levels, allowing a simple heuristic model asking only in a fixed preference-first question ordering to achieve high Qual@K scores. Furthermore, some utility assignments appear inconsistent with the simulated user-answer construction. For example, Q7 is derived from age, yet retains positive utility when age is already provided in the initial context. It would appear Qual@K captures primarily agreement with an offline, context-level question ranking rather than an adaptive explication strategy.

Major Weakness: The instability of the Trend Alignment metric.

The proposed Trend Alignment metric has potential mathematical limitations that raise concerns about its reliability. First, it relies on Pearson correlation, which becomes undefined when exploration rates or question quality scores are constant across context levels. This issue is also reflected in the Oracle baseline in Table 1. Second, Pearson correlation can be highly sensitive to small fluctuations when the underlying values are nearly constant, potentially leading to unstable model rankings. These observations raise concerns about the robustness of the proposed metric and the resulting performance tiers.

Questions for authors

Question 1: In Table 6, some questions appear to receive positive utility scores despite providing little or no additional information beyond the initial context. For example, certain questions ask about demographic attributes that are already provided or can be directly inferred from the available information. Could the authors clarify the rationale behind these utility assignments and explain how the framework accounts for the marginal information gain of each question?

Question 2: The question utility tiers are primarily determined through rankings provided by human experts and GPT-5.5, rather than empirically measured improvements in recommendation performance. Could the authors provide additional evidence that these rankings accurately reflect the actual informativeness of the questions? In particular, would it be possible to validate the utility tiers against downstream recommendation accuracy improvements after obtaining the corresponding user responses?

Question 3: Equation (3) relies on Pearson correlation, which becomes undefined when exploration rates or question quality scores remain constant across context levels. How does the framework handle such cases, particularly when a model consistently selects high-quality questions? Additionally, given the sensitivity of Pearson correlation to small fluctuations in nearly constant values, could the authors investigate whether the model rankings remain stable under alternative, potentially more robust alignment metrics?

1 comment
AuthorsOpenReview ↗

Re: Response to Reviewer NXUr

Dear Reviewer NXUr,

Thank you once again for your review. We have posted a detailed response to your comments and would be grateful if you could take a look. We would like to know whether it addresses your concerns, and we are happy to answer any follow-up questions before the discussion period ends. If our response has resolved your concerns, we would greatly appreciate it if you would consider updating your score. Thank you for your time and feedback.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.