Measuring Meta-Cultural Competency in Action: A Behavioral Framework for LLM-Based Personalization
Abstract
Personalization requires a system to know when it has enough information about a user and when it should ask for more. These abilities correspond to two capacities of meta-cultural competence: variational awareness, or sensitivity to the limits of one's knowledge about a user, and explication strategy, or the ability to elicit missing information efficiently. Existing evaluations of meta-cultural competence mostly measure static model-implied distributions. They do not test whether models can use uncertainty to guide interaction. We propose a dynamic explore-exploit framework for evaluating meta-cultural competence behaviorally in conversational recommender systems. The framework measures whether models explore when initial user context is limited, exploit when context is sufficient, and ask informative questions when they explore. We instantiate it as a multi-turn movie recommendation task using simulated users from MovieLens-1M, and evaluate 11 models from 6 families. While static evaluation reports a plateau in meta-cultural competence beyond a certain model size, our dynamic framework detects statistically significant differences across models above that threshold. This separates them into distinct performance tiers, and we find evidence that model size increases meta-cultural competence in action. These findings suggest that measuring meta-cultural competency through distributions and behavior captures different aspects of model capacity.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.