- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Niche Persona from Observation (NPO), an LLM-based framework for personalized human decision modeling. Motivated by the interdependence between feature selection and behavioral theory selection, NPO introduces a co-evolutionary process in which decision agents are optimized under different behavioral theories, while a selection agent iteratively refines the feature set based on validation feedback. The framework identifies group-specific feature–theory combinations to improve decision prediction. Experiments on three real-world datasets involving job choice, alcohol use, and marijuana use demonstrate improvements in predictive performance and feature efficiency over existing baselines.
Critical strengths
The paper presents an interesting perspective on personalized decision modeling by considering the interdependence between feature selection and behavioral theory selection. The idea of identifying group-specific feature–theory combinations is intuitive and well motivated. I also appreciate the attempt to incorporate established behavioral theories into LLM-based decision agents, which provides a more structured approach to human behavior prediction. The experimental results across three real-world datasets and multiple LLM backbones are encouraging, showing improvements in both predictive performance and feature efficiency.
Critical weaknesses
The paper's motivation that feature selection and behavioral theory selection should be considered jointly is reasonable. Although Tables 2 and 3 demonstrate the benefits of feature selection and theory competition, they do not sufficiently establish whether NPO's specific co-evolutionary design is more effective than simpler joint optimization strategies, such as conventional alternating optimization or direct LLM-based joint search. In particular, since the search space and validation objective are explicitly defined, a natural alternative would be to use a general-purpose LLM agent to directly search for effective feature–theory combinations. Given that NPO already employs LLMs to guide feature selection, such an agentic search approach seems to be a relevant baseline. A comparison under the same evaluation budget would help clarify whether NPO's structured co-evolutionary process provides meaningful benefits over a more straightforward joint search strategy.
Questions for authors
Justification of the Framework:
The motivation for jointly considering feature selection and behavioral theory selection is reasonable, but what justifies the relatively complex design of NPO over simpler joint optimization approaches? For example, why not include a conventional alternating optimization baseline that iteratively updates the feature subset under a fixed behavioral theory, and then selects the best-performing theory under the updated feature set? Such a comparison would help determine whether NPO's improvements arise from the general idea of alternating feature–theory optimization or from its specific LLM-guided search strategy.
Comparison with General Agentic Search:
Since the search space and validation objective are explicitly defined, could a general-purpose LLM agent directly search for effective feature–theory combinations? Why is such an approach not included as a baseline?
6 comments
Response to Reviewer U9iX (Part2)
Q1 Justification of the framework
The main challenge of comparing NPO with conventional numerical alternating optimization lies in the difference of fitting procedures and parameter settings between theory candidates. Our baselines provide two types of approximations: PySR and gplearn adopt iterative evolutionary search based on the results of symbolic regression, while forward selection and recursive feature elimination alternately optimize the model and the feature set. We also set up the LLM-AO as the alternating optimization baseline using GPT-4o-mini. LLM-AO iteratively updates the feature subset under a fixed behavioral theory, and then selects the best-performing theory under the updated feature set.
One major motivation of NPO is to bridge the gap between these well-acknowledged theories and conventional alternating optimization. By leveraging the generalizability of LLMs, NPO can incorporate updated features into each behavioral model in a form appropriate to its underlying theory. It can also draw on feedback from the candidate theories to propose new features for behavioral prediction. Beyond predictive performance, the feature–theory alignment analysis in Figure 3(a) illustrates how prior research can help interpret the features selected by NPO [1,2].
[1] Fergusson, David M., and Joseph M. Boden. "Cannabis use and later life outcomes." Addiction 103.6 (2008): 969-976.
[2] Yamaguchi, Kazuo, and Denise B. Kandel. "On the resolution of role incompatibility: A life event history analysis of family roles and marijuana use." American journal of Sociology 90.6 (1985): 1284-1325.
Thanks for the response from the authors. I appreciate the additional experiments on the coding agents and this is my major concern given the coding agent is really powerful nowadays. I am still wondering if the authors can provide more details of how they conduct the agent experiments. How many iterations can an agent try in your experiments (feedback - submission loop)? What the agent receives as the feedback? How many times have you tried the general coding agent? Since the Codex and Claude Code achieves substantial improvements, what's their limit performance if they were given enough trial budgets (keep iteration until no better solution found)? I am interested in this because if codex and claude can be as good as NPO or even better given enough budget, why we need a relative complex method like NPO?
Response to Reviewer U9iX
We sincerely appreciate the reviewer's insightful comments regarding comparing NPO with simpler search strategies. We evaluated LLM-based alternating optimization and direct joint search with general-purpose agents below.
We agree that the co-evolution of theories and features in NPO introduces additional complexity and we would like to address this concern from the following aspects:
Conventional search strategy. PySR and gplearn provide relevant comparisons with conventional symbolic search: they search over both the variables appearing in an expression and its functional structure. We also included selection strategies such as recursive feature elimination and forward selection, which also jointly conduct feature selection and model evaluation. These baselines therefore address whether a simpler numerical joint search can provide competitive predictive models.
NPO connects alternating optimization and agentic search. We agree that those baselines still differ from the alternating optimization method. However, it is difficult to define an established numerical alternating optimization baseline for this setting. The main challenge is that candidate theories differ in how they specify choice probabilities and estimate parameters, so such a baseline would need to combine theory-specific fitting procedures with a common validation criterion.
This motivates NPO’s design. An LLM uses validation feedback to propose feature updates, and each candidate theory retains its own fitting procedure. We address the general-purpose agentic alternative separately through Direct LLM Joint Search, in which the agent proposes feature–theory pairs directly.
Direct comparisons. To address the challenge discussed above, we evaluated an LLM-driven alternating optimization approach (LLM-AO) as a baseline in Q1. We use GPT-4o-mini as a common selector for theory-feature combinations. In Q2, we further evaluate general-purpose LLM agents that directly propose feature–theory pairs. We will add a section describing both LLM-AO and direct search with general-purpose agents in the revised version.
Following the suggestion to evaluate a more straightforward alternating optimization approach, we implemented LLM Alternating Optimization (LLM-AO) as a baseline for comparison with NPO’s structured co-evolutionary procedure. The results are presented below and we would like to provide more details and further justification in Response Part 2.
We agree that general-purpose LLM agents provide a relevant baseline. To ensure a fair comparison, we packaged the background information and model-fitting process of the five theories into a skill, which we provided to the agents. We chose two representative agents: Codex CLI (version 0.167.0) and Claude Code CLI (version 2.1.311). For Codex, we evaluated GPT-6-Astra and GPT-6-Luna. For Claude Code, we evaluated Claude-Opus5.5 and Claude-Haiku5.5. We set the reasoning effort to medium for all models and used the same search space, validation objective, and other settings as in NPO. All agents searched directly for feature–theory combinations using validation feedback. The detailed results are presented below:
Direct agentic search with Codex and Claude Code achieves substantial improvements over LLM baselines such as vanilla LLM and TextGrad. However, despite using more capable backbone models, these general-purpose agents still underperform NPO with GPT-4o-mini across all three datasets, which shows the value of the co-evolution framework of NPO.