acceptodds
Under review as a conference paper at ICLR 2027

Investor-Specific Multi-Objective Preference Alignment for Personalized Portfolio Optimization

Abstract

Reinforcement learning for portfolio management typically outputs a single strategy for all investors, failing to reflect heterogeneous investor goals. To address this gap, we propose Investor-Type-Wise Multi-Objective Preference Alignment (IMOPA), a framework that accounts for diverse investor profiles and outputs strategies for different investors. Specifically, we integrate preference learning, bandit-based preference weight estimation, and policy optimization into IMOPA. IMOPA first evaluates candidate policies through LLM-generated multi-objective feedback on financial metrics together with an overall evaluation. Utilizing such information, a bandit-based mechanism estimates multi-objective preference weights and selects a candidate policy for each investor type. Finally, IMOPA employs Proximal Policy Optimization (PPO) conditioned on the learned multi-objective preferences to train personalized policies. Experiments on two datasets—S&P 500 and MidCap 100—demonstrate that our estimated preference weights reveal distinct financial trade-offs, leading to customized asset allocations across investor types. Furthermore, ablation studies show that omitting preference alignment narrows or collapses these behavioral differences. Overall, IMOPA enables interpretable, preference-aligned portfolio policies tailored to diverse investor profiles.

Primary area:
reinforcement learning
Keywords:
Multi-objective reinforcement learning, preference-based reinforcement learning, linear bandits, portfolio optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PqHEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript studies the personalized portfolio optimization problem. The proposed IMOPA method generates multi-objective and overall scores from an LLM, estimates preference weights through a bandit-based procedure, and trains investor-specific PPO policies. The motivation of introducing heterogeneous preferences is meaningful. However, a non-negligible gap remains between the observation model required by Proposition 1 and the fixed-score procedure used in the experiments. The validation of preference alignment is also insufficient.

Critical strengths
  • The distinction between objective-level feedback and overall preference is reasonable for describing investor heterogeneity.

  • The theoretical results are comprehensive, which seems to be correct and guarantees the effectiveness of the method.

  • The temporal train/preference/test separation and the initialization ablation help examine where the behavioral differences come from.

Critical weaknesses
  • The validity of the confidence guarantee for the implemented procedure seems to be doubtful. Remark 1 reuses a fixed score Rx=x⊤w+ηxR_x=x^\top w+\eta_x and Appendix C adds Gaussian noise. Repeated observations still contain the same ηx\eta_x. Unless this term is zero or separately controlled, the required conditional zero-mean noise assumption may not hold, and repeated use can shrink the confidence region without removing the bias. Some justifications could be helpful.

  • The experiments establish differences between LLM-defined personas, while the recovery of investor preferences seem to be unreliable. Financial returns are not an independent measure of preference alignment. The comparison is also confounded by the asset universe: Appendix D restricts each baseline to a random 15-stock subset, whereas IMOPA operates on a much larger pool. Thus, the benefit of the proposed preference-learning mechanism seems cannot be independently validated.

Questions for authors

Q1. How is E[ξn∣Fn−1,xn]=0\mathbb E[\xi_n\mid\mathcal F_{n-1},x_n]=0 justified when the same LLM score is used repeatedly? Could Proposition 1 be stated with an explicit misspecification term and the stopping rule corrected accordingly?

Q2. Proposition 1 identifies an arm in the feedback set, while Eq. (8) selects an executable policy through an additional gate. What guarantee transfers to that selected policy?

Q3. Could preference alignment be evaluated using independent feedback or known synthetic utilities, with all methods using the same asset universe and constraints?

3 comments
AuthorsOpenReview ↗

Response 1. Fixed LLM scores and the stopping criterion of Proposition 1

We sincerely appreciate your patient examination of the theoretical assumptions and experimental comparisons. Your comments have helped us clarify the scope of the guarantees, add independent preference comparisons to illustrate out work more clearly.


1. Fixed LLM scores and the stopping criterion of Proposition 1

Weakness 1: "The validity of the confidence guarantee for the implemented procedure seems to be doubtful. ... Some justifications could be helpful."

Q1: "How is E[ξn∣Fn−1,xn]=0\mathbb{E}[\xi_n\mid\mathcal{F}_{n-1},\mathbf{x}_n]=0 justified when the same LLM score is used repeatedly? Could Proposition 1 be stated with an explicit misspecification term and the stopping rule corrected accordingly?"

Response: Proposition 1 in the original manuscript provides a stopping criterion analogous to that in linear BAI. In particular, our preference estimation is inspired by the linear best-arm identification framework of Soare et al. (2014). The confidence guarantee underlying this criterion requires the observation noise ξn(i)\xi_n^{(i)} to be conditionally zero-mean and σ\sigma-sub-Gaussian, as assumed in Proposition 1.

Remark 1 explains the difference between our implementation and standard linear BAI in the construction of observations. Each arm receives a fixed LLM-generated scalar performance rating,

=\mathbf{x}^{\top}\mathbf{w}^{(i)} +\eta_{\mathbf{x}}^{(i)},$$ and each subsequent observation is constructed as $$y_n^{(i)} =R_{\mathbf{x}_n}^{(i)}+\epsilon_n^{(i)},$$ where $\epsilon_n^{(i)}$ is the Gaussian perturbation added at each estimation step. Since the same LLM rating is reused when an arm is selected repeatedly, its evaluation residual is also retained. Thus, the combined residual $\xi_n^{(i)}=\eta_{\mathbf{x}_n}^{(i)}+\epsilon_n^{(i)}$ is not necessarily zero-mean. This distinction is also why we refer to our procedure as a bandit-based method rather than standard linear BAI: it adapts sequential allocation and estimation to fixed offline LLM feedback. Following your comment, we have revised Remark 1 (Page 6) to clarify the two sources of observation error and explicitly state that the confidence guarantee and stopping criterion in Proposition 1 apply only when the combined residual $\xi_n^{(i)}$ satisfies the conditional zero-mean and $\sigma$-sub-Gaussian assumptions in the revised manuscript. They therefore do not provide an unconditional guarantee for the fixed-score implementation. **References** Marta Soare, Alessandro Lazaric, and Rémi Munos. Best-Arm Identification in Linear Bandits. Advances in Neural Information Processing Systems, 828–836, 2014. ---
AuthorsOpenReview ↗

Response 2. Executable-policy selection and gate rule

2. Executable-policy selection and gate rule

Q2: "Proposition 1 identifies an arm in the feedback set, while Eq. (8) selects an executable policy through an additional gate. What guarantee transfers to that selected policy?"

Response: The original manuscript separates preference-optimal arm identification in Proposition 1 from executable policy selection in Eq. (8). Proposition 1 shows that Algorithm 2 produces an optimal arm x^emp(i)\widehat{\mathbf{x}}_{\mathrm{emp}}^{(i)} based on an accurate estimate of the latent weight parameter w(i)w^{(i)}—represented by the preference weight w^(i)\widehat{w}^{(i)} in Algorithm 2. Since the pair (w^(i),x^emp(i))(\widehat{w}^{(i)}, \widehat{\mathbf{x}}_{\mathrm{emp}}^{(i)}) does not uniquely define a single policy in our problem formulation, we leverage them to generate two policies, πa\pi_a and πb\pi_b, and apply a gating rule to obtain the selected policy πsel(i)\pi_{\mathrm{sel}}^{(i)} for each investor type ii. Thus, the estimation guarantee in Proposition 1 directly underpins the quality of the resulting policy. Further details are provided in Appendix B.

Following your suggestion, we also rewrite this distinction in the revised manuscript (Page 6, Lines 319–323, and Page 7, Lines 324–325) to specify the role of this rule.


AuthorsOpenReview ↗

Response 3. Asset-universe comparison and preference validation

3. Asset-universe comparison and preference validation

Weakness 2: "The experiments establish differences between LLM-defined personas, while the recovery of investor preferences seem to be unreliable. ... Thus, the benefit of the proposed preference-learning mechanism seems cannot be independently validated."

Q3: "Could preference alignment be evaluated using independent feedback or known synthetic utilities, with all methods using the same asset universe and constraints?"

Response: We address the comparison design and preference evaluation separately below.

“The comparison is also confounded by the asset universe: Appendix D restricts each baseline to a random 15-stock subset, whereas IMOPA operates on a much larger pool.” We explain the use of ten 15-stock subsets for the portfolio baselines. Regarding the asset datasets, IMOPA operates on the full available stock pool with a constraint of 22--1515 holdings. In contrast, DeepAries, NCMDP, and SIT do not explicitly impose such sparsity constraints. Applying these methods to the full stock pool could result in dense allocations across a large number of assets, making the out-of-sample comparison less informative. We therefore randomly sampled 1010 subsets of 1515 stocks from each dataset, evaluated the three baselines on these subsets, and reported their average performance. This design was intended to control the allocation dimension of the baselines, although it does not constitute a strictly matched asset-universe comparison with IMOPA. We have added this explanation to Appendix C (Comparison Baselines) of the revised manuscript (Page 15, Lines 772--775).

“The experiments establish differences between LLM-defined personas, while the recovery of investor preferences seem to be unreliable. Financial returns are not an independent measure of preference alignment.” As for preference alignment, in original Table 1, we evaluated investor-specific policies using five out-of-sample financial criteria: final wealth, drawdown, Sharpe ratio, turnover, and portfolio concentration. These results have showed behavioral differences consistent with the predefined investor profiles. For example, the conservative profile (PP) exhibits stronger drawdown control and lower turnover, whereas the more active profile (AA) shows greater portfolio concentration and trading activity. These results illustrate how the learned preferences translate into distinct portfolio behaviors.

To address Q3: “Could preference alignment be evaluated using independent feedback or known synthetic utilities, with all methods using the same asset universe and constraints? ” we compare IMOPA with the direct-ridge estimation, which uses the same candidate policy set, asset universe, and portfolio constraints. Their selected policies and out-of-sample financial performance were already reported in Tables F.3--F.5 of the original manuscript. Further details on the estimation procedures and the additional independent preference evaluation can be found in Common Response 1: Bandit-Based Estimation and Direct ridge. We have also evaluated preference alignment against all comparison methods reported in Table E.8 of the revised manuscript (Page 18): Direct ridge, DeepAries, NCMDP, and SIT. For each target investor profile, the independent LLM evaluator compares IMOPA's final policy with each comparison policy using the profile description and anonymized out-of-sample portfolio criteria. The evaluation protocol is described on Page 9, Lines 476--485. The Direct ridge comparison specifically provides preference evidence under the same asset universe and constraints as IMOPA.

Table E.8. Preference win rates of IMOPA against Direct ridge, DeepAries, NCMDP, and SIT (%).

DatasetPersonavs. Direct ridgevs. DeepAriesvs. SITvs. NCMDP
S&P 500PP14.2914.2925.0087.50
S&P 500FF66.6725.0050.0075.00
S&P 500II57.1444.4444.4466.67
S&P 500AA77.7887.5066.6733.33
MidCap100PP50.0055.5662.5071.43
MidCap100FF50.0055.5666.6787.50
MidCap100II66.6755.5657.1442.86
MidCap100AA100.0077.7862.5055.56

Note. A win rate above 50% indicates that the independent LLM evaluator conditioned on the target persona, prefers the policy generated by IMOPA to the policy generated by the other algorithms.

Reviewer_AKaYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This study proposes IMOPA, a personalized portfolio optimization framework that considers different investment goals and risk preferences across investor types. The framework uses an LLM to generate multi-objective evaluations, a bandit-based method to estimate preference weights and select initial policies, and PPO to train personalized investment policies. Experiments on the S&P 500 and MidCap100 datasets demonstrate that the framework produces different preference weights, asset allocations, and investment behaviors across investor types.

Critical strengths

One strength of this study is that it addresses a practical and meaningful problem: portfolio optimization that considers different preferences across investor types. I believe the motivation for this study is reasonable and well justified. The authors also integrate LLM-based multi-objective evaluation, bandit-based preference estimation, and PPO into a single framework to generate interpretable preference weights and personalized investment policies. I believe that appropriately combining existing methods to address a specific problem can make a meaningful contribution to applied machine learning research. In addition, the authors compare the proposed method with existing approaches on two financial datasets and conduct ablation studies. The experimental results show different asset allocations and investment behaviors across investor types. I believe these experiments are appropriately designed to evaluate the proposed hypothesis and provide useful evidence supporting the effectiveness of the framework.

Critical weaknesses

As I understand it, the authors motivate IMOPA by arguing that existing multi-objective reinforcement learning methods have limitations in modeling heterogeneous investor preferences. Specifically, the authors explain that existing methods generally use predefined preference vectors or estimate preference weights from statistics of preference data. I believe this is a reasonable motivation for the study.

However, one weakness of the paper is that it does not sufficiently explain, with supporting comparisons, why these existing approaches are inadequate for modeling heterogeneous preferences. In principle, predefined preference vectors can be set differently for different investor types, and preference estimation methods based on collected feedback could also be applied separately to different investor groups. I would appreciate it if the authors could clarify whether I have misunderstood this point.

The authors emphasize that IMOPA estimates investor-specific preference weights from the relationship between multidimensional evaluations and separate overall evaluations, using a bandit-based procedure. I acknowledge that this design differs from existing methods in terms of feedback representation and estimation procedure. However, it remains unclear what important limitations of existing preference alignment methods are addressed by these differences.

The experiments in Section 5 mainly compare bandit-based estimation with direct ridge regression, evaluate portfolio performance against baselines such as DeepAries, NCMDP, and SIT, and conduct ablation studies by removing individual components of IMOPA. These experiments provide useful evidence about the behavior of the proposed framework. However, it is unclear whether they directly demonstrate that IMOPA can model heterogeneous preferences more effectively than existing preference-based or multi-objective reinforcement learning methods.

I believe that the Conclusion should be included in the main body of the paper rather than in the appendix. From the perspective of ensuring fairness and consistency when evaluating papers, I consider this a significant structural issue that warrants a deduction in the overall evaluation.

In addition, the manuscript contains several minor grammatical and typographical errors that should be corrected to improve readability. For example: • Line 86: "a overall evaluation" should be "an overall evaluation." • Line 396: "Moreover, We" should be "Moreover, we." • Line 422: One of the two periods should be removed. • Table 1: "metriccs" should be corrected to "metrics." I also recommend that the authors carefully review the grammar and language throughout the manuscript.

Questions for authors
  1. In Figure 5, the SP variant shows the largest reduction in behavioral differences across investor types. Does this suggest that the personalization effect depends more on the selection of initial policies than on the subsequent PPO training?

  2. The authors claim to learn investor-specific preferences, but the framework actually learns from LLM-generated evaluations based on predefined investor profiles. How can the authors independently validate whether these LLM-generated evaluations accurately reflect real investor preferences?

  3. The proposed framework combines established techniques, including LLM-based preference evaluation, bandit-based preference estimation, and PPO. I agree that combining existing techniques can also provide a novel methodological contribution in an application-oriented study. However, existing methods also seem capable of assigning different preference vectors to different investor types. Could the authors clarify why the proposed preference learning approach cannot be achieved using existing methods?

5 comments
AuthorsOpenReview ↗

Response 1. Contribution and comparison with existing methods (Part 1/2)

We appreciate your thoughtful assessment of the motivation and experimental design. Our original experiments examined preference estimation, portfolio performance, and the effects of individual components. Your comments have helped us clarify the methodological contribution, add direct preference comparisons, and improve the organization and language of the manuscript.


1. Contribution and comparison with existing methods

Weakness 1: "As I understand it, the authors motivate IMOPA by arguing that existing multi-objective reinforcement learning methods have limitations in modeling heterogeneous investor preferences. ... However, it is unclear whether they directly demonstrate that IMOPA can model heterogeneous preferences more effectively than existing preference-based or multi-objective reinforcement learning methods."

Q3: "The proposed framework combines established techniques, including LLM-based preference evaluation, bandit-based preference estimation, and PPO. I agree that combining existing techniques can also provide a novel methodological contribution in an application-oriented study. However, existing methods also seem capable of assigning different preference vectors to different investor types. Could the authors clarify why the proposed preference learning approach cannot be achieved using existing methods?"

Response: In the Introduction of the original manuscript, we discussed the differences between existing methods and IMOPA. To make the advantages of IMOPA clearer, we have revised the relevant discussion, particularly the third paragraph of the Introduction (Page 2, Lines 57–65):

“ In multi-objective alignment for LLMs, these user preference vectors can be specified in advance, as in the directional preference alignment (DPA) method proposed by Wang et al. (2024), where users specify preference directions to control the relative importance of different attributes of generated responses. Alternatively, preference weights can be estimated from human feedback. Specifically, Xiong & Singh (2025) estimated group-specific preference weights by maximum likelihood from human-reported objective choices in pairwise comparisons of LLM responses. Their method considers different preference weights across groups but optimizes a shared policy rather than generating different policies for each group (Xiong & Singh, 2025) . Therefore, these existing approaches leave the direct optimization of personalized policies for each investor type unaddressed. ”

Our primary contribution is to develop a unified framework that generates distinct personalized policies for multiple investor types based on their respective persona profiles within a single execution. This differs from approaches that require independently running and tuning algorithms for each investor type to obtain different policies, or aggregate heterogeneous preferences into a single consensus policy.

We have also examined the differences and advantages of bandit-based estimation relative to direct ridge estimation in Tables F.3--F.5 of the original manuscript, comparing the selected candidate policies and their out-of-sample financial performance. Further details can be found in Common Response 1: Bandit-Based Estimation and Direct ridge.

AuthorsOpenReview ↗

Response 1. Contribution and comparison with existing methods (Part 2/2)

To supplement the original comparisons, we have added the independent preference evaluation in Table E.8 of the revised manuscript (Page 18), with the evaluation protocol described on Page 9, Lines 476--485. For each of the four target personas, we compare IMOPA's final policy with the policies obtained using Direct ridge, DeepAries, NCMDP, and SIT. This evaluation fixes the target persona and compares methods. We divide the out-of-sample testing data into nine test periods of 6363 trading days for each of the two datasets. For each period, we compute the out-of-sample criteria of the policies being compared: final wealth, volatility, maximum drawdown, Sharpe ratio, turnover, and maximum weight. Gemma 4: 31B receives the target persona description and the anonymized criteria for a pair of policies, and judges which policy the target persona would prefer. Each pair contains one IMOPA policy and one comparison policy, and is evaluated in both presentation orders. A win (WW) is recorded when the evaluator prefers IMOPA in both orders; a loss (LL) is recorded when it prefers the comparison policy in both. We exclude ties and order-inconsistent judgments and compute WR=W/(W+L)\mathrm{WR}=W/(W+L) for each dataset, persona, and comparison method. A win rate above 50% indicates a preference for IMOPA under the target persona. Table E.8 reports these results and is reproduced below. IMOPA is preferred in most cases, with results varying across personas and comparison methods.

Table E.8. Preference win rates of IMOPA against Direct ridge, DeepAries, NCMDP, and SIT (%).

DatasetPersonavs. Direct ridgevs. DeepAriesvs. SITvs. NCMDP
S&P 500PP14.2914.2925.0087.50
S&P 500FF66.6725.0050.0075.00
S&P 500II57.1444.4444.4466.67
S&P 500AA77.7887.5066.6733.33
MidCap100PP50.0055.5662.5071.43
MidCap100FF50.0055.5666.6787.50
MidCap100II66.6755.5657.1442.86
MidCap100AA100.0077.7862.5055.56

Note. A win rate above 50% indicates that the independent LLM evaluator conditioned on the target persona, prefers the policy generated by IMOPA to the policy generated by the other algorithms.

References

Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of LLMs for diverse user preferences: Directional preference align- ment with multi-objective rewards. In Annual Meeting of the Association for Computational Linguistics, 8642–8655, 2024.

Nuoya Xiong and Aarti Singh. Projection optimization: A general framework for multi-objective and multi-group RLHF. In International Conference on Machine Learning, 69017–69058, 2025.


AuthorsOpenReview ↗

Response 2. Shared initialization and PPO

2. Shared initialization and PPO

Q1: "In Figure 5, the SP variant shows the largest reduction in behavioral differences across investor types. Does this suggest that the personalization effect depends more on the selection of initial policies than on the subsequent PPO training?"

Response: We have addressed this point in Common Response 2: Roles of the Initialized Policy and PPO. Please refer to that response for a detailed explanation.


Reviewer_e5LnOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes IMOPA, a framework for personalized portfolio management that combines LLM-generated investor feedback, bandit-based preference estimation, and PPO optimization. It aims to learn policies that reflect different investors’ preferences across several financial objectives.

Critical strengths

The motivation is reasonable: investors may prefer different trade-offs between return, risk, and trading behavior. The framework connects preference elicitation with policy learning, and the experiments demonstrate behavioral differences across investor profiles. The use of chronological data splits and transaction costs also improves the relevance of the evaluation.

Critical weaknesses

My main concern is that the experiments do not directly test the central claim of preference alignment. Differences in concentration, turnover, or returns show that the policies behave differently, but do not establish whether an investor prefers the policy trained for their profile. Since the preference feedback comes from prompted LLM personas, the evidence also has limited scope for claims about actual investors. An independent comparison of the final policies under each target profile would be more convincing.

The benefit of the bandit component remains unclear. Candidate strategy scores are collected in advance, so it is not obvious what is gained over estimating preference weights through regression on the same feedback. The experiments need to establish whether the bandit improves preference satisfaction or reduces the feedback required to select a suitable policy. Producing a different selection does not establish either benefit.

The source of the reported personalization is not sufficiently resolved. Different profiles start from different teacher policies, and subsequent optimization discourages departure from their respective reference policies. Some behavioral differences may therefore be inherited from initialization. The existing ablations are useful, but do not clearly establish how much the preference reward improves the target investor’s evaluation beyond the selected teacher or the distilled policy.

Questions for authors

What evidence shows that each investor profile prefers its own final policy over policies trained for other profiles?

Under the same feedback budget and candidate policy set, does the full method improve independently evaluated preference satisfaction over regression-based selection and direct use of the selected teacher?

3 comments
AuthorsOpenReview ↗

Response 1. Evidence for investor-specific preference alignment

We sincerely appreciate your careful reading of our manuscript and constructive comments, which have helped us improve the evaluation of preference alignment, and clarify the role of each component of our framework.


1. Evidence for investor-specific preference alignment

Weakness 1: "My main concern is that the experiments do not directly test the central claim of preference alignment. ... An independent comparison of the final policies under each target profile would be more convincing."

Q1: "What evidence shows that each investor profile prefers its own final policy over policies trained for other profiles?"

Response: We have already evaluated investor-specific portfolio behavior through the out-of-sample financial criteria reported in Table 1 of the original manuscript. To clarify these findings and their scope, we provide the following explanation. In Table 1, investor-specific policies have been validated using their out-of-sample financial criteria, including final wealth, volatility, maximum drawdown, Sharpe ratio, turnover, and maximum weight. The results are consistent with the descriptions of the corresponding investor types. For example, the more conservative profile (PP) exhibits stronger drawdown control and less active portfolio behavior (lower turnover), whereas the more active profile (AA) shows greater portfolio concentration (higher maximum allocation weight) and trading activity (higher turnover). These results demonstrate that the learned preferences translate into heterogeneous portfolio behaviors consistent with the predefined investor profiles.

The primary contribution of our work is to develop a unified framework that learns heterogeneous investor preferences and generates personalized policies. Directly evaluating whether each investor type prefers its own policy over policies trained for other types is a distinct task of preference evaluation.

Nevertheless, we have added the cross-persona results in Table 2 of the revised manuscript (Page 10) to provide a direct preference evaluation. The evaluation protocol is described on Page 9, Lines 476--485. Specifically, we conduct an independent win-rate evaluation inspired by the directional preference alignment (DPA) study of Wang et al. (2024). We divide the out-of-sample testing data into nine test periods of 6363 trading days for each dataset. For each period, the final policies are evaluated to obtain their out-of-sample portfolio criteria, including final wealth, volatility, maximum drawdown, Sharpe ratio, turnover, and maximum weight. We apply Gemma4:31b as the LLM-evaluator to compare, under each target investor persona ii, its own policy πθ(i)\pi_{\theta}^{(i)} with each policy πθ(j)\pi_{\theta}^{(j)}, j≠ij \ne i, generated for the other personas, using these out-of-sample portfolio criteria. With four personas, nine test periods, and two datasets, this results in 216216 pairwise policy comparisons in total.

To reduce potential position bias in LLM evaluation, each pair of policies is evaluated twice with their presentation order reversed. A win (WW) is recorded if the evaluator prefers persona ii's own policy in both evaluations, and a loss (LL) is recorded if it prefers the other persona's policy in both evaluations. Ties and order-inconsistent judgments are excluded, and the win rate is calculated as WR=W/(W+L)\mathrm{WR}=W/(W+L).

These results are reported in Table 2 of the revised manuscript (Page 10) and reproduced below for convenience. They show that, under each target persona, its own policy is preferred over policies generated for other personas, providing direct evidence of investor-specific preference alignment of our IMOPA beyond differences in portfolio behavior.

Table 2. Cross-persona preference win rates (%).

DatasetPPFFIIAA
S&P 50081.4855.5655.5673.08
MidCap10096.0075.0087.50100.00

References.

Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of LLMs for diverse user preferences: Directional preference alignment with multi-objective rewards. In Annual Meeting of the Association for Computational Linguistics, 8642–8655, 2024.


AuthorsOpenReview ↗

Response 2. Benefit of bandit-based estimation

2. Benefit of bandit-based estimation

Weakness 2: "The benefit of the bandit component remains unclear. ... Producing a different selection does not establish either benefit."

Q2: "Under the same feedback budget and candidate policy set, does the full method improve independently evaluated preference satisfaction over regression-based selection and direct use of the selected teacher?"

Response: We have already compared bandit-based estimation with direct ridge estimation in Tables F.3--F.5 of the original manuscript, which report the selected candidate policies and their out-of-sample financial performance. Below, we explain the differences between the two methods in preference estimation and policy selection.

Further details on the estimation procedure and preference evaluation can be found in Common Response 1: Bandit-Based Estimation and Direct ridge.


AuthorsOpenReview ↗

Response 3. Source of personalization

3. Source of personalization

Weakness 3: "The source of the reported personalization is not sufficiently resolved. ... The existing ablations are useful, but do not clearly establish how much the preference reward improves the target investor’s evaluation beyond the selected teacher or the distilled policy."

Response: In the original manuscript, we have investigated the impact of investor-specific preferences through the component ablation study. For a detailed explanation of the ablation results and the different roles of the initialized policy and PPO, please refer to Common Response 2: Roles of the Initialized Policy and PPO.


Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Common Concerns Raised by Reviewers

We thank all reviewers and the AC for their hard work and valuable feedback. All revisions made in response to the reviewers’ feedback are highlighted in blue in the revised manuscript. Below, we provide a consolidated response to common concerns raised by multiple reviewers.

Common Response 1: Bandit-Based Estimation and Direct ridge

This response addresses the shared questions about the difference between bandit-based estimation and Direct ridge estimation, and the evidence of preference satisfaction from the additional preference evaluation.

Estimation procedure and existing evidence

We have already compared bandit-based estimation with direct ridge estimation in Tables F.3--F.5 of the original manuscript, which report the selected candidate policies and their out-of-sample financial performance. Below, we explain the differences between the two methods in preference estimation and policy selection. Although our bandit-based method also uses constrained ridge regression to update w^\widehat{\mathbf w} at each iteration, it differs from direct ridge estimation in the data used to estimate the preference weights. The LLM-generated multi-objective scores serve as regression inputs, while the separate scalar performance ratings provide the regression targets. Direct ridge estimation fits the collected score--rating pairs directly. The bandit-based method instead uses G-allocation to select a score vector at each step and updates the weights using the stochastic observations derived from the corresponding scalar ratings.

Additional preference evaluation

To assess preference satisfaction, we have added the comparisons in Table E.8 of the revised manuscript (Page 18), with the evaluation protocol described on Page 9, Lines 476--485. For each of the four target personas, we compare the final policy obtained using bandit-based preference estimation with that obtained using direct ridge estimation. This evaluation fixes the target persona and varies the estimation method.

We divide the out-of-sample testing data into nine test periods of 6363 trading days for each of the two datasets. For each period, we compute final wealth, volatility, maximum drawdown, Sharpe ratio, turnover, and maximum weight for both policies. Gemma 4:31B serves as the LLM evaluator, receiving the target persona description and these anonymized portfolio criteria and judging which policy the target persona would prefer. Each pair is evaluated twice with its presentation order reversed. A win (WW) is recorded when the evaluator prefers the bandit-based policy in both orders; a loss (LL) is recorded when it prefers the direct-ridge policy in both. Ties and order-inconsistent judgments are excluded. We compute WR=W/(W+L)\mathrm{WR}=W/(W+L) for each dataset--persona case. A win rate above 50% indicates that the evaluator, conditioned on the target persona, prefers the policy obtained using bandit-based estimation. The results in Table E.8 show this preference in most cases.

Table E.8 (Direct ridge comparison). Preference win rates (%).

DatasetPPFFIIAA
S&P 50014.2966.6757.1477.78
MidCap10050.0050.0066.67100.00

Common Response 2: Roles of the Initialized Policy and Preference reward

This response addresses the shared questions about the source of personalization and the interpretation of ablation study.

In the original manuscript, we have investigated the impact of investor-specific preferences through the component ablation study. We have also rewritten the description around Figure 5 in the revised manuscript (Page 10, Lines 506–511) to improve clarity. In our framework, investor-specific preferences are reflected in the preference weights estimated through bandit-based estimation, the selected policy for PPO initialization, and the multi-objective preference reward. A clear result can be found that the policy differences across investor types are smaller for IMOPA-SP, which uses the same initialized policy across investor types, than for IMOPA-NPR which removes the multi-objective preference reward.

This is because the selected policy for subsequent initialization in PPO and the preference reward play different roles in IMOPA. The selected policy already captures a substantial part of the personalization achieved through preference elicitation, whereas the PPO stage further refines this policy. Specifically, the PPO objective incorporates cumulative return and the learned multi-objective preference reward, while the reference term controls deviations from the initialized policy. Thus, PPO refines the selected policy by balancing financial performance with the learned preference reward, rather than mainly generating investor heterogeneity from scratch. This explains why the selected policy for initialization has a stronger effect on policy differentiation across investor types than the preference reward in the PPO update.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.