acceptodds
Under review as a conference paper at ICLR 2027

Et Tu, Brute? Economic Misalignment in Personal AI Agents

Abstract

Personal AI agents make recommendations and take actions on people's behalf in high-stakes economic contexts, e.g., buying a flight, choosing health insurance, or selecting a graduate program. The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so. In a suite of K experiments on agents across three types of economic decisions (flights, health insurance, and graduate programs), we find that models systematically choose more expensive options for wealthier users when requests are identical. This steering continues even when it directly goes against the user's stated objective: when explicitly instructed to find the cheapest option, some agents still act on the wealth profile they have inferred. It also occurs when wealth is inferred from ambient data, such as emails unrelated to the task. And it persists under privacy controls that block specific attributes: blocking financial attributes largely removes the disparity, but blocking other attributes leaves it unchanged and can increase it by up to 40% for insurance, as agents rely on the remaining signals to infer wealth. Larger and more capable models are no better; Claude Opus 4.8 shows the largest effect. We term this misalignment “adversarial delegation”, in which the very conditions that make a personal AI agent useful — access to personal information — enable it to act against the user’s interests.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
economic misalignment, adversarial delegation, LLM Agents, personalization, privacy, contextual integrity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_s5szOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

Across roughly 325K trials, the paper studies wealth-conditioned recommendations by 13 personal agents. Eight models recommend pricier options to wealthier personas across covered domains. Some gaps persist under explicit cost preferences and email-based inference. This paper is interesting, but a bit far from the real world complex setting.

Critical strengths

There are several key points for the positive side, which are listed below.

  1. Fixed inventories and matched personas separate recommendation steering from price changes. This gives a new insight to the research problem.

  2. Multiple domains, access channels, and explicit-intent conditions provide broad coverage, though still a bit far from the reality.

  3. Attribute blocking and access restrictions test interventions, while retrieval analyses help distinguish access from steering.

Critical weaknesses

The major concern is the fidelity of the emulation. There are other weaknesses as follows.

  1. Intent-pooled gaps may reflect legitimate personalization. Cheap-intent and price-ceiling violations provide stronger evidence of misalignment, but need model-specific uncertainty and violation rates. Smaller gaps are not evidence of absence.

  2. Mean recommended price measures the five-item set, but not whether the first recommendation is cheapest. Top-1 price regret, cheapest-item inclusion, and ceiling compliance would clarify practical consequences. This is really critical in the real world.

  3. GPT-5.5's insurance amplification is reported as both 171 and 151. Table 7 also contains increases above the advertised 40% maximum. Reconcile these and report uncertainty, including potential wealth-dependent inventory exclusions.

  4. Binary wealth extremes and a ten-email inbox establish a controlled effect, not its prevalence in realistic personal contexts.

Questions for authors

Authors are encouraged to clarify some questions below, which are important to improve the readability of this paper.

  1. What are the cheap-intent and ceiling-violation rates, with uncertainty, for each model? Emulating the real world is challenging, but closer to it is still valuable.

  2. Do top-1 regret and cheapest-item coverage support the mean-price findings? This is the major concern from the reading of this submission. Please clarify.

  3. Can you reconcile the amplification numbers and test a direct anti-profiling instruction?

2 comments

We sincerely thank the reviewer for the detailed feedback and for recognizing the strengths of our experimental design. We agree that wealth-conditioned price differences do not necessarily imply misalignment, which is why we distinguish general recommendation steering from violations of explicit user objectives.

  1. Reconciling the amplification figures and inventory exclusions: We thank the reviewer for catching the inconsistency in §5. For GPT-5.5 insurance, blocking employment information increases the wealth-conditioned gap from $122/month to $171/month (+40%, 95% bootstrap CI [129, 211]). Blocking demographic information instead produces a gap of $131/month (+7%). The $151 figure and reference to demographic blocking in §5 were reporting errors, which we will correct.

    Regarding increases above 40% in Table 7, the 40% figure refers to the maximum insurance amplification among models with reliable inventory-valid evaluations and meaningful baseline gaps. Larger percentages arise when the baseline gap is near zero (e.g., $17 for Qwen3.5-2B) or when few trials pass inventory-validity checks (e.g., approximately 18–22% for Gemini 3/3.1 Flash). We will clarify this distinction and report with confidence intervals.

    We also checked for wealth-dependent inventory exclusions by comparing gate-pass rates across wealth groups. These were largely symmetric, with a maximum difference of 2.2% points. For instance, GPT-5.5 insurance had 100% validity for both groups, while Gemini 2.5 Flash had 98.7% versus 99.7%. We thank the reviewer for raising this point and will include this analysis in the revision, as it helps assess potential selection bias introduced by filtering invalid model outputs and strengthens the interpretation of our findings.

  2. Cheap-intent violations, top-1 regret, and cheapest-item coverage: We agree that the mean price of five recommendations may not capture the practical consequences of an agent's decision. We therefore computed the additional metrics suggested by the reviewer. First, we measured top-1 price regret, defined as the price of the first recommendation minus the cheapest available option. For insurance under explicit cheap intent, wealthy personas experience higher regret than low-income personas, with differences of $344/month for Claude Opus 4.8, $337 for Gemini 2.5 Flash, $330 for GPT-5, and $236 for GPT-5.5. The cheapest available plan is the same $85/month for both groups.

    Second, we measured cheapest-item coverage among the five recommendations. For Gemini 2.5 Flash flights, the cheapest option appears in only 33% of wealthy-persona trials versus 83% of low-income trials, despite identical requests and inventories. Thus, the difference is not merely in ranking; the cheapest option is often excluded entirely.

    Finally, we examined explicit price-ceiling violations. While most frontier models respect these constraints, some exhibit wealth-dependent violations. Claude Opus 4.8 breaches the $220 insurance ceiling in 29% of wealthy-persona trials, with plans exceeding the ceiling averaging $569 (2.6× the cap), compared to a 4% breach rate and an average of $322 (1.5× the cap) for low-income personas. For Gemini 2.5 Flash, the corresponding violation rates are 17% versus 4% for insurance and 21% versus 0% for flights.

    These results provide stronger evidence of misalignment beyond potentially legitimate personalization. We will include model-specific results and uncertainty estimates in the revision.

  3. Direct anti-profiling instructions: We thank the reviewer for this suggestion. Our objective is to investigate whether agents infer financial status from personal context and steer recommendations, even when users explicitly request otherwise. In our setting, users simply delegate tasks such as finding the cheapest flight, without asking agents to consider their wealth.

    An explicit anti-profiling instruction would introduce an additional safeguard against the behavior we study. While testing such instructions would be a useful mitigation experiment, our focus is on whether this behavior emerges in ordinary delegation settings without these safeguards.

    We do examine a related intervention through attribute blocking, where restricting financial information reduces the GPT-5.5 insurance gap [gap = mean(rich) − mean(poor)] from $613.79 − $491.53 = +$122/month to $545.67 − $559.79 = −$14/month (both gaps reported in Table 7, Appendix A.8).

(comment 1 of 2)

  1. Binary wealth profiles and external validity: We agree that our study establishes wealth-conditioned steering under controlled conditions rather than its real-world prevalence. As noted in §5 (lines 442–457), "We use personas and mock inventories rather than real users or fieldwork," and "wealth is binary rather than continuous, which may make the signal cleaner than in real populations." We also acknowledge that further validation with real email data and continuous variables remains important. We will emphasize these limitations more clearly in the revision, particularly that our results do not establish the prevalence of this behavior in real-world settings.

(comment 2 of 2)

Reviewer_w3g4OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the idea of adversarial delegation by showing how personal agents can steer recommendations based on personal context (i.e., inferred wealth). Through 325K experiments on 13 agents across multiple economic decision types, the results show that 8 models consistently choose more expensive options for wealthier users.

Critical strengths

This paper is the most interesting one in my batch, and I have really enjoyed reading it. It’s interesting, well written, and easy to follow.

The main strength of the paper is formulating and empirically showing the effect of adversarial delegation. To do this, the authors built personas with different profiles, used multiple models, and tested different methods of acquiring user information (e.g., email, personalized context). Through 325K experiments, they convincingly show that wealth steering occurs without instruction, that agents reconstruct wealth from indirect signals, that the signal overrides explicit user preferences (e.g., find the cheapest flight), and more.

I think a similar effect appears in [1], where agentic choices shifted toward cheaper or more expensive products when prompts included extra user information (e.g., a tight budget). The difference is that their information is explicit in the prompt, whereas here agents pick up implicit insights and use them adversarially.

[1] Cherep, M., Ma, C., Xu, A., Shaked, M., Maes, P., & Singh, N. (2026, April). A framework for studying ai agent behavior: Evidence from consumer choice experiments. In International Conference on Learning Representations (Vol. 2026, pp. 63296-63326).

The limitation section is honest; indeed, using real data would strengthen the claims. However, I think this paper is novel, and future work can focus on showing the welfare impact on real users in real-world applications.

Critical weaknesses

No critical weaknesses.

I’d revise the AI Use Statement and remove: “This section is required and does not count toward the page limit.” and “See the ICLR 2027 AI Policy for Authors for more details. This statement should not be more than 1 page.”

One of the references (Malik, 2025) has an out-of-bounds URL.

Questions for authors
  • Your experiment with explicit preferences is interesting, but I wonder if something even more explicit (e.g., a utility function) would steer models more robustly. As far as I understand, you prompt for the cheapest, which is an explicit utility, but I’m thinking of explicitly describing/showing that the user has a utility function.
  • What do agents mention in their reasoning? For example, what’s their logic around not selecting the actual cheapest flight when stated as a preference?
3 comments

We sincerely thank the reviewer for the thoughtful assessment of our work and the helpful suggestions. We also appreciate the reference to Cherep et al. (2026), particularly the distinction between explicitly provided user preferences and those inferred from personal context.

To address the reviewer's questions:

  1. Agents' reasoning when not selecting the cheapest option: We examined the agents’ recorded rationales in trials where wealthy personas requested the cheapest option but received a more expensive top recommendation and found that agents frequently justify these recommendations using comfort, quality, flexibility, or suitability to the user's circumstances. For example, Gemini 2.5 Flash recommends business class citing comfort, while Claude Opus 4.8 justifies a more expensive refundable flight based on circumstances. A keyword analysis (comfort|premium|business|refundable|flexib|quality|afford|suit|lifestyle|convenien|peace of mind) identified such justifications in 53/54 Gemini 2.5 Flash cases and 59/60 Claude Opus 4.8 cases where the first recommendation (top-1) was not the cheapest available option. Since this analysis uses permissive keyword matching, we interpret these findings as indicative of the agents' stated justifications rather than evidence of their internal reasoning.

We also examined whether this behavior begins during retrieval itself. For Gemini 2.5 Flash, we find that agents issue premium-oriented searches in 52% of wealthy-persona trials compared to only 6% of low-income trials, even when explicitly asked for the cheapest option. This suggests that inferred personal preferences can influence not only the final recommendation but also which options the agent searches for. We will include these additional analyses and representative examples in the revision.

  1. Whether explicit utility functions improve alignment: We agree that explicitly specifying a utility function would be an interesting extension. While requesting the "cheapest" option already expresses an objective to minimize price, we also evaluated a more constrained formulation in which users specify an explicit numeric price ceiling. We find that the wealth-conditioned flight-price gap decreases from $208 to $108 for Gemini 2.5 Flash and from $121 to $10 for Gemini 3.1 Flash Lite when using a numeric ceiling instead of "cheapest." This suggests that more precisely specified cost constraints can reduce wealth-conditioned steering, although they do not eliminate it.

However, we note that a numeric ceiling is not equivalent to a complete utility function. Moreover, requiring users to specify a price ceiling assumes they already know what constitutes a reasonable price, whereas determining this may be precisely why they delegated the task to an agent. We therefore see evaluating explicit utility functions as a useful direction for future work, while our present study focuses on whether agents preserve naturally expressed user preferences.

  1. Editorial corrections: We thank the reviewer for pointing these out. We will remove the template instructions from the AI Use Statement and correct the malformed URL for Malik (2025) in the revision.

Thank you for further evaluating the data; these insights are super interesting!

Personally, I'd include both in the paper. It's quite fascinating to see that agents infer wealth and then decide to maximize attributes such as comfort and quality. In the case of the more explicit utility, it's also nice to see less adversarial delegation. It seems like ambiguity (although "cheapest" should really be unambiguous) leaves the door open for all sorts of behaviors.

This paper is interesting, sound, and introduces the interesting idea of adversarial delegation. I believe it should be accepted, and I'm happy to champion it.

@2VKG I understand your concerns, but I think there's a lot of value in setting up an original experiment that leads to finding something like this adversarial delegation. Future work can do more to figure out the mechanisms behind it, and when it happens or not.

@s5sz I'd encourage you to engage with this paper, instead of submitting fully AI-generated reviews. The writing is identified by Pangram as 100% generated, but the "thinking" seems to be fully delegated as well.

Thank you so much for your thoughtful feedback and support throughout the discussion! We really appreciate the time you've taken to engage with our work and the additional suggestions you've shared. We'll make sure to incorporate both analyses into the revised manuscript, as you suggested. Thank you again for your encouragement and for being willing to champion the paper. It genuinely means a lot to us!

Reviewer_2VKGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether access to personal context causes LLM-based assistants to recommend more expensive options to wealthier users, including when users explicitly request cheap options. The authors construct 32 synthetic personas and evaluate 13 agents on 3 economic decision types. 325K evaluations vary user intent and 14 information-access conditions, including structured profiles, attribute blocking, and synthetic email access. The main outcome is the difference in mean recommended price between high- and low-financial-status personas. The paper reports wealth-conditioned recommendation steering, substantial differences under an explicit cheapest-flight request for one model, and non-monotonic effects of limiting email access.

Critical strengths
  1. The controlled design investigates a consequential interaction between personal context and explicit user objectives: Using fixed inventories and matched financial-s tatus personas helps distinguish changes in the agent’s recommendations from changes in seller prices or available products. The cheapest-flight condition is especially relevant as shown in Figure 4. This provides evidence beyond the potentially benign observation that wealthier users receive higher-quality recommendations. Preserving explicit user objectives is an important aspect of personalized-agent alignment, and this study identifies a concrete setting in which contextual information appears to interfere with that objective.

  2. The range of information-access interventions provides a useful empirical characterization: This study covers 13 agents, three domains, structured and email-based context, and several attribute-blocking and email-budget conditions. The non-monotonic email-access results and the distinction between financial-information retrieval and recommendation steering are useful diagnostic observations.

Critical weaknesses

Limited analytical depth and insufficient investigation of the underlying mechanisms: While this paper identifies an interesting and potentially important phenomenon, its contribution remains largely descriptive, resembling an extensive empirical audit rather than a deeper investigation of the underlying misalignment mechanisms. Although the large-scale evaluation across multiple models and domains is valuable, all experiments rely on synthetic personas, emails, and fixed inventories, leaving the external validity of the findings uncertain. More importantly, the paper does not sufficiently investigate why these behaviors occur.

Questions for authors

One of the key findings in this paper is that certain models continue to recommend substantially more expensive options to wealthier users even when explicitly instructed to find the cheapest option. Have the authors conducted any deeper analysis or targeted experiments to validate the hypothesis that models may interpret "cheapest" relative to the user's inferred purchasing power rather than as an absolute objective? Can this explanation be distinguished from possible alternative mechanisms (I guess), such as biases in product retrieval/recommendation ranking, or general failures to follow explicit cost instructions?

2 comments

We sincerely thank the reviewer for recognizing the importance of the problem and the breadth of our experimental design. We agree that understanding why agents exhibit this behavior is important. To address this, we conducted additional analyses of the agents' trajectories, search behavior, and stated rationales.

  1. Why do agents recommend more expensive options? Our hypothesis is that agents sometimes prioritize inferred user preferences (comfort, convenience, quality) over the stated objective of minimizing price. We examined trajectories where wealthy personas requested the cheapest option but received a more expensive top recommendation. In these cases, agents frequently justified their choices using terms such as “comfort”, “flexibility”, or “suitability”. A keyword analysis (comfort|premium|business|refundable|flexib|quality|afford|suit|lifestyle|convenien|peace of mind) identified such justifications in 53/54 Gemini 2.5 Flash cases and 59/60 Claude Opus 4.8 cases where the first recommendation (top-1) was not the cheapest available option. While these rationales do not establish the models' internal mechanisms, they help explain the observed recommendation behavior.

  2. Retrieval/ranking bias or a general failure to follow instructions: We investigated both explanations through additional analyses of the agents' search behavior and responses to explicit cost constraints.

  • On retrieval bias: We note that our search tool is deterministic and operates on the same fixed inventory for all personas. Given identical search parameters, it returns the same options in the same order, regardless of the user's financial status. However, we find that agents themselves issue different search queries based on personal context. For example, under the explicit cheapest-flight request, Gemini 2.5 Flash searched for premium options in 52% of wealthy-persona trials, compared to only 6% of low-income trials. This suggests that wealth-conditioned steering occurs during the agent's search process itself, rather than being introduced by the retrieval system.

  • On instruction-following failures: We compared the results for the instruction "find the cheapest" against those for an explicit numeric price ceiling. We find that the wealth-conditioned flight-price gap decreases from $208 to $108 for Gemini 2.5 Flash and from $121 to $10 for Gemini 3.1 Flash Lite. Most frontier models also show near-zero violations of the numeric ceiling. This suggests that the behavior is not simply a general failure to follow cost-related instructions. Instead, agents appear more likely to incorporate inferred user preferences when the cost objective is expressed in natural language.

Together, these results suggest that personal context influences both what agents search for and what they ultimately recommend. We will include these additional analyses in the revised manuscript to better distinguish the observed behavior from the alternative explanations raised by the reviewer.

  1. External validity: We agree that synthetic personas and fixed inventories do not establish real-world prevalence. However, this controlled setup allows us to isolate the effect of financial context while keeping prices, available options, and other persona attributes constant. Our ANOVA also shows that task motivation explains only 0.1–0.2% of price variance, compared with up to 37% for persona characteristics.

We will incorporate these additional analyses and clarify the distinction between identifying this failure mode under controlled conditions and estimating its prevalence in real deployments. We hope this addresses the reviewer's central concern regarding analytical depth.

We wanted to kindly follow up as today is the last day of the discussion period. Thank you again for your thoughtful feedback! We’d be happy to clarify any remaining questions or concerns you might have. We also wanted to ask if you would be open to reconsidering your rating in light of our additional analyses and responses. We’d really appreciate your thoughts!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.